Setting up SSL for instance groups: Tier 2 and Tier 3 (external certificates)

Set up SSL for the cluster for tier 2 services and tier 3 Spark drivers so that SSL is turned on by default for instance groups, by using a properly chained certificate from a trusted certificate authority (CA).

Before you begin

You must be a cluster administrator.
You must enable tier 1 SSL files and configuration for the cluster management console and RESTful APIs. See Enabling SSL for the cluster management console using an external certificate and Enabling SSL for the RESTful APIs using an external certificate.

About this task

Follow this task if you want to:

Enable SSL for the first time instance groups (in other words, if you disabled SSL during installation by setting DISABLESSL=Y).
Use a properly chained certificate from a trusted CA that replaces the default self-signed certificate when SSL is enabled during installation.

For your production environment, ensure that you use a properly chained certificate that is issued or signed by a trusted CA.

Note:

This procedure uses keytool and openssl commands solely for purposes of demonstration. For proper certificate handling, follow the instructions from the CA vendor who is generating your certificates.
On a local file system, Tier 2 and 3 files must be generated on all hosts in the cluster. On a shared file system, the files need to be generated on a shared file system location that all hosts in the cluster can access. Also, the subjectAltName (SAN) field must contain all hosts in the cluster.

Procedure

If you want to manage your own certificates, use the keytool utility to clean up the certificate in the default server keystore (tier2and3ServerKeyStore.jks):

Note: This step is required only if SSL was previously enabled (thus, generating the tier2and3ServerKeyStore.jks). In this case, you must clean up the certificate.
1. Back up the default server keystore ($EGO_TOP/security/tier2and3ServerKeyStore.jks).
2. Change to the /bin directory under your Java™ directory:
```
cd $EGO_TOP/jre/8.0.6.0/linux-x86_64/bin
```
3. List the certificates for tier 2 and tier 3 in the keystore and identify the alias'.
  - Tier 2 example:
```
keytool -v -list -keystore tier2and3ServerKeyStore.jks -storepass storepass
```
    where storepass specifies the keystore password.
    In the following example, the generic domain certificate uses the alias tier2alias:
    ... Alias name: tier2alias Creation date: Nov 5, 2015 Entry type: keyEntry Certificate chain length: 2 ...
  - Tier 3 example:
```
keytool -v -list -keystore tier2and3ServerKeyStore.jks -storepass storepass
```
    In the following example, the generic domain certificate uses the alias tier3alias:
    ... Alias name: tier3alias Creation date: Nov 5, 2015 Entry type: keyEntry Certificate chain length: 2 ...
4. Remove the certificate from the keystore for tier 2 and tier 3.
  - Tier 2 example:
```
keytool -delete -alias tier2alias -keystore tier2and3ServerKeyStore.jks -storepass storepass
```
    where storepass specifies the keystore password; for example: Liberty.
  - Tier 3 example:
```
keytool -delete -alias tier3alias -keystore tier2and3ServerKeyStore.jks -storepass storepass
```
Generate one keystore for tier 2 and tier 3 using the keytool utility:
From the /bin directory under your Java directory, run the following command to generate a keystore:
- Tier 2 example:
```
keytool -genkeypair -noprompt -alias tier2alias -dname "CN=*.domainName,O=IBM,C=CA" -keystore keystore_name.jks -storepass storepass -keypass keypass -keyalg rsa -validity 825 -keysize  2048 -sigalg SHA256withRSA -ext "san=dns:hostname"
```
  where:
  - keystore_name is the name of your new keystore. You can reuse the default server keystore (tier2and3ServerKeyStore.jks) after cleanup (see step 1).
  - -dname domainName identifies the fully qualified domain of the cluster management console web server; for example: myhostname.example.com.
  - -storepass storepass specifies the password to protect the integrity of the keystore and -keypass keypass specifies the password for the alias. It is recommended that you do not use the same password for both options.
  - -alias is set to use tier2alias.
  - -ext "san=dns:hostname" specifies all host names in the cluster to which the certificate must apply as the subjectAltName. For example, -ext "san=dns:hostname1,dns:hostname2,...".
- Tier 3 example:
```
keytool -genkeypair -noprompt -alias tier3alias -dname "CN=*.domainName,O=IBM,C=CA" -keystore keystore_name.jks -storepass storepass -keypass keypass -keyalg rsa -validity 825 -keysize  2048 -sigalg SHA256withRSA -ext "san=dns:hostname"
```

Create the certificate-signing request file (.csr file) for tier 2 and tier 3:

Tier 2 example:
```
keytool -certreq -alias tier2alias -file tier2alias.csr -keystore keystore_name.jks -storepass storepass -keypass tier2passwd 
```
where tier2alias.csr is the certificate-signing request file that you require to order an SSL certificate from your CA vendor.

Tier 3 example:

keytool -certreq -alias tier3alias -file tier3alias.csr -keystore keystore_name.jks -storepass storepass -keypass tier3passwd

Contact your CA vendor to create the SSL certificate for tier 2 and tier 3, which is also imported into the keystore_name.jks. Your vendor needs the tier2alias.csr and the tier3alias.csr files that you created in step 3.
Once your vendor approves your request, follow the vendor-provided instructions to download the certificates in .der and .pem formats (for example, cert.der and cert.pem).
If your vendor does not provide the certificate in .pem format, which is required for use in a later step, convert the certificate from .der to .pem format. For example:
```
openssl x509 -inform der -in cert.der -out cert.pem
```
For recovery purposes, back up your SSL files to a secure location. These files include your keystore file (keystore_name.jks), your certificates (for example, cert.der), and optionally, the certificate-signing request files (tier2alias.csr and tier3alias.csr). If you run into any issues when you are configuring SSL for your production environment, restore your backup files and redo the procedure from the next step.
Import your server certificates to the keystore. If your CA vendor provided instructions, you must follow those instructions.
- Tier 2 example:
  Import your server certificates by using the keytool utility, for example, enter:
```
keytool -importcert -trustcacerts -alias tier2alias -keystore keystore_name.jks -file certificate -storepass storepass
```
  where certificate is your certificate in .der format (for example, cert.der), which you downloaded from your CA vendor in step 5.
  If you are prompted to trust the signed certificate when you import the certificate, as follows:
```
Trust this certificate? [no]:
```
  you must additionally import the intermediate or root CA certificates. Your CA vendor must provide instructions on how to obtain the intermediate and root CA certificates; the instructions might be included as part of an email or provided as links.
  If required, use keytool to import the intermediate and root CA certificate. For example:
```
keytool -importcert -noprompt -alias intermediate -file intermediate.der -storepass storepass -keystore caKeyStoreTier2and3.jks 
keytool -importcert -noprompt -alias rootCA -file rootCA.der -storepass storepass -keystore keystore_name.jks
```
  where:
  - intermediate.der and rootCA.der are the certificate names of the intermediate certificate and root CA certificate, respectively.
  - storepass specifies the keystore password.
  - caKeyStoreTier2and3 and keystore_name are the name of your keystores.
- Tier 3 example:
  Import your server certificates by using the keytool utility, for example, enter:
```
keytool -importcert -trustcacerts -alias tier3alias -keystore keystore_name.jks -file certificate -storepass storepass
```
  If you are prompted to trust the signed certificate when you import the certificate, as follows:
```
Trust this certificate? [no]:
```
  If required, use keytool to import the intermediate and root CA certificate. For example:
```
keytool -importcert -noprompt -alias intermediate -file intermediate.der -storepass storepass -keystore caKeyStoreTier2and3.jks 
keytool -importcert -noprompt -alias rootCA -file rootCA.der -storepass storepass -keystore keystore_name.jks
```
Copy this keystore with 444 permissions on all hosts in the cluster.
Optional: Encrypt the passwords that you specified in the previous steps by using the securityUtilities command. For more information, see:
- Creating SSL certificates from the command line.
- Liberty profile: securityUtility command

Edit $EGO_CONFDIR/../../ascd/conf/ssl.conf to modify the following parameters to the values you used in the previous steps:

KEYSTORE_PATH=keystore_name.jks
KEYSTORE_PASSWD={aes}test_storepassword
KEYSTORE_TYPE=JKS
TIER2_ALIAS_NAME=tier2AliasName
TIER2_KEY_PASSWD={aes}password_tier2
TIER3_ALIAS_NAME=tier3AliasName
TIER3_KEY_PASSWD={aes}password_tier3

Optional: If the instance groups support notebooks, contact your CA vendor to generate open SSL certificates for tier 3. If you want to generate your own open SSL certificates from your keystore files, see Setting up SSL for instance groups: Tier 3 notebooks.

Edit $EGO_CONFDIR/../../ascd/conf/ascd.conf to turn SSL on by default for instance groups and to provide the Spark services with the required keystore information:

CONDUCTOR_SPARK_SSL_ENABLED=ON
CA_CERT_KEYSTORE_PATH=$EGO_TOP/wlp/usr/shared/resources/security/caKeyStoreTier2and3.jks
CA_CERT_KEYSTORE_PASSWD={aes}ANr2+3vBND5lpzCl4QWrolYXCy3OggZWgvuDsjD7c5vF

Restart ascd to accept the changes to the properties file:
1. Stop the ascd service by running the following command:
```
egosh service stop ascd
```
1. Start the ascd service by running the following command:
```
egosh service start ascd
```

Results

SSL is now set up for instance groups in the cluster. SSL can be enabled for newly created instance groups by using the cluster management console. See Enabling SSL for the instance group. Existing instance groups can be modified to enable SSL. See Modifying instance groups