Enabling SSL for the RESTful APIs using an external certificate

Configure SSL for the web servers in your production environment, using a properly chained certificate from a trusted certificate authority (CA).

Before you begin

You must be a cluster administrator.
You must ensure that the clock settings on the server and client are identical. Otherwise, errors may occur.

About this task

Follow these steps if you want to:

Enable SSL for the first time (in other words, if you disabled SSL during installation by setting DISABLESSL=Y).
Use a properly chained certificate from a trusted CA that replaces the default self-signed certificate when SSL is enabled during installation.

Configure SSL for the following web servers to safely access the RESTful APIs:

The ascd web server, which hosts the RESTful APIs for instance group management.
The REST web server, which hosts the RESTful APIs for resource management and package deployment.

When SSL is enabled for the REST web services (ascd and REST), a trust relationship between the server and the client is established by sending a server certificate to the client. The client validates the certificates that are signed by a trusted CA. For your production environment, ensure that you use a properly chained certificate that is issued or signed by a trusted CA. By default ascd and REST uses the TLSv1.2 protocol.

Important: You must use the same SSL setting for the cluster management console and the RESTful web servers. If you enable SSL for one, you must also enable SSL for the other; if you disable SSL for one, SSL must be disabled for the other as well. This setting also takes effect for cloud bursting with host factory. Ensure that SSL for all these functions is configured consistently in the cluster; without a uniform configuration, errors occur. Note, however, that when SSL is uniformly enabled, you can use different certificates and keys as required.

Note: On a local file system, Tier 1 files must be generated on all management hosts. On a shared file system, the files need to be generated on a shared file system location that all management hosts can access.

Procedure

Stop the service:
- To stop the ascd service, run:
```
egosh service stop ascd 
```
- To stop the REST service, run:
```
egosh service stop REST
```
If you want to manage your own certificates, use the keytool utility to clean up the certificate in the default server keystore (serverKeyStore.jks):

Note: This step is required only if SSL was previously enabled (thus, generating the serverKeyStore.jks). In this case, you must clean up the certificate.
1. Back up the default server keystore ($EGO_TOP/wlp/usr/shared/resources/security/serverKeyStore.jks).
2. Change to the /bin directory under your Java™ directory:
```
cd $EGO_TOP/jre/8.0.6.0/linux-x86_64/bin
```
3. List the certificates in the keystore and identify its alias. For example:
```
keytool -v -list -keystore serverKeyStore.jks -storepass storepass
```
  where storepass specifies the keystore password; for example: Liberty.
  In the following example, the generic domain certificate uses the alias srvalias:
```
   ...
    Alias name: srvalias
    Creation date: Nov 5, 2015
    Entry type: keyEntry    
    Certificate chain length: 2
   ...
```
4. Remove the certificate from the keystore. For example:
```
keytool -delete -alias srvalias -keystore serverKeyStore.jks -storepass storepass
```
  where storepass specifies the keystore password.
Generate a keystore for the API servers and create a certificate-signing request file by using the keytool utility:

Important: When you run keytool, the server keystore is generated in a local directory, not in a shared directory. If the ascd and REST services are configured to use the ManagementHosts resource group or you have customized the host list, ensure that you complete these steps on all management hosts, so that the keystore is available on all management hosts.
1. From the /bin directory under your Java directory, run the following command to generate a keystore:
```
keytool -genkey -alias server -keyalg RSA -keysize 2048 -keystore keystore_name.jks
```
  where keystore_name is the name of your new keystore. You can optionally reuse the default server keystore (serverKeyStore.jks) after cleanup (see step 2).
2. When prompted for the keystore password, enter a password for the keystore_name.jks file.
3. When prompted for the last and first name, enter the FQDN of the ascd or REST host. For example:
  myascdhost.example.com
  
  myresthost.example.com
  
  Ensure that the domain name does not start with a number.
  Important: Follow the instructions of the CA vendor generating your certificate for specific information on different fields. Some certificate vendors might state that the L (Location), ST (State), and C (Country) fields cannot be blank; other vendors might require the country value (C) to be uppercase. For example:
```
CN=*.eng.platformlab.ibm.com, OU=Platform Computing, O=IBM, L=Markham, ST=ONT, C=CA
```
4. Answer all other questions.
5. Create the certificate-signing request file:
```
keytool -certreq -alias server -file keystore_name.csr -keystore keystore_name.jks
```
  where keystore_name.csr is the certificate-signing request file that you require to order an SSL certificate from your CA vendor.
Contact your CA vendor to create an SSL certificate. Your vendor will need the keystore_name.csr file you created in step 3.

Note: If you add primary-candidates after the certificate is first generated, you must generate a new certificate to include the host names of the newly added primary-candidates.
Once your vendor approves your request, follow the vendor-provided instructions to download the certificate in DER format (for example, cert.der).
For application container workload, if you plan to manage application instances from the command line (using the egosh appinstance command) after enabling SSL and do not want to disable SSL verification, convert the certificate from DER format to PEM format:
```
openssl x509 -inform der -in cert.der -out cert.pem
```
The PEM format is the only format supported for EGO_SSL_EGOSH_CAINFO.
Import your server certificate to the keystore. If your CA vendor provided instructions, you must follow those instructions.
To import your server certificate using the keytool utility, for example, enter:
```
keytool -importcert -trustcacerts -alias server -keystore keystore_name.jks -file certificate
```
where certificate is your certificate (for example, cert.der) which you downloaded from your CA vendor in step 5.
If you are prompted to trust the signed certificate when importing the certificate, as follows:
```
Trust this certificate? [no]:
```
you must additionally import the intermediate or root CA certificates. Your CA vendor must provide instructions on how to obtain the intermediate and root CA certificates; the instructions may be included as part of an email or provided as links.
If required, use keytool to import the intermediate and root CA certificate. For example:
```
keytool -importcert -noprompt -alias intermediate -file intermediate.der -storepass storepass -keystore keystore_name.jks 
keytool -importcert -noprompt -alias rootCA -file rootCA.der -storepass storepass -keystore keystore_name.jks
```
where:
- intermediate.der and rootCA.der are the certificate names of the intermediate certificate and root CA certificate, respectively.
- storepass specifies the keystore password.
- keystore_name is the name of your keystore.
Configure SSL settings in the configuration file:
1. Copy the SSL settings from the *.xml.ssl file:
  - For ascd, copy $EGO_CONFDIR/../../ascd/conf/server_ascd.xml.ssl to $EGO_CONFDIR/../../ascd/conf/server_ascd.xml.
  - For REST, copy $EGO_CONFDIR/server_rest.xml.ssl to $EGO_CONFDIR/server_rest.xml.
2. Open the configuration file:
  - For ascd, open $EGO_CONFDIR/../../ascd/conf/server_ascd.xml.
  - For REST, open $EGO_CONFDIR/server_rest.xml.
3. Add the keystore file location and password to the keyStore element. The keyStore element is called defaultKeyStore and contains the keystore password.
```
<keyStore id ="defaultKeyStore" location="keystore_name.jks_location" password="keypass"/>
```
  where:
  - keystore_name.jks_location is the path to keystore_name.jks.
  - keypass is the password set for keystore_name.jks.
4. Optional: Encrypt the password using the securityUtilities command. For more information, see:
  - Liberty profile: securityUtility command
5. Save your changes to server_ascd.xml or server_rest.xml.
If you are enabling SSL for the first time or you are using custom certificates, then you need to copy the CA public certificate to the same location on all hosts.
1. Edit the CA_CERT_KEYSTORE_PASSWD, CA_CERT_KEYSTORE_PATH, and CA_CERT_PATH parameters in $EGO_CONFDIR/../../ascd/conf/ascd.conf. For example,
```
CA_CERT_KEYSTORE_PASSWD={aes}ANr2+3vBND5lpzCl4QWrolYXCy3OggZWgvuDsjD7c5vF
CA_CERT_KEYSTORE_PATH=$EGO_TOP/security/caKeyStore.jks
CA_CERT_PATH=$EGO_TOP/wlp/usr/shared/resources/security/cacert.pem
```
2. Double check the value of the CA_CERT_PATH parameter in $EGO_CONFDIR/../../integration/elk/conf/elk.conf.
  By default this value is set to CA_CERT_PATH=${EGO_TOP}/wlp/usr/shared/resources/security/cacert.pem.
3. Restart the following services if you changed the value of CA_CERT_PATH in elk.conf:
  - elk-manager
  - elk-elasticsearch
  - elk-elasticsearch-master
  - elk-elasticsearch-data
For the ascd service with application container workload, if you want to manage application instances from the command line (using the egosh appinstance command) with SSL, you must define the following environment variables for your CA-signed certificate or optionally disable verification.
Important: Disabling SSL verification is not secure and is not recommended.

For Linux® hosts, these variables are also defined by default in the environment profile scripts ($EGO_TOP/ascd/conf/profile.ascd and $EGO_TOP/ascd/conf/cshrc.ascd). Change the settings using the scripts, if required.
- EGO_SSL_EGOSH_CAINFO: Specify the path to the CA certificate file. The supported file format is .pem.
- EGO_SSL_EGOSH_VERIFYHOST: If the Common Name (CN) field in the certificate does not match the host name (displayed in the Description field of the egosh client view ASC_REST_URL_1 command), set this variable to N.
To disable verification, define the following variables:
- EGO_SSL_EGOSH_VERIFYHOST: Set this variable to N to disable verification of the Common Name (CN) field in the certificate against the host name.
- EGO_SSL_EGOSH_VERIFYPEER: Set this variable to N to disable verification of the peer SSL certificate.
For more information on the environment variables, see appinstance.
Start the service:
- To start the ascd service, run:
```
egosh service start ascd 
```
- To start the REST service, run:
```
egosh service start REST
```
Restart the instance groups in the cluster.
Update the following environment variables that are used by the SparkCleanup service:
- ASCD_REST_URL: Defines the base URL for the ascd web server and is required by the SparkCleanup service to periodically clean up the instance groups in your cluster.
- ASCD_REST_CACERT_PATH: If you are enabling SSL for the first time or you are using custom certificates, you need to edit this environment variable to the path that you specify.
1. cluster management console, select System & Services > EGO Services > Service Profiles.
2. In the Other Services tab, click SparkCleanup.
3. In the service profile dialog, edit the ASCD_REST_URL and the ASCD_REST_CACERT_PATH environment variables under sc:ActivityDescription > ego:ActivitySpecification. For example:
  - For ASCD_REST_URL with SSL enabled, the default URL is https://${SERVICE_ascd_LOCATION}:8643/platform/rest.
  - For ASCD_REST_CACERT_PATH: $EGO_TOP/wlp/usr/shared/resources/security/cacert.pem
4. Click Save.
Launch a browser to access the RESTful APIs over SSL.
- To access the APIs for application instance management, enter (for example) https://ascrestserver.example.com:8643/platform/asc/v1.
  If you set EGO_SSL_EGOSH_VERIFYHOST and EGO_SSL_EGOSH_VERIFYPEER to N, you can access the RESTful APIs. Otherwise, continue with the rest of this procedure.
- To access the APIs for instance group management, enter (for example) https://ascrestserver.example.com:8643/platform/rest/conductor/v1.
- To access the APIs for resource management, enter (for example) https://restserver.example.com:8543/platform/rest/ego/v1.
- To access the APIs for package deployment, enter (for example) https:/restserver.example.com:8543/platform/rest/deployment/v1.
When the browser prompts you about an untrusted connection, follow the instructions provided by your CA to install the server certificate.

Results

You can now securely access the RESTful APIs over SSL in your production environment.

What to do next

If the cluster contains instance groups with notebooks or instance groups that are configured to use GPU executors and have the SPARK_EGO_AUTOSCALE_SLOTS_PER_TASK parameter set, you or the consumer administrator must modify these instance groups so that ascd can apply the appropriate service profile changes. To modify, stop the instance group, and click Configure > Modify Instance Group.