Configuring SSL and TLS by using self-signed certificates

Configure Secure Sockets Layer (SSL) and Transport Layer Security (TLS) on the IBM Spectrum Protect™ server, backup-archive client, and storage agent to ensure that data is encrypted during communication. You can use a self-signed SSL certificate to verify an SSL communication request between the server, client, and storage agent.

Before you begin

To use SSL to secure communications between the Operations Center and the hub server, see Securing communications between the Operations Center and the hub server.

Before you set up the server certificate on the client, follow these steps:

Open a command window and change the directory to your IBM Spectrum Protect client directory, for example: cd "C:\Program Files\Tivoli\TSM\baclient"
Append the GSKit binary path and library path to the PATH environment variable, for example:
```
set PATH=x:\Program Files\Common Files\Tivoli\tsm\api64\gsk8\bin\;
 x:\Program Files\Common Files\Tivoli\tsm\api64\gsk8\lib64;%PATH%
```
where x: is the system drive where IBM Spectrum Protect is installed.

If your client operating system is 32-bit, replace the gsk8capicmd_64 command with gsk8capicmd in all GSKit commands.

About this task

TLS 1.2 is a more secure communication protocol than previous TLS protocol levels. For successful TLS communication, configure the client that communicates with the target server or storage agent to use TLS 1.2.

The image is a graphical depiction of how you configure SSL by using self-signed certificates, and provides the number for each task step.

Enter commands on one line. In the following steps, commands are displayed on multiple lines to make them easier to read. Ensure that you enter a space after each command.

Procedure

Specify the TCP/IP port on which the server waits for client communications that are enabled for SSL or TLS.
- Server: Update the dsmserv.opt file in the server instance directory by specifying the SSLTCPADMINPORT and SSLTCPPORT options, or both. The SSLTLS12 YES server option is automatically set to specify the use of TLS 1.2.
- Client: Specify the ssl yes and tcpport options in the client options file:
  - dsm.sys
  - dsm.opt
  The server is typically set up for SSL or TLS connections on a different port. If you use an SSL or TLS connection, two ports are open on the server. One port accepts regular non-SSL or non-TLS client connections and the other port only accepts SSL or TLS connections.
- Storage agent: In the dsmsta.opt file, specify the SSLTCPADMINPORT and SSTCPPORT option, or both.
Create the key database file and establish SSL communication between the server and the storage agent if necessary:
- Server: Start the server. This action creates the server key database file, cert.kdb, and stores it in the server instance directory. To enable the TLS 1.2 communication protocol, the following changes occur when a key database file is created:
  - The default label is automatically set as "TSM Server SelfSigned SHA Key".
  - The SSLHIDELEGACY and SSLDISABLELEGACYTLS server options are updated to YES in the server options file.
  If a password exists for the server database, it is reused for the cert.kdb key database. After you create the database, the key database access password is generated and stored. To establish communication between the server and the storage agent, issue the following command:
```
define server sta
hla=ip_address
lla=ssl_port
serverpa=password
ssl=yes
```
- Client: Create the key database file, dsmcert.kdb, by issuing the following command. Issue the command from the bin directory:
```
gsk8capicmd_64 -keydb 
-create -populate
-db dsmcert.kdb 
-pw password -stash
```
  Tip: The bin directory for the client is installed to the client system directory during client installation. For example, the bin directory for the client is installed in the following path:
```
system directory\Tivoli\TSM\api64\gsk8\bin
```
- Storage agent: Initialize the storage agent and add communication information to the device configuration file and the storage agent options file dsmsta.opt by issuing the DSMSTA SETSTORAGESERVER command. You must specify the SSL=YES and STAKEYDBPW=password parameters to create the key database file in dsmsta.opt. All passwords are encrypted in dsmsta.opt.
```
LDR_CNTRL=TEXTPSIZE=
64K@DATAPSIZE=64K@STACKPSIZE=64K@SHMPSIZE=64K
dsmsta setstorageserver myname=storage_agent_name
mypa=sta_password
myhla=ip_address
servername=server_name
serverpa=server_password
hla=ip_address
lla=ssl_port
STAKEYDBPW=password
ssl=yes
```
```
dsmsta setstorageserver 
myname=storage_agent_name
mypa=sta_password
myhla=ip_address
servername=server_name
serverpa=server_password
hla=ip_address
lla=ssl_port
STAKEYDBPW=password
ssl=yes
```
  Tip: Specify the SSLTCPADMIN port for the LLA parameter. If you do not specify the SSLTCPADMIN port, use the SSLTCPPORT port instead.
Create the key database certificate and default certificates by starting the storage agent.
Stop the server and the storage agent.
Enable SSL or TLS communication by importing the .arm files. You must use the same communication protocol between the server, client, and storage agent.
- Server and storage agent: Import the storage agent certificate to the key database file for the server and import the server certificate to the key database file for the storage agent:
```
gsk8capicmd_64 -cert 
-add -label 
server_example_name
-db cert.kdb -stashed 
-file cert256.arm 
-format ascii
```
- Client: Import the cert256.arm file to the key database file:
```
gsk8capicmd_64 -cert -add -db dsmcert.kdb 
-stashed -label "TSM server server_name 
self-signed key" -file path_to_cert256.arm 
-format ascii
```
  To change the default certificate in the cert.kdb key ring database file to the "TSM Server SelfSigned SHA Key" label, issue the following command from the server instance directory:
```
gsk8capicmd_64 -cert 
-setdefault -db cert.kdb 
-stashed -label 
"TSM Server SelfSigned 
SHA Key"
```
  Tip: To use TLS 1.2 communication between the client and the storage agent, the default label must be "TSM Server SelfSigned SHA Key."
Restart the storage agent and the server.