Enabling SSL for agent communications

Build Forge components are set up by default to use certain ports and security settings when SSL is enabled.

To enable SSL communication between Build Forge and agents, you need to do the following:

On UNIX computers, ensure that the GCC library is installed on the computer hosting the agent.
Prerequisite: enable SSL for client and internal communications. See Enabling SSL for client and internal connections.
Configure each agent. This task includes:
- Adding certificates to the agent host
- Editing the bfagent.conf file for the agent.
In the console, enable SSL in each Server definition that connects to the agent.

Configuring agents for SSL

If the agent is running, stop it.
Place .PEM files for the certificates in the root installation directory for the agent.
To implement and test SSL quickly you can copy the .PEM files from the Build Forge installation. The files are in <bfinstall>/keystore.

The best practice for SSL is to use a separate certificate for each agent:
1. Create separate keystore files (.PEM) for each of the following:
  - private key (key)
  - public certificate for the private key (cert)
  - trusted signers (ca or certificate authority)
2. If you are using an unique certificate for the agent (instead of a copy of the Build Forge engine's certificate), then add the certificate for the agent to the certificate authority keystore for Build Forge, <bfinstall>/keystore/buildForgeCA.pem. If you are running multiple engines (redundant engines), add the certificate to each engine's certificate authority keystore.
Edit BFAgent.conf. The following lines are commented out in the file. Remove the comment prefix.
```
ssl_key_location buildForgeKey.pem
ssl_key_password password
ssl_cert_location buildForgeCert.pem
ssl_ca_location buildForgeCA.pem
ssl_protocol TLSv1
ssl_cipher_group ALL
```
The password is for the buildForgeKey.pem keystore. If you want to encrypt it, see Encrypting passwords in buildforge.conf and bfagent.conf.
If you want to require client authentication when a connection is made to the agent, uncomment the following line:
```
ssl_client_authentication true
```
This setting requires that the engine certificate be added in the agent's certificate authority keystore, buildForgeCA.pem.
If you want to use specific ciphers, uncomment this line and add your cipher list:
```
ssl_cipher_override cipher_list
```
Start the agent. The agent must be running to test the connection from the console.

Enabling SSL in Server definitions

The console uses Server definitions to connect to agents.

For each Server definition that is connected to an SSL-enabled agent, do the following:

In the console, go to the Servers panel.
Click the server definition name.
In the Details tab for the server definition:
- Set SSL Enabled to Yes.
Click Save.
Click Test Connection.

Troubleshooting SSL communication with agents

The following checklist describes common issues when enabling SSL:

Agent
- Agent SSL is not configured, though SSL is enabled in Security and in the Server definition.
- The certificate for the agent is not trusted by the Build Forge engine. The agent certificate needs to be added to the engine's CA keystore:
```
<bfinstall>/keystore/buildForgeCA.pem
```
- Password for the keystore specified in BFAgent.conf is incorrect.
- Client authentication is specified in BFAgent.conf but the engine's certificate has not been added to the agent's certificate authority, buildForgeCA.pem.
Build Forge console
- SSL was enabled on the console. It was not saved, or Update Master Bfclient.conf was clicked before saving, or Update Master Bfclient.conf was never clicked after saving.
Server definition
- SSL was not successfully enabled. You must click Save before Test Connection.
Engine and Agent settings match
- Handshake protocol mismatch. The handshake protocol must be set to the same value for both the engine SSL configuration and agent SSL configuration, either TLSv1 or SSLv3. The default value is TLSv1.
- Cipher suite mismatch. The cipher suites specified in the engine SSL configuration and agent SSL configuration must have ciphers in common. The default cipher suite group is ALL.

Enabling debugging messages

You can enable debugging in the engine and the agent. When debugging is enabled, additional detailed output is produced that can help identify problems in the configuration.

Enabling engine debugging:
1. In an environment used by an SSL-enabled server definition, add the following variable:
```
BFDEBUG_SECURITY=1
```
2. Stop the engine and restart it.
  - On Windows, start the engine in the foreground. Output appears in a command window.
  - On UNIX or Linux, start the engine with debugging turned on:
```
cd <bfinstall>/rc
./buildforge start
```
    Output from the engine goes to the engine log file in <bfinstall>/log.
Enabling agent debugging:
1. Stop the agent.
2. Add the following line to BFAgent.conf:
```
activity_log bfagent.log
```
  In this example, the agent writes output to bfagent.log. You can specify a different filename.
  Note: If the agent runs as a service, specify an absolute path.
3. Start the agent.

Example engine debug output for a successful SSL connection

An engine produces the following output when it connects successfully to an agent.

SSL_ca_file: ./keystore/buildForgeCA.pem
SSL_cert_file: ./keystore/buildForgeCert.pem
SSL_key_file: ./keystore/buildForgeKey.pem
SSL_verify_mode: 0x01
SSL_version: TLSv1
SSL_cipher_list: ALL
SSL_use_cert: 1
Making as SSL connection using socket IO::Socket::INET=GLOB(0x1e8f0f4).
SSL connection to agent.
DEBUG: .../IO/Socket/SSL.pm:1387: new ctx 80662848
DEBUG: .../IO/Socket/SSL.pm:880: dont start handshake: IO::Socket::SSL=GLOB(0x1e8f0f4)
DEBUG: .../IO/Socket/SSL.pm:284: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:327: Net::SSLeay::connect -> 1
DEBUG: .../IO/Socket/SSL.pm:382: ssl handshake done
Socket is of type: ref(IO::Socket::SSL=GLOB(0x1e8f0f4))
ReadyLine: 202 HELLO TLS - BuildForge Agent v_VERSION_
.
Storing Agent Version [999.999.999.999-999-9999] for [08974C8E-6C3B-1014-972D-D9B2901D9F42]
cmd ping
username pbirk
encpass c1713f4a31af3f1300f7b2414a24559c4d6097e07310cf9c412e
go
Sending agent request...

Example agent debug output for a successful SSL connection

An agent running normally produces the following output when it establishes an SSL connection.

[2256] main.c          : 409: === NEW AGENT ===
[2256] io.c            : 264: In start_SSL
[2256] io.c            :  89: Key location: buildForgeKey.pem
[2256] bfpwdlocloader.c: 134: Looking for password locator: ssl_key_password_locator
[2256] bfpwdlocloader.c: 244: Looking for password for prop 
                              ssl_key_password from bfagent.conf.
[2256] bfcryptloader.c : 202: Loading password encryption module.
[2256] bfcryptloader.c : 276: Password encryption property 
                              password_encrypt_module is not configured.
[2256] bfcryptloader.c : 539: Password decoded.
[2256] io.c            :  98: Cert location: buildForgeCert.pem
[2256] bfpwdlocloader.c: 134: Looking for password locator: 
                              ssl_cert_password_locator
[2256] bfpwdlocloader.c: 244: Looking for password for prop 
                              ssl_cert_password from bfagent.conf.
[2256] io.c            : 153: Setting key password in default userdata.
[2256] io.c            : 160: Getting private key from PEM.
[2256] io.c            : 166: Checking private key from PEM.
[2256] io.c            : 172: Getting CA store information.
[2256] bfpwdlocloader.c: 134: Looking for password locator: 
                              ssl_ca_password_locator
[2256] bfpwdlocloader.c: 244: Looking for password for prop 
                              ssl_ca_password from bfagent.conf.
[2256] io.c            : 178: CA location: buildForgeCert.pem
[2256] io.c            : 184: Checking the CA store.
[2256] io.c            : 230: Returning from init_CTX.
[2256] io.c            : 281: Calling SSL_new
[2256] io.c            : 294: Calling SSL_accept.
[2256] io.c            : 346: Cipher chosen: AES256-SHA
[2256] io.c            : 367: ssl_state = SS_CERTIFIED

Example output for bad keystore password on the agent

If the keystore password configured on the agent side is wrong, it shows up in both engine and agent output.

Engine output (excerpt):

SSL_use_cert: 1
Making as SSL connection using socket IO::Socket::INET=GLOB(0x1e8f0f4).
SSL connection to agent.
DEBUG: .../IO/Socket/SSL.pm:1387: new ctx 80662848
DEBUG: .../IO/Socket/SSL.pm:880: dont start handshake: IO::Socket::SSL=GLOB(0x1e8f0f4)
DEBUG: .../IO/Socket/SSL.pm:284: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:327: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1135: SSL connect attempt failed with unknown error
error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

DEBUG: .../IO/Socket/SSL.pm:333: fatal SSL error: SSL connect attempt failed with unknown error
error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
DEBUG: .../IO/Socket/SSL.pm:1422: free ctx 80662848 open=80662848 80566656
DEBUG: .../IO/Socket/SSL.pm:1425: OK free ctx 80662848

Agent output (excerpt):

[    5272] io.c            :  98: Cert location: buildForgeCert.pem
[    5272] bfpwdlocloader.c: 134: Looking for password locator: ssl_cert_passwor
d_locator
[    5272] bfpwdlocloader.c: 244: Looking for password for prop ssl_cert_password from bfagent.conf.
[    5272] io.c            : 153: Setting key password in default userdata.
[    5272] io.c            : 160: Getting private key from PEM.
[    5272] io.c            : 218: Failure reason: SSLErrorBadPKeyFile
[    5272] io.c            : 221: OpenSSL error string: error:00000000:lib(0):func(0):reason(0)
[    5272] io.c            : 281: Calling SSL_new
[    5272] platform.c      :2693: platform_release_credentials
[    5272] main.c          : 412: --- EXITING ---

Error codes in agent output

This list includes some of the other error codes you may encounter and their causes:

SSLErrorBadCA: Problem loading the signers in buildForgeCA.pem. It could be a problem in the format of the file resulting from how the signers were added.
SSLErrorBadCert: Problem loading the certificate in the buildForgeCert.pem. Either the certificate does not match the private key or it is corrupted in the PEM.
SSLErrorBadPKeyFile: Problem with the buildForgeKey.pem password specified for the ssl_key_password property in BFAgent.conf.
SSLErrorBadPKey: SSL_CTX_check_private_key returned a value other than 1. The private key format is not valid or it does not match the certificate.
SSLErrorFIPSEnablement: An error occurred during FIPS enablement. This typically is due to an issue encountered during the FIPS self-check. This is likely an internal error.
SSLErrorInvalidCipher: A cipher specification does not match what OpenSSL allows. Check the ciphers specified on properties ssl_cipher_group or ssl_cipher_override in BFAgent.conf.
SSLErrorNoCtx: Problem creating a new SSL CTX object. This is likely an internal error.