IBM Support

How to fix this SSL error javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name in Websphere Application Server?

Question & Answer


Question

I am using WASv9.0.0.7 Java version = 1.8.0_171, Java Runtime Version = 8.0.5.16 - pxa6480sr5fp16-20180524_01(SR5 FP16)

From the WAS console, I am trying to retrieve signers from a remote SSL port and it failed to retrieve the certificate with following error

javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name

Answer

From Java 7 on-wards Server Name Indication (SNI) extension was introduced as a default in the JSSE client. see the following link under JSSE enhancements

https://www.ibm.com/support/knowledgecenter/SSYKE2_7.0.0/com.ibm.java.security.component.70.doc/security-component/whats_new/security_changes_70/security_whatsnew.html

SNI stands for Server Name Indication and is an extension of the TLS protocol. It indicates which hostname is being contacted by SSL client during the initial SSL handshake process after clienthello message.

More detail about SNI see the following info center link

https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/sni_extension.html

 How to identify from the SSL debug traces

    [9/4/18 8:16:55:464 PDT] 000000eb SystemOut O *** ClientHello, TLSv1.2
    [9/4/18 8:16:55:465 PDT] 000000eb SystemOut O Session ID: {}
    [9/4/18 8:16:55:465 PDT] 000000eb SystemOut O Cipher Suites: [SSL_RSA_WITH_AES_256_GCM_SHA384, SSL_RSA_WITH_AES_256_CBC_SHA256, SSL_RSA_WITH_AES_128_CBC_SHA256, SSL_RSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA]
    [9/4/18 8:16:55:465 PDT] 000000eb SystemOut O Compression Methods: { 0 }
    [9/4/18 8:16:55:466 PDT] 000000eb SystemOut O Extension renegotiation_info, ri_length: 0, ri_connection_data: { null }
    [9/4/18 8:16:55:466 PDT] 000000eb SystemOut O Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA256withDSA, SHA1withDSA
    [9/4/18 8:16:55:466 PDT] 000000eb SystemOut O Extension extended_master_secret
    [9/4/18 8:16:55:466 PDT] 000000eb SystemOut O Extension server_name, server_name: [type=host_name (0), value=ramserver.austin.ibm.com]
    [9/4/18 8:16:55:466 PDT] 000000eb SystemOut O ***
    [9/4/18 8:16:55:467 PDT] 000000eb SystemOut O [write] MD5 and SHA1 hashes: len = 128
    [9/4/18 8:16:55:467 PDT] 000000eb SystemOut O 0000: 01 00 00 7c 03 03 5b 8e a1 e7 2b fb 76 c7 54 ac ............v.T. 0010: c0 a7 17 20 2a bc 1e 98 82 59 eb 2b ad cb 28 25 .........Y...... 0020: 3f c5 3d 3a 5f fa 00 00 0c 00 9d 00 3d 00 3c 00 ................ 0030: 35 00 2f 00 0a 01 00 00 47 ff 01 00 01 00 00 0d 5.......G....... 0040: 00 1a 00 18 06 03 06 01 05 03 05 01 04 03 04 01 ................ 0050: 03 03 03 01 02 03 02 01 04 02 02 02 00 17 00 00 ................ 0060: 00 00 00 1c 00 1a 00 00 17 61 77 73 63 61 74 32 .........ramserver 0070: 2e 68 73 62 63 2e 73 79 6d 63 6f 72 2e 62 69 7a .austin.ibm.com
    [9/4/18 8:16:55:578 PDT] 000000eb SystemOut O WebContainer : 2, READ: TLSv1.2 Alert, length = 2
    [9/4/18 8:16:55:578 PDT] 000000eb SystemOut O WebContainer : 2, RECV TLSv1.2 ALERT: warning, unrecognized_name
    [9/4/18 8:16:55:578 PDT] 000000eb SystemOut O SSL - handshake alert: unrecognized_name
    [9/4/18 8:16:55:579 PDT] 000000eb SystemOut O WebContainer : 2, handling exception: javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name

The server does not recognize the hostname that the client is setting ramserver.austin.ibm.com. You can disable SNI support per JVM

Standalone Application Server

  • In the Administration Console select Servers Expand Server Type and select WebSphere application servers
  • Click on the name of your server server1
  • Expand Java and Process Management and select Process Definition
  • Under the Additional Properties section, click Java Virtual Machine
  • Scroll down and locate the textbox for Generic JVM arguments
  • -Djsse.enableSNIExtension=false
  • Click on Apply and Save the change
  • Restart the WAS and Test it again
Deployment Manager
  • In the Administration Console, select System Administration Select Deployment Manager In the Server Infrastructure section
  • Expand Java and Process Management and select Process Definition
  • Under the Additional Properties section
  • click Java Virtual Machine.
  • Scroll down and locate the textbox for Generic JVM arguments
  • -Djsse.enableSNIExtension=false
  • Click on Apply and Save the change
  • Restart the dmgr and Test it again
For more details see about SNI custom property the following link:

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Component":"SSL","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
30 January 2020

UID

ibm11285270

Page Feedback