IBM Support

How to troubleshoot and debug the javax.crypto.AEADBadTagException

Troubleshooting


Problem

The AEADBadTagException is thrown when the cryptographic integrity check on the data fails during decryption. This is commonly seen with AEAD block ciphers like AES in GCM mode. (Example cipher SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256)
 SystemErr                                                    R javax.crypto.AEADBadTagException
 SystemErr                                                    R     at com.ibm.crypto.provider.GCTRInHardware.gcm_ad(Unknown Source)
 SystemErr                                                    R     at com.ibm.crypto.provider.aI.c(Unknown Source)
 SystemErr                                                    R     at com.ibm.crypto.provider.AESGCMCipher.engineDoFinal(Unknown Source)
 SystemErr                                                    R     at javax.crypto.Cipher.doFinal(Unknown Source)
 SystemErr                                                    R     at com.ibm.jsse2.n.a(n.java:279)
 SystemErr                                                    R     at com.ibm.jsse2.b.a(b.java:12)
 SystemErr                                                    R     at com.ibm.jsse2.av.a(av.java:346)
 SystemErr                                                    R     at com.ibm.jsse2.av.i(av.java:282)

Symptom

Sysmptom for this error causes:
    a) Data Corruption: Occurs if the ciphertext or associated data is modified or corrupted.
    b) Key Mismatch: Using a different key for decryption than was used for encryption.
    c) Nonce/IV Reuse: Reusing the nonce or initialization vector for multiple encryptions with the same key.
    d) Incorrect Tag Handling: The authentication tag is either incorrect or improperly handled.

Cause

AEAD ciphers provide both confidentiality and integrity assurances for encrypted data, ensuring that the data hasn't been tampered with or altered. The exception AEADBadTagException is thrown when the authentication tag in an AEAD cipher does not match the expected value. This usually means that the data has been tampered with, the key used for encryption/decryption is incorrect, or the data is corrupted somehow over the network.
This exception typically occurs when decrypting data. If the decryption process fails to verify the integrity of the data (using the authentication tag), this exception is thrown to indicate that the decrypted data might be corrupt or that the authentication failed due to incorrect keys, tampering, or other issues.

Diagnosing The Problem

 Troubleshooting Tips
Please make sure to capture tcdump + SSL traces on the SSL Client side and SSL Server side.
 
     The recommendation to engage with the network team and verify the server-side encryption process is a crucial step in troubleshooting the javax.crypto.AEADBadTagException is checking with your network team about any data corruption or unintended modification that might have occurred during transmission or Ensure that the data has not been altered on each endpoint over the network.  They need to check on both sides and also ensure the server side team is encrypting data correctly with the expected tag like Verifying that the encryption and decryption processes use the same cryptographic keys, algorithms, and verifying the supplied authentication tag.
     Setting up Wireshark on Both sides, SSL client side and SSL server side:
1.    Start Wireshark and begin capturing traffic on the interface communicating with the server.
2.    Use the display filter ssl (or tls depending on your Wireshark version) to focus on encrypted traffic.
3.    If possible, configure Wireshark with the server's private key (if it's RSA) or the pre-master secret (if it's some other form of encryption). This will allow Wireshark to decrypt the TLS traffic for easier analysis.
4. After the handshake, focus on the application data where the actual encrypted content is.
How to Identifying the Problematic Packets
After enabling debugging, the server will log detailed information about the SSL/TLS handshakes. Please be sure to look for the ClientHello message with the random value you've identified in the client-side packet capture.
Tracking a specific SSL handshake in server-side logs using the ClientHello random value can be quite useful in debugging scenarios, especially in environments with high volumes of traffic. The ClientHello random value is a unique value sent by the client during the initial stage of an SSL/TLS handshake. You can correlate specific sessions by identifying this value in both client-side packet captures and server-side packet logs.
Enabled SSL traces using the following mustgather
MustGather: SSL problems on WebSphere Traditional
https://www.ibm.com/support/pages/node/707663
MustGather: SSL problems on WebSphere Liberty
https://www.ibm.com/support/pages/node/561067
From the provided SSL trace, it's clear that you are analyzing the SSL/TLS handshake process, specifically focusing on the client's random key.  The following SSL trace is an example.
The server_name extension shows the hostname being connected to, in this case, remoteSSLserver.xyz.com. This is part of the Server Name Indication (SNI), which allows the client to specify the hostname to connect to at the start of the handshake process.
The client provides 32 bytes of random data, which is crucial in the SSL/TLS handshake. This randomness is used to ensure the uniqueness of each session and contribute to the security of the key exchange process.
The client random key in your trace is
78497893c41e93712d7f54ebe2b3a08f4a3206d97b98a5157e61fff65e21643b
[11/6/23 23:36:51:944 UTC] 00000079 SystemOut     O   Extension server_name, server_name: [type=host_name (0), value=remoteSSLserver.xyz.com]
[11/6/23 23:36:51:945 UTC] 00000079 SystemOut     O   ***
[11/6/23 23:36:51:958 UTC] 00000079 SystemOut     O   0000: 16 03 03 00 ec 01 00 00  e8 03 03 78 49 78 93 c4  ...........eIx..
0010: 1e 93 71 2d 7f 54 eb e2  b3 a0 8f 4a 32 06 d9 7b  ..q..T.....J2...
0020: 98 a5 15 7e 61 ff f6 5e  21 64 3b 00 00 54 c0 24  ....a....d...T..
0030: c0 28 00 3d c0 26 c0 2a  00 6b 00 6a c0 0a c0 14  .........k.j....
0040: 00 35 c0 05 c0 0f 00 39  00 38 c0 23 c0 27 00 3c  .5.....9.8......
0050: c0 25 c0 29 00 67 00 40  c0 09 c0 13 00 2f c0 04  .....g..........
0060: c0 0e 00 33 00 32 c0 2c  c0 2b c0 30 00 9d c0 2e  ...3.2.....0....
To locate a packet from  Wireshark you might use a filter like ssl.handshake.random or tls.handshake.random in the filter expression. Example sl.handshake.random contains 78:49:78:93:c4 or tls.handshake.random == 78:49:78:93:c4:1e:93 with complete random key number.
In your exception message, it refers to a Java class com.ibm.crypto.provider.GCTRInHardware, It deals with GCM (Galois/Counter Mode) authenticated encryption with associated data (AEAD). This particular class name suggests it is designed to take advantage of Just-In-Time (JIT) compilation for performance optimization and possibly hardware acceleration for cryptographic functions.
 SystemErr                                                    R javax.crypto.AEADBadTagException
 SystemErr                                                    R     at com.ibm.crypto.provider.GCTRInHardware.gcm_ad(Unknown Source)
 SystemErr                                                    R     at com.ibm.crypto.provider.aI.c(Unknown Source)
 SystemErr                                                    R     at com.ibm.crypto.provider.AESGCMCipher.engineDoFinal(Unknown Source)
 SystemErr                                                    R     at javax.crypto.Cipher.doFinal(Unknown Source)
 SystemErr                                                    R     at com.ibm.jsse2.n.a(n.java:279)
 SystemErr                                                    R     at com.ibm.jsse2.b.a(b.java:12)
 SystemErr                                                    R     at com.ibm.jsse2.av.a(av.java:346)
 SystemErr                                                    R     at com.ibm.jsse2.av.i(av.java:282)

 
If data transmission is good over the network on both sides (no data corruption or alteration).  There is a chance that the exception is occurring due to an issue in the hardware acceleration used by the IBM Java JIT runtime during the decryption process. To determine whether this is contributing to the problem. Please make sure you are using the latest IBM JDK to avoid any known issues with JIT.
To narrow down the problem, run a test with hardware acceleration disabled.
In Liberty Side, along with SSL traces
Hardware acceleration can be disabled using the following JVM properties under jvm.options file.

-Dcom.ibm.jit.crypto.nohardwaredigest=true
-Dcom.ibm.jit.crypto.nohardwarecrypt=true
image-20231119102013-2
In  WebSphere Traditional, along with SSL traces
Hardware acceleration can be disabled using the following JVM properties under Servers > Server Types > WebSphere Application Servers > server_name > Expand Java and Process Management (under Server Infrastructure) >Process definition > Java Virtual Machine > add the following properties.
-Dcom.ibm.jit.crypto.nohardwaredigest=true
-Dcom.ibm.jit.crypto.nohardwarecrypt=true
image-20231119101818-1

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m50000000CdL6AAK","label":"WebSphere Application Server traditional-All Platforms-\u003ESecurity-\u003ESSL-\u003ESSL - SDK-JAVA"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
19 November 2023

UID

ibm17080017

Page Feedback