Troubleshooting
Problem
During the execution of the testcon20 or testconn40 program with the "-dtc" option, an error occurs while creating the XA connection. The error is "SQL0998N Error occurred during transaction or heuristic processing. Reason Code = "16". Subcode = "2-FFFFFFFF80004005".".
Symptom
1) You might get error messages in the db2diag.log
2014-04-21-12.24.43.626000-240 I3739F406 LEVEL: Severe
PID : 4432 TID : 8140 PROC : testconn20.exe
INSTANCE: DB2 NODE : 000
APPID : 109.109.109.109.690637.140421162441
EDUID : 8140
FUNCTION: DB2 UDB, XA DTP Support, MicrosoftXARMCreate, probe:364
DATA #1 : String, 69 bytes
HRESULT: 80004005
HRESULT Message: Unspecified error
ITransaction:
2014-04-21-12.24.43.626000-240 I4147F748 LEVEL: Info
PID : 4432 TID : 8140 PROC : testconn20.exe
INSTANCE: DB2 NODE : 000
APPID : 109.109.109.109.60637.140421162441
EDUID : 8140
FUNCTION: DB2 UDB, oper system services, sqlofica, probe:10
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -998 sqlerrml: 21
sqlerrmc: 16 2-FFFFFFFF80004005
sqlerrp : SQLXACOM
sqlerrd : (1) 0x00000000 (2) 0x00000010 (3) 0x00000000
(4) 0x00000003 (5) 0x00000000 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate: 58005
2) You are able to reproduce the error by running testconn utility with -dtc option:
testconn40 -dtc "database=dbName;server=serverName:portNumber;uid=userName;pwd=password"
Adding the MSDTC step:
Step 1: Printing version info
.NET Framework version: 2.0.50727.5477
64-bit
DB2 .NET provider version: 9.0.0.2
DB2 .NET file version: 9.7.5.2
Capability bits: ALLDEFINED
Build: 20111018
Factory for invariant name IBM.Data.DB2 verified
Factory for invariant name IBM.Data.Informix verified
IBM.Data.Informix from DbFactory is the Common Informix .NET provider
Elapsed: 0.6084117
Step 2: Validating db2dsdriver.cfg against db2dsdriver.xsd schema file
File C:\ProgramData\IBM\DB2\DB2COPY1\cfg\db2dsdriver.cfg missing. Nothing to validate
Elapsed: 0
Step 3: Connecting using "database=dbName;server=serverName:portNumber;uid=userName;pwd=password"
Server type and version: OS/390 10.01.0005
Elapsed: 2.028039
Step 4: Selecting rows from SYSIBM.systables to validate existence of packages
SELECT * FROM SYSIBM.systables FETCH FIRST 5 rows only
Elapsed: 0.3276063
Step 5: Calling GetSchema for tables to validate existence of schema functions
Elapsed: 3.1512606
Step 6: Creating XA connection
DB2TransactionScope: Failed to open connection to database!
IBM.Data.DB2.DB2Exception: ERROR [58005] [IBM][DB2] SQL0998N Error occurred during transaction or heuristic processing. Reason Code = "16". Subcode = "2-FFFFFFFF80004005".
at IBM.Data.DB2.DB2ConnPool.Open(DB2Connection connection, String& szConnectionString, DB2ConnSettings& ppSettings, Object& ppConn)
at IBM.Data.DB2.DB2Connection.Open()
at TestConn.DB2TransactionScope.Open(String connstr)
Looking for XADLL key for C:\Program Files\IBM\SQLLIB\bin\db2app.dll
Correct entry found
Check if XA Transactions are enabled:
Windows 7, 2008, Vista -
Component Services, Computers, My Computer,
Distributed Transaction Coordinator, Local DTC
Properties, Security tab, Enable XA Transactions
Other windows versions -
Component Services, Computers, My computer, Properties
MSDTC tab, Security configuration, Enable XA Transactions
Test failed.
Cause
Error comes during loading of library db2app.dll (db2app64.dll on 64-bit machines). The library is expected in <install_path>\BIN directory. Assuming the file is there. It is possible the MSDTC process cannot load it due to access permissions or if DB2PATH variable is not set properly.
Diagnosing The Problem
- Check Windows event viewer for any error messages at the time when application is giving SQL0998N.
- Check whether the system PATH points to the right SQLLIB\bin directory.
- Check whether the DB2PATH contains backslash at the end.
- Check whether the DB2INSTANCE environment variable points to the right instance.
- Check whether the MSDTC process can write to the db2diag.log.
- Check whether the NETWORK SERVICE account has read-execute permissions to the folder containing the XA DLL.
- Check whether the NETWORK SERVICE account has access to the Db2 Windows registry keys.
- Check whether Db2 32-bit is installed on a 64-bit Windows operating system.
- Check whether MSDTC is returning an error code of 8004D01B as it might need to be reinstalled.
- Check that "Enable XA transactions","Network DTC access" and "Outbound connections" are selected in the MSDTC component security tab.
- Check whether the MSDTC process has read-execute privilege to the SSL stash and key files and check the length of path to these files.
- If connections with SSL are used, check whether the path to the GSK and ICC libraries was added to the Windows environment PATH variable.
Resolving The Problem
- Check the Windows Event Viewer.
If you are seeing any error from MSDTC, fix the issue and run testconn20 or testconn40. - Check the system PATH points to the right SQLLIB\bin directory
The system PATH (not current user's one) has to point to the right SQLLIB\bin directory so the MSDTC service can load the db2app.dll library. If you make any changes to the system PATH, either restart the MSDTC service or the whole server. - Check the DB2PATH does not contain backslash at the end.
Check the DB2PATH registry variable by using the "db2set -all" command. DB2PATH cannot contain backslash at the end. This can prevent the MSDTC service to load the db2app.dll library.
db2set -all
[e] DB2PATH=C:\Program Files\IBM\SQLLIB\
[i] DB2INSTPROF=C:\Documents and Settings\All Users\Application Data\IBM\DB2\DB2 COPY 1
[i] DB2COMM=TCPIP
[g] DB2PATH=C:\Program Files\IBM\SQLLIB\
[g] DB2INSTDEF=DB2
To update the DB2PATH setting, open the Windows registry by using the "regedit" program and change the value of DB2PATH found under HKEY_LOCAL_MACHINE\SOFTWARE\IBM\DB2\InstalledCopies\<DB2 Copy Name>\GLOBAL_PROFILE - Check the DB2INSTANCE points to the right instance.
For installations where there is more than one DB2 instance present, make sure the system environment variable DB2INSTANCE points to the right instance so the MSDTC process can work with the correct libraries. - Allow the MSDTC to write to the db2diag.log.
In order for the MSDTC process to report any errors during the establishment of a XA connection using the Db2 XA DLL, the MSDTC process must have write access to the db2diag.log file. Without this access, any error originating in the Db2 library invoked from within the MSDTC is not logged, which could lead to false assumption that Db2 is configured properly. - Allow the NetworkService account access to the folder containing the XA DLL.
If no error or cannot load db2app.dll is reported by the Windows Event Viewer, allow the NetworkService account access to the folder containing the XA DLL. Since NetworkService a restricted account, you need to provide this account with access to the folder where the XA DLL is located; for instance by modifying the access control list (ACL) for the Db2 folder SQLLIB where db2app.dll is located.
If Extended Security enabled, add the NT AUTHORITY\NetworkService account to either the DB2USERS or DB2ADMNS group.
Otherwise, Granting File Access to Network Service manually.
To provide access to an ASP.NET application running as Network Service, you must grant access to the Network Service account:
a. In Windows Explorer, locate and SQLLIB folder (the db2level command can also give that)
b. Right-click the file and then click Properties.
c. In the Properties dialog box, click the Security tab.
d. On the Security tab, examine the list of users. If the Network Service account is not listed, add it.
e. In the Properties dialog box, click the Network Service user name, and in the Permissions for NETWORK SERVICE section, select the Read, Write, and Modify permissions.
f. Click Apply, and then click OK. - Allow the NetworkService account access to the Db2 Windows registry keys.
The MSDTC process requires read access to the Db2 Windows registry subtree HKEY_LOCAL_MACHINE\SOFTWARE\IBM\DB2 to properly initialize Db2 environment when the testconn20 or testconn40 is initiating a XA connection. Insufficient access permissions manifest in Db2 trace errors like this:
| sqloInitEnvironment entry
| | sqloPRegQueryDefaultValue entry
| | | sqloGetDefGlobProfKey entry
| | | | sqloGetCurrentInstallationName entry
| | | | | sqloGetThisDllPathFromDllInit entry
| | | | | sqloGetThisDllPathFromDllInit data [probe 80]
| | | | | sqloGetThisDllPathFromDllInit exit
| | | | sqloGetCurrentInstallationName data [probe 10]
| | | | | sqloGetInstallNameFromDllPath entry
| | | | | sqloGetInstallNameFromDllPath data [probe 10]
| | | | | sqloGetInstallNameFromDllPath exit [rc = 0x840F0001 = -2079391743 = SQLO_ACCD]
| | | | sqloGetCurrentInstallationName exit [rc = 0x870F0150 = SQLO_INSTALLATION_NAME_NOT_FOUND]
To fix this problem grant read access to the account used for running the MSDTC process, which is typically "NT AUTHORITY\NetworkService". - MSDTC is 64-bit on a 64-bit Windows and is 32-bit on a 32-bit Windows. Therefore, the Db2 driver that matches the bitness of the operating system is required in order to create a XA connection. If 32-bit Db2 is installed on a 64-bit Windows platform, the MSDTC connection fails as the 64-bit DLL is not available. Ensure that 32-bit DB2 is installed on 32-bit Windows and 64-bit Db2 is installed on 64-bit Windows. To verify whether Db2 is 32-bit or 64-bit, issue the "db2level" command, which shows if the instance is 32-bit or 64-bit. Note: A 32-bit application running on 64-bit Windows can create a XA connection when using a 64-bit Db2 installation as it ships the 64-bit driver DLL required by MSDTC.
- MSDTC might return a return code of 8004D01B.
For example,
2014-11-07-13.07.00.655000-240 I13459876F464 LEVEL: Severe
PID : 35476 TID : 25116 PROC : TransactionTest.exe
INSTANCE: DB2 NODE : 000 APPID : 9.111.123.50.50268.160130544299
HOSTNAME: TESTSERV
EDUID : 25116
FUNCTION: DB2 UDB, XA DTP Support, MicrosoftXARMCreate, probe:364
DATA #1 : String, 93 bytes
HRESULT: 8004D01B
HRESULT Message: The Transaction Manager is not available.
If so, reinstalling MSDTC might help resolve the problem. Instructions on how to reinstall MSDTC can be found here. Before reinstalling MSDTC, you will want to back up the registry key HKLM\SOFTWARE\Microsoft\MSDTC\XADLL as the DB2 specific keys will need to be restored after MSDTC is reinstalled. - Check that "Enable XA transactions","Network DTC access" and "Outbound connections" are selected in the MSDTC component security tab for all local or clustered MSDTC.
The Component Services -> Computer -> My Computer -> Distributed Transaction Coordinator lists local and clustered MSDTC. Make sure to checkmark "Network DTC access", "Outbound connections" and "Enable XA transactions" in the security tab of the "Local DTC" and clustered DTC properties. - Check whether the MSDTC process has read-execute privilege to the SSL stash and key files and check the length of path to these files.
When XA is configured to use the SSL connections, the SSL stash and key files needs to be accessible by the MSDTC process to successfully establish the SSL connection.
In addition, the same error is returned if the length of the first parameter (XA_OPEN string) of MS API XARMCreate() exceeds the API limit.
Before Windows 10, the maximum length of the first parameter to Microsoft's XARMCreate() function is 256 characters. On Windows 10, Microsoft increased the XA_OPEN string limit up to 3072 characters. The XA_OPEN string is the parameter into which all the DSN information is passed from Db2 to MSDTC. All SSL, DTC, and connection-related properties and few other parameters essential to MSDTC must fit within the Windows 7 256 character limit.
https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ms679248(v%3Dvs.85)
Anything that can reduce duplication of information (such as removing information already in the db2dsdriver.cfg from the application connection string) or reduce information that is not essential (such as fully qualified host name where a shorter host alias can be obtained from DNS lookup) allows the user to work with as much room within that 256 character limit as possible. Reduce the path length of the stash and key file paths to the minimum possible as these are also part of the XA_OPEN string.
Since Windows 10 the XA_OPEN string limit increased up to 3072 characters, this issue does not occur on Windows 10 platforms. - If connections with SSL are used, check whether the path to the GSK and ICC libraries was added to the Windows environment PATH variable.
If testconn20 or testconn40 is run more than once, a known issue can be triggered resulting in the error. The initial run is successful but the subsequent execution fails with the error. Upon restarting MSDTC, the execution works again once.
The issue is that after the first execution some libraries are not properly unloaded from the MSDTC, which triggers a code issue in the handling of the ICC and GSK libraries needed to handle SSL connections.
The workaround is adding the path to the ICC and GSK libraries to the PATH environment variable (and restart MSDTC). The path is usually C:\Program Files\IBM\<install name>\BIN\ICC64 for the 64-bit version. For the 32-bit version, the path is usually C:\Program Files\IBM\<install name>\BIN\ICC. C:\Program Files\IBM\<install name>\BIN is normally already be part of the PATH environment variable (depending on the type of the installation).
Related Information
Was this topic helpful?
Document Information
Modified date:
07 October 2022
UID
swg21673341