Question & Answer
Question
What is the work around when DB2 Text Search gets IQQG0020E java.lang.OutOfMemoryError: Java heap space?
Answer
While running DB2 Text Search, "db2ts START FOR TEXT", it may fail with OutOfMemoryError and the db2diag.log file has the CIE00760 message, as shown below:
---------------------
2015-04-23-13.01.09.364000+540 I72992120F587 LEVEL: Severe
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 192.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, Text Search, CieError::makeDiagnosticMessage, probe:1
MESSAGE : CIE reason
DATA #1 : Hexdump, 4 bytes
0x0000009D499A71B8 : F802 0000 ....
---------------------
2015-04-23-13.01.09.364000+540 I72992120F587 LEVEL: Severe
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 192.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, Text Search, CieError::makeDiagnosticMessage, probe:1
MESSAGE : CIE reason
DATA #1 : Hexdump, 4 bytes
0x0000009D499A71B8 : F802 0000 ....
2015-04-23-13.01.09.380000+540 I72992709F565 LEVEL: Severe
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 19.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, trace services, sqlt_logerr_string (secondary logging function), probe:30
MESSAGE : Message
DATA #1 : String, 29 bytes
CIE00760 .v...g.R...E.G...[.B
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 19.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, trace services, sqlt_logerr_string (secondary logging function), probe:30
MESSAGE : Message
DATA #1 : String, 29 bytes
CIE00760 .v...g.R...E.G...[.B
2015-04-23-13.01.09.427000+540 I72993276F708 LEVEL: Warning
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 192.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, oper system services, sqloInvokeInterruptible, probe:15
DATA #1 : String, 208 bytes
An unfenced User Defined Function (UDF) was abnormally terminated.
This may leave the DB2 engine in inconsistent state.
It is recommended that DB2 server instance is stopped and restarted as soon as possible.
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 192.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, oper system services, sqloInvokeInterruptible, probe:15
DATA #1 : String, 208 bytes
An unfenced User Defined Function (UDF) was abnormally terminated.
This may leave the DB2 engine in inconsistent state.
It is recommended that DB2 server instance is stopped and restarted as soon as possible.
2015-04-23-13.01.09.458000+540 I72993986F592 LEVEL: Error
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 192.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, runtime interpreter, sqlri_tfcls, probe:191
RETCODE : ZRC=0xFFFFFE52=-430
SQL0430N User defined function "" (specific name "") has abnormally
terminated.
PID : 1172 TID : 2656 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : SAMPLE
APPHDL : 0-2761 APPID: 192.168.0.0.xxxx
AUTHID : DB2INST1 HOSTNAME: MYHOST1
EDUID : 2656 EDUNAME: db2agent (SAMPLE) 0
FUNCTION: DB2 UDB, runtime interpreter, sqlri_tfcls, probe:191
RETCODE : ZRC=0xFFFFFE52=-430
SQL0430N User defined function "" (specific name "") has abnormally
terminated.
---------------------
From db2tss/log/default0.log
---------------------
<record>
<date>2015-04-23T13:01:09</date>
<millis>1429761669239</millis>
:
<message>Throwable while running listener: IQQG0020E java.lang.OutOfMemoryError: Java heap space
:
</record>
---------------------
---------------------
<record>
<date>2015-04-23T13:01:09</date>
<millis>1429761669239</millis>
:
<message>Throwable while running listener: IQQG0020E java.lang.OutOfMemoryError: Java heap space
:
</record>
---------------------
From db2tss/log/serverConfiguration.log
---------------------
maxHeapSize = 1500M
---------------------
---------------------
maxHeapSize = 1500M
---------------------
From the command below, date 2015-04-23, which is the day of OutOfMemoryError, can be found at monitor0.csv file under db2tss/log directory.
$ grep -l '2015-04-23' monitor*
monitor0.csv
$ grep -l '2015-04-23' monitor*
monitor0.csv
MonitorX.csv file contains below information and 17th column shows Used Heap Memory (MB).
---------------------
Time(sec),
Total number of processed documents,
Total size of processed documents (KB),
Total number of add requests that failed,
Total number of successful add requests,
Total size of successful add requests (KB),
Total number of delete requests that failed,
Total number of successful delete requests,
Documents in the input queue,
Input queue size (in bytes),
Documents in the output queue,
Output queue size (in bytes),
Documents waiting for preprocessing,
Documents currently in preprocessing,
Documents waiting for indexing,
Documents currently being indexed,
Used Heap Memory (MB),
Thread Count,
System Load,
Open File Descriptors,
Free Physical Memory,
Batches,
activeMerges,
mergesSizeMB
---------------------
---------------------
Time(sec),
Total number of processed documents,
Total size of processed documents (KB),
Total number of add requests that failed,
Total number of successful add requests,
Total size of successful add requests (KB),
Total number of delete requests that failed,
Total number of successful delete requests,
Documents in the input queue,
Input queue size (in bytes),
Documents in the output queue,
Output queue size (in bytes),
Documents waiting for preprocessing,
Documents currently in preprocessing,
Documents waiting for indexing,
Documents currently being indexed,
Used Heap Memory (MB),
Thread Count,
System Load,
Open File Descriptors,
Free Physical Memory,
Batches,
activeMerges,
mergesSizeMB
---------------------
From the command below, monitor0.csv shows the top of five peaks of Used Heap Memory (MB) on 2015-04-23.
$ grep '2015-04-23' monitor0.csv | awk -F',' '{print $17}' | uniq | sort -rn | head -5
1493
1492
1492
1491
1488
$ grep '2015-04-23' monitor0.csv | awk -F',' '{print $17}' | uniq | sort -rn | head -5
1493
1492
1492
1491
1488
Top peak 1493 is nearly equal to the size of maxHeapSize = 1500M, can be found at serverConfiguration.log
So IQQG0020E java.lang.OutOfMemoryError could be caused by this limit.
So IQQG0020E java.lang.OutOfMemoryError could be caused by this limit.
Try to tune maxHeapSize, try double (or more) and run it again to see if it is caused by this limit or not.
If it works, confirm the command below, find out the new value testing timestamp.
$ grep -l '2015-04-25' monitor*
monitor3.csv
If it works, confirm the command below, find out the new value testing timestamp.
$ grep -l '2015-04-25' monitor*
monitor3.csv
From the command below, monitor3.csv shows the top of five peaks of Used Heap Memory (MB) on 2015-04-25.
$ grep '2015-04-25' monitor3.csv | awk -F',' '{print $17}' | uniq | sort -rn | head -5
2714
2527
2526
2525
2524
$ grep '2015-04-25' monitor3.csv | awk -F',' '{print $17}' | uniq | sort -rn | head -5
2714
2527
2526
2525
2524
As above, the system needed 2714 MB but there was only 1500 MB so it failed on 2015-04-23.
It this particular case, maxHeapSize = 3000M should work around. Here is a sample command to
apply 3000M. Before run it, need to run 'db2ts "stop for text"'.
$ configTool set -system -maxHeapSize 3000M -configPath /home/db2inst1/sqllib/db2tss/config
It this particular case, maxHeapSize = 3000M should work around. Here is a sample command to
apply 3000M. Before run it, need to run 'db2ts "stop for text"'.
$ configTool set -system -maxHeapSize 3000M -configPath /home/db2inst1/sqllib/db2tss/config
Unfortunately, there is no calculating formula for maxHeapSize and it depends on number of indexes and its size. For preventing this type of OutOfMemory error, may need to monitor monitorX.csv files and confirm whether enough space or not.
Note:
If this technote could not help your situation, please contact technical service and ask some help.
monitorX.csv is an IBM internal file and its format may be changed without notice in the future.
If its changed, then users will have to adjust their procedures or will not apply this method.
This behavior might be changed without notice in the future. We can confirm whether this technote
is valid or not by following the above steps.
If this technote could not help your situation, please contact technical service and ask some help.
monitorX.csv is an IBM internal file and its format may be changed without notice in the future.
If its changed, then users will have to adjust their procedures or will not apply this method.
This behavior might be changed without notice in the future. We can confirm whether this technote
is valid or not by following the above steps.
Related Information
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSEPGG","label":"Db2 for Linux, UNIX and Windows"},"ARM Category":[{"code":"a8m500000008PlsAAE","label":"Extenders-\u003EText Search"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"All Version(s)","Line of Business":{"code":"LOB10","label":"Data and AI"}}]
Was this topic helpful?
Document Information
Modified date:
07 December 2022
UID
swg21902600