IBM Support

[Db2] Db2 Text Search rarely gets Out Of Memory Error even not hit maxHeapSize yet

Question & Answer


Question

Why Db2 Text Search got Out Of Memory Error even maxHeapSize 4000M and not hit max yet?

Cause

It is an known limitation of Db2 V11.1 GA to V11.1 Mod 4 Fix Pack 5, and V11.5 GA to V11.5 Mod 4 Fix Pack 0 Text Search. 
This limitation will be removed on Db2 V11.1 Mod 4 Fix Pack 6 (planning as of date Nov 2020) and V11.5 Mod 5 Fix Pack 0 (released on Nov 2020) because both have newer ECMTS version.

Answer

In heavy workload situation, Db2 Text Search maxHeapSize can be increased to 4000M by similar as below command for stable system:
$ configTool set -system -maxHeapSize 4000M -configPath /home/db2inst1/sqllib/db2tss/config
But Used Heap Memory size in monitorX.csv files tells it did not reach the configured maxHeapSize yet.
(See DB2 Text Search gets IQQG0020E java.lang.OutOfMemoryError how to find it did not reach yet)
And there is an Out Of Memory Error (OOME) message in defaultX.log as similar as below:
-----
<record>
    <date>2020-11-09T07:41:36</date>
    <millis>1604875296406</millis>
    <sequence>80338</sequence>
    <logger>com.ibm.es.nuvo.inyo.util.OOMEHandler</logger>
    <level>WARNING</level>
    <class>com.ibm.es.nuvo.inyo.util.OOMEHandler</class>
    <method>handleNotification</method>
    <thread>61</thread>
    <message>Running low on heap space. Calling the Out Of Memory Error callback method in an attempt to prevent an Out Of Memory Error from occurring</message>
</record>
<record>
    <date>2020-11-09T07:41:36</date>
    <millis>1604875296407</millis>
    <sequence>80339</sequence>
    <logger>com.ibm.es.nuvo.inyo.util.OOMEShutDown</logger>
    <level>WARNING</level>
    <class>com.ibm.es.nuvo.inyo.util.OOMEShutDown</class>
    <method>shutDown</method>
    <thread>61</thread>
    <message>Memory Pool Information:
</record>
-----
Also, below message can be found in traceX.log when the server is restarted:
-----
<record>
    <date>2020-11-09T08:19:33</date>
    <millis>1604877573060</millis>
    <sequence>11</sequence>
    <logger>com.ibm.es.nuvo.inyo.util.OOMEHandler</logger>
    <level>INFO</level>
    <class>com.ibm.es.nuvo.inyo.util.OOMEHandler</class>
    <method>listen</method>
    <thread>1</thread>
    <message>OOMEHandler will not listen to the Heap memory - tenured-LOA memory pool because its maximum size is 200 megabytes and the threshold requested is 50 megabytes. The maximum allowed threshold is 10.0% of the heap's size (approximately 20 megabytes).</message>
</record>
-----
This symptom may be caused by an known Content Search Services (CSS) problem as described at below technote :
OOMEHandler will not listen to the Heap memory - tenured-LOA memory pool 
From Db2 Text Search perspective, it can be worked around by below steps.
1. db2ts "stop for text"
2. Edit startup.sh/bat, insert -XX:+HeapManagementMXBeanCompatibility after HEAP_SIZE as shown above technote
3. db2ts "start for text"
We will see one line "pool=Java heap" line at MemoryPoolMXBean information tag when it's applying as similar as below:
    <message>MemoryPoolMXBean information:
pool=Java heap, type=Heap memory, usage=init ...
pool=class storage, type=Non-heap memory, usage=init ...
pool=miscellaneous non-heap storage, usage=init ...
pool=JIT code cache, type=Non-heap memory, usage=init ...
pool=JIT data cache, type=Non-heap memory, usage=init ...
</message>
For our reference, here is a sample "before" the parameter applied MemoryPoolMXBean information tag:
    <message>MemoryPoolMXBean information:
pool=tenured-SOA, type=Heap memory, usage=init ...
pool=tenured-LOA, type=Heap memory, usage=init ...
pool=nursery-allocate, type=Heap memory, usage=init ...
pool=nursery-survivor, type=Heap memory, usage=init ...
pool=class storage, type=Non-heap memory, usage=init ...
pool=miscellaneous non-heap storage, type=Non-heap memory, usage=init ...
pool=JIT code cache, type=Non-heap memory, usage=init ...
pool=JIT data cache, type=Non-heap memory, usage=init ...
</message>
 

[{"Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSEPGG","label":"Db2 for Linux, UNIX and Windows"},"ARM Category":[{"code":"a8m500000008PlsAAE","label":"Extenders->Text Search"}],"ARM Case Number":"TS004472234","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"11.1.0;11.5.0"}]

Document Information

Modified date:
06 December 2020

UID

ibm16373462

Page Feedback