APAR status
Closed as program error.
Error description
IBM MQ processes may be seen to be using a large amount of virtual memory (VSZ), for example: S 100 15572 1 814m 30888 0:0 04:31 00:00:00 /opt/mqm/bin/amqzxma0 -m S 100 15580 15572 353m 8376 0:0 04:31 00:00:00 /opt/mqm/bin/amqzfuma -m S 100 15585 15572 1962m 14476 0:0 04:31 00:00:00 /opt/mqm/bin/amqzmuc0 -m S 100 15604 15572 735m 7748 0:0 04:31 00:00:00 /opt/mqm/bin/amqzmur0 -m S 100 15609 15572 803m 10360 0:0 04:31 00:00:00 /opt/mqm/bin/amqzmuf0 -m S 100 15613 15572 289m 10616 0:0 04:31 00:00:00 /opt/mqm/bin/amqrrmfa -m S 100 15630 15572 185m 6004 0:0 04:31 00:00:00 /opt/mqm/bin/amqzmgr0 -m S 100 15632 15609 416m 10324 0:0 04:31 00:00:00 /opt/mqm/bin/amqfqpub -mT S 100 15642 15630 343m 6856 0:0 04:31 00:00:00 /opt/mqm/bin/runmqchi -m S 100 15643 15630 201m 6288 0:0 04:31 00:00:00 /opt/mqm/bin/amqpcsea TES S 100 15644 15632 673m 11160 0:0 04:31 00:00:00 /opt/mqm/bin/amqfcxba -m S 100 15649 15572 797m 9964 0:0 04:31 00:00:00 /opt/mqm/bin/amqzlaa0 -mT
Local fix
It may be possible to work around the issue by exporting this environment variable before starting an IBM MQ queue manager: MALLOC_ARENA_MAX=1 Some malloc() implementations however will ignore the environment variable for setuid/setgid binaries such as those used by IBM MQ.
Problem summary
**************************************************************** USERS AFFECTED: Users running IBM MQ on an OS that uses an implementation of malloc() which utilises per-thread memory arenas may be affected by this issue. This issue may particularly affect customers that run IBM MQ on a machine with many CPU cores. Platforms affected: Linux on x86-64, Linux on x86, Linux on S390, Linux on Power, Linux on zSeries **************************************************************** PROBLEM DESCRIPTION: This URL gives an overview of the issue caused by assigning threads their own memory pools: https://www.ibm.com/developerworks/community/blogs/kevgrig/entry /linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memor y_usage?lang=en IBM MQ processes such as the IBM MQ agent (amqzlaa0) and channel (amqrmppa) are highly multi-threaded. This makes IBM MQ processes particularly vulnerable to this issue on a machine with a large number of cores as there is high contention for the malloc() lock across the threads in an IBM MQ process. When this issue occurs then pmap output will show many "arena" memory segments and these are normally 64MB in size, for example: Address	 Kbytes RSS Dirty Swap Mode Mapping ... 80064000 65136 0 0 0 ---p [ anon ] 84000000 132 12 12 0 rw-p [ anon ] 84021000 65404 0 0 0 ---p [ anon ] 88000000 132 16 16 0 rw-p [ anon ] 88021000 65404 0 0 0 ---p [ anon ] 8c000000 132 12 12 0 rw-p [ anon ] 8c021000 65404 0 0 0 ---p [ anon ] ... -------- ------ ------ ------ ------ total	 2009720 5924 3224 0 The majority of the addressed memory segments should not be in the resident set (RSS) and should therefore not contribute to the commit charge of the process. IBM has however seen evidence that some customers with particular memory allocation patterns and large numbers of arenas may suffer from fragmentation within arenas and this can lead to actual memory resource issues.
Problem conclusion
The majority of users should not be affected by this issue and so the default behaviour of the IBM MQ code is unchanged. Two new environment variables have been added which can be used to tune the behaviour of malloc(): AMQ_MALLOC_ARENA_SET - if this is set, then MQ will call mallopt(M_ARENA_MAX, 1) when MQ processes start, in order to force a single arena AMQ_MALLOC_ARENA_COUNT - this can be set to a number in order to tune the number of arenas in use by the mallopt() call. The default value is 1 and the maximum value is 8. Performance testing did not show any degradation with the number of arenas set to 1 but users should run their own tests before committing this change in a production environment. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v8.0 8.0.0.8 v9.0 CD 9.0.5 v9.0 LTS 9.0.0.3 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT20636
Reported component name
IBM MQ APPL M20
Reported component ID
5725S1400
Reported release
800
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2017-05-17
Closed date
2017-10-25
Last modified date
2018-02-06
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
IBM MQ APPL M20
Fixed component ID
5725S1400
Applicable component levels
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS5K6E","label":"IBM MQ Appliance"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]
Document Information
Modified date:
06 February 2018