Technical Blog Post
Abstract
High number of getrlimit() call observed from db2ckpwd after Spectre/Meltdown Patches applied
Body
After applying the Spectre/Meltdown patches in RedHat Linux system hosting Db2LUW database connections were timing out.
From Linux trace it was observed db2's db2ckpwd process was issuing getrlimit() system call many number of times.
Long loop of getrlimit and close.
Example,
30607 13:48:58.283090 close(4) = 0 <0.000021>
30607 13:48:58.283159 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000019>
30607 13:48:58.283247 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000024>
30607 13:48:58.283353 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000017>
30607 13:48:58.283412 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000016>
30607 13:48:58.283463 close(3) = 0 <0.000018>
30607 13:48:58.283515 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000016>
30607 13:48:58.283566 close(4) = -1 EBADF (Bad file descriptor) <0.000017>
30607 13:48:58.283620 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000016>
30607 13:48:58.283672 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000017>
30607 13:48:58.283722 close(6) = -1 EBADF (Bad file descriptor) <0.000016>
30607 13:48:58.283770 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000017>
30607 13:48:58.283820 close(7) = -1 EBADF (Bad file descriptor) <0.000016>
30607 13:48:58.283868 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0 <0.000017>
These getrlimit calls were not coming from DB2. This was most likely a consequence of the Spectre/Meltdown patches.
Likely glibc's close() is now calling getrlimit() for some reason.
As per Db2's check password code we can see that Db2 loop through the first 1024 file descriptors and close them, so this behavior is expected.
But, Db2 do not call getrlimit() directly.
The "looping through all file descriptors" code was removed in Db2 v97fp10 / v101fp5 / v105fp5 as it is incompatible with RedHat System Security Service Daemon (SSSD).
So, anybody face similar observation can apply those minimum fixpack level of Db2 versions which should avoid this specific issue.
UID
ibm11139866