APAR status
Closed as program error.
Error description
IBM Spectrum Protect Plus hit Out-Of-Memory issue during file restore with following error in virgo log: ERROR http-nio-8082-exec-1314 c.s.dp.xsb.api.commons.exception.ExceptionMessageTranslator org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space Then file indexing starts failing due to write.lock acquire error: ERROR pool-18-thread-1 com.syncsort.dp.xsb.index.impl.CatalogIndexerImpl FAILED to index session 1637380800026 com.syncsort.dp.xsb.commons.XSBException: Lock obtain timed out: NativeFSLock@/data3/lucene/lucidx/write.lock It is possible that there is some sort of Lucene index corruption after Out-Of-Memory. After that, virgo service receives error if it is restarted: /osgi/service/blueprint/container/FAILURE' with properties ' {bundle.version=3.8.0, bundle=com.syncsort.dp.xsb.lucene_3.8.0 [334], bundle.symbolicName=com.syncsort.dp.xsb.lucene, exceptio n=org.springframework.beans.factory.BeanCreationException:Error of init method failed; nested exception is com.syncsort.dp.xsb.commons.XSBException: Too many documents, composite IndexReaders cannot exceed 2147483647, type=5, timestamp=1638147088758, bundle.id=334} All above unexpected behaviors are related with 2 billion Lucene index limit. IBM Spectrum Protect Plus has the provision of 2 types of indexes: high-level object indexes and file indexes. These indexes data should get generated inside /data/lucene/lucidx/ and /data/lucene/lucfileidx/ respectively. However, during file indexing, all of the indexing data is always generated inside /data/lucene/lucidx/ only instead of being populated into both directories. This causes the 2 billion limit easy to reach. Both types of indexes should be handled in different structures and get separate 2 billion limits. Versions affected: 10.1.*
Local fix
N/A
Problem summary
**************************************************************** * USERS AFFECTED: * * IBM Spectrum Protect Plus level 10.1.8 and 10.1.9 * **************************************************************** * PROBLEM DESCRIPTION: * * see ERROR Description * **************************************************************** * RECOMMENDATION: * * Apply the fixing level when available. This problem is * * currently projected to be fixed in IBM Spectrum Protect Plus * * level 10.1.10. Note that this is subject to change at the * * discretion of IBM. * ****************************************************************
Problem conclusion
Originally, the High-Level Object (HO or HLO) and file indices were cobbled into the same data structure, which limited the combined size of the HO and file index to the maximum for a single data structure. The HO and file indices have now been separated into two different data structures, which ensures that both the HO and file indices can individually grow to the maximum for a data structure. For example, if the maximum index size for a single data structure is 2 billion, the HO index can now grow to 2 billion and the file index can now grow to 2 billion, rather than the previous maximum of 2 billion combined.
Temporary fix
Comments
APAR Information
APAR number
IT39591
Reported component name
SP PLUS
Reported component ID
5737SPLUS
Reported release
A18
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2022-01-08
Closed date
2022-02-14
Last modified date
2022-02-14
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Modules/Macros
Platform Indexing
Fix information
Fixed component name
SP PLUS
Fixed component ID
5737SPLUS
Applicable component levels
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNQFQ","label":"IBM Spectrum Protect Plus"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"A18","Line of Business":{"code":"LOB26","label":"Storage"}}]
Document Information
Modified date:
31 January 2024