IT39591: UNEXPECTED BEHAVIORS LIKE OUTOFMEMORYERROR, CORRUPTED LUCENE INDEX AND VIRGO STARTUP FAILURE DUE TO LUCENE INDEX OVERFLOW

APAR status

• Closed as program error.

Error description

• IBM Spectrum Protect Plus hit Out-Of-Memory issue during file
  restore with following error in virgo log:
  
  ERROR http-nio-8082-exec-1314
  c.s.dp.xsb.api.commons.exception.ExceptionMessageTranslator
     org.springframework.web.util.NestedServletException: Handler
  dispatch failed; nested exception is java.lang.OutOfMemoryError:
  Java heap space
  
  
  Then file indexing starts failing due to write.lock acquire
  error:
  
  ERROR pool-18-thread-1
  com.syncsort.dp.xsb.index.impl.CatalogIndexerImpl FAILED to
  index session 1637380800026
  com.syncsort.dp.xsb.commons.XSBException: Lock obtain timed out:
  
  NativeFSLock@/data3/lucene/lucidx/write.lock
  
  It is possible that there is some sort of Lucene index
  corruption after Out-Of-Memory. After that, virgo service
  receives error if it is restarted:
  
  /osgi/service/blueprint/container/FAILURE' with properties '
  {bundle.version=3.8.0, bundle=com.syncsort.dp.xsb.lucene_3.8.0
  [334], bundle.symbolicName=com.syncsort.dp.xsb.lucene, exceptio
  n=org.springframework.beans.factory.BeanCreationException:Error
  of init method failed; nested exception is
  com.syncsort.dp.xsb.commons.XSBException: Too many documents,
  composite IndexReaders cannot exceed 2147483647, type=5,
  timestamp=1638147088758, bundle.id=334}
  
  All above unexpected behaviors are related with 2 billion
  Lucene index limit. IBM Spectrum Protect Plus has the provision
  of 2 types of indexes: high-level object indexes and file
  indexes. These indexes data should get generated inside
  /data/lucene/lucidx/ and /data/lucene/lucfileidx/ respectively.
  However, during file indexing, all of the indexing data is
  always generated inside /data/lucene/lucidx/ only instead of
  being populated into both directories. This causes the 2 billion
  limit easy to reach. Both types of indexes should be handled in
  different structures and get separate 2 billion limits.
  
  
  Versions affected: 10.1.*
  

Local fix

• N/A
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  *  IBM Spectrum Protect Plus level 10.1.8 and 10.1.9           *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * see ERROR Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem is       *
  * currently projected to be fixed in IBM Spectrum Protect Plus *
  * level 10.1.10. Note that this is subject to change at the    *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• Originally, the High-Level Object (HO or HLO) and file indices
  were cobbled into the same data structure, which limited the
  combined size of the HO and file index to the maximum for a
  single data structure. The HO and file indices have now been
  separated into two different data structures, which ensures that
  both the HO and file indices can individually grow to the
  maximum for a data structure. For example, if the maximum index
  size for a single data structure is 2 billion, the HO index can
  now grow to 2 billion and the file index can now grow to 2
  billion, rather than the previous maximum of 2 billion combined.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Platform Indexing
  

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels