Troubleshooting
Problem
Analytics records API events in indices named using the syntax apic-api-yyyy.mm.dd-nnnnnn, for example apic-api-2023.11.20-000005.
When rollover runs, it rolls over the active index i.e. the one that's being written to. When it's rolled over, a new index is created, the new index becomes the active index, and the old active index becomes read-only. The yyyy.mmm.dd in the new index name will be the date that the rollover ran, and the nnnnnn will be 1 more than the sequence number of the previous active index.
For example, rolling over index apic-api-2023.11.20-000005 will create new index apic-api-2023.11.21-000006
Cause
Known causes
Java heap exhaustion in storage pods. This is frequently indicated by a log message similar to:
[2024-01-15T15:43:46,427][WARN ][o.o.i.i.s.r.AttemptRolloverStep] [analytics-storage-2] Failed to evaluate conditions for rollover [index=apic-api-2024.01.14-000008] - INTERNAL_SERVER_ERROR
...
Caused by: org.opensearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [indices:data/write/bulk[s]] would be [4125599524/3.8gb], which is larger than the limit of [4080218931/3.7gb], real usage: [4125594376/3.8gb], new bytes reserved: [5148/5kb], usages [request=0/0b, fielddata=2566/2.5kb, in_flight_requests=16614980/15.8mb]
Resolving The Problem
As a temporary solution, you can manually rollover using:
alias stcurl="kubectl -n <ns> exec -it analytics-storage-0 -- curl -k --cert /etc/velox/certs/client/tls.crt --key /etc/velox/certs/client/tls.key" stcurl "https://analytics-storage:9200/apic-api-w/_rollover" -XPOST
- As a long term fix, you have to increase the memory by switching to a larger profile which gives more heap memory to storage.
- Reduce retention period.
- Disable payload logging
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000CeBlAAK","label":"API Connect-\u003EManagement and Monitoring (MM)-\u003EAnalytics"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
19 April 2024
UID
ibm17148798