Troubleshooting
Problem
Analytics is not working however output of 'system show analytics health' is green
Symptom
Analytics is not working
Diagnosing The Problem
If you run 'debug tail file /var/log/logstash.log' and if it reports error and you need to reboot the logstash servers to see whether it helps.
Sample error but not limited to
[2019-05-29T13:16:56,647][ERROR][logstash.outputs.elasticsearch] Encountered an unexpected error submitting a bulk request! Will retry. {:error_message=>"undefined method `respon
se' for #<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError:0x2082a861>", :class=>"NoMethodError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems
/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:223:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsea
rch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:187:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash
/outputs/elasticsearch/common.rb:109:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.
rb:76:in `retrying_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:27:in `multi_receiv
e'", "/opt/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:12:in `multi_receive'", "/opt/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in `
multi_receive'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:331:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/logstash-core/lib/logstash
/pipeline.rb:330:in `output_batch'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:288:in `worker_loop'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:258:in `st
art_workers'"]}
Resolving The Problem
User can restart logstash server by restarting Logstash using following curl command from Linux or Portal server where you can run curl command and have access to server.
curl https://hostnameofmgmtserver/v1/analytics_ops/logstash_restart -u cmc/admin -k
While you doing so you can check debug tail file /var/log/logstash.log to make sure logstash gets rebooted and comes up without issue.
Some example of good line after it finishes coming up
[2021-10-13T18:45:20,638][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb"}
[2021-10-13T18:45:20,643][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2021-10-13T18:45:20,645][INFO ][logstash.pipeline ] Pipeline main started
[2021-10-13T18:45:20,678][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9800}
^C
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSMNED","label":"IBM API Connect"},"ARM Category":[{"code":"a8m50000000CeBlAAK","label":"API Connect->Management and Monitoring (MM)->Analytics"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.0.8"}]
Was this topic helpful?
Document Information
Modified date:
14 October 2021
UID
ibm16498663