Use the Logstash collector feature in Liberty to collect logs and other events from
your Liberty servers and send them to a
remote Logstash server. The collected events can be used for log analysis and troubleshooting
purposes.
The most current
documentation for using Logstash collector with Liberty is available on the Open Liberty website.
Before you begin
The logstashCollector-1.0
feature was tested with the following products:
- Logstash V2.x, Elasticsearch V2.x, and Kibana V4.x
- Logstash V5.3.x, Elasticsearch V5.3.x, and Kibana
V5.3.x
- Logstash V6.4.x, Elasticsearch V6.4.x, and Kibana V6.4.x
- Logstash V7.x, Elasticsearch V7.x, and Kibana V7.x
You can use the logstashCollector-1.0
feature with a Logstash server that runs
with any of the available output plug-ins from Logstash. However, many users choose to use Logstash
with Elasticsearch and Kibana to provide a complete log consolidation and analysis facility. For
more information, see the Elastic website.
Logstash collector garbage collection events are obtained from IBM Health Center in IBM JDKs. IBM
JDKs usually include IBM Health Center with the exception of IBM Semeru JDKs.
Procedure
-
Set up Logstash by following the instructions from Elastic.
-
Create or acquire certificate and key pair files for SSL for Logstash.
The following example command for openSSL generates a certificate and key pair. Customize the
number of days the keys are valid as needed.
openssl req -x509 -newkey rsa:2048 -keyout logstash.key -out logstash.crt -days 365 -nodes
-
For Logstash and Elasticsearch users, download a sample Logstash configuration file and an
index template file from this repository.
-
Download the Logstash configuration file, liberty_logstash.conf, and the
index template file, liberty_logstash_template.json, for your Elastic stack
version according to the readme file.
-
In the liberty_logstash.conf file, customize the lumberjack
ssl_certificate
and ssl_key
paths and the Elasticsearch
hosts
(Elasticsearch_host_name:port_number
)
value.
-
For Elasticsearch V2.x users, optionally customize the
_ttl
defaults to
indicate the number of milliseconds to keep records of each event type in the
liberty_logstash_template.json file.
-
Complete the following steps for each Liberty server from which you want to collect
events.
-
Acquire or create a keystore for the Liberty server. To create a self-signed
certificate, use the following command. Customize the server name, password, and subject as
needed.
d:\wlp\bin\securityUtility createSSLCertificate --server=myServerName --password="Liberty" --subject=CN=myHostname,OU=defaultServer,O=ibm,C=us
-
Import the logstash.crt file from step 2 into the
trust.jks file in your server. Customize the
wlp_install_dir and server name as needed. When prompted for a password, use
the certificate password from step 4a.
d:\java\bin\keytool -import -noprompt -alias logstash -file logstash.crt -keystore wlp_install_dir\usr\servers\myServerName\resources\security\trust.jks -storepass Liberty
-
Run the following command to install the
logstashcollector-1.0
feature:
d:\wlp\bin\installUtility install logstashcollector-1.0
-
Configure Logstash collector in the server.xml file in Liberty by adding the following content.
Customize the logstashCollector list of sources, host name, and port as needed.
You can change the maximum number of characters that are
allowed for the message field in a message or trace event or for the stack trace in an FFDC event.
The default maximum is 2048 characters. The following example sets the
maxFieldLength parameter to 3000 characters in the
logstashCollector
element. For an unlimited number of characters, set
maxFieldLength to 0
(zero) or -1
.
You can limit the maximum number of events that are sent for
each type of event (message, trace, accessLog, ffdc, garbageCollection, audit) per second. The
default maximum number of events is 0
, which means that no throttling is applied.
If maxEvents is set to greater than 0
, there is a limit to the
number of events that are sent per second. The maxEvents value must be an
integer between 0
and 2147483647
.
Also, you can add custom tags to decorate events that
are sent from the server. The following example has two
tag
elements in the
logstashCollector
element.
Note: Tags that are used in
Admin Center have no relation to tags that are used in
logstashCollector/bluemixLogCollector
. If you are using the audit source, then
enable the audit feature. For more information, see the
Audit topic.
<featureManager>
<feature>logstashCollector-1.0</feature>
</featureManager>
<keyStore id="defaultKeyStore" password="Liberty" />
<keyStore id="defaultTrustStore" location="trust.jks" password="Liberty" />
<ssl id="mySSLConfig" trustStoreRef="defaultTrustStore" keyStoreRef="defaultKeyStore" />
<logstashCollector
source="message,trace,garbageCollection,ffdc,accessLog,audit"
hostName="localhost"
port="5043"
sslRef="mySSLConfig"
maxEvents="10"
maxFieldLength="3000">
<tag>serverRackA5</tag>
<tag>billingAppTeam</tag>
</logstashCollector>
Trace and access logs are high volume logs and require more network, CPU, and storage resources
to collect.
-
Enable HTTP access logging and use the
'%h %u %t "%r" %s %b %D %{User-agent}i'
log format.
-
Start Elasticsearch, Logstash, and Kibana. See the Elastic website for instructions.
-
Start the Liberty server and generate some events.
-
Open Kibana in a browser and create an index.
- For Kibana 7, 6, or 5.6, click .
- Enter
logstash-*
as the Index Pattern.
- Click Advanced Options, and enter
logstash-*
as the
Index Pattern ID.
- Select
datetime
as the Time filter field name, and click
Create.
- For Kibana 5.0-5.5, click . and select
datetime
as the Time filter field
name. Click Create.
- For Kibana 4, click , and select
datetime
as the Time filter field
name. Click Create.
-
Download a sample dashboard from this repository.
-
Import the dashboard into Kibana.
- For Kibana 7, 6, or 5 , click .
- For Kibana 4, click .
-
View the dashboard.
- For Kibana 7, 6, or 5 , click and then select the dashboard.
- For Kibana 4, click and then select the dashboard.
Results
You configured your Liberty servers to
send events to your Logstash server and can now view your events in the Liberty dashboard by using Kibana.
If you enable Logstash collector to
acquire garbage collection events, late attach might not enable properly for IBM® tools against the z/OS®
Java™ virtual machine (JVM). To fix the problem, add the system
property and value -Dcom.ibm.tools.attach.enable=yes
for the JVM invocation. Either
add it directly as part of the Java execution or within a
jvm.options file.