Analyzing application Logs on Red Hat OpenShift Container Platform with Elasticsearch, Fluentd, and Kibana

Installing Red Hat OpenShift Container Platform logging

The Red Hat OpenShift logging instance also exposes a route for external access to the Kibana console, as shown in the following example.

oc get routes -n openshift-logging

NAME     HOST/PORT                                               PATH   SERVICES   PORT    TERMINATION          WILDCARD
kibana   kibana-openshift-logging.apps.host.kabanero.com                kibana     <all>   reencrypt/Redirect   None

Parsing JSON container logs

In Red Hat OpenShift, the Red Hat OpenShift logging Fluentd collectors capture the application container logs and puts each log in a message field of a Fluentd JSON document as a string. If you output the logs in JSON format, they are nested in the message field of the Fluentd JSON document. To use the JSON log data inside a Kibana dashboard, the individual fields inside the nested JSON logs must be parsed.

You can parse these nested JSON application container logs by deploying a Cluster Log Forwarder instance. The deployed Cluster Log Forwarder instance copies the nested JSON logs in to a separate structured field inside the Fluentd JSON document. The individual fields from the JSON container log can be accessed in the structured.<field_name> format.

Different products or applications can use the same JSON field names to represent different data types. To avoid conflicting JSON fields, the Cluster Log Forwarder instance requires JSON container logs from different products or applications to be separated into unique indexes. In the following instructions, the ClusterLogForwarder CR creates these unique indexes by using a label that is attached to the service of your application.

1. Add the logFormat: liberty label to your WebSphereLibertyApplication CR. The Cluster log Forwarder instance uses this label later to create a unique index for the application logs of the container.

   kind: WebSphereLibertyApplication
   apiVersion: v1
   metadata:
     name: <your-liberty-app>
     labels:
       logFormat: liberty
   ....

2. Restart your application deployment to include the updated label in the service and pod of your application.
3. Create the following cluster-logging-forwarder.yaml file to configure a Cluster log Forwarder instance that parses your JSON container logs.

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     namespace: openshift-logging
     name: instance
   spec:
     inputs:
       - name: liberty-logs
         application:
           namespaces:
             - liberty-app   #modify this value to be your own app namespace
     outputDefaults:
       elasticsearch:
         structuredTypeKey: kubernetes.labels.logFormat
         structuredTypeName: nologformat
     outputs:
     pipelines:
       - name: parse-liberty-json
         inputRefs:
           - liberty-logs
         outputRefs:
           - default
         parse: json

The .yaml file creates a parse-liberty-json pipeline for the ClusterLogForwarder kind. This pipeline takes an input from the liberty_logs input reference all the container logs from the liberty-app namespace. The pipeline outputs the container logs to the Red Hat OpenShift default Elasticsearch log store for the cluster. The parse: json definition enables the JSON log parsing.

The configured outputDefaults.elasticsearch.structuredTypeKey parameter builds a unique index for the container logs by adding the app- prefix to the logFormat label in the container. Previously, the logFormat: liberty label was added to the service of your WebSphereLibertyApplication CR. Therefore, the log files that are forwarded to the Elasticsearch default log store follow the app-liberty-* index pattern. If no logFormat label exists in your application container, the outputDefaults.elasticsearch.structuredTypeName parameter provides a fallback index name.

oc create -f cluster-logging-forwarder.yaml

For more information about parsing JSON logs, see the Red Hat OpenShift guide on enabling JSON logging.

Viewing application logs in Kibana

The following image displays the logs on an imported Kibana dashboard.

Configuring and uninstalling Red Hat OpenShift logging

To change the installed EFK stack, edit the ClusterLogging CR of the deployed Red Hat OpenShift logging instance.

To uninstall the EFK stack, remove the Red Hat OpenShift logging instance from the Red Hat OpenShift Logging Operator Details page.