Connecting to Elastic Stack in IBM Cloud Private

IBM® Cloud Private Cloud Foundry provides a Helm chart that is called ibm-cflogging. This Helm chart can be installed in IBM Cloud Private to allow logs from both platforms to be collected and analyzed in a single Elastic Stack installation on IBM Cloud Private. This integration relies on the cfp-ext-syslog-forwarder and cfp-ext-applog-forwarder extensions for IBM Cloud Private Cloud Foundry, which can be automatically configured during installation of the Helm chart. If you choose not to apply the configuration during installation, it can be retrieved later and applied to the extensions manually. You can also use the extensions to forward Cloud Foundry platform system logs and application logs to an endpoint of your choice. For more information, see Configuring platform system log forwarding and Configuring application log forwarding.

• Prerequisites
• Loading the chart archive
• Installing the chart
• Activating log forwarding
• Log processing

Prerequisites

You must have an IBM Cloud Private 3.1 or later environment with the chart ibm-icplogging 2.0.0 or later installed. (This chart is installed by default during IBM® Cloud Private installation as Helm release logging.) You must also have IBM Cloud Private Cloud Foundry installed because the ibm-cflogging Helm chart is exported during installation.

• Install the cloudctl IBM Cloud Private CLI and the kubectl CLI. Log in to your cluster. Make sure that you target the namespace where the ibm-icplogging chart is installed, which is kube-system by default. The kubectl CLI is automatically configured when you log in using cloudctl. See Installing the IBM® Cloud Private CLI.
• Install and configure the Docker CLI for use with your cluster. See Configuring authentication for the Docker CLI

Loading the chart archive

After you install IBM Cloud Private Cloud Foundry, you can find the chart archive in the <data_directory>/IBMCloudPrivate directory on the system where you ran the Cloud Foundry installer. (<data_directory> is the directory that you supplied to the launch.sh script by using the -c option.)

The chart archive is named ibm-cflogging-1.1.0-archive.tgz and it contains both the Helm chart and a required image. If you installed and configured the CLIs as described in Prerequisites on a different system, copy the archive to that system. Make sure that you are logged in to your IBM® Cloud Private cluster and targeting the same namespace as the release of ibm-icplogging you want to use. Make sure that your Docker CLI is logged in to the private image registry for your cluster. Then, run the following command:

cloudctl catalog load-archive --archive ibm-cflogging-1.1.0-archive.tgz

By default, this command loads the chart into the local-charts Helm repository and the image into the private image registry where it is accessible only by charts that are installed in the current target namespace. For more information, see cloudctl catalog load-archive command and Managing images.

Clusters in an airgap environment

The chart archive includes a required image but also depends on other images that are publicly available on the Internet. If your cluster does not have Internet connectivity, additional steps are required after you load the chart archive.

1. Locate and click the ibm-cflogging chart in the catalog. Download the chart .tgz file from the Source & TAR Files section.
2. On a computer with Internet connectivity and Docker, run the following command.

   cloudctl catalog create-archive --chart ibm-cflogging-1.1.0.tgz --archive ibm-cflogging-1.1.0-archive-offline.tgz
   

3. Load this new archive into the catalog, replacing the original chart.

   cloudctl catalog load-archive --archive ibm-cflogging-1.1.0-archive-offline.tgz
   

The additional images are now loaded into the private image repository and the default values of the chart are updated to refer to these images.

Installing the chart

The Helm chart requires no configuration to install, but to avoid manual setup, enable automatic configuration of the Cloud Foundry extensions. To do so, follow the instructions under Enabling automatic configuration before installing the chart.

The installation can completed by using either the IBM Cloud Private catalog or the Helm CLI.

Enabling automatic configuration

To enable automatic configuration, you need the Configuration Manager URL and Configuration Manager Token. These are the same values that you enter when you configure the IBM Cloud Private Cloud Foundry deployment tool. See Installing IBM Cloud Private Cloud Foundry with the user interface.

You must create a Kubernetes secret that contains the Configuration Manager token in the field configManagerToken. It must be created in the target namespace for the Helm chart. For example, create the file cf-config-manager-secret.yaml with the following contents, replacing {{token}} with the actual value of the token.

apiVersion: v1
kind: Secret
metadata:
  name: cf-config-manager-secret
type: Opaque
stringData:
  configManagerToken: {{token}}

Then create the secret:

kubectl create -f cf-config-manager-secret.yaml

You need to provide the name of the secret when you follow the instructions for installation by using the IBM Cloud Private catalog or the Helm CLI.

IBM Cloud Private catalog

Locate and click the ibm-cflogging chart in the catalog. The overview contains detailed information about all of the configuration parameters of the chart. Switch to the Configuration tab or click Configure.

• Enter a unique name for Helm release name.
• Select the Target namespace (must match the namespace of the ibm-icplogging release, normally kube-system).
• Accept the License.
• By default, Logstash is configured to receive both system and application logs. In the All parameters section, you can change this parameter by clearing the Enable check box under either cfp-ext-syslog-forwarder extension or cfp-ext-applog-forwarder extension. You must have at least one of these extensions enabled.
• If you want to enable automatic configuration of the Cloud Foundry extensions:
  • In the All parameters section, under IBM Cloud Private configuration check the Enable Automatic Configuration check box and complete the Configuration Manager URL and Configuration Manager Token Secret fields.
• Click Install.

Helm CLI

If you prefer to install the chart by using the helm CLI, first see Installing the Helm CLI (helm) for instructions on installing the helm CLI. Helm is automatically configured when you log in using cloudctl.

If you performed the setup that is described in Adding the internal Helm repository to Helm CLI, you can simply run the following command.

helm install local-charts/ibm-cflogging --version 1.1.0 --name <release_name> --namespace kube-system --tls

Alternatively, you can download the chart and install it from the TAR file. View the chart overview in the IBM® Cloud Private catalog for details on the configuration parameters and download the chart .tgz file from the Source & TAR Files section. Minimally, all that is required is the name for the release.

helm install ibm-cflogging-1.1.0.tgz --name <release_name> --namespace kube-system --tls

If you want to enable automatic configuration of the Cloud Foundry extensions, set cloudFoundry.configureExtensions to true and provide values for cloudFoundry.configManagerUrl and cloudFoundry.configManagerTokenSecret.

To disable forwarding of system logs, set syslogForwarder.enabled to false. To disable forwarding of application logs, set applogForwarder.enabled to false.

If you want to set these or other configuration parameters, create a YAML file containing the values and supply it by using the --values option.

Activating log forwarding

After installing the chart, you must launch a Cloud Foundry deployment to update the Cloud Foundry component instances so that they begin forwarding logs.

If you enabled automatic configuration, the cfp-ext-syslog-forwarder extension, cfp-ext-applog-forwarder extension, or both, are inserted into the deployment states and configured. Click the Start deployment button in the Cloud Foundry deployment tool or run the launch_deployment.sh script. For more information, see Installing IBM Cloud Private Cloud Foundry with the user interface or Installing IBM Cloud Private Cloud Foundry.

If you did not enable automatic configuration, the notes for the Helm release that are created by your installation specify the name of the Kubernetes secret where you can find the configuration for cfp-ext-syslog-forwarder, cfp-ext-applog-forwarder, or both. Follow the instructions to install and configure these extensions, then update your Cloud Foundry deployment.

• Configuring platform system log forwarding
• Configuring application log forwarding

Log processing

All system logs that are received from the cfp-ext-syslog-forwarder extension are output by Logstash to a dated index in Elasticsearch. The index is named by using the value supplied for syslogForwarder.logstash.index, which is cloudfoundry by default. For example, using the default value, index names have the form cloudfoundry-YYYY.MM.DD. Create an index pattern upon first use of Kibana with the pattern cloudfoundry-*.

The document output to Elasticsearch includes the following fields.

For more information, see RFC 5424 - The Syslog Protocol .

All application logs that are received from the cfp-ext-applog-forwarder extension are output by Logstash to a dated index in Elasticsearch. The index is named by using the value supplied for applogForwarder.logstash.index, which is cloudfoundry-apps by default. For example, using the default value, index names have the form cloudfoundry-apps-YYYY.MM.DD. Create an index pattern upon first use of Kibana with the pattern cloudfoundry-apps-*.

The document output to Elasticsearch includes the following fields.

Log Retention

Log retention depends on the target Logstash index that you supply for syslog and applog forwarding through the syslogForwarder.logstash.index and applogForwarder.logstash.index values. With the default values of cloudfoundry and cloudfoundry-apps, all logs are retained indefinitely, and this can quickly exhaust the storage space that is allocated to Elastic Stack.

By default, the ibm-icplogging chart curates the logs for the logstash-YYYY.MM.DD indexes by using a daily job that deletes all but the last one day of logs. For simple curation of Cloud Foundry logs, you can set syslogForwarder.logstash.index and applogForwarder.logstash.index to logstash.

Alternatively, to use the cloudfoundry and cloudfoundry-apps target indexes and keep Cloud Foundry syslogs and applog separated from other IBM Cloud Private logs, you can add these custom indexes to the list that are cleaned by the curator with the following steps. Modifications to the curator configuration, however, are lost if the logging release is upgraded or reinstalled.

1. Save the current curator configmap (which is called logging-elk-elasticsearch-curator-config or <release-name>-ibm-icplogging-elasticsearch-curator-config if you deployed a customized logging stack) to a YAML file, and create a backup copy of that file.

   kubectl -n kube-system get cm logging-elk-elasticsearch-curator-config -o yaml > logging-elk-elasticsearch-curator-config.yaml
   

2. Modify the file by adding a sixth prefix for the curator to clean under the actions key of the action.yml section. If you are using the target indexes cloudfoundry and cloudfoundry-apps, one prefix covers both of these cases. The following sample shows a default retention period of one day, but you can customize this sample depending on the volume of logs that are generated by your Cloud Foundry deployment. Be sure to match the indentation of the other actions for properly formatted YAML.

      6:
        action: delete_indices
        description: "Delete Cloud Foundry log indices that are older than 1 days. Cron schedule: 30 23 * * *"
        options:
          timeout_override:
          continue_if_exception: True
          ignore_empty_list: True
          disable_action: False
        filters:
        - filtertype: pattern
          kind: prefix
          value: cloudfoundry-
        - filtertype: age
          source: name
          direction: older
          timestring: '%Y.%m.%d'
          unit: days
          unit_count: 1
   

3. Replace the configmap with your modifications:

   kubectl -n kube-system replace -f logging-elk-elasticsearch-curator-config.yaml