Audit logging adoption guide

Configure your IBM® Cloud Private cluster to generate audit logs and route the logs to your Security information and event management (SIEM).

IBM Cloud Private audit logging architecture

Audit logging components AA - Audit Agent; LA - Application Log Agent

Audit container

Following are the key components of the audit logging architecture:

Audit container

Audit container is a sidecar container. It tails the audit.log file and pipes it by using the system-cat command. It sends the audit logs to the systemd journal.

Every service that generates audit logs writes the logs to /var/log/audit/<service_name>-audit.log file. A service container must share /var/log/audit with the audit container.

An emptyDir volume is used for sharing the directory /var/log/audit among the service containers to audit the sidecar container in a pod.

As the audit container must write to the systemd journal, it also needs to mount the host file system where the system journal exists.

The logrotate tool is used to monitor the logs in the /var/log/audit directory for size, rotate period, and other parameters, and to recycle the audit logs as specified in the configuration.

Journald (systemd journal)

systemd is a service that runs on a node. Audit logs that are generated by IBM Cloud Private services that run in pods that have audit sidecar container, are sent to systemd journal. The systemd journal stores the data in binary format. Data can be only appended. After systemd journal receives the audit data, it is picked up by fluentd and then sent to Elasticsearch or SIEM.

Fluentd

Fluentd is a log collector that uses input and output plug-ins to collect data from multiple sources and to distribute or send data to multiple destinations.

Fluentd collects audit logs from systemd journal by using the fluent-plugin-systemd input plug-in. This plug-in has data-filtering ability, and it collects only audit logs from the systemd journal.

Fluentd containers mount a host file system where the journal log data is stored. The default location is /run/log/journal.

Fluentd, by default, sends the audit logs to the Elasticsearch Logstash Kibana (ELK) stack by using the fluent-plugin-elasticsearch output plug-in. Fluentd can be configured to send logs to an enterprise SIEM tool such as QRadar.

Following configuration options are important for output plug-in from audit logging point of view:

Client or host certificate options. These options enable the Public-key Infrastructure (PKI) authentication with ELK.

  ca_file /path/to/your/ca/cert

  client_cert /path/to/your/client/cert
  client_key /path/to/your/private/key
  client_key_pass password

  ssl_version TLSv1_2

Note: You can feed the password from a secret by using the following statement:

  client_key_pass "#{ENV["APP_KEYSTORE_PASSWORD"]}"

Logstash option. This option puts audit logging data in its bucket to isolate audit logs from other debugging logs.

  logstash_format true # defaults to false
  logstash_prefix mylogs # defaults to "logstash"
  logstash_prefix_separator _ # defaults to "-"
  logstash_dateformat %Y.%m. # defaults to "%Y.%m.%d"

ELK

ELK stack is used for storing, indexing, and representing IBM Cloud Private logs.

(For proof of concept only.) ELK is also used for audit logging. The data that is received by ELK from Fluentd is parsed and then put into separate buckets: one for logging and one for audit data.

Audit log events

Following events can be logged as audit events:

Authentication events (successful or failed login attempt)
Authorization events (successful or failed attempt to access resources or data)
System configuration modification
Create, read, delete, or update resources or data
Failure of session management
Changes to user privileges
Database access (success or failure)
Network or application firewall, Intrusion Detection System (IDS), or Intrusion Prevention System (IPS) events
Amount of system use
System start, shutdown, or reboot events
Application or system failure events

Following data must not be included in the audit logs:

Sensitive information
User credentials
Passwords
Bank account details
Access token
Authentication token
File system path or information
Database query or string
Encryption or decryption keys

Audit logging format

IBM Cloud Private follows the Cloud Auditing Data Federation (CADF) standards. The standard defines an event model to collect the required data for auditing. IBM Cloud Private adds some custom fields to generate comprehensive logs.

For more information about CADF, see Cloud Auditing Data Federation Opens in a new tab .

The following fields are important:

{
    "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
    "eventType": "activity",
    "id": "icp:e97c7b00-e215-11e8-abf8-79cb75b57820",
    "action": "create",
    "requestPath": "/identity/api/v1/teams",
    "initiator": {
        "typeURI": "service/security/account/user",
        "name": "admin",
        "credential": {
            "type": "token"
        },
        "host": {
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15",
            "address": "icp-management-ingress:8443"
        }
    },
    "target": {
        "id": "c4e8170e90a7c01a228fbef74c22245d2665cefffac1662907a6a75e82319a74",
        "name": "icp-testing-audit-logs",
        "actions": {
            "name": "icp-testing-audit-logs",
            "teamId": "icp-testing-audit-logs",
            "users": [],
            "usergroups": [],
            "directoryList": []
        },
        "typeURI": "service/security/group"
    },
    "observer": {
        "id": "target"
    },
    "severity": "normal",
    "outcome": "success",
    "reason": {
        "reasonType": "HTTP",
        "reasonCode": 200
    },
    "eventTime": "2018-11-06T22:47:17.424Z",
    "kubernetes.container_id": "c4e8170e90a7c01a228fbef74c22245d2665cefffac1662907a6a75e82319a74",
    "kubernetes.container_name": "platform-identity-management",
    "kubernetes.pod": "auth-idp-mw2x9",
    "kubernetes.namespace": "kube-system",
    "origination": "ui",
    "version": "v1.0"
}

Go language CADF structure

type CADF struct {
    TypeURI   string `json:"typeURI"`
    Action    string `json:"action"`
    ID        string `json:"id"`
    Initiator struct {
        Name       string `json:"name"`
        TypeURI    string `json:"typeURI"`
        Credential struct {
            Type string `json:"type"`
        } `json:"credential"`
    } `json:"initiator"`
    Target struct {
        ID      string `json:"id"`
        Name    string `json:"name"`
        TypeURI string `json:"typeURI"`
    } `json:"target"`
    RequestPath             string `json:"requestPath"`
    EventType               string `json:"eventType"`
    Severity                string `json:"severity"`
    Outcome                 string `json:"outcome"`
    EventTime               string `json:"eventTime"`
    KubernetesContainerID   string `json:"kubernetes.container_id"`
    KubernetesContainerName string `json:"kubernetes.container_name"`
    KubernetesPod           string `json:"kubernetes.pod"`
    KubernetesNamespace     string `json:"kubernetes.namespace"`
    Observer                struct {
        ID string `json:"id"`
    } `json:"observer"`
    Origination string `json:"origination"`
    Version     string `json:"version"`
}

NodeJs CADF structure

 let cadf = {
      "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
      "eventType": "activity",
      "id": "icp:"+uuid1,
      "action": action,
      "requestPath": path,
      "initiator": {
    "typeURI": (user ? "service/security/account/user": ''),
        "name": user,
    "credential": {
      "type":"token"
        },
    "host": {
          "user-agent": req.headers['user-agent'],
          "address": req.headers['host']         
    }
      },
      "target": {
    "id": cont_id,      
    "name": res,
    "actions": actions,
        "typeURI": (map ? map: parseUrl(path)) // pretend this app is a service
      },
      "observer": {
        "id": "target"
      },
      "severity" : severity,
      "outcome": outcome,        
      "reason": {
    "reasonType":"HTTP",
        "reasonCode": status, // like 200 or 400 
      },
      "eventTime": expT,
      "kubernetes.container_id": cont_id, 
      "kubernetes.container_name": process.env.SERVICE_NAME, 
      "kubernetes.pod": process.env.POD_NAME || process.env.HOSTNAME,
      "kubernetes.namespace": process.env.POD_NAMESPACE, 
      "origination": identifyOrig(req.headers['referer'] || req.headers['user-agent']), 
      "version": "v1.0"
    };

Deployment file modification

Add an audit logging sidecar container at the first position under container section in the deployment file:

      - name: icp-audit-service
        image: mycluster.icp:8500/ibmcom/icp-audit-service:3.1.1
        imagePullPolicy: IfNotPresent
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /run/systemd/journal
          name: journal
        - mountPath: /var/log/audit
          name: shared

Note: The image path depends on the IBM Cloud Private version and installation type. Following example is of an offline installation on IBM Cloud Private version 3.1.1:

Mount the /var/log/audit/ shared volume on all containers in a pod:

        volumeMounts:
        - mountPath: /var/log/audit
          name: shared

Some of the custom fields in the CADF format can be provided in a container by using the environment variables kubernetes.container_name,kubernetes.pod, and kubernetes.namespace.

      - env:
        - name: SERVICE_NAME
          value: key-management-lifecycle
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CONFIG_PATH
          value: /opt/keyprotect/config/
        - name: ICP_NAMESPACE
          value: kube-system
        - name: CLUSTER_NAME
          valueFrom:
            configMapKeyRef:
              key: CLUSTER_NAME
              name: platform-auth-idp

In a service container, the application that is running creates a <service_name>-audit.log file in the directory that is mounted on /var/log/audit/.
Write or append all audit logs that are generated by the application or service to the <service_name>-audit.log file. Audit sidecar container forwards the logs to journald, and journald forwards to fluentd.

The audit logs must be in JSON format to be correctly parse by Fluentd and a SIEM tool. See the following sample:

 {"typeURI":"http://schemas.dmtf.org/cloud/audit/1.0/event","eventType":"activity","id":"icp:16d5af60-4cc2-11e9-9451-57b95a7e8968","action":"update","requestPath":"/identity/api/v1/teams/test-team/resources","initiator":{"typeURI":"service/security/account/user","name":"admin","credential":{"type":"token"},"host":{"user-agent":"curl/7.47.0","address":"<cluster-ip>:8443"}},"target":{"id":"4b1871f2f163856e3e3f56723fa16c543af3b1386588d311c2b4a07436122671","name":"test-team","actions":"crn:v1:icp:private:k8:mycluster.icp:n/default:::","typeURI":"service/security/group"},"observer":{"id":"target"},"severity":"normal","outcome":"success","reason":{"reasonType":"HTTP","reasonCode":200},"eventTime":"2019-03-22T16:46:50.198Z","kubernetes.container_id":"4b1871f2f163856e3e3f56723fa16c543af3b1386588d311c2b4a07436122671","kubernetes.container_name":"platform-identity-management","kubernetes.pod":"auth-idp-8jdlx","kubernetes.namespace":"kube-system","origination":"cli","version":"v1.0"}

If there are multiple service containers in a single pod, there is a separate audit log file for each service container in the /var/log/audit/ directory and only one file for audit sidecar container. The volume of the /var/log/audit/ directory is shared by all containers in a pod.
You can enable or disable audit logs for a service. Add the AUDIT_ENABLED flag in the ConfigMap. The flag can be added as an environment variable so that the application code has access to enable or disable generation of audit records.

Following is a deployment file example:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.17.0 ()
  creationTimestamp: null
  labels:
    io.kompose.service: think-blue-demo-app
  name: think-blue-demo-app
  namespace: jkstore
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: think-blue-demo-app
    spec:
      containers:
      - name: icp-audit-service
        image: mycluster.icp:8500/ibmcom/icp-audit-service:3.1.2
        imagePullPolicy: IfNotPresent
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /run/systemd/journal
          name: journal
        - mountPath: /var/log/audit
          name: shared
      - name: think-blue-demo-app
        image: mycluster.icp:8500/kube-system/think-blue-demo_app:0.1
        imagePullPolicy: IfNotPresent
        args:
        - npm
        - start
        ports:
        - containerPort: 30003
        env:
        - name: CLIENT_ID
          value: "9b570f23952a45099966ffa2cf7b6355"
        - name: CLIENT_SECRET
          value: "3rQwon8BrfpOEqzX2RttUXAt4CVkf8S2WuBQKiE5wPHKudMGX0FIlARejVf9"
        - name: AUTH_URL
          value: "https://<CLUSTER_IP>:8443/idprovider/v1/auth/authorize"
        - name: TOKEN_URL
          value: "https://<CLUSTER_IP>:8443/v1/auth/token"
        - name: ISSUER_ID
          value: "https://mycluster.icp:9443/oidc/endpoint/OP"
        - name: CALLBACK_URL
          value: "https://bobcat.rtp.raleigh.ibm.com/auth/liberty/callback"    
        - name: LOGOUT_URL
          value: "https://<CLUSTER_IP>:8443/v1/auth/logout"  
        - name: AUDIT_ENABLED
          value: "true"
        - name: CONTAINER_ID
          value: "think-blue-demo-app"
        - name: SERVICE_NAME
          value: "think-blue-demo-app"
        - name: AUTHZ_URL
          value: "https://<CLUSTER_IP>:8443/iam-pdp/v1/authz"  
        - name: PAYMENT_URL
          value: "https://<CLUSTER_IP>:8443/payments/payment/"
        - name: AUDIT_ENABLED
          valueFrom:
            configMapKeyRef:
              name: "think-blue-demo-ConfigMap"
              key: AUDIT_ENABLED
        resources: {}
        volumeMounts:
        - mountPath: /var/log/audit
          name: shared
      restartPolicy: Always
      volumes:
      - hostPath:
         path: /run/systemd/journal
         type: ""
        name: journal
      - emptyDir: {}
        name: shared
status: {}

Note: icp-audit-service is at first position. The /var/log/audit volume is mounted and shared between application container and audit sidecar container. AUDIT_ENABLED flag is imported from the ConfigMap file.