Architecture for Audit logging version 3.7.0
Following are the key components of the audit logging architecture:
- Rsyslog Sidecar for forwarding audit records over TLS syslog connections
- CommonAudit Custom Resource (CR)
- Fluentd
- Integrating your cluster with IBM QRadar
- Integrating your cluster with Splunk Enterprise
- Audit log events
- Audit logging format
- Authentication and authorization audit logs
- Audit logging in your cluster
- Audit logging data statistics
Rsyslog Sidecar for forwarding audit records over TLS syslog connections
Audit logging version 3.7.0 introduces support for sending audit logs over Rsyslog. Rsyslog is a dynamic log processing tool that is built on top of the standard Linux® syslogd. Similar to the deprecated audit logging sidecar, a Rsyslog sidecar offers easy adoption of audit logging and shifts the burden of transmitting the messages to the sidecar. The audit logging Rsyslog sidecar can be adopted by IBM Cloud Pak® foundational services and IBM Cloud Pak® applications to forward generated audit record JSON files to the audit logging service over a TLS syslog connection.
The Rsyslog sidecar can be deployed in parallel with a service container on the same pod. The service application container and the Rsyslog sidecar container are configured to share the emptyDir
volume /var/log/audit
.
Every service that generates audit logs writes the logs to the /var/log/audit/<service_name>-audit.log
file. The audit sidecar monitors the file for audit records to forward onto the audit logging service over a TLS syslog connection.
The Rsyslog sidecar uses the Logrotate tool to monitor the logs in the /var/log/audit
directory. It monitors size, rotate period, and other parameters, and also recycles the audit logs as specified in the configuration.
The audit container sidecar is used by audit logging adopters in their service offerings. The audit container sidecar is not included in the audit logging service.
Note: Adopters need to indicate to their customers that data in the sidecar container might include Sensitive Private Information (SPI) in the audit logs.
Steps for adoption
- Add the audit logging Rsyslog sidecar container at the first position under container section in the deployment file. As an example, image,
quay.io/opencloudio/audit-sidecar-syslog:1.0.0
includes the image for the 3.7.0 release. -
Audit logging Rsyslog sidecar containers use a UID. Both the Rsyslog sidecar and the service container must run as the same user. Only the following UIDs are supported:
1000 21000 55555 65534 65535
-
Mount the
/var/log/audit
shared volume on all containers in your pod. - Mount the
audit-server-certs
secret to the sidecar container in your pod with the mountPathetc/audit-tls
. Set this volume tooptional: true
. If your service runs outside of the namespace where audit logging fluentd is running, you must copy the secret to your namespace. For more information about how to request that your secret is copied to your namespace, see Bindings for the ibm-auditlogging-operator. - Add the environment variable
AUDIT_URL
by using the value from the keyAuditLoggingSyslogIngestURL
in the configmap,audit-logging-fluentd-ds-http-ingesturl
to the sidecar container in your pod. Link OperandBindInfo for audit logging. -
Configure Logrotate. The following example resembles the default configuration.
/var/log/audit/*.log { copytruncate rotate 24 hourly missingok notifempty }
If you prefer to use a custom Logrotate configuration, you can use a
ConfigMap
and mount it to the Rsyslog sidecar. For example,kind: ConfigMap apiVersion: v1 metadata: name: logrotate labels: app: "logrotate" data: LOGROTATE: | /var/log/audit/*.log { copytruncate rotate 5 weekly missingok notifempty } LOGROTATE_CONF: |- include /etc/logrotate.d
volumeMounts: - mountPath: /var/log/audit name: shared - mountPath: /etc/audit-ingest/ name: audit-ingest - mountPath: /etc/audit-tls name: audit-server-certs - mountPath: /etc/logrotate.d/audit name: logrotate subPath: audit - mountPath: /etc/logrotate.conf name: logrotate-conf subPath: logrotate.conf volumes: - name: audit-ingest configMap: name: audit-logging-fluentd-ds-http-ingesturl optional: true items: - key: AuditLoggingSyslogIngestURL path: auditurl - name: audit-server-certs secret: secretName: audit-server-certs optional: true - configMap: defaultMode: 420 items: - key: LOGROTATE path: audit name: logrotate name: logrotate - configMap: defaultMode: 420 items: - key: LOGROTATE_CONF path: logrotate.conf name: logrotate name: logrotate-conf
-
You can enable or disable audit logs for a service. Add the
AUDIT_ENABLED
flag to theConfigMap
of one of your services. You can add the flag as an environment variable so that the application code has access to enable or disable generation of audit records. For more information about theAUDIT_ENABLED
flag, see Audit logging policy controller.Note: You can enable the audit logs for the Identity and Access management (IAM) service. For more information, see Auditing IAM service.
-
Your final deployment resembles the following example.
Note: Your service and the rsyslog sidecar must run in the same namespace as your fluentd instance. Complete sample deployment follows:
apiVersion: apps/v1 kind: Deployment metadata: labels: run: audit-log-test name: audit-log-test namespace: common-test spec: replicas: 1 selector: matchLabels: run: audit-log-test strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: run: audit-log-test spec: containers: - name: audit-service-syslog env: - name: DEBUG_SYSLOG value: "true" - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: SERVICE_NAME value: audit-log-test-service image: quay.io/opencloudio/audit-syslog-service:1.0.3 imagePullPolicy: Always securityContext: runAsUser: 1000 name: audit-service-syslog resources: limits: cpu: 100m memory: 128Mi requests: cpu: 50m memory: 128Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/log/audit name: shared - mountPath: /etc/audit-ingest/ name: audit-ingest - mountPath: /etc/audit-tls name: audit-server-certs - mountPath: /etc/logrotate.d/audit name: logrotate subPath: audit - mountPath: /etc/logrotate.conf name: logrotate-conf subPath: logrotate.conf - name: audit-log-test image: myrepository/audit-log-test:syslog imagePullPolicy: Always securityContext: runAsUser: 1000 volumeMounts: - mountPath: /etc/audit-ingest/ name: audit-ingest - mountPath: /var/log/audit name: shared env: - name: SERVICE_NAME value: audit-log-test - name: EVENT_COUNT value: "10" - name: CONTAINER_ID value: audit-log-test - name: SLEEP_TIME value: "900" - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumes: - name: audit-ingest configMap: name: audit-logging-fluentd-ds-http-ingesturl optional: true items: - key: AuditLoggingSyslogIngestURL path: auditurl - name: audit-server-certs secret: secretName: audit-server-certs optional: true - name: shared emptyDir: {} - configMap: defaultMode: 420 items: - key: LOGROTATE path: audit name: logrotate name: logrotate - configMap: defaultMode: 420 items: - key: LOGROTATE_CONF path: logrotate.conf name: logrotate name: logrotate-conf serviceAccount: test-sa serviceAccountName: test-sa
In a service container, the application that is running creates a <service_name>-audit.log
file in the directory that is mounted on /var/log/audit
.
Write or append all audit logs that are generated by the application or service to the <service_name>-audit.log file
. The Rsyslog sidecar container forwards the logs to fluentd over a TLS syslog connection.
The audit logs must be in JSON format to be correctly parsed by Fluentd and a SIEM tool.
CommonAudit Custom Resource (CR)
Audit logging service version 3.7.0 introduces the CommonAudit
CR. The CommonAudit
CR configures a namespace scoped fluentd
. In a multi-tenant configuration, CommonAudit
configures the audit logging
service and runs independently from audit logging services in other namespaces. Audit records are collected and forwarded only to SIEM tools that are configured within a namespace where a CommonAudit
CR is deployed.
AA = Audit Agent. For more information about audit agents, see Rsyslog Sidecar for forwarding audit records over TLS syslog connections.
In the case where a cluster scoped instance of fluentd
is needed, you can deploy a single CommonAudit
CR to the ibm-common-services
namespace. You can then request an OperandBindInfo
for services across namespaces to forward logs to the audit logging service.
Note: A CommonAudit
CR and AuditLogging
CR (deprecated) cannot run together in the same namespace. By default, the Operand Deployment Lifecycle Manager deploys an AuditLogging
CR to the ibm-common-services
namespace. To deploy a CommonAudit
CR in the ibm-common-services
namespace, you must first remove the AuditLogging
CR.
The CommonAudit
fluentd
deployment supports multiple replicas. By default, the fluentd
deployment is configured as a single replica with minimal memory and CPU load. However, fluentd
can be scaled
up and out as needed. For more information, see Hardware requirements and recommendations for IBM Cloud Pak foundational services.
The audit logging service supports HTTPS ingest endpoints and Rsyslog forwarding collection and ingestion mechanisms. IBM Cloud Pak foundational services and IBM Cloud Pak® can route selected service audit records to the audit logging services.
Configuring CommonAudit
- Log in to your cluster.
- From the Red Hat® OpenShift® Container Platform console, click Operators > Installed Operators.
- From the Project drop-down list, select the
ibm-common-services
namespace. - Select
IBM Audit Logging Operator
. - Select the CommonAudit tab.
- Create or edit a
CommonAudit
instance. Note:CommonAudit
instances that are deployed outside of theibm-common-services
namespace do not appear under the CommonAudit tab. You can use theoc get commonaudit --all-namespaces
command to view them from your infrastructure node.
CommonAudit settings
Parameter | Description | Default value | Syntax |
---|---|---|---|
enabled |
Set this parameter to true to enable forwarding of audit logs. |
false |
Boolean |
issuer |
A central authority to obtain certificates from. | cs-ca-issuer |
String |
replicas |
Number of replicated fluentd pods. | 1 |
int |
fluentd.imageRegistry |
Sets the registry to pull the fluentd image from. | quay.io/opencloudio/ |
String |
fluentd.pullPolicy |
Sets the pullPolicy for the fluentd image. | IfNotPresent |
"IfNotPresent", "PullNever", or "Always" |
fluentd.resources.limits.cpu |
Sets the CPU limit for Fluentd. | 300 m | Kubernetes CPU units (String) |
fluentd.resources.limits.memory |
Sets the memory limit for Fluentd. | 400Mi | Bytes (String) |
fluentd.resources.requests.cpu |
Sets the CPU request for Fluentd | 25 m | Kubernetes CPU units (String) |
fluentd.resources.requests.memory |
Sets the memory request for Fluentd. | 100Mi | Bytes (String) |
outputs.splunk.enableSIEM |
Enables forwarding audit logs to Splunk. | false |
Boolean |
outputs.splunk.host |
Splunk server hostname. | SPLUNK_SERVER_HOSTNAME |
String |
outputs.splunk.port |
The port number for the HEC token or the HEC load balancer. | 8088 |
int |
outputs.splunk.token |
Identifier for the HEC token. | SPLUNK_HEC_TOKEN |
String |
outputs.splunk.enableTLS |
Enable https for calling the HEC api. | false |
Boolean |
outputs.qradar.enableSIEM |
Enables forwarding audit logs to IBM QRadar. | false |
Boolean |
outputs.qradar.host |
IBM QRadar server hostname. | QRADAR_SERVER_HOSTNAME |
String |
outputs.qradar.port |
Syslog target port. | 514 |
int |
outputs.qradar.hostname |
IBM QRadar log source identifier. | QRADAR_LOG_SOURCE_IDENTIFIER |
String |
outputs.qradar.enableTLS |
Enable TLS over TCP. | false |
Boolean |
outputs.hostAliases |
Hostnames and IP addresses for SIEM servers. | hostAliases:[{ip:QRADAR_SERVER_IP,hostnames:[QRADAR_SERVER_HOSTNAME]}] |
Kubernetes hostAliases |
Sample CommonAudit CR
apiVersion: operator.ibm.com/v1
kind: CommonAudit
metadata:
name: example-commonaudit
labels:
app.kubernetes.io/instance: common-audit-logging
app.kubernetes.io/managed-by: ibm-auditlogging-operator
app.kubernetes.io/name: common-audit-logging
spec:
enabled: true
issuer: cs-ca-issuer
replicas: 1
fluentd:
imageRegistry: quay.io/opencloudio/
pullPolicy: IfNotPresent
resources:
requests:
cpu: 100m
memory: 200Mi
limits:
cpu: 200m
memory: 300Mi
outputs:
splunk:
enableSIEM: true
host: splunk.fyre.ibm.com
port: 8088
token: abc-123
enableTLS: true
syslog:
enableSIEM: false
host: qradar.fyre.ibm.com
port: 514
hostname: qradar_source_id
enableTLS: true
hostAliases:
- ip: 9.12.34.56
hostnames:
- qradar.fyre.ibm.com
- ip: 9.65.43.21
hostnames:
- splunk.fyre.ibm.com
Sample AuditLogging CR (deprecated)
apiVersion: operator.ibm.com/v1alpha1
kind: AuditLogging
metadata:
creationTimestamp: "2021-09-30T04:11:13Z"
generation: 2
labels:
app.kubernetes.io/instance: common-audit-logging
app.kubernetes.io/managed-by: ibm-auditlogging-operator
app.kubernetes.io/name: common-audit-logging
operator.ibm.com/opreq-control: "true"
name: example-auditlogging
resourceVersion: "18197844"
uid: 39c37997-f3d8-4042-ba07-074718922527
spec:
fluentd:
enabled: true
imageRegistry: quay.io/opencloudio/
journalPath: /run/log/journal
pullPolicy: IfNotPresent
resources:
limits:
cpu: 50m
memory: 150Mi
requests:
cpu: 20m
memory: 20Mi
policyController:
enabled: "true"
status:
nodes:
- audit-logging-fluentd-ds-8k72m
- audit-logging-fluentd-ds-qcdjm
- audit-logging-fluentd-ds-rxprv
- audit-policy-controller-678b44d788-4kgtl
versions:
reconciled: 3.20.0
Fluentd
Fluentd is a log collector that uses input and output plug-ins to collect data from multiple sources and to distribute or send data to multiple destinations.
With Audit logging, fluentd collects audit logs by using the following sources:
- Journald by using the
fluent-plugin-systemd
input plug-in (deprecated). - HTTP requests by using the
in_http
plug-in. - Syslog protocol with TLS by using the
in_syslog
input plug-in.
Fluentd containers mount a host file system where the journal log data is stored. The default location is /run/log/journal
.
Fluentd can be configured to send logs to an enterprise SIEM tool such as IBM QRadar.
Integrating your cluster with IBM QRadar
- IBM QRadar log source extension to parse your cluster audit logs
- Configuring IBM QRadar to receive your cluster audit logs over TLS
- Configuring CommonAudit to send audit logs over TLS to IBM QRadar
- Configuring IBM QRadar rules