Announcing the new IBM Cloud Activity Tracker and IBM Log Analysis streaming feature as a public beta.
The ability to share data and insights between services is a strength of cloud-architected applications. IBM Log Analysis and IBM Cloud Activity Tracker services on IBM Cloud are expanding their ability to share data with other cloud services and applications, and both services are introducing the ability to stream log and event data to IBM Cloud Event Streams. IBM Cloud Event Streams supports connectors to a wide range of systems, and the Kafka API allows you easily share data and insights within your broader application workspaces.
The new IBM Cloud Activity Tracker and IBM Log Analysis streaming feature is announced as a public beta today and initially available in the Dallas and Frankfurt regions.
This new feature enables and simplifies use cases for your IBM Cloud applications:
- Forwarding cloud activity event data to a SIEM
- Forwarding specific log and event data to other corporate tools while maintaining a more complete copy of data within IBM Cloud.
- Aggregating then forwarding data to data lakes and other analysis tools.
The following diagram is an example of an end-to-end streaming use case:
This new Streaming feature is offered as an alternative to the existing Export API feature. The Streaming feature is an ideal alternative for scenarios where data forwarding needs are continuous, high volume and where the configuration should be managed at the service instance level. Data forwarded is a copy of the data which is processed within the service.
The Export feature continues to be an ideal solution when relying on the service’s template parsing tools defining custom fields. Export processes are activated by the user or programmatically and good for ad-hoc, reporting, and time-range specific use cases.
Configure Streaming
The Streaming feature is currently accessible from IBM Log Analysis and IBM Cloud Activity Tracker instances hosted in the Dallas and Frankfurt regions. It is a feature offered as part of the 7-day, 14-day, 30-day and HIPAA premium search plans. To configure Streaming, you will first need to define an IBM Cloud Event Streams instance, then, configure your IBM Log Analysis or IBM Cloud Activity Tracker instance to stream the data. The same IBM Cloud Event Streams instance can be used to receive data from multiple IBM Log Analysis and IBM Cloud Activity Tracker instances.
Once a streaming connection has been established, it is next important to configure monitoring of the streaming connection and establish meaningful alerts. The service documentation describes a recommended Cloud Monitoring configuration helping you observe your streaming performance. Set up alerts tailored to your application’s performance to detect if data is streaming as expected.
Optimize the scope of data to be streamed by setting up conditional streaming filters through streaming exclusion rules. Data streamed is a filtered subset of data retained within the IBM Log Analysis and IBM Cloud Activity Tracker instances.
Over the course of this beta, we will be preparing other regions to support the new streaming feature.
Learn more
If you are new to IBM Cloud Activity Tracker and IBM Log Analysis, we encourage you learn more about the services:
- IBM Cloud Activity Tracker allows you to capture cloud activity events from your applications and users use of IBM Cloud and its Cloud Services.
- IBM Log Analysis aggregates application log data from both agents and IBM Cloud services.
- Both services offer the ability to alert, search and visualize log and event data for operational analysis and insight.
- Both services offer the ability to archive and maintain compliance with SOC2, PCI and HIPAA.