There is a dilemma facing infrastructure and app performance—as workloads generate an expanding amount of observability data, it puts increased pressure on collection tool abilities to process it all. The resulting data stress becomes expensive to manage and makes it harder to obtain actionable insights from the data itself, making it harder to have fast, effective, and cost-efficient performance management.

A recent IDC study (link resides outside ibm.com) found that 57% of large enterprises are either collecting too much or too little observability data. Clients seek to right-size their data while ever conscious to security and compliance requirements to retain more.

To help clients address observability data challenges, IBM Cloud® will make its next-gen cloud logging solution, IBM Cloud Logs, available in the coming months. IBM Cloud Logs is designed to help users to take control of their observability data and expedite insights to reduce application downtime.

IBM Cloud Logs will become generally available during the summer of 2024 in Frankfurt and Madrid with day-one support for EU-Managed controls. The service will continue its worldwide multizone region (MZR) roll-out through 3Q 2024. IBM Cloud Logs will be provisioned through the Cloud Catalog and provision can also be executed through API and Terraform enabling users to codify their environments.

Comprehensive environments include many sources of data and, IBM Cloud Logs will support integration with common workload environments on IBM Cloud including Red Hat Openshift® on IBM Cloud, IBM Cloud Kubernetes Services (IKS), VSI and Power, with the list continuing to grow.

With IBM Cloud Logs, users may flow their application log and IBM Cloud Activity Tracker events into the service giving users flexibility in how they handle your data. Users may flow the data to separate instances or combine the data into a single instance to expand observability insights.

IBM Cloud Logs processes incoming data and applies machine learning algorithms including log aggregation and anomaly detection to help users cut through data noise to hone into the root cause of the issue. Sophisticated alert rules may be configured in the tool by users to help reduce triage time. Examples include:

• The ability to be notified when a combination of alert events happens within a defined set of criteria.
• Receiving alerts when new errors or log types are detected, or anomalous values on established data.

To further aid in processing the logs and amplifying the value of the data incoming data, IBM Cloud Logs can be parsed to turn log data into actionable metrics.

• IBM Cloud Logs parsing tools help users evaluate which data is less essential or redundant. Restructuring the data makes it easier to aggregate dissimilar sources into signals of information teams need to find and address incidents fast.
• Oftentimes a simple spark-line of frequency over time will provide the needed insight to quickly narrow in on the problem. IBM Cloud Logs is designed to convert log data into metrics to summarize what is happening in a summarized format. Metrics from log data are a great way look at vast amounts of data quickly when searching on different data sources.

The next generation of logging on IBM Cloud enables users to search all of their retained data and is easily queried within the service. Data is stored on IBM Cloud Object Storage in a search-friendly format to allow for rapid search results. When logs’ rapid-search results are needed, data can be additionally hosted using hot storage for priority insights into users’ data. Query results also appear in the same tool and configured set of dashboards used for operations.

IBM Cloud Logs offers the tools to query data effectively from the simplest of queries using Boolean search to advanced queries using Regex and queries technology-based on Apache Lucene®. IBM Cloud Logs enhances the query experience with its in-service query language and build queries with prompted add-on help to construct queries that can handle more complex analysis needs.

IBM Cloud Logs also offers tools to quickly assemble dashboards to better visualize users’ environments. Preconfigured alerts and dashboards for common application environments can be tailored to users’ specific environment needs. Dashboard insights paired with IBM Cloud Logs’ machine learning analytics give SREs the opportunity to quickly identify the start of an incident before it becomes a multi-alarm fire.

IBM Cloud Logs supports alert incident management within the service helping to manage operational control of workloads and comprehensive environments with maintenance windows can be managed within the tool. When more complex incidents occur and trigger layers of alarms, users can gain fast command and visualize the situation within the tool, even helping to suppress alerts to downstream alert management solutions to help address alert fatigue.

IBM Cloud Logs as a service is designed to integrate with the most common applications and systems management tools to fit within users’ toolchains. Sharing data with other operational tooling is built-in by design:

• Integrate with alert management tools: IBM Cloud Logs supports webhook values within alert messages enabling information to be shared within the alert and allowing users to quickly connect to the source of the trigger.
  • Share alert data with the IBM Event Notification service for a comprehensive IBM Cloud alert management visibility and control.
  • Share alert data with PagerDuty and other specialized alert management tools.
• Integrate with downstream observability, SIEM, and data analysis tools: IBM Cloud Logs will send data to IBM Event Streams, a Kafka service implementation, where data may be shared with a wide variety of tools and applications.
• Integrate with your workloads and bespoke tools: IBM Cloud Logs supports launching into and out of the service with a defined set of parameters enabling you to automate and streamline your SRE or users’ ability to nimbly navigate the comprehensive workloads and maintain smooth context-switching between tools.

All data is not valued equally, and IBM Cloud Logs helps manage this to optimize the value of log data users keep. To guide users while reviewing their observability needs and budget, the solution offers three tiers of log and event processing to select from:

• Store and search: Data retained primarily for compliance obligations may be stored and searched as necessary at a low cost/GB.
• Analyze and alert: Log and event data with analysis and alert value is processed as a mid-tier cost/GB. The mid-tier includes adding definition of metrics from logs, allowing visualization of trends and preparation for future incidents quickly.
• Priority insights: Select and configure most critical and highest value data to users’ operations for priority query results. Data logs in this tier are retained in hot storage.

IBM Cloud Logs is IBM’s strategic cloud logging platform for the future and provides replacement functions for both the IBM Log Analysis and IBM Cloud Activity Tracker services. IBM Log Analysis and IBM Cloud Activity Tracker services will be deprecated and will have an end of service and end of life on March 30, 2025.

A migration tool will be released in 2Q 2024 for clients migrating from IBM Log Analysis and IBM Cloud Activity Tracker services. During the migration period, clients may be using both sets of tools and will be charged accordingly. The tool will migrate currently configured alerts, dashboards, views, and other settings into an IBM Cloud Logs instance. Terraform output will also be supported for users needing to codify their deployments and configurations.

IBM Cloud’s next-gen logging solution is designed to help users to collect and aggregate data within and outside IBM Cloud, quickly cut through data noise, search on all retained data, visualize environments, integrate into multi-tools and optimize log value to match observability budgets.

Gain logging insights to improve the performance of your infrastructure and apps.