Logging Smart Alerts

Using Smart Alerts

From the navigation menu in the Instana UI, select Logs > Smart Alerts tab.

The Smart Alert tab offers you a quick overview of the configured Smart Alert for logs in a sortable table. You can sort the table by:

• Name
• Enabled
• Disabled
• Date created
• Date changed

Change the sorting order or use the search field to find specific alerts.

At the end of each row, use the toggle button to quickly enable or disable a logging Smart Alert. Click overflow menu to access the following actions:

• Edit
• Duplicate
• Delete

Click the name of a configured alert to view an overview of the logging Smart Alert. The overview displays the Alert Configuration panel and Alerts Created list. Select an item from the Alerts Created list to access the associated issue.

On Logs Smart Alert associated issue provides detailed information to help resolve it, including:

• Start and end timestamps
• Duration
• Severity
• Description
• Scope
• Automation Policies
• Recommended Actions - generated with Watsonx AI
• Action History

Adding an alert

To add an alert, complete the following steps:

1. From the navigation menu in the Instana UI, select Logs.
2. Select the Smart Alerts tab.
3. Click Create Smart Alert.

The Create Smart Alert opens the alert configuration dialog where you can configure Smart Alerts.

The alert configuration process includes the following steps:

1. Define the scope
2. Define the threshold for violations
3. Define the time threshold about when to be alerted
4. Select the alert channels that are to be notified
5. Define the alert properties
6. Add custom payloads to be included in alerts

Defining the scope

In the Scope section, the Log count metric is selected by default. You can narrow down the scope by adding filters based on log content or the underlying infrastructure. The filters also support the RegEx operator, enabling you to define regular expression-based conditions for more flexible pattern matching.

The metric results can be grouped with the available grouping tags. Currently, multiple grouping tags are not supported in log Smart Alerts.

Defining the threshold

When you set up a Smart Alert for logs, you can choose to use Static or Adaptive thresholds.

Static thresholds

Static thresholds remain fixed and are defined when you create or modify a Smart Alert. The threshold takes a constant value that does not change after the Smart Alert is created.

However, a static threshold can become less relevant if the underlying metric changes significantly over time. In such cases, you can manually adjust the threshold whenever needed.

When to use static threshold

Static thresholds are ideal when it is undesirable for the metric to exceed or fall below a specific constant value, irrespective of any seasonality of the underlying metric.

Adaptive thresholds

Adaptive thresholds continuously evolve and adjust themselves based on new data that is observed by Instana. The threshold continuously accounts for seasonal changes to the underlying metric without any human intervention. Baseline predictions are updated automatically with each evaluation window, enabling the threshold to adapt to changing patterns in your log data over time.

When to use adaptive threshold

Adaptive thresholds are ideal in the following situations:

• The underlying metric is not seasonal. The threshold is expected to gradually change over time, but any sudden deviation from this trend is undesirable.
• The underlying metric is seasonal and different thresholds exist for different times of the day or week. The thresholds themselves are expected to gradually change over time, but any sudden deviation from this trend is undesirable.
• You want the system to automatically learn and adapt to evolving log patterns without manual intervention.

Advanced settings for adaptive thresholds

When you configure an adaptive threshold, you can optionally adjust the following advanced settings:

• Seasonality: Defines how the threshold accounts for recurring patterns in your data. Options include:

  • Auto: Automatically detects patterns (suggested)
  • None: Ignores seasonal patterns
  • Daily: Uses 24-hour cycles
  • Weekly: Uses 7-day cycles

• Adaptability: Controls how quickly the threshold adapts to changes in the underlying metric. Higher values make the threshold more responsive to recent data, while lower values provide more stable thresholds based on longer-term patterns.

Configuring threshold values

To configure threshold values, complete the following steps:

1. Select a threshold operator to define how the threshold condition is evaluated.

2. Select a severity level by checking the corresponding checkboxes: Warning, Critical, or both.

3. After you select one or more severity levels, enter the threshold value for each. This configuration enables the definition of multiple severity levels within a single alert, each with its own threshold condition.

   • For Static threshhold, Instana provides a suggested threshold value. You can either use this suggested value or define a custom value manually.

   • For Adaptive threshold, you can adjust the sensitivity level by using a slider. Increasing the sensitivity narrows the upper and lower boundaries for anomaly detection, which results in more alert notifications. Decreasing the sensitivity broadens these boundaries, reducing the number of alerts. This adjustment sets the expected value range for the metric.

Based on the threshold operator in use, a metric that exceeds either boundary is considered as a violation and might trigger an alert.

After both the scope and threshold are defined, a chart is plotted based on historical data for the selected metrics:

• Static threshold: The chart displays up to 7 days of historical data for visualization. You can toggle the chart view between the last 24 hours and 7 days to better understand historical variations in metric values over time.
• Adaptive threshold: The chart preview shows the last 6 hours of data.

The following image shows a chart that illustrates potential alert triggers based on the current threshold configuration and available historical data:

If you select any grouping options, the grouping results might appear as a table just after the chart. To analyze the metric data trends in the chart against each grouping, select the respective rows in the table as displayed in the following image:

Defining the time threshold

For the alert that is triggered, you can add more conditions in the Time Threshold section when the defined threshold for the selected metric is violated.

The following typical conditions, often used in practice, are as follows:

• Persistence over time: Select a time window and the number of consecutive times of violation as shown in the following image. You receive an alert when the metric violates a defined threshold over the defined time window.

Adding alert channels

You can configure different alert channels for both warning and critical severity level in Smart Alerts for Logs. To add alert channels, complete the following steps:

1. Click Select Alert Channel.
2. From the list of preconfigured channels, select the channels from which you want to receive the alerts.

If a threshold value is set for warning and critical severities, you can set the alert channels for each severity. If a threshold value is set for both severities, all the alert channels are selected for the warning severity by default.

The following image shows alert channels with both severities configured:

If a threshold value is set only for one severity, the severity is displayed for every alert channel as the Alert Level.

The following image shows alert channels with one severity configured:

For more information about creating channels, see Alert channels.

Selecting alert properties

Adding more alert properties is optional, but it provides more configurations to better suit your requirements. You can edit the default alert title and description, use placeholders to create dynamic titles and descriptions, and select whether the alert triggers an incident. For more information, see Alerting.

Adding custom payloads

You can customize alert notifications by adding the following custom payloads:

• Global custom payloads: These payloads are relevant in all alert notifications that are sent by Instana.
• Alert-specific custom payloads: These payloads are relevant in alert notifications for a specific alert configuration that is sent by Instana.

An alert notification can include both global and alert-specific custom payloads (if applicable), but the alert-specific configuration is prioritized over the global configuration. As a result, if you use the same key, the value of the global custom payload field is overridden by the alert-specific one.

To add global custom payloads, see Configure custom payload globally.

The following image shows globally defined custom payloads that are used in the alert configuration:

To add alert-specific custom payloads, complete the following steps:

1. Click Add Row in the Custom Payloads section.
2. Enter a key to identify the custom payload entry.
3. Select the value type of the custom payload: Static or Dynamic
4. Define the value of the payload entry:

   1. For Static payload, enter the value.

   2. For Dynamic payload, click Select tag and choose a dynamic tag. You can use the suggestions to select the correct key for the selected dynamic tag or add it manually.

      The following image shows how to select a dynamic tag:

      

      The following image shows suggestions to select the correct key for the selected dynamic tag: