Splunk

Instana supports integrating logs with Splunk. After you enable the integration, you can be redirected to the Splunk UI from the Instana UI to view logs that are stored in Splunk. This integration enables a fluid workflow between Instana and Splunk.

You can also send alert notifications to Splunk in real time by creating the Splunk alert channel or view Instana application and service metrics in Splunk.

Integrating logs from Splunk

Configuring Instana

To integrate logs with Splunk, you need to configure Instana by following the steps:

  1. Click Settings > Log management > Splunk in the Instana UI.

    Splunk Form

  2. By default, Splunk log management is not enabled. To enable the Splunk log integration, set the toggle to show Splunk link on Hosts, Container and Pods.

  3. Enter the following information:

    • In the Splunk Instance field, enter the URL or the IP address (including the port number) of the deployed instance where the logs are stored.
    • In the Index field, enter the name of the index you have configured in the Splunk platform. This is an optional parameter.

Accessing Splunk from the Instana UI

To access Splunk, click Splunk, which is at the upper left of each of these dashboards:

  • Kubernetes:
    • Host
    • Pod
    • Docker container
  • Host
  • Docker container

Splunk Toggle

If multiple log providers are integrated, click Go to Logs > Splunk.

Splunk Buttons

Accessing Instana from Splunk

Use either of the following options to enable accessing Instana-related entities (such as host and container metrics) from your Splunk logs:

  • Configure your existing Splunk dashboards. Add the _raw field to the panel's query, and add the drilldown to the panel section.

    <drilldown>
        <link target="_blank">https://<ENVIRONMENT_URL_HERE>/#/integration/landing;config=$row._raw$</link>
</drilldown>

  • Create a dashboard. Go to the Dashboards section in Splunk. Click Create New Dashboard, enter a name, and click Save. Click Edit Dashboard, select Source, and paste the following contents:

    <form theme="light">
  <label>Instana</label>
  <fieldset submitButton="false" autoRun="true">
      <input type="time" token="myTime" searchWhenChanged="true">
          <label></label>
          <default>
              <earliest>-1@h</earliest>
              <latest>now</latest>
          </default>
      </input>
  </fieldset>
  <row>
      <panel>
          <title>Events</title>
          <table>
              <search>
                  <query>sourcetype = * | table host docker.container_id kubernetes.pod_name _raw
                  </query>
                  <earliest>$myTime.earliest$</earliest>
                  <latest>$myTime.latest$</latest>
              </search>
              <option name="count">15</option>
              <option name="drilldown">row</option>
              <option name="refresh.display">progressbar</option>
              <option name="rowNumbers">false</option>
              <option name="totalsRow">true</option>
              <option name="wrap">false</option>
              <fields>["host","docker.container_id","kubernetes.pod_name","_raw"]</fields>
              <drilldown>
                  <link target="_blank">https://<ENVIRONMENT_URL_HERE>/#/integration/landing;config=$row._raw$</link>
              </drilldown>
          </table>
      </panel>
  </row>
</form>

Creating the Splunk alert channel

To send alert notifications from Instana to Splunk in real time, install the add-on and app from Splunkbase, and then create the Splunk alert channel.

To create a Splunk alert channel, click Settings > Team Settings > Events & Alerts > Alert Channels > Add Alert Channel.

splunk alert channel

The following Splunk events are received as an HTTP POST to the configured URLs (HTTP or HTTPS), the address which the alerts will be sent to.

See the following examples:

On open issues or incidents

{
  "issue": {
    "id": "53650436-8e35-49a3-a610-56b442ae7620",
    "type": "issue",
    "state": "OPEN",
    "start": 1460537793322,
    "severity": 5,
    "text": "Garbage Collection Activity High (11%)",
    "suggestion": "Tune your Garbage Collector, reduce allocation rate through code changes",
    "link": "https://XXXXXXX/#/?snapshotId=rjhkZXdNzegliVVEswMScGNn0YY",
    "zone": "prod",
    "fqdn": "host1.demo.com",
    "entity": "jvm",
    "entityLabel": "Test jvm",
    "tags": "production, documents, elasticsearch",
    "container": "test-container"
  }
}

On close issues or incidents

{
  "issue": {
    "id": "6596e1c9-d6e4-4a8e-85fd-432432eddac3",
    "state": "CLOSED",
    "end": 1460537777478
  }
}

On offline, on online, or on change events

{
  "issue": {
    "id": "53650436-8e35-49a3-a610-56b442ae7620",
    "type": "presence",
    "start": 1460537793322,
    "text": "online",
    "description": "Java virtual machine on Host host1.demo.com",
    "link": "https://XXXXXXX/#/?snapshotId=rjhkZXdNzegliVVEswMScGNn0YY",
    "zone": "prod",
    "fqdn": "host1.demo.com",
    "entity": "jvm",
    "entityLabel": "Test jvm",
    "tags": "production, documents, elasticsearch",
    "container": "test-container"
  }
}