Monitoring AWS S3

The AWS S3 sensor is automatically deployed and installed after you install the Instana agent.

Learn about the other supported AWS services with our AWS documentation.

Supported information

Supported client-side tracing

For this technology, Instana supports client-side tracing for the following languages and runtimes:

Prerequisites

For the Instana agent to collect metrics from an S3 instance, you need to enable paid (request and data transfer) metrics. To create a CloudWatch metrics configuration for all the objects in your S3 bucket, see the AWS documentation.

Sensor (Data Collection)

Tracked Configuration

  • S3 Bucket Details
  • Bucket Name
  • Bucket Owner Id
  • Bucket Owner Name
  • Bucket Created At
  • Grouping zone (region)

Metrics

Requests

Metric Description
All (Request) The total number of HTTP requests made to an Amazon S3 bucket, regardless of type. If you’re using a metrics configuration with a filter, then this metric only returns the HTTP requests made to the objects in the bucket that meet the filter's requirements.
Get (Request) The number of HTTP GET requests made for objects in an Amazon S3 bucket. This doesn't include list operations.
Put (Request) The number of HTTP PUT requests made for objects in an Amazon S3 bucket.
Delete (Request) The number of HTTP DELETE requests made for objects in an Amazon S3 bucket. This also includes Delete Multiple Objects requests. This metric shows the number of requests, not the number of objects deleted.
Head (Request) The number of HTTP HEAD requests made to an Amazon S3 bucket.
Post (Request) The number of HTTP POST requests made to an Amazon S3 bucket.
List (Request) The number of HTTP requests that list the contents of a bucket.

Traffic

Metric Description
Bytes Downloaded The number bytes downloaded for requests made to an Amazon S3 bucket, where the response includes a body.
Bytes Uploaded The number bytes uploaded that contain a request body, made to an Amazon S3 bucket.

Errors

Metric Description
4xx Errors The number of HTTP 4xx client error status code requests made to an Amazon S3 bucket with a value of either 0 or 1. The average statistic shows the error rate, and the sum statistic shows the count of that type of error, during each period.
5xx Errors The number of HTTP 5xx server error status code requests made to an Amazon S3 bucket with a value of either 0 or 1. The average statistic shows the error rate, and the sum statistic shows the count of that type of error, during each period.

Latency

Metric Description
First Byte Latency The per-request time from the complete request being received by an Amazon S3 bucket to when the response starts to be returned.
Total Request Latency The elapsed per-request time from the first byte received to the last byte sent to an Amazon S3 bucket. This includes the time taken to receive the request body and send the response body, which is not included in FirstByteLatency.

Required Permissions

  • cloudwatch:GetMetricStatistics
  • cloudwatch:GetMetricData
  • s3:GetBucketTagging
  • s3:ListAllMyBuckets
  • s3:GetBucketLocation

In order to have an indicator for public buckets, the following permission needs to be added:

  • s3:GetBucketPolicyStatus

A bucket policy for each specific S3 bucket should also be defined. For more information on how to add the bucket policy, see the AWS documentation.

Configuration

Metrics for S3 are pulled every 60 seconds, this can be changed via agent configuration in <agent_install_dir>/etc/instana/configuration.yml:

com.instana.plugin.aws.s3:
  cloudwatch_period: 300

To disable monitoring of S3 instances use the following configuration:

com.instana.plugin.aws.s3:
  enabled: false

Proxy configuration

To configure the specific AWS Sensor to use proxy configuration, add the following agent configuration settings:

com.instana.plugin.aws.s3:
  proxy_host: 'example.com' # proxy host name or ip address
  proxy_port: 3128 # proxy port
  proxy_protocol: 'HTTP' # proxy protocol: HTTP or HTTPS
  proxy_username: 'username' # OPTIONAL: proxy username
  proxy_password: 'password' # OPTIONAL: proxy password

Monitoring multiple AWS accounts

Refer to the Monitoring multiple AWS accounts documentation to set up monitoring of multiple AWS accounts with one AWS agent in the same region.

AWS named profiles approach

To override which profiles should be used to monitor S3, use the following configuration:

com.instana.plugin.aws.s3:
  profile_names:
    - 'profile2'
    - 'profile3'

Defining profiles on service level will override the global AWS profiles configuration.

AWS STS approach

To override which IAM Roles should be used to monitor S3, use the following configuration:

com.instana.plugin.aws.s3:
  role_arns:
    - 'arn:aws:iam::<account_1_id>:role/<role_1_name>'
    - 'arn:aws:iam::<account_2_id>:role/<role_2_name>'

Defining IAM roles on service level will override the global AWS IAM roles configuration.

Filtering

Multiple tags can be defined, separated by a comma. Tags should be provided as a key-value pair separated by :. In order to make configuration easier, it is possible to define which tags you want to include in discovery or exclude from discovery. In case of defining tag in both lists (include and exclude), exclude list has higher priority. If there is no need for services filtering, the configuration should not be defined. It’s not mandatory to define all values in order to enable filtering.

Users are able to specify how often sensors will poll the AWS tagged resources using the tagged-services-poll-rate configuration property (default 300 seconds).

Tags are only available in conjunction with the AWS Agent.

To define how often sensors will poll the tagged resources use following configuration:

com.instana.plugin.aws:
  tagged-services-poll-rate: 60 #default 300

To include services by tags into discovery use following configuration:

com.instana.plugin.aws.s3:
  include_tags: # Comma separated list of tags in key:value format (e.g. env:prod,env:staging)

To exclude services by tags from discovery use following configuration:

com.instana.plugin.aws.s3:
  exclude_tags: # Comma separated list of tags in key:value format (e.g. env:dev,env:test)

AWS services without tags will be monitored by default but can be excluded by setting the include_untagged field to false:

com.instana.plugin.aws.s3:
  include_untagged: false # True value by default

Instana Agent Tags

Note that tags are only available in conjunction with the AWS Agent. More details on using tags are described here.