Website Monitoring FAQ

What is the difference between page views, loads, and transitions?

Single-page applications and multi-page applications work differently from a technical perspective. This difference is essential to judge performance and thus, user experience. These terms help to communicate how a website is used. communicate how a website is used.

• Page load: The retrieval of the initial HTML document and all subsequent actions until the next navigation in the browser. For example, a navigation that requires a new HTML document to be loaded. Typically the website content is rendered as HTML on the server and then delivered to the user. A multi-page application has only page loads. An example of a multi-page application website is Wikipedia.

• Page transitions: Websites might change the content that the users are looking at with JavaScript. In contrast to page loads, page transitions do not use classical browser navigation. Typically new website content is loaded with JavaScript and then turned into HTML. This HTML is then placed into the document to enrich or replace the existing document. Single-page applications use this technique. For single-page applications, the number of page transitions are larger than page loads. An example of a website that implements this architecture is Instana.

  A page transition is triggered when the page name is changed by using API.

• Page view: The term page view is introduced to judge general activity on a website irrespective of the architecture that is implemented. Page views are the sum of the page loads and page transitions.

What is a beacon?

The JavaScript agent transmits small monitoring payloads to the Instana servers that model specific events that occur within the lifecycle of a page view of a website (for example, page loading, resource retrieval, and HTTP requests). The term beacon comes from the W3C Beacon specification.

What is an origin?

Origins are a central part of the web security mechanisms. For more information about origins, see cross-origin resource sharing (CORS). The combination of the scheme (also called a protocol), hostname, and port form an origin. See the following example URL:

https://shop.example.com/articles/hoverboard/ratings

The preceding URL has the origin https://shop.example.com.

- Scheme: `https`
- Hostname: `shop.example.com`
- Port: 443 (based on the default for the HTTPS protocol)

Two origins are the same when all three parts (scheme, hostname, and port) are equal. The following examples are all unequal to each other:

• https://shop.example.com
• http://shop.example.com
• https://example.com
• http://example.com

Same-origin and cross-origin calls or resource sharing is a terminology that is frequently used when you mention origins. The web security mechanism same-origin policy and the cross-origin resource sharing mechanisms are commonly known. When the source origin (the origin of the HTML document) and the target origin (the origin of an API server) are different, a call is considered to be cross origin and not same origin.

The concept of origins must not be confused with the similar, but notably different in implementation and intention concept of sites.

What do the website metrics mean?

Most of the website metrics are received from the various W3C specifications. Specifically

• Resource Timing,
• Navigation Timing,
• Paint Timing and
• Network Information API

Instana uses common web terminology from the web vitals project.

• Largest Contentful Paint (LCP)
• First Input Delay (FID)
• Cumulative Layout Shift (CLS)
• Time to First Byte (TTFB)

Instana adheres to this terminology as much as possible. Instana uses the following metrics:

• onLoad Time: This timing exists for each page load and models the time until a navigation is complete (that is, the loading spinner stops). It is defined as loadEventEnd - fetchStart (see navigation timing). Within the user interface, onLoad Time and onLoad Event Time are distinguished to be explicit about the differences in terminology between the Instana user interface and the navigation timing specification.
• DOM: A variation of the timing defined in navigation timing. This metric is defined as domContentLoadedEventStart - domLoading (see navigation timing). It is considered a more useful timing breakdown than the navigation timing's processing time. See the following diagram to understand DOM time.
• Children: A variation of the timing defined in navigation timing. This metric is defined as loadEventEnd - domContentLoadedEventStart (see navigation timing). It is considered a more useful timing breakdown than navigation timing's onLoad times. See the following diagram to understand Children time.

For users of the Instana web REST API, an API endpoint is displayed to learn about the technical metric IDs (metricId). The metric IDs that is referenced in this endpoint's response can be used to query metrics with various Instana website monitoring web REST APIs.

Why are some website metrics not always available or have weird values?

The most important factors are user interaction and web browser support.

User interaction is required for some metrics within the time frame in which the metrics are collected, for example, First-Input Delay. Without user interaction, no metric value is available.

The second important factor is web browser support. Some metrics require web browser features that are not widely supported yet, for example, the modern web vitals. Unsupported web browser feature means that some websites as a whole, some pages, or single page loads might not contain any information about a metric (for example, about Cumulative Layout Shift). Varying web browser support can also create some confusing situations. For example, First-Contentful Paint (FCP) is more widely supported than Largest-Contentful Paint (LCP). As a result, statistical aggregations across a set of observations cause smaller LCP than FCP values. Although observing for a single page load is difficult, statistical aggregations can occur for a set of page loads given a difference in web browser support as the following examples shows.

• Observed First-Contentful Paint values: 1000 ms, 1400 ms, 1600 ms. Mean: 1333ms.
• Observed Largest-Contentful Paint values: 1000 ms, UNSUPPORTED METRIC, 1600 ms. Mean: 1300ms.

Why are the recorded timings for some of my HTTP calls so unusually huge?

Instana website monitoring records the time between the start of the request (XMLHttpRequest#send) and the success event (XMLHttpRequest#onreadystatechange) with readyState === 4. The following factors affect the time between the preceding two events:

• HTTP redirects
• DNS lookup time
• Time to establish TCP or TLS connections
• Server response times
• Request queuing times
• Latency and throughput
• Request and response sizes
• Page throttling

For page throttling, browsers might decide to throttle down or even stop processing for web pages that are not visible. Page throttling is most likely the reason for huge timings. For more information, see Google's blog post about this subject or the respective Mozilla Developer Network Page Visibility API page.

Why is the geographic information missing?

Instana's website monitoring collects the geographic information based on IP-address/ or Geo-location mapping. The IP address information is collected by using the reverse proxy server. If the geographic information is missing on the website monitoring user interface, you can check whether the reporting URL of the JavaScript agent is configured to bypass the reverse proxy server and send data to the internal monitoring endpoint (usually 2999) directly.

The reverse proxy server is installed automatically as part of the Instana self-hosted (on-premises) setup. For more information, see Exposing the monitoring endpoint to end-users.

How are you gathering this information?

Instana's website monitoring is based on an open-source library called weasel. Weasel gathers the information from the Browser Navigation Timing API and transmits it in an efficient form. For more information about weasel, see its source code on GitHub.

Which browsers are supported?

The JavaScript agent supports all commonly used browsers. However, some APIs and JavaScript language features aren't supported in older browsers. In those cases, the JavaScript agent might not be able to collect all wanted data points, such as navigation, resource, and paint timing information.

In old browsers (all browsers before Internet Explorer 6), the JavaScript agent can fail to load. As a result, no data is collected.

How do you handle browsers that do not support the navigation timing API?

For browsers that do not support the navigation timing API, Instana provides approximate timings. These timings are not reliable, and so do not rely on them for these traces too much. When Instana must resort to approximations for page load times, the values are excluded from statistics, that is, approximations are not part of aggregated page load times like mean, min, and max.

What kind of ineum calls can I make?

For more information about the global ineum function, see website monitoring API documentation.

Why am I blocked by AdBlock or similar browser extensions?

While most of the ad-blocking extensions are created not to display advertisements, most of them evolve into extensions that prevent website owners from tracking their users. The Instana monitoring script ends up in many of the ad-blocking extensions. If you have control over your users, you can ask them to allow the EUM script in their ad-blocking extension.

How does backend correlation work with Ajax calls?

For more information about backend correlation mechanisms, see Backend correlation.

What errors might occur with an HTTP request?

The JavaScript agent runs on the web browser that issues HTTP requests. The JavaScript agent collects the common HTTP errors that include client error with HTTP response code 4XX and server error with HTTP response code 5XX, which are identified with the HTTP response code. The agent can also detect the following errors when the web browser handles the incomplete HTTP requests:

• Timeout
• Request aborted
• Open XHR object error
• Non-HTTP error

Which HTTP headers are used?

The JavaScript agent makes use of the following HTTP headers to achieve backend correlation.

• Request Headers (to the backend):
  • X-INSTANA-T
  • X-INSTANA-S
  • X-INSTANA-L
• Response Headers (to the front end):
  • Server-Timing

Why is there no data for GoogleBot and others?

Due to bots that manipulate JavaScript APIs to return predictable or reproducible results, the Instana JavaScript agent and servers identify and block data collection for various bots. Blocking data collection has a positive impact when scraping the web, but unfortunately causes problems when wanting to monitor user experiences. For example, the GoogleBot's JavaScript API:

• Date.now() does not return the current time in milliseconds, but instead some seemingly fixed timestamps. As a result, durations recorded by diffing two timestamps that are acquired by Date.now() cannot be trusted.
• Math.random() and crypto.getRandomValues() return values out of a pool of pre-defined values. This causes problems with client-side generated IDs, such as causing the wrong backend correlation references.

Which HTTP endpoints do users make calls to (for SaaS)?

You needn't configure anything for correct data transmission to Instana's SaaS platform. The Instana user interface always presents the correct tracking snippet, which includes the necessary URLs. For more information about endpoints, see website monitoring endpoints.

Is it possible to proxy the HTTP endpoints (for SaaS)?

Do not attempt proxying of the HTTP endpoints. Instana does not provide support for any proxy setups or for any issues that might arise due to the usage of a proxy. If you still want (or have) to do setup proxy, you might find the following tips helpful:

• Set proper Host HTTP headers.
• Respect the difference between the eum.instana.io and eum-{region}.instana.io servers.
• Make sure that Instana servers are aware of the user IPs. Send an X-FORWARDED-FOR header to Instana servers with the user's IP. Alternatively, send a X-REALER-IP HTTP header (not X-REAL-IP) to the Instana servers that contains the user's IP.
• Pass through all the HTTP headers that the Instana servers include in the response body.
• Don't do any caching in the proxy.

Are you collecting any data from WebSockets?

Instana's JavaScript agent does not collect any information about WebSockets. WebSockets have no semantic model that Instana can effectively monitor aside from messages flowing between client and server. Because WebSocket messages have arbitrary formats (just character sequences or bytes), Instana cannot deduce any type of request, response, success, or failure state.

Although you can theoretically transmit every message transmitted or received with WebSockets to Instana's servers, such transmission has several problems:

• High probability of collection of sensitive data that must never land in a monitoring system.
• Large amounts of data are collected from every user, which results in overhead for users.
• Typically a low signal-to-noise ratio in the collected data.
• Lack of standardized semantic model in the data means that Instana cannot optimize access to the data for you.

For these reasons, Instana does not automatically collect any data about WebSockets. However, some customers implement request or response and subscription mechanisms on WebSockets. For more information, see custom event API. With the custom event API, you can realize backend correlation for WebSocket-based request or response systems.

Why is eum.js the initiator of all XMLHttpRequest and Fetch calls?

Instana JavaScript agent instruments the XMLHttpRequest and fetch APIs of the web browsers to learn about calls that websites are making. This instrumentation causes the JavaScript agent (eum.js) to appear as the initiator within the web browser developer tools as the following screenshot shows. The Instana JavaScript agent seems to be making these calls. However, these calls are actually made by the website that Instana monitors.

Instana's JavaScript agent communicates directly only with servers that are defined within the JavaScript snippet that are shown by the Instana user interface or as adapted for on-premises deployments.

What happens for users with bad internet or network connections?

Two cases can occur:

1. Website visitors might not be able to download our JavaScript agent. In these cases, no data can be collected.
2. Data transmission requests from your website visitors to our servers might not be working. Instana does not reattempt delivery in these cases, nor does Instana store the data for later delivery.

Why does your script tag carry the defer attribute?

Questions about JavaScript snippet, its structure, and Instana's chosen approach are frequently asked. The snippet is carefully designed to balance the customers' needs. The following section provide some insights into these questions.

Enable debug logging for the JavaScript agent

Instana's JavaScript supports debug logging that you can use to gain extra insights into misconfiguration, data transmission, limits, and more. To enable debug logging for the JavaScript agent, change the URL to the JavaScript agent within the monitoring snippet.

Edit the monitoring snippet and replace the file references from eum.min.js to eum.debug.js, as shown in the following snippet.

<!-- before -->
<script defer crossorigin="anonymous" src="https://{{hostname}}/eum.min.js"></script>

<!-- after -->
<script defer crossorigin="anonymous" src="https://{{hostname}}/eum.debug.js"></script>

Are there any limits to the amount of data collected?

The JavaScript agent has a data collection limit per browser tab to protect Instana systems and users from misconfiguration and circular monitoring patterns. The limits are as follows:

• Any transmitted beacon (except for page resources, rules apply in addition to the more specific rules mentioned later)
  • max per 10s: 128
  • max per 10 min: 4096
  • max per page load: 8096
• Page changes
  • max per 10s: 32
  • max per 10 min: 128
• Custom events
  • max per 10s: 32
  • max per 10 min: 128
• HTTP call beacons that are initiated by using window.fetch
  • max per 10s: 32
  • max per 10 min: 256
• HTTP call beacons that are initiated by using window.XMLHttpRequest
  • max per 10s: 32
  • max per 10 min: 256

Are you collecting data that can uniquely identify users?

By default, the Instana JavaScript agent does not include data that can uniquely identify users. Also, the Instana JavaScript agent also does not apply techniques such as device or browser fingerprinting.

User-specific data can be made available to Instana by using the user API.

What are you doing with the user data transmitted to Instana?

The user API can be configured by customers to transmit user identifying information to Instana. This information is used only to provide the features visible within the product. Instana does not interpret this data in any other way, nor is it correlated across multiple customers.

Is it possible to delete user data after it is transmitted to Instana?

Instana supports infrequent deletion requests, for example, to comply with GDPR. If you expect frequent or periodic deletion requests, instead transmit anonymized data to Instana (for example, hashed user IDs).

Are IPs anonymized?

IPs are anonymized. By default, the last octet of IPv4 addresses and the last 80 bits of IPv6 addresses are set to zeros. Stricter anonymization rules can be configured by using the configuration tab in a website's dashboard within the Instana user interface.

How do you gain access to user IPs?

The JavaScript agent does not have access to IP addresses. User IP addresses are accessed through the network connections that they establish to the servers. The data that is received from the JavaScript agents is enriched by the servers with anonymized IP addresses.

Does Instana use cookies?

The website monitoring itself does not directly use cookies. However, Instana's Content-Delivery Network (Cloudflare) sets a cookie that is named __cfduid for security purposes. For more information about the __cfduid cookie, see the Cloudflare FAQ article.

Does Instana use localStorage or sessionStorage?

By default, the Instana JavaScript agent does not use the localStorage and sessionStorage technologies. However, localStorage is used when you turn on session tracking. By default, the session tracking feature is unavailable in the Instana JavaScript agent.

Where does Instana store website monitoring data?

Where the website monitoring data is stored depends on the specific Instana installation. Generally, the data is stored in either the Amazon or Google cloud within either their European or US region. You can identify where your data is stored based on the reportingUrl shown in the Instana tracking snippet and the list of reporting endpoints for JavaScript agents.

Does CDN (Cloudflare) store user data?

Users download the JavaScript agent by using the Content-Delivery Network (CDN) provided by Cloudflare. Cloudflare keeps user data (including IP addresses) for a small amount of time within their logs. For more information, see Cloudflare FAQ.

I have a Content-Security-Policy in place, is there anything I need to do?

The Instana JS agent is asynchronously loaded from eum.instana.io and can be loaded with HTTP(S). Check that loading scripts from this domain is possible.

Data is transmitted to Instana with image loads, HTTP GET, and POST requests (with XMLHttpRequest). The origins that are used for data transmission are displayed in the tracking snippet.

The following Content-Security-Policy definition shows what is required for Instana's SaaS product:

script-src *.instana.io;
img-src *.instana.io;
connect-src *.instana.io;

What is the impact of same-origin policy on website monitoring?

The same-origin policy is one of the most fundamental website security concepts. Every website is subject to the policy because all web browsers enforce it. As a website monitoring provider, Instana cannot control your website or browser security. Instana can work only within the imposed security constraints. Unfortunately, this restricts Instana monitoring abilities.

• Browsers restrict access to error messages and stack traces to scripts of the same origin.
• Browsers restrict allowed HTTP headers for cross-origin requests. Therefore, backend correlation is not always possible.

To unlock these features when multiple origins are involved, you can use cross-origin resource sharing (CORS). CORS is a mechanism with which small controlled holes are opened within the same-origin policy security mechanism. The following image describes in detail what needs to be done to address the same-origin policy-imposed restrictions.

For more information about Instana backend correlation mechanisms and how web security is impacting these correlation mechanisms, see Backend correlation.

Why are detailed resource retrieval breakdowns not always available?

The availability of insights into network times, caching statistics, and asset sizes rely on resource timing capabilities. These capabilities are available in most modern web browsers when allowed by the same-origin policy.

To enable insights into resources of cross-origin resources (for example, origin https://cdn.example.com:443 for an HTML document that is loaded from https://example.com:443), you can use the Timing-Allow-Origin HTTP header. The following image shows how to set the header. Also, for more information, see the resource timing specification.

How can I gain insights into Script Errors?

Websites that embed many third-party scripts typically encounter a steady number of script errors. Script Error indicates JavaScript error in third-party scripts. For security purposes, web browsers restrict access to third-party scripts' JavaScript errors by replacing the real error message with Script Error and clearing the stack trace.

On the web, you can use the following ways to gain insights into Script Error:

• You can instruct web browsers to expose error messages and stack traces by indicating that the third-party script does not contain sensitive data.
• You can use loopholes in the web security model (discouraged and not guaranteed to work).

You are recommended to use the first option because this is the designed and proper way to achieve Script Error insights. To implement the first option, do the following actions:

• Add a crossorigin="anonymous" attribute to script tags.
• Make sure that the HTTP response carrying the script's source is sending a Access-Control-Allow-Origin response header.

Why do I see same-site cookie warnings when I use the JavaScript agent?

When you analyze same-site cookie warnings, you must differentiate between the following two cases:

• Users signed into Instana's product (either SAAS or self-hosted), which have cookies set related to their sign in and Instana product usage.
• Your website visitors who have no affiliation with Instana and are, typically, not aware of Instana.

For the former group (that is, Instana users), Instana and its partners are setting cookies. Web browsers transmit these cookies to Instana servers as part of website monitoring. Instana servers never expect these cookies to exist for website monitoring. Therefore, blocking of cookies by web browsers is not an issue. The only cookie that is used for website monitoring is the one by Cloudflare.

For the latter group (that is, your website's visitors), the SameSite cookie has no known issues. Instana's CDN provider sets a cookie with a valid SameSite=Lax cookie configuration.

Check for potential SameSite cookie issues by using an incognito or private browser window. Through this incognito or InPrivate browser window, you can experience your website like a visitor.

What is JavaScript stack trace translation?

The JavaScript stack trace translation provides clear and more actionable stack traces within Instana.

Before:
at http://shop-demo-app.instana.io/static/js/main.b1510333.chunk.js:1:1559

After:
at ProductDetails.js 26:11

The problem with untranslated stack trace lines is that the errors aren't actionable. Developers work with many (typically small) files. For performance reasons, these files are shipped to user browsers in a bundled and minified format that result in file names, lines, and column numbers in stack traces that are not human-readable or actionable.

With a translated stack trace, it's clear that the error occurring at …/main.b1510333.chunk.js:1:1559 is in fact at ProductDetails.js 26:11.

How does JavaScript Stack Trace Translation work?

The translation works by using source maps. Source maps enable us to translate unactionable stack traces to actionable stack traces. More specifically, they allow us to translate references to files, names, lines, and columns to their actual source code counterparts.

To achieve this, Instana runs the following steps:

1. The JavaScript agent reports JavaScript errors to Instana's servers. For example, let's assume the stack trace contains the line at http://shop-demo-app.instana.io/static/js/main.b1510333.chunk.js:1:1559.
2. Instana's servers attempt to download the JavaScript file to identify the source map responsible for this file.
3. The HTTP response is parsed, and Instana looks for references to source maps.
4. When a source map file is referenced, Instana downloads the source map file by using an HTTP GET request or searches in the uploaded source map files by the user.
5. When the download or the lookup is successful, the source map file is used to translate the file, name, line, and column references.

See the following steps for the stack trace line at http://shop-demo-app.instana.io/static/js/main.b1510333.chunk.js:1:1559.

1. An error is reported to Instana's servers.
2. Instana's servers issue an HTTP GET request to http://shop-demo-app.instana.io/static/js/main.b1510333.chunk.js.
3. The source map reference //# sourceMappingURL=main.b1510333.chunk.js.map is located in the JavaScript file.
4. The source map file http://shop-demo-app.instana.io/static/js/main.b1510333.chunk.js.map is downloaded by using an HTTP GET request or can be found in the uploaded source map files.
5. The source map is parsed, and the stack trace made readable.

How exactly does Instana retrieve files from our servers?

When Instana learns about the stack trace, it automatically issues HTTP requests to retrieve the JavaScript and source map files. The following call shows the closest representation of the calls made:

curl -H 'Accept: */*' \
  # Use a fake user-agent to bypass simple bot blockers
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36' \
  {{url of JavaScript or source map files}}

You can use the previous command to identify what kind of configuration Instana needs to download the JavaScript and source map files from your servers. The HTTP requests are issued from servers that are running inside AWS (Amazon Web Services) or Google Cloud. Advanced bot detection mechanisms might block requests coming from AWS or Google Cloud. Therefore, consider configuring extra headers that Instana needs to send to your servers to circumvent bot detection mechanisms.

How can I make sure that the Instana servers can establish a TCP or TLS connection?

The Instana servers are issuing requests from AWS or Google Cloud servers. Therefore, the JavaScript and source map files are accessible from the internet. Also you need to make sure that your server has a working TLS configuration with a complete certificate chain. You can check for any TLS issues by using the free SSL Test from SSL Labs/Qualys or by running the following commands:

openssl s_client -showcerts -connect {{YOUR_DOMAIN_HERE}}:443

How can I upload source map files to Instana?

For information about uploading source map files to Instana, see uploading source map files to Instana.