Enable JWT security instead of mTLS

When and how to use JWT security instead of mTLS.

Most of the network interactions between API Connect subsystems use mTLS.The communication flows in API Connect that are secured with mTLS are as follows:

Management to portal.
Management to analytics.
Management to API gateway.
Management to v5 compatible gateway.
Management to Event gateway. Cloud Pak for Integration only.
API gateway to analytics.
V5 compatible gateway to analytics.
Event gateway to analytics.

All other inter-subsystem communication uses TLS. For the communication routes that use mTLS, it is a requirement that all network infrastructure such as load-balancers that are located between API Connect subsystems implement TLS passthrough (and not TLS termination).

Use of JSON Web Token (JWT) security instead of mTLS allows for the deployment of API Connect in network infrastructures where TLS termination is enabled on load-balancers that are located between subsystems. JWT provides application layer security between API Connect subsystems. The typical use-case for this feature is when payload inspection of all network communications is required at network boundaries. The communication flows that support JWT security are:

Management to portal.
Management to analytics.
Management to API gateway.
Management to v5 compatible gateway.
API gateway to analytics.
Note: It is not possible to use JWT and disable mTLS on the V5 compatible gateway to analytics message flow.

The management subsystem always sends a JWT when it initiates communication with the other subsystems, but by default the subsystems do not verify this JWT. When JWT security is enabled, the subsystems verify the JWT by making a REST call to the JSON Web Key Set (JWKS) URL. The JWKS URL is hosted on the management subsystem's platform REST API.

To enable JWT verification on management initiated communication, update the subsystem CRs to specify the JWKS URL. To disable JWT verification, update the subsystem CRs and delete the value of the JWKS URL (leave it unset).

To enable JWT verification on gateway to analytics communication, enable the Use JWT switch when you register the gateway in the Cloud Manager UI.

Note: For OpenShift users: The example steps in this topic use the Kubernetes kubectl command. On OpenShift, use the equivalent oc command in its place. If you are using a top-level CR you must edit the APIConnectCluster CR (the top-level CR), instead of directly in the subsystem CRs. If the subsystem section is not included in the top-level CR, copy and paste the section from the subsystem CR to the APIConnectCluster CR.

Disabling mTLS and enabling JSON Web Token (JWT) security

If you disable mTLS, you must enable JWT. It is not possible to configure API Connect with both mTLS and JWT disabled.

You disable mTLS on the portal, gateway (for management to gateway communication), and analytics subsystems by setting the mtlsValidateClient property to false in the Custom Resources (CRs) of each subsystem.

When mtlsValidateClient is set to false, the subsystem does not verify the client's TLS credentials, but it does check for a valid JWT configuration. For example, to disable mTLS and enable JWT for management to portal communication, edit the portal CR and set:

spec:
  mtlsValidateClient: false
  jwksUrl: <JWKS URL>

where <JWKS URL> is the jwksUrl specified in the management subsystem CR status.endpoints section.

Using registration of the portal service in the Cloud Manager UI as an example, JWT works as follows:

Management subsystem establishes a TLS connection to the portal subsystem on the portal director admin endpoint, and sends a JWT to the portal.
Before the portal trusts the management subsystem, it first verifies the JWT it received. The JWT is verified by a call to a JSON Web Key Set (JWKS) URL.
The management subsystem receives the JWT verification request from the portal on its platform REST API, and responds that the JWT is valid.
The portal receives confirmation that the JWT is valid, and so it can trust the management subsystem and continue the registration process.

To disable JWT security for management initiated communication, either remove the jwksUrl property from the CR, or leave its value empty. If you disable JWT, you must enable mTLS by setting mtlsValidateClient to true.

To disable JWT security for gateway to analytics communication, disable the Use JWT switch for the registered gateway in the Topology page of the Cloud Manager UI.

Configuring the JWKS URL

The JWKS URL is defined as the platform API hostname with the following subpath appended: api/cloud/oauth2/certs. You can see what the JWKS URL is from the status.endpoints section of the management CR, for example:

status:
...
  endpoints:
...
    - name: jwksUrl
      secretName: api-endpoint
      type: API
      uri: https://api.apic.acme.com/api/cloud/oauth2/certs

Note: The JWT feature is intended for environments where the are load-balancers located between subsystems. If you are using the in-cluster communications feature Use services instead of routes (OpenShift) or ingresses (Kubernetes), then you cannot have load-balancers between subsystems, and so disabling mTLS and enabling JWT is not necessary. However, if you do want to use the JWT feature with in-cluster communications then the JWKS URL can be obtained following these steps:

Describe any API Connect pod:
```
kubectl describe pod <apic-pod>
```

In the pod describe output, search for APIC_PLATFORM_API_SVC_ENDPOINT:

Containers:
  apim:
    Environment:
      APIC_PLATFORM_API_SVC_ENDPOINT:  https://management-juhu.acme-v1005-020323.svc:2000/api

The JWKS URL is the value of APIC_PLATFORM_API_SVC_ENDPOINT with /cloud/oauth2/certs appended to it. If the APIC_PLATFORM_API_SVC_ENDPOINT is missing the /api at the end, add api/cloud/oauth2/certs instead. The path of the JWKS URL must always be api/cloud/oauth2/certs.

Key points and limitations of JWT security

Changing the JWKS URL on portal, and analytics subsystems causes pods to restart, and so a short subsystem outage. The gateway pod does not restart when the JWKS URL is changed.
On new installs of Cloud Pak for Integration and top-level CR OpenShift deployments, the JWKS URL is set automatically for the portal, gateway, and analytics subsystems.
You can have both mTLS and JWT enabled, although the analytics subsystem does not verify the JWT if mTLS is enabled.
The v5c gateway does not send a JWT to the analytics subsystem. This flow is secured by mTLS, which cannot be disabled.
If you enable JWT for gateway to analytics communication, the ingestion of analytics data becomes dependent on the availability of the management subsystem (so that the JWT can be verified). For more information about this limitation, see: JWT usage between gateway and analytics subsystems.
Under high analytics ingestion loads, JWT enablement might impact the performance of the analytics subsystem. It is recommended to complete a performance test under your expected analytics load before you decide to use JWT for gateway to analytics communication.

JWT usage between gateway and analytics subsystems

Because the analytics subsystem makes a REST call to the management subsystems to verify the JWT it receives from the gateway, analytics ingestion depends on the management subsystem when JWT is enabled. The analytics subsystem caches the response from the JWKS URL so that it does not need to make a REST call to the management subsystem every time API event data is received from the gateway. But the JWT verification cache does expire after 24 hours, and a new REST call to the management subsystem is made to refresh the cache. The JWT dependency on the platform REST API means that an outage of the management subsystem does eventually prevent the analytics subsystem from accepting new analytics API event records.