Istio-pilot pods have high CPU usage

Symptoms

• istio-pilot pods have high CPU usage.
• Frequent 503 Service Unavailable error from istio-ingressgateway in the Ingress pod logs
• Almost all microservices are down.

Resolving the problem

Set the PILOT_PUSH_THROTTLE and PILOT_DEBOUNCE_AFTER environment variables to reduce the CPU usage for istio-pilot. The variables need to be set in the env: section of container discovery in deployment istio-pilot.

Complete the following steps:

1. Create istio-patch.yaml with the following content:

    spec:
      template:
        spec:
          containers:
          - name: discovery
            env:
            - name: PILOT_PUSH_THROTTLE
              value: "1"
            - name: PILOT_DEBOUNCE_AFTER
              value: "10s"
   

2. Run the patch command to update the deployment:

    kubectl --namespace=istio-system patch deployment istio-pilot --patch "$(cat istio-patch.yaml)"
   

   After running the patch command, the env: section of container discovery in deployment istio-pilot should include these values:

            - name: PILOT_PUSH_THROTTLE
              value: "1"
            - name: PILOT_DEBOUNCE_AFTER
              value: 10s