Adjusting the CPU and memory reservations for integration server pods with the Vertical Pod Autoscaler

The Vertical Pod Autoscaler (VPA) is a Kubernetes resource that automates the process for setting resource limits and requests for the pods in a cluster. You can install the VPA in your cluster and then use it to configure autoscaling for your running integration server pods to ensure that they are not over resourced, or can be scaled up when demand increases.

You configure autoscaling for an integration server by deploying a VPA object that defines minimum and maximum CPU and memory limits for pod containers, with an update policy to automatically modify resource requests. The VPA will then monitor CPU and memory usage over an extended period of time, generate recommendations to apply to the integration server pods, and dynamically scale up or scale down as required.

Installing the VPA or assigning permissions to an existing VPA

If the VPA is not yet installed in your cluster, or if you want to update the existing version, you can clone the contents of the Kubernetes Autoscaler GitHub repository, and then use the supplied files and scripts to assign role-based access control (RBAC) permissions and complete the installation. If you want to use an existing deployment of the VPA in your cluster, you must ensure that the required RBAC permissions are assigned to the cluster roles. These permissions will enable the VPA to access the integration server custom resource and integration server scale API to perform autoscaling actions.

Note: Ensure that you have cluster administrator permissions.

Installing or updating the VPA in your cluster

To install the VPA or update an existing version that is installed in your cluster, complete the following steps:

  1. Ensure that you are logged in to your Red Hat OpenShift cluster by using the oc login command.
  2. From the GitHub command-line interface, clone the Kubernetes Autoscaler repository at https://github.com/kubernetes/autoscaler.git. If necessary, switch (cd) to a preferred directory where you want to save the clone before you run this command:
    git clone https://github.com/kubernetes/autoscaler.git
  3. From the cloned autoscaler directory, switch to the vertical-pod-autoscaler subdirectory:
    cd autoscaler/vertical-pod-autoscaler
  4. Run the following script to delete any existing version of the VPA in your cluster, which might cause conflicts with certificates, webhooks, or obsolete objects: 
    ./hack/vpa-down.sh
  5. Specify which autoscaling actions can be performed on deployed integration servers by assigning RBAC permissions to the cluster roles:
    1. Navigate to the cloned autoscaler/vertical-pod-autoscaler/deploy subdirectory and locate the file named vpa-rbac.yaml.
    2. Copy and paste the following custom resource (CR) settings at the end of each of the kind: ClusterRole sections in the file. 
        - apiGroups:
      - appconnect.ibm.com
    resources:
      - integrationservers
      - integrationservers/scale
    verbs:
      - get
      - list
      - watch

      For example, the first section should look like this after you add the CR settings:

      ---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-reader
rules:
  - apiGroups:
      - "metrics.k8s.io"
    resources:
      - pods
    verbs:
      - get
      - list
  - apiGroups:
      - appconnect.ibm.com
    resources:
      - integrationservers
      - integrationservers/scale
    verbs:
      - get
      - list
      - watch
    3. Save and close the file.
  6. From the cloned autoscaler/vertical-pod-autoscaler directory, run the following script to install the VPA in your cluster. The VPA is installed into the kube-system Kubernetes namespace by default. 
    ./hack/vpa-up.sh

    The VPA comprises three components (admission-controller, recommender, and updater), which should be identified as created in the command output. 

    customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictionter-binding created
serviceaccount/vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
serviceaccount/vpa-updater created
deployment.apps/vpa-updater created
serviceaccount/vpa-recommender created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus (2 primes)
...
    Note: If you see an unknown option -addext error message, this identifies an openssl issue, which you can resolve by using the ./hack/vpa-up.sh script on the 0.8 release branch on GitHub, or by upgrading to openssl V1.1.1 or higher.
  7. To verify the installation, run the following command to check that a deployment exists for each of the VPA components and that the pods have started successfully:
    oc get deployment -n kube-system

    You should see output similar to this:

    NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
vpa-admission-controller   1/1     1            1           25m
vpa-recommender            1/1     1            1           25m
vpa-updater                1/1     1            1           25m

If there are no issues with the pods, you can begin to configure autoscaling for your running integration servers.

Assigning permissions to an existing deployment of the VPA

If you have already installed the VPA in your cluster and want to use that deployment, you must ensure that autoscaling RBAC permissions are assigned to the cluster roles as follows:

  1. Ensure that you are logged in to your Red Hat OpenShift cluster by using the oc login command.
  2. To view a list of cluster roles that are defined for the VPA, run the following command:. 
    oc get clusterrole | grep vpa

    You should see output similar to this:

    system:vpa-actor                                                            2020-11-06T17:51:59Z
system:vpa-admission-controller                                             2020-11-06T17:52:01Z
system:vpa-checkpoint-actor                                                 2020-11-06T17:51:59Z
system:vpa-status-reader                                                    2020-11-06T17:52:01Z
system:vpa-target-reader                                                    2020-11-06T17:52:00Z
  3. Edit each of these cluster role definitions to add the supplied CR settings for the permissions. The following steps use the oc edit command, but you can use your preferred method to add the supplied CR settings.
    1. Run the following command:
      oc edit clusterrole clusterRoleDefinition

      For example:

      oc edit clusterrole system:vpa-actor

      The edit command will automatically open the YAML file in the default text editor for your operating system or another configured editor.

    2. Update the file by adding the following CR settings at the end of each of the kind: ClusterRole sections. 
        - apiGroups:
      - appconnect.ibm.com
    resources:
      - integrationservers
      - integrationservers/scale
    verbs:
      - get
      - list
      - watch
    3. Save the file, and then close the text editor to apply the changes.

Configuring autoscaling for your running integration servers

After you've installed the VPA and set ClusterRole permissions, you can configure the VPA to target your running integration servers. To configure autoscaling for an integration server, you must deploy a custom resource object of kind VerticalPodAutoscaler, which identifies the containers, the minimum and maximum limits, and the update policy for autoscaling.

Complete the following steps for any running integration server:

  1. From your local computer, create a YAML file by copying the following VerticalPodAutoscaler custom resource, where:
    • metadata.name is a unique name for this instance of the VerticalPodAutoscaler custom resource. (In this example, the value is shown as toolkit-vpa.)
    • spec.targetRef.name is set to the name of your integration server. (In this example, the value is shown as is-toolkit.)
    • spec.resourcePolicy.containerPolicies.minAllowed and spec.resourcePolicy.containerPolicies.maxAllowed are set to the lower and upper resource limits for containers that are defined for the integration server. 
    apiVersion: "autoscaling.k8s.io/v1"
kind: VerticalPodAutoscaler
metadata:
  name: toolkit-vpa
spec:
  targetRef:
    apiVersion: appconnect.ibm.com/v1beta1
    kind: IntegrationServer
    name: is-toolkit
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 50Mi
        maxAllowed:
          cpu: 2
          memory: 500Mi
        controlledResources: ["cpu", "memory"]
    Note: An update policy is not explicitly set in the supplied YAML because by default, the VPA will automatically apply resource recommendations by using this spec.updatePolicy.updateMode setting: 
    spec:
...
  updatePolicy:
    updateMode: "Auto"
  2. Save the file with a .yaml extension; for example, toolkit_vpa.yaml.
  3. From the command line, log in to your Red Hat OpenShift cluster by using the oc login command (if not already logged in).
  4. Run the following command to deploy a VerticalPodAutoscaler object. (Use the name of the .yaml file that you created.)
    oc apply -f toolkit_vpa.yaml

    The VPA will begin to query for metrics on the pods and after approximately 30 seconds, should begin to generate recommendations based on current and previous usage. (Under certain circumstances, this process might take longer.)

  5. Run the following command to view detailed information about the VerticalPodAutoscaler object, where vpaObjectName is the name that was specified as the metadata.name value:
    oc describe vpa vpaObjectName

    The following example shows the output of the oc describe command for a sample VerticalPodAutoscaler object called toolkit-vpa. The Status section of the output indicates when the recommendations were last provided, and shows the generated recommendations for containers belonging to an integration server called is-toolkit, after querying the resource metrics. 

    oc describe vpa toolkit-vpa 
Name:         toolkit-vpa
Namespace:    ace-cam
API Version:  autoscaling.k8s.io/v1
Kind:         VerticalPodAutoscaler
Metadata:
...
Spec:
  Resource Policy:
    Container Policies:
      Container Name:  *
      Controlled Resources:
        cpu
        memory
      Max Allowed:
        Cpu:     2
        Memory:  500Mi
      Min Allowed:
        Cpu:     100m
        Memory:  50Mi
  Target Ref:
    API Version:  appconnect.ibm.com/v1beta1
    Kind:         IntegrationServer
    Name:         is-toolkit
  Update Policy:
    Update Mode:  Auto
Status:
  Conditions:
    Last Transition Time:  2020-11-03T17:46:15Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  is-toolkit
      Lower Bound:
        Cpu:     100m
        Memory:  262144k
      Target:
        Cpu:     182m
        Memory:  272061154
      Uncapped Target:
        Cpu:     182m
        Memory:  272061154
      Upper Bound:
        Cpu:     2
        Memory:  500Mi
Events:          <none>

    A pod with resource requests that are less than the Lower Bound or greater than the Upper Bound recommendations will be recreated with the Target (optimal) recommendation. The Uncapped Target values show the most recent resource recommendations.

For more information about the Vertical Pod Autoscaler, see the README.md file in the GitHub repository.

Example: Testing autoscaling

If you have integration servers that are running close to the limits, the VPA should readily spring into action. However, if all your integration servers are well within the limits, you can test autoscaling by using a sample load. In this example, we will illustrate autoscaling for a sample integration server that is running close to the CPU limit.

  1. Download the attached perf-rating.bar.zip file.
  2. Extract the contents of the ZIP file to a directory on your local computer. This ZIP file contains a PerfRating.bar file for a Toolkit integration.
  3. From the App Connect Dashboard, deploy the PerfRating.bar file to an integration server.
    Deployed integration server
  4. Optional. Use Grafana to monitor the CPU usage for the pod. Or monitor CPU usage from the Red Hat OpenShift web console (Monitoring > Dashboards ); for example:
    Red Hat web console Dashboard for monitoring CPU usage of the pod
  5. Run the following command to obtain the HTTP route for the integration server:
    oc get routes

    In the output, take note of the entry that shows the integration server name appended by -http; for example, vpa-is-toolkit-http. You will need to specify its HOST/PORT value in the next step.

    Sample output of the oc get routes command
  6. To invoke the flow, run the following command, where BASEURL is the HOST/PORT value of the integrationServerName-http entry: 
    curl --request GET --url 'BASEURL:80/perf/rating?SequenceNumber=50&NumberOfIterations=2&NumberOfThreads=1' --header 'accept: application/json'

    On Windows, you can run the command as follows:

    curl --request GET --url "BASEURL:80/perf/rating?SequenceNumber=50&NumberOfIterations=2&NumberOfThreads=1" --header "accept: application/json"
  7. When the pod starts requesting resource beyond the lower limit, the VPA will revise the figures and proceed with cycling the pod to apply the updated figures.