Deployment requirements

Describes the system requirements for deploying into the Kubernetes, OpenShift, and Cloud Pak for Integration runtime environments.

1. Verify that you have supported software.
   • Refer to the IBM Software Product Compatibility Reports site for detailed requirements for operating systems, supporting software, and other prerequisites. See Detailed system requirements for a specific product
   • For new installations, the API Connect operators and operands must be from the same version of API Connect. This requirement applies to native Kubernetes and OpenShift installations.
     • Note that the API Connect operator version number does not match the API Connect release version number. You can view a mapping of API Connect release number to API Connect operator version numbers in Deploying operators and cert-manager.

     • For API Connect operands (Management, Portal, Analytics, and Gateway subsystems), the version number matches the API Connect release version number.

2. Verify that your Kubernetes environment is running and meets the configuration requirements for API Connect:
   • When using IBM Cloud Kubernetes Service, ingress-nginx is required.
   • Specify settings for the ingress controller for a Kubernetes runtime environment. See Kubernetes ingress controller prerequisites.
   • The timezone for API Connect pods is set to UTC. Do not change the timezone for the pods.
   • To ensure the API Connect services have time to start, we recommend increasing the proxy-read-timeout and proxy-send-timeout values in the config.map used to configure the kubernetes/ingress-nginx ingress controller. The following settings should be increased:
     • proxy-read-timeout: "240"
     • proxy-send-timeout: "240"
     240 seconds (4 minutes) is a recommended minimum; actual value will vary depending upon your environment. If there is a load balancer in front of the worker node(s), then the load balancer configuration may also need to have extended timeouts.
   • Storage type:

     When selecting storage type, review the following considerations:

     • Management subsystem requires block storage.
     • Developer Portal subsystem supports block storage or local volume storage.
     • Analytics subsystem supports block storage or local volume storage.

     Supported storage types

     • Block storage

       Select a block storage format of your choosing in order for API Connect components to be supported. Supported block storage types include, but are not limited to:

       • Rook Ceph

         Note: cephrbd (RADOS Block Device) is recommended over cephfs file system

       • OpenShift Container Storage
       • Portworx
       • IBM Cloud Block Storage
       • AWS: the gp2, gp3, io1, and io2 types of Elastic Block Storage (EBS) are supported.
       • IBM Spectrum Scale with Block Storage
       • Azure Premium Storage
       • GCE Persistent Disk

       See also the API Connect v10 WhitePaper 1.5 for information on limitations observed with other block storage solutions: https://community.ibm.com/community/user/integration/viewdocument/api-connect-deplyoment-whitepaper-v?CommunityKey=2106cca0-a9f9-45c6-9b28-01a28f4ce947&tab=librarydocuments&LibraryFolderKey=&DefaultView=folder

     • Local volume storage

       Management subsystem and Developer Portal subsystem

       All subsystems will run on local volume storage but it is not recommended for the Management subsystem and Developer Portal subsystem because it has severe performance impacts for high availability deployments, and for pod allocation.

       Note also that local volume storage is not available in most cloud environments, and might incur limitations in disaster recovery scenarios, since it is tied to the node on which the pod is run. This means that use of LVS might be adequate for deployments with a one replicaprofile, but inadequate for deployments with a three replica profile.

       Analytics subsystem

       For Analytics subsystems, transactions per second (TPS) is three times faster with local volume storage than with Ceph block storage. In a production scenario, you will have multiple analytics-storage (OpenSearch) nodes forming a storage cluster. OpenSearch will manage data replication across its cluster and ensure high availability, thus mitigating possible local volume storage drawbacks with disaster recovery.

     Non-supported storage

     • NFS

       API Connect v10 cannot be deployed on NFS.

     • GlusterFS

       GlusterFS is not recommended as a storage option due to severe performance degradation and in some cases, loss of data. There are known GlusterFS issues that affect OpenSearch (which is used in the API Connect analytics service), the Developer Portal and the Management subsystem

   • Storage class configuration:

     In your Kubernetes cluster configuration, you must create storage classes with the setting:

     volumeBindingMode: WaitForFirstConsumer

     This setting is needed to avoid volume node affinity conflicts, such as those caused by requiredPodAntiAffinity restrictions. If WaitForFirstConsumer is the binding mode, PVCs (PersistentVolumeClaims) are only bound once the pod is actually created.

     By contrast, when the binding mode is immediate (volumeBindingMode: Immediate), the PVCs are bound to PVs (PersistentVolumes) as soon as the PV is created.

     • Example:

       The Postgres operator creates the PVC first and then deploys the deployments. In immediate mode, PVCs are bound to PVs as soon as the PV is created. The pgbackrest pod starts successfully, but because requiredPodAntiAffinity restrictions are set, and the postgres pod is NOT ALLOWED to schedule on the same node as pgbackrest pod, Kubernetes cannot schedule the pod in node eu-west-1c. Also, because the volume attached to the pod is at eu-west-1c, Kubernetes cannot move the pod to any other node, thus causing volume node affinity conflict.

     • Example:

       When deploying to a multi-zone cluster, you must use StorageClass with WaitForFirstConsumer because when binding mode is immediate, successful deployment is indeterminate because of interactions between pod/node affinity and the Kubernetes provisioning of the PV on the right node where the pod is being provisioned.

     For more information, see "Volume Binding Mode" on https://kubernetes.io/docs/concepts/storage/storage-classes/.

     Note: During deployment, you are required to specify the name of the storage class you created. You can either use the setting storageClassName: $STORAGE_CLASS in the applicable CR (such as apiconnect top-level CR or an individual subsystem CR), or you can use a configuration UI such as Platform Navigator UI or OpenShift Container Platform (OCP) UI.
   • When the Analytics service is configured to store data, (that is, it is not configured for ingestion-only), the service depends on OpenSearch, which requires map counts higher than the operating system defaults. See: Installing the Analytics subsystem.

     If you are deploying on OpenShift, the map count is preconfigured (vm.max_map_count = 1027580) with a value that is sufficiently large for OpenSearch so you do not need to modify this setting.

   • Ensure that your Kubernetes deployment addresses known security vulnerabilities with SSH to prevent the use of weak SSH Message Authentication Code (MAC) and Key exchange algorithms (KexAlgorithms) with TCP over port 22. Use of weak algorithms can allow an attacker to recover the plain text message from the encrypted text. If you have not yet addressed the security vulnerabilities with weak MACs and KexAlgorithms, update your configuration manually by adding the following lines to /etc/ssh/sshd_config:

     KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
     MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

   • IBM API Connect requires a number of container registry repositories to be added to your container registry service if automatic repository creation is not supported. Follow the procedure in Step 4 of Obtaining product files.

   • Logging:
     • Configure a remote server for logging. See Logging. Pods are terminated during an upgrade, and the logs will be lost if not stored remotely.
     • Configure the log rotation by setting the max-size and max-files values to a setting appropriate for your deployment. In a Kubernetes deployment, insufficient values for max-size and max-files can cause logs to grow too large and eventually force pods to restart.

       For more information, visit the Docker documentation and search for "JSON File logging driver".

3. Ensure your deployment meets the DNS requirements for API Connect domain names, host names, and endpoints.

   You specify endpoints when you edit the custom resource templates for each subsystem, and when you use the Cloud Manager UI to configure the services for your cloud. The domain names you use for the endpoints must be recognized by your DNS configuration. DNS entries must meet these requirements:

   Important: For API Connect on native Kubernetes and OpenShift, do not use coredns 1.8.1 through 1.8.3. These releases have a defect that can prevent pods for Developer Portal from starting. See https://coredns.io/.

   • Endpoints cannot contain the underscore character "_" or uppercase characters.
   • Host names that are used to create endpoints cannot contain the underscore character "_" or uppercase letters.
   • DNS must be configured with domain names that correspond to the endpoints configured at installation time for each subsystem.
   • Host names and DNS entries may not be changed on a cluster after the initial installation.
   • Kubernetes ingress limits the character set for DNS names to not support the underscore character "_". This means you cannot specify underscores in domain names that are used as endpoints. For example, my_domain.abc.com and my.domain_abc.com are not supported for <xxx>.<hostname>.<domainname>, and will cause an error:

     Invalid value: "my_domain.abc.com": a DNS-1123 subdomain must consist of lowercase alphanumeric characters, '-' or '.', 
     and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is
      '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

   • We recommend that you keep a record of the DNS entries and endpoint names, as you will need to map the DNS entries and use the same endpoints when restoring a backup of the management database. Note that endpoints are FQDNs, entered in all lowercase. For an overview on endpoints, review Deployment overview for endpoints and certificates
   • Specifying domain names when deploying CRs (custom resources) for each subsystem:

     The custom resource templates provide endpoint settings that consist of a default endpoint prefix and a variable $STACK_HOST that you set to specify your domain name, as part of Installing the API Connect subsystems.

     Table 1. Endpoints with domain names that must be configured in DNS

     Default endpoint name	Description
     admin.$STACK_HOST	Cloud Manager URL endpoint for the Management subsystem
     manager.$STACK_HOST	API Manager URL endpoint for the Management subsystem
     api.$STACK_HOST	Platform REST API Endpoint for admin and provider APIs
     consumer.$STACK_HOST	Platform REST API Endpoint for consumer APIs for the Management subsystem.
     rgw.$STACK_HOST	Gateway Endpoint for API invocation for DataPower API Gateway
     rgwd.$STACK_HOST	Gateway Management Endpoint for DataPower API Gateway
     gw.$STACK_HOST	Gateway Endpoint for API invocation for DataPower Multi-Protocol Gateway (v5-compatible)
     gwd.$STACK_HOST	Gateway Management Endpoint for DataPower Multi-Protocol Gateway (v5-compatible)
     ai.$STACK_HOST	Analytics ingestion endpoint for the Analytics subsystem
     api.portal.$STACK_HOST	Developer Portal admin endpoint
     portal.$STACK_HOST	Developer Portal UI endpoint

   • Specifying endpoints using the Cloud Manager interface

     After initial deployment of the Kubernetes custom resource for each subystem, you will also enter the endpoints in the Cloud Manager user interface when defining each service as part of Defining your topology.

4. The Cloud Provider for the target Kubernetes platform must provide Ingress Controller support for SSL Passthrough.

   For example, if the Cloud Provider uses or suggests the use of the NGINX Community Ingress, then the --enable-ssl-passthrough flag would need to be provided at runtime, as described at https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough. For IBM Cloud Kubernetes Service, see https://cloud.ibm.com/docs/containers?topic=containers-ingress-user_managed..

5. Determine your strategy for using certificates and cert-manager with API Connect.

   If you haven't already, review the topics in the section Certificates in a Kubernetes environment.

   Consider the following:

   • Will you use a cert-manager (recommended) or manually manage your certificates?

     API Connect provides a cert-manager and default certificates. If you use these, you can follow default configuration steps. See Deploying operators and cert-manager.

   • Will you use custom certificates rather than default certificates? See Custom certificates on Kubernetes.
6. If you plan to deploy API Connect on Kubernetes with DataPower Gateway in a non-Kubernetes environment, such as a DataPower appliance, review the API Connect requirements for operation with DataPower Gateway:
   • When deploying with a DataPower appliance, configure a Gateway Service Management Endpoint and API invocation Endpoint in DataPower. See Configuring the API Connect Gateway Service in the appropriate version of the DataPower documentation.
   • Ensure that DataPower Gateway firmware version you plan to install is compatible with the API Connect Management server version.
     Note:

     You can use any combination of API Connect 10.0.5.x with DataPower Gateway 10.5.0.x or DataPower API Gateway 10.5.0.x.

     Before you install, best practice is to review the latest compatibility support for your version of API Connect. To view compatibility support, follow the instructions in IBM API Connect Version 10 software product compatibility requirements to access API Connect information on the Software Product Compatibility Reports website. Once you access information for your version of API Connect, select Supported Software > Integration Middleware, and view the list of compatible DataPower Gateway versions.

7. If you plan to deploy the API Connect management subsystem and the DataPower Gateway subsystem on a Kubernetes deployment, managed by the DataPower and API Connect Operators, you can use any combination of API Connect 10.0.5.x with DataPower Gateway 10.5.0.x or DataPower API Gateway 10.5.0.x.
8. Observe namespace restrictions:
   • OpenShift

     Only one top level CR (APIConnectCluster) can be deployed in each namespace.

   • Cloud Pak for Integration

     In the Platform Navigator UI, only one APIConnect capability can be deployed in each namespace.

   • Native Kubernetes

     Only one instance of a subsystem can be deployed in each namespace. Deploying multiple similar subsystems in a single namespace is not supported. For example, there is no support for deploying two management subsystems, mgmt1, mgmt2 in a single namespace where apiconnect operator is installed.

9. The following kernel or Kubernetes node limits for the Developer Portal subsystem must be set correctly:
   • Ensure that your kernel or Kubernetes node has the value of its inotify watches set high enough so that the Developer Portal can monitor and maintain the files for each Developer Portal site. If set too low, the Developer Portal containers might fail to start or go into a non-ready state when this limit is reached. If you have many Developer Portal sites, or if your sites contain a lot of content, for example, many custom modules and themes, then a larger number of inotify watches are required. You can start with a value of 65,000, but for large deployments, this value might need to go up as high as 1,000,000. The Developer Portal containers take inotify watches only when they need them. The full number is not reserved or held, so it is acceptable to set this value high.
   • Ensure that your kernel or Kubernetes node has a value of nproc (maximum number of processes) that applies to the user ID of the Portal pods that have been assigned, and that it is high enough to allow all of the Portal processes to execute. For smaller installations this might be as low as 16384, but for larger installations that have more concurrent web calls, you might need as many as 125205. If the number is too low, you will see errors like "fork: retry: Resource temporarily unavailable" in the Portal logs. Note that this value might need to be even higher if other, non-Portal, pods are sharing the same user ID.
   • Ensure that your kernel or Kubernetes node has a value of nofiles (maximum number of open file descriptors) that applies to the user ID of the Portal pods that have been assigned, and that it is high enough to allow the Portal to open all of the files that it requires. For smaller installations this might be as low as 16384, but for larger installations that have more concurrent web calls and Portal web sites, you might need as many as 1048576. If the number is too low, you will see errors like "too many open files" in the Portal logs. Note that this value might need to be even higher if other, non-Portal, pods are sharing the same user ID.