Installing IBM Cloud Pak foundational services in an air-gapped environment by using a portable compute or storage device with ibm-pak plug-in

If your cluster is not connected to the internet, you can install IBM Cloud Pak foundational services in your cluster by using a file system. The portable storage and compute scenarios are enabled with a file system.

Before you begin

You must complete the steps in the following sections before you begin generating mirror manifests:

• Prerequisites
• Prepare a host
• Set environment variables and download CASE files

Prerequisites

No matter what medium you choose for your air-gapped installation, you must satisfy the following prerequisites:

• Red Hat® OpenShift® Container Platform requires you to have cluster admin access to run the install-operator command.

• An Red Hat® OpenShift® Container Platform cluster must be installed. For the supported Red Hat® OpenShift® Container Platform versions, see Supported OpenShift versions and platforms.

• Access to the following sites and ports:

  • *.docker.io and *.docker.com: For more information about specific sites to allow access to, see Docker Hub Hosts for Firewalls and HTTP Proxy Servers
  • icr.io:443 for IBM Entitled Registry and IBM Cloud Pak foundational services catalog source
  • quay.io:443 for IBM Cloud Pak foundational services catalog and images
  • github.com for CASEs and tools
  • redhat.com for Red Hat OpenShift Container Platform upgrades

Prepare a host

Regardless of which type of host you're using, you must be able to connect it to the internet and to the air-gapped network with access to the Red Hat® OpenShift® Container Platform cluster and the local, intranet Docker registry. Your host must be on a Linux® x86_64, Windows or Mac platform with any operating system that the ibm-pak plug-in, and the Red Hat® OpenShift® Container Platform CLI support. If you are on a Windows platform, you must execute oc ibm-pak launch in a Linux® x86_64 VM or from a Windows Subsystem for Linux (WSL) terminal.

The following table explains the software requirements for installing IBM Cloud Pak foundational services in an air-gapped environment:

Complete the following steps on your host:

1. Install Docker or Podman.

   Notes:

   • If you want to install a version of the foundational services prior to version 3.7.3 or ibm-cp-common-services-1.3.3.tgz (not included), install the Docker version between 18.x.x - 19.x.x, and run yum install <package name>-<version info>.
   • You can use Podman to install foundational services on Red Hat Enterprise Linux 8.x or later. Docker is not supported.

   To install Docker (for example, on Red Hat® Enterprise Linux®), run the following commands:

   Note: If you are installing as non-root you must use sudo. For more information, refer to the Podman or Docker documentation for installing as a non-root user.

   yum check-update
   yum install docker
   

   To install Podman, see Podman Installation Instructions .

2. Install the oc Red Hat® OpenShift® Container Platform CLI tool. For more information, see Red Hat® OpenShift® Container Platform CLI tools.

3. Download and install the most recent version of IBM Catalog Management Plug-in for IBM Cloud Paks from the IBM/ibm-pak-plugin. Extract the binary file by entering the following command:

   tar -xf oc-ibm_pak-linux-amd64.tar.gz
   

   Run the following command to move the file to the /usr/local/bin directory:

   Note: If you are installing as non-root you must use sudo. For more information, refer to the Podman or Docker documentation for installing as a non-root user.

   mv oc-ibm_pak-linux-amd64 /usr/local/bin/oc-ibm_pak
   

   Note: Download the plug-in based on the host operating system. You can confirm that oc ibm-pak -h is installed by running the following command:

   oc ibm-pak --help
   

   The plug-in usage is displayed.

   The plug-in is also provided in a container image cp.icr.io/cpopen/cpfs/ibm-pak:TAG where TAG should be replaced with the corresponding plugin version, for example cp.icr.io/cpopen/cpfs/ibm-pak:v1.2.0 will have v1.2.0 of the plug-in.

   The following command will create a container and copy the plug-ins for all the supported platforms in a directory, plugin-dir. You can specify any directory name and it will be created while copying. After copying, it will delete the temporary container. The plugin-dir will have all the binaries and other artifacts you find in a GitHub release and repo at IBM/ibm-pak-plugin.

   id=$(docker create cp.icr.io/cpopen/cpfs/ibm-pak:TAG - )
   docker cp $id:/ibm-pak-plugin plugin-dir
   docker rm -v $id
   cd plugin-dir
   

If you are installing foundational services by using a bastion host, the bastion server must be configured.

Creating registry namespaces

Top-level namespaces are the namespaces which appear at the root path of your private registry. For example, if your registry is hosted at myregistry.com:5000, then mynamespace in myregistry.com:5000/mynamespace is defined as a top-level namespace. There can be many top-level namespaces.

When the images are mirrored to your private registry, it is required that the top-level namespace where images are getting mirrored already exists or can be automatically created during the image push. If your registry does not allow automatic creation of top-level namespaces, you must create them manually.

In section 3.1 when you generate mirror manifests, you can specify the top-level namespace where you want to mirror the images by setting TARGET_REGISTRY to myregistry.com:5000/mynamespace which has the benefit of needing to create only one namespace mynamespace in your registry if it does not allow automatic creation of namespaces.

If you do not specify your own top-level namespace, the mirroring process will use the ones which are specified by the CASEs. For example, it will try to mirror the images at myregistry.com:5000/cp, myregistry.com:5000/cpopen etc.

So if your registry doesn't allow automatic creation of top level namespaces and you are not going to use your own during generation of mirror manifests then you must create the following namespaces at the root of your registry.

• cp
• cpopen

There can be more top-level namespaces which you might need to create. See section on Generate mirror manifests for more information on how to use the oc ibm-pak describe command to list all the top-level namespaces.

Set environment variables and download CASE files

If your bastion host, portable compute device, or portable storage device must connect to the internet via a proxy, you must set environment variables on the machine that accesses the internet via the proxy server. For detailed information, see Setting up proxy environment variables.

Before mirroring your images, you can set the environment variables on your mirroring device, and connect to the internet so that you can download the corresponding CASE files. To finish preparing your host, complete the following steps:

Note: Save a copy of your environment variable values to a text editor. You can use that file as a reference to cut and paste from when completing your air-gapped installation tasks.

1. Create the following environment variables with the installer image name and the version.

   export CASE_NAME=ibm-cp-common-services
   export CASE_VERSION=1.9.0
   

2. Connect your host to the internet and disconnect it from the local air-gapped network.

3. The plug-in can detect the locale of your environment and provide textual helps and messages accordingly. You can optionally set the locale by running the following command:

   oc ibm-pak config locale -l LOCALE
   

   where LOCALE can be one of de_DE, en_US, es_ES, fr_FR, it_IT, ja_JP, ko_KR, pt_BR, zh_Hans, zh_Hant.

4. Download the IBM Cloud Pak foundational services installer and image inventory to your host.

   Tip: If you do not specify the CASE version, it will download the latest CASE.

   oc ibm-pak get \
   $CASE_NAME \
   --version $CASE_VERSION
   

Notes:

• The plug in supports downloading the CASEs from cp.icr.io/cpopen since v1.2.0. You can issue the following command to configure a repository which will download the CASEs from cp.icr.io registry (an OCI-compliant registry) before running the oc ibm-pak get command to download the CASEs:

   oc ibm-pak config repo 'IBM Cloud-Pak OCI registry' -r oci:cp.icr.io/cpopen --enable
  

  • The CASEs are stored as artifacts in this OCI-compliant registry. The plug in downloads the CASEs from a GitHub repository by default. The preceding command will set 'IBM Cloud-Pak OCI registry' as default. You can view the current config by issuing the following command:

     oc ibm-pak config
    

  • The output lists all the configured repositories. The default repository from where the CASEs are downloaded has an asterisk mark (*) against the Name field.

• You can list all the available CASEs to download by running the following command:

   oc ibm-pak list
  

• To list the versions of all downloaded CASEs you can run the following command:

   oc ibm-pak list --downloaded
  

• Get more help about the list command by running the following command:

   oc ibm-pak list --help
  

By default, the root directory used by plug-in is ~/.ibm-pak. This means that the preceding command will download the CASE under ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION. You can configure this root directory by setting the IBMPAK_HOME environment variable. Assuming IBMPAK_HOME is set, the preceding command will download the CASE under $IBMPAK_HOME/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION.

The logs files will be available at $IBMPAK_HOME/.ibm-pak/logs/oc-ibm_pak.log.

Your host is now configured and you are ready to mirror your images.

1. Generate mirror manifests

1. Define the environment variable $TARGET_REGISTRY by running the following command:

   export TARGET_REGISTRY=<target-registry>
   

   The <target-registry> refers to the registry (hostname and port) where your images will be mirrored to and accessed by the oc cluster. For example: 172.16.0.10:5000

2. Generate mirror manifests by running the following command:

   oc ibm-pak generate mirror-manifests \
      $CASE_NAME \
      file://cpfs \
      --version $CASE_VERSION \
      --final-registry $TARGET_REGISTRY/cpfs
   

   The command will generate the following files at ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION:

   • images-mapping-to-filesystem.txt
   • images-mapping-from-filesystem.txt
   • image-content-source-policy.yaml
   • catalog-sourcesxxx.yaml

Tip: If you do not know the value of the final registry where the images will be mirrored, you can provide a placeholder value of TARGET_REGISTRY. For example: oc ibm-pak generate mirror-manifests $CASE_NAME file://cpfs --version $CASE_VERSION --final-registry TARGET_REGISTRY. Note that TARGET_REGISTRY used without any environment variable expansion is just a plain string that will be replaced later with the actual image registry URL when it is known to you.

Notes:

• Recommended: You can use the following command to list all the images that will be mirrored and the publicly accessible registries from where those images will be pulled from:

   oc ibm-pak describe $CASE_NAME --version $CASE_VERSION --list-mirror-images
  

  Tip: Note down the Registries found section at the end of output from the preceding command. You will need to log in to those registries so that the images can be pulled and mirrored to your local target registry. Top level namespaces found section shows the list of namespaces under which the images will be mirrored. These namespaces should be created manually in your registry root path if your registry doesn't allow automatic creation of the namespaces.

• Some products support the ability to generate mirror manifests only for a subset of images using the --filter argument and image grouping. The --filter argument provides the ability to customize which images are mirrored during an air-gapped installation. The foundational services CASE (ibm-cp-common-services) does not support filtering. As an example for this functionality ibm-cloud-native-postgresql CASE can be used, which contains groups that allow mirroring specific variant of ibm-cloud-native-postgresql (Standard or Enterprise). Use the --filter argument to target a variant of ibm-cloud-native-postgresql to mirror rather than the entire library. The filtering can be applied for groups and architectures.

  Consider the following command:

   oc ibm-pak generate mirror-manifests \
      ibm-cloud-native-postgresql \
      file://cpfs \
      --final-registry $TARGET_REGISTRY/cpfs \
      --filter $GROUPS
  

  The command was updated with a --filter argument. For example, for $GROUPS equal to ibmEdbStandard the mirror manifests will be generated only for the images associated with ibm-cloud-native-postgresql in its Standard variant. The resulting image group consists of images in the ibm-cloud-native-postgresql image group as well as any images that are not associated with any groups. This allows products to include common images as well as the ability to reduce the number of images that you need to mirror.

2. Mirror the images to the host

Complete these steps to mirror the images from the internet to your host:

1. You must run the following command to configure credentials for all target registries that require authentication. Run the command separately for each registry:

   export REGISTRY_AUTH_FILE=<path to the file which will store the auth credentials generated on podman login>
   podman login cp.icr.io
   podman login <TARGET_REGISTRY>
   

   Important: When you log in to cp.icr.io, you must specify the user as cp and the password which is your Entitlement key from the IBM Container registry. For example:

   podman login cp.icr.io
   Username: cp
   Password:
   Login Succeeded!
   

   For example, if you export REGISTRY_AUTH_FILE=~/.ibm-pak/auth.json, then after performing podman login, you can see that the file is populated with registry credentials.

2. If you use docker login, the authentication file is typically located at $HOME/.docker/config.json on Linux or %USERPROFILE%/.docker/config.json on Windows. After docker login, export REGISTRY_AUTH_FILE to point to that location. For example in Linux you can issue the following command:

export REGISTRY_AUTH_FILE=$HOME/.docker/config.json

Example ~/.ibm-pak directory structure

The ~/.ibm-pak directory structure is built over time as you save CASEs and mirror. The following tree shows an example of the ~/.ibm-pak directory structure:

 tree ~/.ibm-pak
/root/.ibm-pak
├── config
│   └── config.yaml
├── data
│   ├── cases
│   │   └── ibm-cp-common-services
│   │       └── 1.9.0
│   │           ├── XXXX
│   │           ├── XXXX
│   └── mirror
│       └── ibm-cp-common-services
│           └── 1.9.0
│               ├── catalog-sources.yaml
│               ├── image-content-source-policy.yaml
│               ├── images-mapping-to-filesystem.txt
│               └── images-mapping-from-filesystem.txt
└── logs
    └── oc-ibm_pak.log

3. Mirror images to the file system

1. Mirror images to the file system.

   export IMAGE_PATH=<image-path>
   oc image mirror \
     -f ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-to-filesystem.txt \
     --filter-by-os '.*' \
     -a $REGISTRY_AUTH_FILE \
     --insecure \
     --skip-multiple-scopes \
     --max-per-registry=1 \
     --continue-on-error=true \
     --dir "$IMAGE_PATH"
   

   The <image-path> refers to the local path to store the images. For example, in the previous section if you provided file://cpfs as input during generate mirror-manifests, the preceding command will create a subdirectory cpfs under <image-path> and copy the images under it.

oc image mirror --help

The above command can be used to see all the options available on the mirror command. Note that we use continue-on-error to indicate that command should try to mirror as much as possible and continue on errors.

NOTE - Sometimes based on the number and size of images to be mirrored, the oc image mirror might take longer. If you are issuing the command on a remote machine it is recommended that you run the command in the background with a nohup so even if network connection to your remote machine is lost or you close the terminal the mirroring will continue. For example, the below command will start the mirroring process in background and write the log to my-mirror-progress.txt.

export IMAGE_PATH=<image-path>
 nohup oc image mirror \
   -f ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-to-filesystem.txt \
   --filter-by-os '.*' \
   -a $REGISTRY_AUTH_FILE \
   --insecure \
   --skip-multiple-scopes \
   --max-per-registry=1 \
   --continue-on-error=true \
   --dir "$IMAGE_PATH" > my-mirror-progress.txt  2>&1 &

You can view the progress of the mirror by issuing the below command on the remote machine.

tail -f my-mirror-progress.txt

1. Continue to move the following items to your file system:

   • The <image-path> directory you specified in the previous step
   • The auth file referred by $REGISTSRY_AUTH_FILE
   • ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-from-filesystem.txt

4. Mirror images to the target registry from file system

Complete the steps in this section on your file system to copy the images from the file system to the $TARGET_REGISTRY. Your file system must be connected to both the target docker registry and the Red Hat® OpenShift® Container Platform cluster.

Important: If you used the placeholder value of TARGET_REGISTRY as a parameter to --final-registry at the time of generating mirror manifests, then before running the following command, find and replace the placeholder value of TARGET_REGISTRY in the file, images-mapping-from-filesystem.txt, with the actual registry where you want to mirror the images. For example, if you want to mirror images to myregistry.com/mynamespace then replace TARGET_REGISTRY with myregistry.com/mynamespace.

1. Run the following command to copy the images (referred in the images-mapping-from-filesystem.txt file) from the v2 directory to the final target registry:

   export IMAGE_PATH=<image-path>
   oc image mirror \
     -f ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-from-filesystem.txt \
     -a $REGISTRY_AUTH_FILE \
     --from-dir "$IMAGE_PATH" \
     --filter-by-os '.*' \
     --insecure \
     --skip-multiple-scopes \
     --max-per-registry=1 \
     --continue-on-error=true
   

1. Update the global image pull secret for your Red Hat OpenShift cluster. Follow the steps in Updating the global cluster pull secret. The documented steps in the link enable your cluster to have proper authentication credentials in place to pull images from your TARGET_REGISTRY as specified in the image-content-source-policy.yaml which you will apply to your cluster in the next step.

2. Create ImageContentSourcePolicy

   Important:

   • Before you run the command in this step, you must be logged into your OpenShift cluster. Using the oc login command, log in to the Red Hat OpenShift Container Platform cluster where your final location resides. You can identify your specific oc login by clicking the user drop-down menu in the Red Hat OpenShift Container Platform console, then clicking Copy Login Command.

   • If you used the placeholder value of TARGET_REGISTRY as a parameter to --final-registry at the time of generating mirror manifests, then before running the following command, find and replace the placeholder value of TARGET_REGISTRY in file, ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml with the actual registry where you want to mirror the images. For example, replace TARGET_REGISTRY with myregistry.com/mynamespace.

   oc apply -f  ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml
   

   If you are using Red Hat OpenShift Container Platform version 4.7 or earlier, this step might cause your cluster nodes to drain and restart sequentially to apply the configuration changes.

3. Verify that the ImageContentSourcePolicy resource is created.

   oc get imageContentSourcePolicy
   

4. Verify your cluster node status and wait for all the nodes to be restarted before proceeding.

   oc get MachineConfigPool -w
   

5. Create a new project for the CASE commands by running the following commands:

   Note: You must be logged in to a cluster before performing the following steps.

   export NAMESPACE=ibm-common-services
   

   oc new-project $NAMESPACE
   

6. Optional: If you use an insecure registry, you must add the target registry to the cluster insecureRegistries list.

   oc patch image.config.openshift.io/cluster --type=merge \
   -p '{"spec":{"registrySources":{"insecureRegistries":["'${TARGET_REGISTRY}'"]}}}'
   

7. Verify your cluster node status and wait for all the nodes to be restarted before proceeding.

   oc get MachineConfigPool -w
   

   After the ImageContentsourcePolicy and global image pull secret are applied, the configuration of your nodes will be updated sequentially. Wait until all MachineConfigPools are updated.

5. Install IBM Cloud Pak foundational services by way of Red Hat OpenShift Container Platform

Now that your images are mirrored to your air-gapped environment, you can deploy your IBM Cloud® Paks to that environment. When you mirrored your environment, you created a parallel offline version of everything that you needed to install an operator into Red Hat® OpenShift® Container Platform. To install the IBM Cloud Pak foundational services, complete the following steps:

• 5.1. Create the catalog source and install the foundational services
• 5.2. Access the console
• 5.3. Retrieve your console username and password

5.1. Create the catalog source and install the IBM Cloud Pak foundational services

1. Set the namespace to install the foundational services catalog:

   export NAMESPACE=ibm-common-services
   

2. Set the environment variable of the --inventory parameter:

   export CASE_INVENTORY_SETUP=ibmCommonServiceOperatorSetup
   

3. Create and configure a catalog source.

   Important: Before you run the command in this step, you must be logged into your OpenShift cluster.

   Using the oc login command, log in to the Red Hat OpenShift Container Platform cluster where your final location resides. You can identify your specific oc login by clicking the user drop-down menu in the Red Hat OpenShift Container Platform console, then clicking Copy Login Command.

   The recommended way to install the catalog is to run the following command:

   oc apply -f ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/catalog-sources.yaml
   

   The below command can also be used to install the catalog.

   oc ibm-pak launch \
   $CASE_NAME \
     --version $CASE_VERSION \
     --action install-catalog \
     --inventory $CASE_INVENTORY_SETUP \
     --namespace $NAMESPACE \
     --args "--registry $TARGET_REGISTRY/cpfs --recursive --inputDir ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION"
   

   Note: In foundational services version 3.13 and previous versions, the install-catalog command will deploy catalogsource with the latest tag. Starting from foundational services version 3.14, install-catalog deploys the catalog source ,opencloud-operators, with catalogsource image digest.

4. Verify that the CatalogSource for foundational services installer operator is created.

   oc get pods -n openshift-marketplace
   oc get catalogsource -n openshift-marketplace
   

5. Optional: Create a configmap if you want to install foundational services in a custom namespace. By default, the foundational services are installed in the ibm-common-services namespace. For more information, see Installing IBM Cloud Pak foundational services in a custom namespace.

   Important: You can install foundational services in only one namespace in your cluster.

6. Install the foundational services operators.

   Note: You must have cluster admin access to run the install-operator command. However, you do not need cluster admin access for mirroring.

   oc ibm-pak launch \
      $CASE_NAME \
      --version $CASE_VERSION \
      --inventory $CASE_INVENTORY_SETUP \
      --action install-operator \
      --namespace $NAMESPACE
   

   Note: The foundational services are by default installed with the starterset deployment profile. You can change the profile to small by adding --args "--size small", or to large by adding --args "--size large", to the command. See the following example:

   oc ibm-pak launch \
      $CASE_NAME \
      --version $CASE_VERSION \
      --inventory $CASE_INVENTORY_SETUP \
      --action install-operator \
      --namespace $NAMESPACE \
      --args "--size small"
   

7. Using the oc login command, log in to the Red Hat® OpenShift® Container Platform cluster where your final location resides. You can identify your specific oc login by clicking the user drop-down menu in the Red Hat OpenShift Container Platform console, then clicking Copy Login Command.

8. Verify that the foundational services are installed:

   oc get pod -n ibm-common-services
   

   It might take up to 15 minutes for all the pods to show the Running status.

   Note: If you want to deploy Db2, you must load the Db2 CASE bundle and update the OperandRegistry before you create OperandRequest.

   oc edit opreg common-service -n ibm-common-services
   

   Replace the catalog source value from ibm-operator-catalog to ibm-db2uoperator-catalog such that the catalog source looks like the following example:

   - channel: v1.0
     name: ibm-db2u-operator
     namespace: placeholder
     packageName: db2u-operator
     scope: public
     sourceName: ibm-db2uoperator-catalog
     sourceNamespace: openshift-marketplace
   

5.2. Access the console

Use the following command to get the URL to access the console:

oc get route -n ibm-common-services cp-console -o jsonpath=‘{.spec.host}’

The preceding command returns the following output:

cp-console.apps.mycluster.mydomain.com

Based on the example output, your console URL would be https://cp-console.apps.mycluster.mydomain.com.

5.3. Retrieve your console username and password

The default username to access the console is admin.

You can get the password for the admin username by running the following command:

oc -n ibm-common-services get secret platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -d

The following password is an example of the output of the preceding command:

EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsA

Based on the example output, you would then use EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsA as the password.

You can change the default password at any time. For more information, see Changing the cluster administrator password

Note: Any user with access to the ibm-common-services namespace can retrieve this password since it is stored in a secret in the ibm-common-services namespace. To minimize password exposure, allow limited users to access the ibm-common-services namespace.

You have now successfully created and deployed your air-gapped instance of IBM Cloud Pak foundational services.

For more information about troubleshooting your air-gapped installation, see Troubleshooting an air-gapped installation.