Air-gapped installs

In production, it is common to have a cluster that does not have internet access. If your Red Hat OpenShift Container Platform cluster is air-gapped (otherwise known as offline or disconnected), you can install IBM Storage Scale container native by mirroring images. These tasks must be ran by a Red Hat OpenShift administrator.

A bastion host is a device that has access to both the public internet and the network-restricted environment where a local registry and Red Hat OpenShift Container Platform clusters reside. Using the bastion host, you can mirror your images directly to the local registry.

You need to do the Air gap setup if the worker nodes are not able to access the repository due to network and firewall restrictions.

Prerequisites

A production-grade Docker V2 registry that is available and accessible from the OpenShift Container Platform cluster nodes such as Quay Enterprise, JFrog Artifactory, or Docker Registry. The Red Hat OpenShift Internal Registry is not supported.
A bastion node that:
- Access to the public internet.
- Access to the network-restricted environment where the local registry exists.
- Access to the entitled registry: icr.io:443, cp.icr.io:443.
- skopeo command is installed.
Access to the Red Hat OpenShift Container Platform cluster as a user with the cluster-admin role.

Do not use Red Hat OpenShift internal image registry as your local registry. The Red Hat OpenShift registry does not support multi-architecture images or path separators in the image name.

Create an ImageContentSourcePolicy resource

Create a ImageContentSourcePolicy resource on your Red Hat OpenShift cluster to enable the redirection of requests to pull images from a repository on a mirrored image registry. Complete the following steps from the bastion host:

Create the sample registrymirror.yaml. Edit and replace example.io/<path_to_images> to match the path structure of your internal image registry.

 cat << EOF > registrymirror.yaml
 apiVersion: operator.openshift.io/v1alpha1
 kind: ImageContentSourcePolicy
 metadata:
   name: icr-mirror
 spec:
   repositoryDigestMirrors:
   - mirrors:
     - example.io/<path_to_images>
     source: cp.icr.io/cp/spectrum/scale
   - mirrors:
     - example.io/<path_to_images>
     source: icr.io/cpopen
 EOF

Do not prefix mirrors with http:// or https:// and make sure that there is no trailing / character.

Apply the sample to create an ImageContentSourcePolicy named icr-mirror:
```
 oc apply -f registrymirror.yaml
```
Verify that the ImageContentSourcePolicy resource is created:
```
 oc get imageContentSourcePolicy
```
This update is rolled out to all nodes. Verify your cluster node status and wait for all nodes to be updated before proceeding:
```
 oc get MachineConfigPool
```

For more information, see About disconnected installation mirroring in Red Hat OpenShift documentation.

Mirror the images

Mirror the images by copying images from the source registry to the internal image registry.

Complete the following steps from the bastion node:

Log in to the IBM Entitled Container Registry with the skopeo command:
```
 skopeo login cp.icr.io
```
Log in to the internal image registry with the skopeo command:
```
 skopeo login example.io
```
Use skopeo to copy the images from the IBM Entitled Container Registry to your internal image registry.

The image listing for each release is available on GitHub. See Container image list for IBM Storage Scale container native in the GitHub repository.

Copy the air-gapped images from GitHub to a file called images.txt. You can use the following helper script to get started in creating the skopeo copy commands to run:
```
 # Set the INTERNAL registry name
 export INTERNAL=example.io
 for image in `cat images.txt | grep -v \#`; do
     if [[ $image == *"cpopen"* ]]; then
         echo "skopeo copy --all docker://${image} docker://$INTERNAL/${image#*icr.io/}"
     else
         echo "skopeo copy --all docker://${image} docker://$INTERNAL/${image#*cp.icr.io/}"
     fi
 done
```
The helper script is provided as an example. Use at your own risk.

A generic skopeo copy command is:
```
 skopeo copy --all docker://<source image registry>/<image> docker://<internal image registry>/<image>
```
For more information, see Skopeo Copy to the Rescue.
Log out of the IBM Entitled Container Registry by entering the skopeo command.
```
 skopeo logout cp.icr.io
```
Log out of your internal image registry by entering the skopeo command.
```
 skopeo logout example.io
```

Red Hat OpenShift Container Registry pull secret

For images to be properly pulled at the pod level, the Red Hat OpenShift global pull secrets must be modified to contain credentials to access your internal Container Registry.

Complete the following steps:

Create a base64 encoded string of the credentials used to access your internal Container Registry.

The following example uses example.io/subdir as the internal Container Registry.
- Use the credentials to access your example.io/subdir internal Container Registry.
```
echo -n "<username>:<password>" | base64 -w0
```
Create an authority.json to include the base64 encoded string of your credentials. Use your username and password to access the internal Container Registry example.io/subdir.
```
{
  "auth": "<base64 encoded string from previous step>",
  "username":"<example.io username>",
  "password":"<example.io generated entitlement key>"
}
```

Enter the following command to include the authority.json as a new authority in your .dockerconfigjson and store it as temp_config.json:

If the internal Container Registry is example.io/subdir, use example.io as the input key for the contents of authority.json.

# Set the INTERNAL registry name
export INTERNAL=example.io

oc get secret/pull-secret -n openshift-config -ojson | \
jq -r '.data[".dockerconfigjson"]' | \
base64 -d - | \
jq --arg key $INTERNAL '.[] | .[$key] += input' - authority.json > temp_config.json

This command is supported by jq version 1.5 or higher.

Enter the following command to verify that your authority credentials were created in the resulting file:

  cat temp_config.json

The content of temp_config.json displays all your existing credentials for global pull secrets with example.io appended at the end:

   # cat temp_config.json
   {
       ...
       ...
         "example.io": {
            "auth": "<base64 encoded string created in previous step>",
            "username": "<example.io username>",
            "password": "<example.io password>"
         }
      }
   }

Apply the pull secret configuration to the Red Hat OpenShift cluster.

 oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=temp_config.json

Verify that your pull-secret is updated with your new authority, enter the following command and confirm that your authority is present:
```
 oc get secret/pull-secret -n openshift-config -ojson | \
 jq -r '.data[".dockerconfigjson"]' | \
 base64 -d -
```
This update is rolled out to all nodes. Verify your cluster node status and wait for all nodes to be updated before proceeding:
```
 oc get MachineConfigPool
```
Remove the temporary files that were created.
```
rm authority.json temp_config.json
```

Testing the pull of images from the mirrored registry

Complete the following steps from the bastion node:

Pick a worker node from oc get nodes and start a debug pod:
```
 oc debug node/<worker node>
```

When you see a prompt, use host binaries by entering the chroot /host command.

 # oc debug node/worker0.example.com
 Starting pod/worker0examplecom-debug ...
 To use host binaries, run `chroot /host`
 Pod IP: 12.34.56.78
 If you don't see a command prompt, try pressing enter.
 # chroot /host

Use the podman login command to authenticate your mirrored image registry.
```
 podman login example.io
```
Attempt to pull one of the images from the source image registry by using podman. The Red Hat OpenShift cluster must be able to redirect the request from the external image registry to the internal image registry and successfully pull the image.
```
 podman pull cp.icr.io/cp/spectrum/scale/ibm-spectrum-scale-gui@sha256:78216ac2a8157753bd70f273f5e696479538007f43312bacd8e7f8e30f146276
```
A valid image from the image.txt can be used here to verify that things are working.

Verify that the image is pulled.

 # podman images | grep cp.icr.io/cp/spectrum/scale/ibm-spectrum-scale-gui
 cp.icr.io/cp/spectrum/scale/ibm-spectrum-scale-gui    <none>     9c215ae62f37   22 hours ago    851 MB