Installing IBM Cloud Pak foundational services in an air-gapped environment using a Bastion host
If your cluster is not connected to the internet, you can install IBM Cloud Pak foundational services in your cluster by using either a bastion host, portable compute device, or a portable storage device.
1. Set up your mirroring environment
Before you install any IBM Cloud Pak® on an air-gapped environment, you must set up a host that can be connected to the internet to complete configuring your mirroring environment. To set up your mirroring environment, complete the following steps:
1.1. Prerequisites
No matter what medium you choose for your air-gapped installation, you must satisfy the following prerequisites:
-
OpenShift Container Platform requires you to have cluster admin access to run the
configure-cluster-airgap
andinstall-operator
commands. -
An OpenShift Container Platform cluster must be installed. For the supported OpenShift Container Platform versions, see Supported OpenShift versions and platforms.
-
A Docker V2 registry must be available and accessible from the OpenShift Container Platform cluster nodes. Registry is available to aid in mirroring to final location using portable options. For more information, see 1.2 Set up local image registry and access.
-
Access to the following sites and ports:
*.docker.io
and*.docker.com
: For more information about specific sites to allow access to, see Docker Hub Hosts for Firewalls and HTTP Proxy Serversicr.io:443
for IBM Entitled Registry and IBM Cloud Pak foundational services catalog sourcequay.io:443
for IBM Cloud Pak foundational services catalog and imagesgithub.com
for CASEs and toolsredhat.com
for Red Hat OpenShift Container Platform upgrades
Prepare a host
Regardless of which type of host you're using, you must be able to connect it to the internet and to the air-gapped network with access to the OpenShift Container Platform cluster and the local, intranet Docker registry. Your host must be on a Linux® x86_64 or Mac platform with any operating system that the IBM Cloud Pak® CLI and the OpenShift Container Platform CLI support. If you are on a Windows platform, you must execute the actions in a Linux® x86_64 VM or from a Windows Subsystem for Linux (WSL) terminal.
Your host must have sufficient storage to hold all of the software that is to be transferred to the local, intranet Docker registry.
The following table explains the software requirements for installing IBM Cloud Pak foundational services in an air-gapped environment:
Software | Purpose |
---|---|
OpenSSL | Validating certificates when you run the air-gapped install scripts |
Docker | Container management |
Podman | Container management |
Apache httpd-tools | Creating an account when you run the air-gapped install scripts |
IBM Cloud Pak® CLI (cloudctl) | Running CASE commands |
Red Hat OpenShift CLI (oc) | Red Hat OpenShift Container Platform administration |
Skopeo | Working with container images and registries in an air-gapped environment |
Complete the following steps on your host:
-
Install OpenSSL version 1.1.1 or higher.
-
Install Docker or Podman.
Notes:
- If you want to install a version of the foundational services prior to 3.7.3 or ibm-cp-common-services-1.3.3.tgz (not included), install the docker version within
18.x.x - 19.x.x
, and runyum install <package name>-<version info>
. - If you want to install foundational services on Red Hat Enterprise Linux 8.x or later, Docker is not supported, and you can use Podman instead.
To install Docker (for example, on Red Hat® Enterprise Linux®), run the following commands:
```cmd yum check-update yum install docker ```
To install Podman, see Podman Installation Instructions
.
- If you want to install a version of the foundational services prior to 3.7.3 or ibm-cp-common-services-1.3.3.tgz (not included), install the docker version within
-
Install
httpd-tools
.yum install httpd-tools
-
Download and install the most recent version of
cloudctl-linux-amd64.tar.gz
from the IBM/cloud-pak-cli repo. Extract the binary file by entering the following command:tar -xf cloudctl-linux-amd64.tar.gz
Run the following command to modify the file:
chmod 755 cloudctl-linux-amd64
Run the following command to move the file to the
/usr/local/bin
directory:mv cloudctl-linux-amd64 /usr/local/bin/cloudctl
Note: You can confirm that
cloudctl
is installed by entering the following command:cloudctl --help
The cloudctl usage is displayed.
-
Install the
oc
OpenShift Container Platform CLI tool. For more information, see OpenShift Container Platform CLI tools. -
Install the
skopeo
CLI version1.0.0
or higher. For more information, selectinstall.md
in the Skopeo repo.
-
Create a directory that serves as the offline store.
Following is an example directory, which is used in the subsequent steps.
mkdir $HOME/offline
Note: This offline store must be persistent to avoid transferring data more than once. The persistence also helps to run the mirroring process multiple times or on a schedule.
If you are installing foundational services by using a bastion host, the bastion server must be configured.
1.2. Set up local image registry and access
You must use a local Docker registry to store all of your images in your intranet network. Many customers have one or more centralized, corporate registry servers to store production container images. If a local registry is not available, a production-grade registry will need to be installed and configured. To access your registries during an air-gapped installation, you must use an account that the username and password are associated with a user who can write to the target local registry from a bastion host and read from the target local registry that is on the OpenShift cluster nodes. You must create such a registry and the registry must meet the following requirements:
- Supports [Docker Manifest V2, Schema 2](https://docs.docker.com/registry/spec/manifest-v2-2/).
- Supports multi-architecture images.
- Is accessible from the OpenShift Container Platform cluster nodes.
- Allows path separators in the image name.
**Note:** Do not use OpenShift image registry as your local, intranet registry. The OpenShift registry does not support multi-architecture images or path separators in the image name.
After you create the internal Docker registry, you must configure the registry:
-
Create registry namespaces. If you are using Podman or Docker for your internal registry, you can skip this step. Docker creates these namespaces for you when you mirror the images.
A registry namespace is the first component in the image name. For example: in the image name,
icr.io/cpopen/myproduct
, the namespace portion of the image name iscpopen
.If you are using different registry providers, you must create separate registry Kubernetes namespaces for each public registry source. These Kubernetes namespaces are used as the location that the contents of the CASE file gets pushed into when you run your CASE commands.
The following registry namespaces might be used by the CASE command:
-
cp
- Namespace to store the IBM images from thecp.icr.io/cp
repository.The
cp
namespace is for the images in the IBM Entitled Registry that require a product entitlement key and credentials to pull. -
cpopen
- Namespace to store the operator related IBM images from theicr.io/cpopen
repository.The
cpopen
namespace is for publicly available images hosted by IBM® that don't require credentials to pull. -
opencloudio
- Namespace to store the images fromquay.io/opencloudio
.The
opencloudio
namespace is for select IBM open source component images that are available on quay.io. The IBM Cloud Pak foundational services images are hosted onopencloudio
.
-
2. Set environment variables and download CASE files
Notes:
- For all cloudctl commands, if the environment variable
TMPDIR
is set, then it will use that value as the location of a temporary directory for its processing. IfTMPDIR
is not set, then it will use the/tmp
directory. - The following proxy environment variables are supported on cloudctl version 3.12.1 and later.
If your bastion host, portable compute device, or portable storage device must connect to the internet via a proxy, you must set environment variables on the machine that accesses the internet via the proxy server. For detailed information, see Setting up proxy environment variables.
Before mirroring your images, you can set the environment variables on your mirroring device, and connect to the internet so that you can download the corresponding CASE files. To finish preparing your host, complete the following steps:
Note: Save a copy of your environment variable values to a text editor. You can use that file as a reference to cut and paste from when completing your air-gapped installation tasks.
-
Create the following environment variables with the installer image name and the image inventory on your host.
Important: To install IBM Cloud Pak foundational services version 3.14.0, use the
ibm-cp-common-services-1.19.11.tgz
CASE archive. If you want to install an earlier version of IBM Cloud Pak foundational services, see Table 1. Image versions for offline installation for the CASE versions that you can use.```cmd export CASE_NAME=ibm-cp-common-services export CASE_VERSION=1.19.11 export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz export CASE_INVENTORY_SETUP=ibmCommonServiceOperatorSetup export OFFLINEDIR=$HOME/offline export CASE_REPO_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE ```
-
Connect your host to the internet and disconnect it from the local air-gapped network.
-
Download the IBM Cloud Pak foundational services installer and image inventory to your host.
```cmd cloudctl case save \ --repo $CASE_REPO_PATH \ --case $CASE_NAME \ --version $CASE_VERSION \ --outputdir $OFFLINEDIR ```
Your host is now configured and you are ready to mirror your images.
Note: There is an optional parameter to cloudctl case save
that allows you to specify unique log directories (--log-dir
). If you are setting up in a scenario where multiple users are running on same bastion
servers and need separate logs to prevent collision with permissions, use the optional parameter as shown in the following example:
```cmd
$cloudctl case save -h
NAME:
save - Save and parse a CASE file in preparation for installation.
USAGE:
cloudctl case save —repo
OPTIONS:
--case value, -c value The local path or URL containing the CASE file to parse. If repo is specified, then this flag should be the name of the CASE to be retrieved from the repo
--log-dir value Output directory to store logs for CASE commands
--no-dependency If set, disables resolution of dependencies
--outputdir value, -o value The output directory to which the CASE resources will be placed. The output directory will be created if it does not exist
--repo value, -r value The location of the CASE repo, e.g. raw Github URL
--tls-skip-verify, --tls If set, TLS/SSL verification will be skipped
--tolerance value, -t value Tolerance levels for validating the CASE
0 - maximum validation
1 - reduced validation
--version value, -v value The semantic version range specifying the CASE to grab, e.g. '2.0.0' or '>=1.0.0, <=3.0.0'. Ignored if repo is not specified
```
3. Mirror images to your final location
The process of mirroring images takes the image from the internet to your host, then effectively copies that image on to your air-gapped environment. After you mirror your images, you can configure your cluster and complete air-gapped installation.
Notes:
- The only step in this section required for the bastion host scenario is storing the authentication credentials for all of the source Docker registries.
- If you want to install subsequent updates to your air-gapped environment, you must do a CASE save to get the image list when performing those updates.
- For details on ensuring you do not introduce changes to multiple environments, see Setting up a repeatable air-gap process.
-
If your CASE file references images in Docker, you can create a
registries.conf
file to redirect and pull the images from IBM Cloud Container Registry. The air-gapped installation script uses theoc image mirror
command to mirror images. Theoc
command does not support usingregistries.conf
files. Setting theUSE_SKOPEO
environment variable totrue
(USE_SKOPEO=true
) enables the script to use Skopeo to mirror images instead. Theregistries.conf
file must be in V2 format. Other formats, such as V1, cause errors.[[registry]] location = "docker.io/ibmcom" insecure = false blocked = false mirror-by-digest-only = true prefix = "" [[registry.mirror]] location = "icr.io/cpopen" insecure = false
Complete the following steps to mirror your images from your host to your air-gapped environment:
3.1. Mirror the images to the host
Complete these steps to mirror the images from the internet to your host:
Notes: Don't use the tilde within double quotation marks in any command. For example, don't use args "--registry <registry> --user <registry userid> --pass {registry password} --inputDir ~/offline"
.
The tilde does not expand and your commands might fail.
-
Store authentication credentials for all source Docker registries.
All IBM Cloud Pak foundational services credentials are stored in public registries that don't require authentication. However, other products and third-party components require one or more authenticated registries. The following registries require authentication:
cp.icr.io
registry.redhat.io
registry.access.redhat.com
For more information about these registries, see Create registry namespaces.
You must run the following command to configure credentials for all registries that require authentication. Run the command separately for each such registry:
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action configure-creds-airgap \ --args "--registry <registry> --user $REGISTRY_USER --pass $REGISTRY_PASSWORD" \
The command stores and caches the registry credentials in a file on your file system in the
$HOME/.airgap/secrets
location.
3.2. Mirror images to final location and configure the cluster
Complete these steps on your host that is connected to both the local docker registry and the OpenShift Container Platform cluster:
-
Create environment variables with the local Docker registry connection information.
export CASE_NAME=ibm-cp-common-services export CASE_VERSION=1.19.11 export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz export CASE_INVENTORY_SETUP=ibmCommonServiceOperatorSetup export OFFLINEDIR=$HOME/offline export OFFLINEDIR_ARCHIVE=offline.tgz export CASE_REPO_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE export LOCAL_DOCKER_REGISTRY_HOST=<IP_or_FQDN_of_local_docker_registry> export LOCAL_DOCKER_REGISTRY_PORT=443 export LOCAL_DOCKER_REGISTRY=$LOCAL_DOCKER_REGISTRY_HOST:$LOCAL_DOCKER_REGISTRY_PORT export LOCAL_DOCKER_USER=<username> export LOCAL_DOCKER_PASSWORD=<password>
Note: The Docker registry uses standard ports such as
80
or443
. If your Docker registry uses a non-standard port, specify the port by using the syntax<host>:<port>
. For example,export LOCAL_DOCKER_REGISTRY=myregistry.local:5000
. -
Configure an authentication secret for the Docker registry. Then, store the credentials of the registry that is going to serve images to the cluster or workloads
Note: This step needs to be done only one time.
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action configure-creds-airgap \ --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_USER --pass $LOCAL_DOCKER_PASSWORD" \
The command stores and caches the registry credentials in a file on your file system in the
$HOME/.airgap/secrets
location. -
Mirror images to the local image registry:
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action mirror-images \ --args "--registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR" \
Notes:
-
If you want to mirror images to an alternative namespace, you can use the
--nsPrefix
argument. You can copy images to your mirror registry in a specific location and not to the default path by providing the location in the--nsPrefix
argument. For example, the following CASE command implements the--nsPrefix
argument forlocationX/
:cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action mirror-images \ --args " --registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR --nsPrefix locationX/" \
-
Some products support the ability to mirror a subset of images using the
--filter
argument and image grouping. The--filter
argument provides the ability to customize which images are mirrored during an air-gapped installation. For example, you have an Elasticsearch CASE that contains groups that allow you to mirror specific versions of Elasticsearch. Use the--filter
argument to target one or more versions of Elasticsearch to mirror rather than the entire library.Consider the following command:
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action mirror-images \ --args " --registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR" \
Update the
--args
argument to include a--filter
argument. For example, use--args "--registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINE_DIR --filter ibm_es_7"
in the preceding example,ibm_es_7
will mirror only the images associated with Elasticsearch version 7.0. The resulting image group consists of images in theibm_es_7
image group as well as any images that are not associated with any groups. This allows products to include common images as well as the ability to reduce the number of images that you need to mirror.
-
-
Create a new project for the CASE commands by running the following commands:
export NAMESPACE=ibm-common-services
oc new-project ibm-common-services
-
Configure a global image pull secret and ImageContentSourcePolicy.
Note: You must have cluster admin access to run the
configure-cluster-airgap
command. However, you do not need cluster admin access for mirroring.cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action configure-cluster-airgap \ --namespace $NAMESPACE \ --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_USER --pass $LOCAL_DOCKER_PASSWORD --inputDir $OFFLINEDIR" \ --tolerance 1
Note: If you include the
--nsPrefix
argument in yourmirror-image
command, you must also include it in yourconfigure-cluster-airgap
command to set up the image redirection. For example, the following command adds the--nsPrefix
argument to the preceding command:cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action configure-cluster-airgap \ --namespace $NAMESPACE \ --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_USER --pass $LOCAL_DOCKER_PASSWORD --inputDir $OFFLINEDIR --nsPrefix $NSPREFIX" \ --tolerance 1
If you are using Red Hat OpenShift Container Platform version 4.7 or earlier, this step might cause your cluster nodes to drain and reboot sequentially to apply the configuration changes.
-
Verify that the ImageContentSourcePolicy resource is created.
oc get imageContentSourcePolicy
-
Optional: If you use an insecure registry, you must add the local registry to the cluster insecureRegistries list.
oc patch image.config.openshift.io/cluster --type=merge \ -p '{"spec":{"registrySources":{"insecureRegistries":["'${LOCAL_DOCKER_REGISTRY}'"]}}}'
-
Verify your cluster node status.
oc get MachineConfigPool -w
After the
ImageContentsourcePolicy
and global image pull secret are applied, the configuration of your nodes will be updated sequentially. Wait until allMachineConfigPools
are updated.
4. Install IBM Cloud Pak foundational services by way of Red Hat OpenShift Container Platform
Now that your images are mirrored to your air-gapped environment, you can deploy your IBM Cloud Paks® to that that environment. When you mirrored your environment, you created a parallel offline version of everything that you needed to install an operator into OpenShift Container Platform. To install the IBM Cloud Pak foundational services, complete the following steps:
- 4.1. Create the catalog source and install the foundational services
- 4.2. Access the console
- 4.3. Retrieve your console username and password
4.1. Create the catalog source and install the IBM Cloud Pak foundational services
-
Set the namespace to install the foundational services catalog:
export NAMESPACE=ibm-common-services
-
Create and configure a catalog source.
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action install-catalog \ --namespace $NAMESPACE \ --args "--registry $LOCAL_DOCKER_REGISTRY" \ --recursive \ --inputDir $OFFLINEDIR \ --tolerance 1
Note: In foundational services version 3.13 and previous versions, the
install-catalog
command deployscatalogsource
with the latest tag. Starting from foundational services version 3.14,install-catalog
deploys the catalog source ,opencloud-operators
, withcatalogsource image digest
. -
Verify that the
CatalogSource
for foundational services installer operator is created.oc get pods -n openshift-marketplace oc get catalogsource -n openshift-marketplace
-
(Optional) Create a configmap if you want to install foundational services in a custom namespace. By default, the foundational services are installed in the
ibm-common-services
namespace. For more information, see Installing IBM Cloud Pak foundational services in a custom namespace.Important: You can install foundational services in only one namespace in your cluster.
-
Install the foundational services operators.
Note: You must have cluster admin access to run the
install-operator
command. However, you do not need cluster admin access for mirroring.cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action install-operator \ --namespace $NAMESPACE \ --tolerance 1
Note: The foundational services are by default installed with the
starterset
deployment profile. You can change the profile tosmall
by adding--args "--size small"
, or tolarge
by adding--args "--size large"
, to the command. See the following example:```cmd cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action install-operator \ --namespace $NAMESPACE \ --args "--size small" \ --tolerance 1 ```
-
Using the
oc login
command, log in to the OpenShift Container Platform cluster where your final location resides. You can identify your specificoc login
by clicking the user drop-down menu in the Red Hat OpenShift Container Platform console, then clicking Copy Login Command. -
Verify that the foundational services are installed:
oc get pod -n ibm-common-services
It might take up to 15 minutes for all the pods to show the Running
status.
Note: If you want to deploy Db2, you must load the Db2 CASE bundle and update the OperandRegistry before you create OperandRequest.
```cmd
oc edit opreg common-service -n ibm-common-services
```
Replace the catalog source value from ibm-operator-catalog
to ibm-db2uoperator-catalog
such that the catalog source looks like the following example:
```cmd
- channel: v1.0
name: ibm-db2u-operator
namespace: placeholder
packageName: db2u-operator
scope: public
sourceName: ibm-db2uoperator-catalog
sourceNamespace: openshift-marketplace
```
4.2. Access the console
Use the following command to get the URL to access the console:
```cmd
oc get route -n ibm-common-services cp-console -o jsonpath=‘{.spec.host}’
```
This command results in the following output:
```cmd
cp-console.apps.mycluster.mydomain.com
```
Based on the example output, your console URL would be https://cp-console.apps.mycluster.mydomain.com
.
4.3. Retrieve your console username and password
The default username to access the console is admin
.
You can get the password for the admin
username by running the following command:
```cmd
oc -n ibm-common-services get secret platform-auth-idp-credentials -o jsonpath='{.data.admin_password}' | base64 -d
```
The following password is an example of the output of the preceding command:
```cmd
EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsA
```
Based on the example output, you would then use EwK9dj9fwPZHyHTyu9TyIgh9klZSzVsA
as the password.
You can change the default password at any time. For more information, see Changing the cluster administrator password
Note: Any user with access to the ibm-common-services
namespace can retrieve this password since it is stored in a secret in the ibm-common-services
namespace. To minimize password exposure, allow limited
users to access the ibm-common-services
namespace.
You have now successfully created and deployed your air-gapped instance of IBM Cloud Pak foundational services.
For more information about troubleshooting your air-gapped installation, see Troubleshooting an air-gapped installation.
Setting up a repeatable air-gap process
Once you completed a CASE
save, you can mirror the CASE
as many times as you want to. This approach allows you to airgap a specific version of the Cloud Pak into development, test, and production stages.
Follow the steps in this section if you want to save the CASE
to multiple registries (per environment) once and be able to run the CASE
in the future without repeating the CASE
save process.
- Run the following command to save the
CASE
toOFFLINEDIR
, which can be used as an input during theCASE
launch:
cloudctl case save \
--repo $CASE_REPO_PATH \
--case $CASE_NAME \
--version $CASE_VERSION \
--outputdir $OFFLINEDIR
- Execute the
CASE
launch by running the following command:
cloudctl case launch \
--case $CASE_LOCAL_PATH \
--inventory $CASE_INVENTORY_SETUP \
--action mirror-images \
--args "--registry $PORTABLE_DOCKER_REGISTRY --inputDir $OFFLINEDIR" \
--tolerance 1
If you want to make this repeatable across environments, you can reuse the same saved CASE
cache ($OFFLINEDIR
) instead of executing a CASE
save again in other environments. You do not have to worry about updated
versions of dependencies being brought into the saved cache.