Installing Netcool Operations Insight in an offline environment (airgap)

Follow these instructions to deploy an installation of Netcool® Operations Insight® on an offline environment, by using Container Application Software for Enterprises (CASE) and the Red Hat® OpenShift® Operator Lifecycle Manager (OLM).

1. Set up the following variables:

   export CASE_NAME=ibm-netcool-prod
   export CASE_VERSION=1.3.6
   export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
   export OFFLINEDIR=$HOME/offline
   export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE
   export CASE_REPO_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case
   export LOCAL_DOCKER_REGISTRY_HOST=<IP_or_FQDN_of_local_docker_registry>
   export LOCAL_DOCKER_REGISTRY_PORT=<port_number_of_local_docker_registry>
   export LOCAL_DOCKER_REGISTRY=$LOCAL_DOCKER_REGISTRY_HOST:$LOCAL_DOCKER_REGISTRY_PORT
   export LOCAL_DOCKER_REGISTRY_USER=<user>
   export LOCAL_DOCKER_REGISTRY_PASSWORD=<password>
   export TARGET_NAMESPACE=<Event Manager namespace>
   export IBM_ENTITLEMENT_KEY=<your entitlement key>

Create a target registry to store all the images locally.

2. Install and start a production grade Docker V2 compatible registry, such as Quay Enterprise, JFrog Artifactory, or Docker Registry.
   The target registry must be accessible by the Red Hat OpenShift cluster and the bastion host. The Red Hat OpenShift internal registry is not supported.
3. Create a secret for access to the target registry
   Run the following command on your Red Hat OpenShift cluster.

   oc create secret docker-registry target-registry-secret  \
       --docker-server=$LOCAL_DOCKER_REGISTRY                    \
       --docker-username=$LOCAL_DOCKER_REGISTRY_USER                         \
       --docker-password=$LOCAL_DOCKER_REGISTRY_PASSWORD                    \                      
       --namespace=$TARGET_NAMESPACE

   Where:

   • target-registry-secret is the name of the secret that you are creating. Suggested value is target-registry-secret.
   • $LOCAL_DOCKER_REGISTRY is the target registry that you created.
   • $LOCAL_DOCKER_REGISTRY_USER and $LOCAL_DOCKER_REGISTRY_PASSWORD are the credentials to access your target registry.
   • $TARGET_NAMESPACE is the namespace that you want to deploy Netcool Operations Insight in.

Prepare the bastion server.

4. Verify the bastion server's access.
   Log on to the bastion server and verify that it has access to the following components:

   • public internet - to download the Netcool Operations Insight CASE and images from the source registries.
   • target registry - where the images are mirrored.
   • target Red Hat OpenShift cluster - to install the Netcool Operations Insight operator.
5. Download and install the following items onto the bastion server.
   • cloudctl- Download IBM® Cloud Pak CLI (cloudctl) version 3.12.

   Download IBM® Cloud Pak CLI (cloudctl) from https://github.com/IBM/cloud-pak-cli/releases. cloudctl verifies the integrity of the Netcool Operations Insight CASE's digital signature by default. If you want to verify the cloudctl binary, follow the instructions in https://github.com/IBM/cloud-pak-cli#check-certificatekey-validity. Extract the cloudctl binary, give it executable permissions, and ensure that it is in your PATH.
   • oc - Download and install the Red Hat OpenShift CLI (oc), V4.6.x or higher. For more information, see https://docs.openshift.com/container-platform/4.8/cli_reference/openshift_cli/getting-started-cli.html#installing-the-cli.
   • Docker - Install docker version 1.13.1 or higher, and start the docker daemon. For more information, see https://docs.docker.com/install/.

Download the CASE bundle onto the bastion server.

6. Download the Netcool Operations Insight CASE bundle (ibm-netcool-prod), into a local directory on your bastion server.

   mkdir $OFFLINEDIR

   cloudctl case save \
     --case $CASE_NAME \
     --outputdir $OFFLINEDIR \
     --repo $CASE_REPO_PATH

   1. Filter out the unwanted digests from the images.csv file. Remove ppc64le and s390x architecture images and opencontent-redis-5 images with version tag=2.0.13 from the images.csv file.
7. Extract the Netcool Operations Insight CASE bundle.

   tar -xvf $OFFLINEDIR/$CASE_ARCHIVE -C $OFFLINEDIR

8. Verify that the Netcool Operations Insight CASE bundle, images.csv, and charts.csv are successfully downloaded on your bastion server, with the following command:

   find $OFFLINEDIR -type f

Configure bastion server authentication

9. Set up access to the IBM Entitled Registry, cp.icr.io, which you pull images from.
   Run the following command on your bastion server:

   cloudctl case launch \
       --case $CASE_LOCAL_PATH \
       --namespace $TARGET_NAMESPACE \
       --inventory noiOperatorSetup \
       --action configure-creds-airgap \
       --args "--registry cp.icr.io --user cp --pass $IBM_ENTITLEMENT_KEY"

   Where

   • $TARGET_NAMESPACE is the custom namespace that you want to deploy Netcool Operations Insight into.
   • $IBM_ENTITLEMENT_KEY is your IBM Entitled Registry entitlement key, as found when you prepared your cluster.
10. Set up access to the target image registry, which you copy images into.
    Run the following command on your bastion server:

    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --namespace $TARGET_NAMESPACE \
        --inventory noiOperatorSetup \
        --action configure-creds-airgap \
        --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD"

    The credentials are saved to ~/.airgap/secrets/<registry-name>.json

Mirror images from CASE to the target registry in the airgap environment

11. Before mirroring images, set CLOUDCTL_CASE_USE_CATALOG_DIGEST by running the command:

    export CLOUDCTL_CASE_USE_CATALOG_DIGEST=1

12. Note: Split the images.csv file into multiple images.csv files, with fewer than 15 images each. Place the images.csv files in separate input directories.
    Mirror images from CASE to the target registry. This can take up to 2 hours.
    Run the following command for each input directory on your bastion server:

    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --namespace $TARGET_NAMESPACE \
        --inventory noiOperatorSetup \
        --action mirror-images \
        --args "--registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR"

    The images that are listed in the downloaded CASE, (images.csv), are copied to the target registry in the airgap environment.

Configure Red Hat OpenShift cluster for airgap.

13. Warning:

    • Cluster resources must adjust to the new pull secret, which can temporarily limit the usability of the cluster. Authorization credentials are stored in $HOME/.airgap/secrets and /tmp/airgap* to support this action.
    • Applying ImageSourceContentPolicy causes cluster nodes to recycle for OpenShift versions 4.7 or earlier.
    • Applying ImageSourceContentPolicy causes cluster nodes to temporarily have a STATUS SchedulingDisabled status for OpenShift versions 4.8 or later.
    Configure your Red Hat OpenShift cluster for airgap. This step can take 90+ minutes.
    Insecure registry: If your offline registry is insecure, complete the following steps for registry access:

    • Login as kubeadmin with the following command:

      oc login -u kubeadmin -p <password>

    • Run the following commands on your bastion server to create a global image pull secret for the target registry, and create an ImageSourceContentPolicy.

      oc patch image.config.openshift.io/cluster --type=merge -p '{"spec":{"registrySources":{"insecureRegistries":["'${LOCAL_DOCKER_REGISTRY}'"]}}}'

    Secure registry: If your offline registry is secure, complete the following steps for registry access. You can add certificate authorities (CA) to the cluster for use when pushing and pulling images. You must have cluster administrator privileges. You must have access to the public certificates of the registry, usually a hostname/ca.crt file located in the /etc/docker/certs.d/ directory.

    • Create a configmap in the openshift-config namespace containing the trusted certificates for the registries that use self-signed certificates. For each CA file, ensure that the key in the configmap is the hostname of the registry, in the hostname[..port] format:

      oc create configmap registry-cas -n openshift-config \
      --from-file=myregistry.corp.com..5000=/etc/docker/certs.d/myregistry.corp.com:5000/ca.crt \

    • Update the cluster image configuration:

      oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge

    Run the following command for your secure or insecure registry:

    cloudctl case launch \
      --case $OFFLINEDIR/$CASE_ARCHIVE \
      --namespace $TARGET_NAMESPACE \
      --inventory noiOperatorSetup \
      --action configure-cluster-airgap \
      --args "--registry $LOCAL_DOCKER_REGISTRY --inputDir $OFFLINEDIR --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD" 

Install the Netcool Operations Insight catalog and operator.

14. Install the catalog by using CASE. Run the following command on your bastion server:
    Set the project (namespace) to install the catalog:

    export CATALOG_NAMESPACE=openshift-marketplace

    Create the catalog source:

    cloudctl case launch \
      --case $CASE_LOCAL_PATH \
      --inventory noiOperatorSetup \
      --action install-catalog \
      --namespace $CATALOG_NAMESPACE \
      --args "--registry $LOCAL_DOCKER_REGISTRY"

15. Verify the Netcool Operations Insight catalog source.
    From the Red Hat OpenShift OLM UI, go to Administration > Cluster Settings, and then select the OperatorHub configuration resource under the Global Configuration tab. Verify that the ibm-noi-catalog catalog source is present.
16. Install the Netcool Operations Insight operator by using CASE. Run the following command on your bastion server:

    cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --namespace $TARGET_NAMESPACE \
        --inventory noiOperatorSetup \
        --action install-operator \
        --args "--secret target-registry-secret"

    Where

    • namespace is the custom namespace to be used for your deployment.
    • target-registry-secret is the secret for accessing the target registry that you created in step 2.
17. Verify the status of IBM Cloud Pak for Watson™ AIOps Event Manager.
    From the Red Hat OpenShift OLM UI, go to Operators > Installed Operators, and verify that the status of IBM Cloud Pak for Watson AIOps Event Manager is Succeeded.

Create a Netcool Operations Insight instance.

18. From the Red Hat OpenShift OLM UI, go to Operators > Installed Operators, and select IBM Cloud Pak for Watson AIOps Event Manager. Under Provided APIs > NOI, select Create Instance.
19. From the Red Hat OpenShift OLM UI, use the YAML view or the Form view to configure the properties for the Netcool Operations Insight deployment. For more information about configurable properties for a cloud only deployment, see Cloud operator properties.
    Note: Ensure that the name of the Netcool Operations Insight instance does not exceed 10 characters.
20. Edit the Netcool Operations Insight properties to provide access to the target registry.
    1. Update $TARGET_REGISTRY/noi so that it points to target_registry/noi.
    2. Set spec.entitlementSecret to the target registry secret.
21. Select Create.
22. Under the All Instances tab, a Netcool Operations Insight instance appears.
    To monitor the status of the installation, see Monitoring cloud installation progress.
    Note:

    • Changing an existing deployment from a Trial deployment type to a Production deployment type is not supported.
    • Changing an instance's deployment parameters in the Form view is not supported post deployment.
    • If you update custom secrets in the OLM console, the crypto key is corrupted and the command to encrypt passwords does not work. Update custom secrets only with the CLI. For more information about storing a certificate as a secret, see https://www.ibm.com/docs/en/SS9LQB_1.1.10/LoadingData/t_asm_obs_configuringsecurity.html