health storage-validation

Evaluate whether the storage on your cluster includes the required features and is correctly configured and for use with IBM Software Hub.

Extended description

The cpd-cli health storage-validation command uses the Storage Validation tool for IBM Cloud Paks to validate storage on ReadWriteOnce and ReadWriteMany volumes on your Red Hat® OpenShift® cluster.

The cpd-cli health storage-validation command performs the following health checks:

  • Dynamic provisioning of a volume
  • Mounting volume from a node
  • Sequential read/write consistency from single and multiple nodes
  • Parallel read/write consistency from single and multiple nodes
  • Parallel read/write consistency across multiple threads
  • File permissions on mounted volumes
  • Accessibility based on POSIX compliance group ID permissions
  • SubPath test for volumes
  • File locking test
Important: The cpd-cli health storage-validation command creates resources on your cluster. You can run the cleanup.sh script to list and delete the resources.

Prerequisites

Before you run the cpd-cli health storage-validation command:

  1. Source your installation environment variables script.
  2. Log in to Red Hat OpenShift Container Platform as a cluster administrator.
    ${OC_LOGIN}
    Remember: OC_LOGIN is an alias for the oc login command.
  3. If you pull images from a private container registry, complete Pulling the required image from a private container registry.
  4. Set the STORAGE_TEST_NS environment variable to the name of the project where you want to create the resources that are required for the storage validation tests.

    The recommended project name is ibm-storage-test.

    export STORAGE_TEST_NS=<project-name>

    If you specify a project that does not exist, the storage-validation command creates the project on your cluster.

  5. Set the following environment variables based on the credentials that you want to use to authenticate.
    Important: You must provide the credentials for a cluster administrator.
    Username and password
    export STG_TEST_USERNAME=${OCP_USERNAME}
export STG_TEST_PASSWORD=${OCP_PASSWORD}
export STG_TEST_TOKEN="<not-specified>"
    Token
    export STG_TEST_USERNAME="<required>"
export STG_TEST_PASSWORD="<required>"
export STG_TEST_TOKEN=${OCP_TOKEN}
  6. Change to the directory on your workstation where you want to create the param.yml file.
  7. Create a param.yml file:
    cat << EOF > ./param.yml
# OCP Parameters
ocp_url: ${OCP_URL}
ocp_username: ${STG_TEST_USERNAME}
ocp_password: ${STG_TEST_PASSWORD}
ocp_token: ${STG_TEST_TOKEN}

storageClass_ReadWriteOnce: ${STG_CLASS_BLOCK}
storageClass_ReadWriteMany: ${STG_CLASS_FILE}
arch: ${CPU_ARCH}
imageurl: ${IMAGE_LOCATION}/cpopen/cpd/k8s-storage-test:${VERSION}.${IMAGE_ARCH}

run_storage_readiness: true  # true = run the "storage-validation" tests

############################ STORAGE VALIDATION PARAMETERS START ########################

# project to create test resources in, will be created if not already present
storage_validation_namespace: ${STORAGE_PERF_NS} 
# name prefix for cluster resources created by validation playbook (changing not recommended)
prefix: "readiness"
# size of pvcs created for storage tests
storageSize: "1Gi"

# OPTIONAL PARAMETERS 
options: ""
backoffLimit: 5

############################ STORAGE VALIDATION PARAMETERS END ##########################
EOF
  8. Ensure that the container runtime on the client workstation runs processes with root privileges.
    Docker
    By default, Docker containers run processes as the root user.
    Podman
    Run the appropriate command based on the operating system on the client workstation:
    Linux®
    If you are not logged in as root, run:
    sudo cpd-cli health storage-validation
    MacOS
    podman machine stop
podman machine set --rootful=true
podman machine start
    Windows
    podman machine stop
podman machine set --rootful=true
podman machine start

Pulling the required image from a private container registry

The k8s-storage-test image is automatically mirrored when you mirror the IBM® Software Hub images to your private container registry.

Before you run the cpd-cli health storage-validation command, you must authorize the client runtime environment on your workstation to pull the k8s-storage-test image from the private container registry:

  1. Ensure that Docker or Podman is running on the client workstation.
  2. Source your installation environment variables script.
  3. Authenticate to your private container registry:
    • Docker
      docker login ${PRIVATE_REGISTRY_LOCATION} \
-u ${PRIVATE_REGISTRY_PULL_USER} \
-p ${PRIVATE_REGISTRY_PULL_PASSWORD}
    • Podman
      podman login ${PRIVATE_REGISTRY_LOCATION} \
-u ${PRIVATE_REGISTRY_PULL_USER} \
-p ${PRIVATE_REGISTRY_PULL_PASSWORD}

To use the k8s-storage-test image from your private container registry, you must specify the following options when you run the cpd-cli health storage-validation command:

  • --image-prefix
  • --image-tag

For example:

cpd-cli health storage-validation \
--image-prefix=${PRIVATE_REGISTRY_LOCATION}/cpopen/cpd \
--image-tag=${VERSION}.${IMAGE_ARCH} \
--param=param.yml

Syntax

cpd-cli health storage-validation \
--param <path of param.yml file> \
[--image-prefix=<image-registry-prefix>] \
[--image-tag=<image-tag>

Arguments

The command has no arguments.

Options

Table 1: Command options
Option Description
--image-prefix Specify the registry that the image should be pulled from.
Status
Required if your cluster pulls images from a private container registry.
Syntax
--image-prefix=<image-registry-prefix>
Default value
icr.io/cpopen/cpd

If you omit this option, the default value is used.

Valid values
IBM Entitled Registry
icr.io/cpopen/cpd
Private container registry
${PRIVATE_REGISTRY_LOCATION}
--image-tag Specify the tag that identifies the image in the repository.

The tag must exist on the k8s-storage-test image in the repository that you specified for the --image-prefix option.

Status
Required if your cluster pulls images from a private container registry.
Syntax
--image-tag=<image-tag>
Default value
No default.
Valid values
The value depends on which image you want to pull.
Tip: If you mirrored the images to a private container registry using the --arch flag, use the single-arch image.
Multi-arch image
${VERSION}
Single-arch image
${VERSION}.IMAGE_ARCH
--param The path of the YAML that defines the parameters required to run the storage validation test.
Status
Required.
Syntax
--param <path of param.yml file>
Default value
No default.
Valid values
The fully qualified path of the param.yml file.

Cleanup

You can run a cleanup script to remove any resources that were created after you ran the cpd-cli health storage-validation command.

Follow these steps to run the cleanup.sh script:
  1. Save the following script file as a shell script, cleanup.sh : 
    #!/usr/bin/env sh

HELP="
This script lists storage test resources in the supplied namespace. If the '--delete' flag is provided, it removes those resources.

USAGE:
  cleanup.sh --namespace <namespace> <--delete>

FLAGS:
  -c | --command:           (Required) The cpd-cli storage command resources to list and/or clean up, storage-performance|storage-validation.
  --delete:                 Delete storage test resources created in --namespace.
  -n | --namespace:         (Required) The namespace specified in param.yml, storage_validation_namespace|storage_perf_namespace.
  -h | --help:              Show help for cleanup.sh
"
NAMESPACE=
COMMAND=
DELETE_RESOURCES="0"

CM_SEARCH=
JOB_SEARCH=
PVC_SEARCH=
LOCAL_CONTAINER=

log() {
  prefix=""
  case "${1}" in
  info)
    prefix="[INFO ]"
    ;;
  debug)
    prefix="[DEBUG]"
    ;;
  error)
    prefix="[ERROR]"
    ;;
  *)
    prefix=""
    ;;
  esac
  echo "${prefix} ${2}"
}

print_usage() {
  log noop "${HELP}"
}

set_vars() {
  if [ "${COMMAND}" == "storage-performance" ]; then
    PVC_SEARCH="pvc-sysbench-rwo\|pvc-sysbench-rwx"
    JOB_SEARCH="sysbench-random-\|sysbench-sequential-"
    LOCAL_CONTAINER="k8s-storage-perf"
  fi

  if [ "${COMMAND}" == "storage-validation" ]; then
    CM_SEARCH="consumer\|producer"
    PVC_SEARCH="readiness-readwritemany\|readiness-readwriteonce"
    JOB_SEARCH="readiness-consumer\|readiness-create\|readiness-edit\|readiness-file\|readiness-mount\|readiness-producer\|readiness-read"
    LOCAL_CONTAINER="k8s-storage-val"
  fi
}

list() {
  echo "\nListing ${COMMAND} cluster resources in namespace: ${NAMESPACE}"
  
  if [ -n "${CM_SEARCH}" ]; then
    echo "\nCONFIGMAPS:"
    oc get cm -n ${NAMESPACE} | sed -n "1p;/${CM_SEARCH}/p"
  fi
  
  echo "\nPVCS:"
  oc get pvc -n ${NAMESPACE} | sed -n "1p;/${PVC_SEARCH}/p"
  echo "\nJOBS:"
  oc get job -n ${NAMESPACE} | sed -n "1p;/${JOB_SEARCH}/p"

  echo "\nLocal ${COMMAND} playbook container:"
  podman ps -a --filter="name=${LOCAL_CONTAINER}"
  echo ""
}

delete_resources() {
    echo "\nDeleting ${COMMAND} cluster resources from namespace: ${NAMESPACE}"
    oc get job -n ${NAMESPACE} -o name | grep "${JOB_SEARCH}" | xargs -I % -n 1 oc delete % -n ${NAMESPACE}

    if [ -n "${CM_SEARCH}" ]; then
      oc get cm -n ${NAMESPACE} -o name | grep "${CM_SEARCH}" | xargs -I % -n 1 oc delete % -n ${NAMESPACE}
    fi
    
    sleep 10
    oc get pvc -n ${NAMESPACE} -o name | grep "${PVC_SEARCH}" | xargs -I % -n 1 oc delete % -n ${NAMESPACE}

    echo "\nRemoving local ${COMMAND} container:"
    podman rm -f ${LOCAL_CONTAINER}
    echo ""
}

run() {
  # confirm required flags have been provided  
  if [ -z "${NAMESPACE}" ]; then
    log error "You must specify the --namespace option. For example: --namespace <project-name>"
    print_usage
    exit 1
  fi
  if [ "${COMMAND}" != "storage-performance" ] && [ "${COMMAND}" != "storage-validation" ]; then
    log error "You must specify the --command option. For example: --command <storage-performance|storage-validation>"
    print_usage
    exit 1
  fi

  # set command-specific variables
  set_vars

  # show resources
  list

  # delete storage test resources and re-run list to confirm deletion
  if [ "${DELETE_RESOURCES}" -eq "1" ]; then
    delete_resources
  fi

  exit 0
}

# Parse CLI args
while [ "${1-}" != "" ]; do
    case $1 in
    --namespace | -n)
        shift
        NAMESPACE="${1}"
        ;;
    --command | -c)
        shift
        COMMAND="${1}"
        ;;    
    --delete)
        shift
        DELETE_RESOURCES="1"
        ;;
    --help | -h)
        print_usage 0
        exit 0
        ;;
    *)
        echo "Invalid Option ${1}" >&2
        exit 1
        ;;
    esac
    shift
done

run
  2. If you are using a Mac or Podman client workstation, update the file permissions so you can run the script: 
    chmod +x % ./cleanup.sh
Table: Arguments for cleanup.sh script
Argument Description
-c--command
Description
The cpd-cli storage command, storage-performance|storage-validation, that created resources to list or clean up.
Status
Required.
--delete
Description
Removes storage test resources that were created in --namespace.
Status
Optional.
-n--namespace
Description
The namespace specified in the param.yml file: storage_validation_namespace|storage_perf_namespace.
Status
Optional.
-h--help Show help for cleanup.sh

Examples

Standard run.

cpd-cli health storage-validation \
--param param.yml

Sample output.

./cpd-cli health storage-validation --param param.yml
INFO[0000] manage(olm-utils)                             event=Information info="Checking podman or docker" trace=09f88cd2-d748-48c1-95ac-3333e2756601
[INFO] 2023-09-06T21:42:46.785603Z Checking podman or docker
INFO[0000] health                                        event=Information info="Dockerexe: podman" trace=09f88cd2-d748-48c1-95ac-3333e2756601
[INFO] 2023-09-06T21:42:46.832035Z Dockerexe: podman

PLAY [localhost] ***************************************************************

TASK [ocp login using creds] ***************************************************
changed: [localhost]

TASK [ocp login using token] ***************************************************
skipping: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "login_creds.stdout_lines": [
        "Login successful.",
        "",
        "You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
        "",
        "Using project \"default\"."
    ]
}

TASK [debug] *******************************************************************
skipping: [localhost]

TASK [Storage Readiness] *******************************************************

TASK [storage-readiness : Create namespace managed-nfs-storage if not present] ***
ok: [localhost]

TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"
}

TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteMany] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteMany Volume  #################################"
}

TASK [storage-readiness : Run sequential RW tests for ReadWriteOnce] ***********
included: /opt/ansible/roles/storage-readiness/tasks/sequential-rw.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Sequential read/write ReadWriteOnce volumes for readiness] ***
ok: [localhost] => (item={'name': 'producer.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SEQUENTIAL READ WRITE TEST PASSED FOR ReadWriteOnce Volume #################################"
}

TASK [storage-readiness : Run sequential RW tests for ReadWriteMany] ***********
included: /opt/ansible/roles/storage-readiness/tasks/sequential-rw.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Sequential read/write ReadWriteMany volumes for readiness] ***
ok: [localhost] => (item={'name': 'producer.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SEQUENTIAL READ WRITE TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : Run parallel rw single thread tests  ReadWriteOnce] ***
included: /opt/ansible/roles/storage-readiness/tasks/parallel-rw-single-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Parallel read/write  ReadWriteOnce volumes for readiness] ***
ok: [localhost] => (item={'name': 'producer.yaml.j2'})

TASK [storage-readiness : Parallel concurrency read while producer is running] ***
ok: [localhost] => (item={'name': 'consumer-nocheck.yaml.j2'})

TASK [storage-readiness : Assert that producer is still running] ***************
ok: [localhost]

TASK [storage-readiness : Warn if producer job is already completed] ***********
skipping: [localhost]

TASK [storage-readiness : Wait until producer job is done] *********************
ok: [localhost]

TASK [storage-readiness : Parallel read  ReadWriteOnce volumes for readiness] ***
ok: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SINGLE THREAD PARALLEL READ WRITE TEST PASSED for ReadWriteOnce #################################"
}

TASK [storage-readiness : Run parallel rw single thread tests for ReadWriteMany] ***
included: /opt/ansible/roles/storage-readiness/tasks/parallel-rw-single-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Parallel read/write  ReadWriteMany volumes for readiness] ***
ok: [localhost] => (item={'name': 'producer.yaml.j2'})

TASK [storage-readiness : Parallel concurrency read while producer is running] ***
ok: [localhost] => (item={'name': 'consumer-nocheck.yaml.j2'})

TASK [storage-readiness : Assert that producer is still running] ***************
ok: [localhost]

TASK [storage-readiness : Warn if producer job is already completed] ***********
skipping: [localhost]

TASK [storage-readiness : Wait until producer job is done] *********************
ok: [localhost]

TASK [storage-readiness : Parallel read  ReadWriteMany volumes for readiness] ***
ok: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SINGLE THREAD PARALLEL READ WRITE TEST PASSED for ReadWriteMany #################################"
}

TASK [storage-readiness : Run parallel rw multi thread tests for ReadWriteOnce] ***
included: /opt/ansible/roles/storage-readiness/tasks/parallel-rw-multi-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Get node count] **************************************
ok: [localhost]

TASK [storage-readiness : set_fact] ********************************************
ok: [localhost]

TASK [storage-readiness : Parallel write from same/separate nodes] *************
ok: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel write no wait from same/separate nodes] *****
ok: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel read no wait from same/separate nodes] ******
ok: [localhost] => (item={'name': 'parallel-consumer.yaml.j2'})

TASK [storage-readiness : Wait Until parallel producer job is done] ************
ok: [localhost]

TASK [storage-readiness : Wait Until parallel consumer job is done] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MULTI NODE PARALLEL READ WRTIE TEST PASSED FOR ReadWriteOnce #################################"
}

TASK [storage-readiness : Run parallel rw multi thread tests  for ReadWriteMany] ***
included: /opt/ansible/roles/storage-readiness/tasks/parallel-rw-multi-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Get node count] **************************************
ok: [localhost]

TASK [storage-readiness : set_fact] ********************************************
ok: [localhost]

TASK [storage-readiness : Parallel write from same/separate nodes] *************
ok: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel write no wait from same/separate nodes] *****
ok: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel read no wait from same/separate nodes] ******
ok: [localhost] => (item={'name': 'parallel-consumer.yaml.j2'})

TASK [storage-readiness : Wait Until parallel producer job is done] ************
ok: [localhost]

TASK [storage-readiness : Wait Until parallel consumer job is done] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MULTI NODE PARALLEL READ WRTIE TEST PASSED FOR ReadWriteMany #################################"
}

TASK [storage-readiness : File UID permission tests for ReadWriteMany] *********
included: /opt/ansible/roles/storage-readiness/tasks/file-uid-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : create custom scc] ***********************************
ok: [localhost]

TASK [storage-readiness : create custom sa] ************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc create sa uid-sa -n managed-nfs-storage", "delta": "0:00:00.100040", "end": "2023-09-07 04:43:56.675346", "msg": "non-zero return code", "rc": 1, "start": "2023-09-07 04:43:56.575306", "stderr": "error: failed to create serviceaccount: serviceaccounts \"uid-sa\" already exists", "stderr_lines": ["error: failed to create serviceaccount: serviceaccounts \"uid-sa\" already exists"], "stdout": "", "stdout_lines": []}

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "set up scc and sa for file uid test failed"
}

TASK [storage-readiness : File uid create ReadWriteMany volumes for readiness] ***
ok: [localhost] => (item={'name': 'file-owner-uid-test.yaml.j2'})
ok: [localhost] => (item={'name': 'file-create-uid.yaml.j2'})
ok: [localhost] => (item={'name': 'file-edit-uid.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE UID TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : SupplementalGroup GID permission tests for ReadWriteMany] ***
included: /opt/ansible/roles/storage-readiness/tasks/file-permissions-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Get first uid from project range] ********************
changed: [localhost]

TASK [storage-readiness : Set target uid to set for the reader job to be different than the writer] ***
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "The target_uid_reader is 1000690100"
}

TASK [storage-readiness : create custom scc] ***********************************
skipping: [localhost]

TASK [storage-readiness : create custom sa] ************************************
skipping: [localhost]

TASK [storage-readiness : add scc to custom sa] ********************************
skipping: [localhost]

TASK [storage-readiness : File Permissions GID test for ReadWriteMany volumes for readiness] ***
ok: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : File Permissions Supplemental Group test for ReadWriteMany volumes for readiness] ***
ok: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE PERMISSIONS TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : FSGroup GID permission tests for ReadWriteOnce] ******
included: /opt/ansible/roles/storage-readiness/tasks/file-permissions-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Get first uid from project range] ********************
changed: [localhost]

TASK [storage-readiness : Set target uid to set for the reader job to be different than the writer] ***
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "The target_uid_reader is 1000690100"
}

TASK [storage-readiness : create custom scc] ***********************************
ok: [localhost]

TASK [storage-readiness : create custom sa] ************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc create sa fsgroup-sa -n managed-nfs-storage", "delta": "0:00:00.140013", "end": "2023-09-07 04:44:10.257889", "msg": "non-zero return code", "rc": 1, "start": "2023-09-07 04:44:10.117876", "stderr": "error: failed to create serviceaccount: serviceaccounts \"fsgroup-sa\" already exists", "stderr_lines": ["error: failed to create serviceaccount: serviceaccounts \"fsgroup-sa\" already exists"], "stdout": "", "stdout_lines": []}

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "set up scc and sa for fsgroup test failed"
}

TASK [storage-readiness : File Permissions GID test for ReadWriteOnce volumes for readiness] ***
ok: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : File Permissions Supplemental Group test for ReadWriteOnce volumes for readiness] ***
ok: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
ok: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE PERMISSIONS TEST PASSED FOR ReadWriteOnce Volume #################################"
}

TASK [storage-readiness : Subpath tests ReadWriteMany] *************************
included: /opt/ansible/roles/storage-readiness/tasks/volume-subpath-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Subpath tests for ReadWriteMany volumes for readiness] ***
ok: [localhost] => (item={'name': 'producer-subpath.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-subpath.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SUB PATH TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : File locks tests] ************************************
included: /opt/ansible/roles/storage-readiness/tasks/file-lock-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})

TASK [storage-readiness : File lock test for readiness] ************************
ok: [localhost] => (item={'name': 'file-lock.yaml.j2'})

TASK [storage-readiness : Wait few seconds] ************************************
Pausing for 20 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]

TASK [storage-readiness : Negative file lock test for readiness] ***************
ok: [localhost] => (item={'name': 'file-lock.yaml.j2'})

TASK [storage-readiness : Wait for lock to be released] ************************
ok: [localhost] => (item={'name': 'file-lock.yaml.j2'})

TASK [storage-readiness : Positive lock test for readiness] ********************
ok: [localhost] => (item={'name': 'file-lock.yaml.j2'})

TASK [storage-readiness : Fcntl test for readiness] ****************************
ok: [localhost] => (item={'name': 'file-lock-fcntl.yaml.j2'})

TASK [storage-readiness : Wait few seconds] ************************************
Pausing for 20 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]

TASK [storage-readiness : Negative Fcntl test for readiness] *******************
ok: [localhost] => (item={'name': 'file-lock-fcntl.yaml.j2'})

TASK [storage-readiness : Wait for Fcntl lock to be released] ******************
ok: [localhost] => (item={'name': 'file-lock-fcntl.yaml.j2'})

TASK [storage-readiness : Positive Fcntl test for readiness] *******************
ok: [localhost] => (item={'name': 'file-lock-fcntl.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE LOCK TESTS PASSED FOR ReadWriteMany Volume #################################"
}

PLAY RECAP *********************************************************************
localhost                  : ok=107  changed=3    unreachable=0    failed=0    skipped=7    rescued=2    ignored=0
Example of the cleanup.sh script to list resources

The storage_validation_namespace namespace ibm-stor-val is the namespace provided in the param.yml file:

% ./cleanup.sh --namespace ibm-stor-val --command storage-validation
Example of the cleanup.sh script to delete resources
% ./cleanup.sh --namespace ibm-stor-val --command storage-validation --delete
Example output from the cleanup.sh script to delete resources
% ./cleanup.sh --namespace ibm-stor-val --command storage-validation --delete

Listing storage-validation cluster resources in namespace: ibm-stor-val

CONFIGMAPS:
NAME                       DATA   AGE
consumer-cm                4      3h49m
consumer-nocheck-cm        1      3h48m
producer-cm                8      3h49m

PVCS:
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  VOLUMEATTRIBUTESCLASS   AGE
readiness-readwritemany   Bound    pvc-e7c9cc0d-f274-47f0-8db5-4b7f7ddfeab3   1Gi        RWX            ocs-storagecluster-cephfs     <unset>                 3h50m
readiness-readwriteonce   Bound    pvc-6a3b1781-e5b5-4390-9818-58ded69932e4   1Gi        RWO            ocs-storagecluster-ceph-rbd   <unset>                 3h50m

JOBS:
NAME                                                          STATUS     COMPLETIONS   DURATION   AGE
readiness-consumer--subpath-job-readwritemany-sequential      Complete   1/1           7s         3h35m
readiness-consumer-job-readwritemany-parallel-final           Complete   1/1           12s        3h46m
readiness-consumer-job-readwritemany-parallel-multi           Complete   5/5           11s        3h41m
readiness-consumer-job-readwritemany-parallel-nocheck         Complete   1/1           5s         3h47m
readiness-consumer-job-readwritemany-sequential               Complete   1/1           7s         3h48m
readiness-consumer-job-readwriteonce-parallel-final           Complete   1/1           17s        3h47m
readiness-consumer-job-readwriteonce-parallel-multi           Complete   5/5           11s        3h44m
readiness-consumer-job-readwriteonce-parallel-nocheck         Complete   1/1           5s         3h48m
readiness-consumer-job-readwriteonce-sequential               Complete   1/1           18s        3h49m
readiness-create-file-job-readwritemany-gid                   Complete   1/1           7s         3h38m
readiness-create-file-job-readwritemany-sg                    Complete   1/1           6s         3h37m
readiness-create-file-job-readwriteonce-gid                   Complete   1/1           15s        3h37m
readiness-create-file-job-readwriteonce-sg                    Complete   1/1           14s        3h36m
readiness-edit-file-job-readwritemany-gid                     Complete   1/1           8s         3h38m
readiness-edit-file-job-readwritemany-sg                      Complete   1/1           6s         3h37m
readiness-edit-file-job-readwriteonce-gid                     Complete   1/1           16s        3h36m
readiness-edit-file-job-readwriteonce-sg                      Complete   1/1           14s        3h35m
readiness-file-lock-fcntl-job-create-lock                     Complete   1/1           68s        3h32m
readiness-file-lock-fcntl-job-create-lock2                    Failed     0/1           3h31m      3h31m
readiness-file-lock-fcntl-job-create-lock3                    Complete   1/1           65s        3h31m
readiness-file-lock-job-create-lock                           Complete   1/1           73s        3h34m
readiness-file-lock-job-create-lock2                          Failed     0/1           3h34m      3h34m
readiness-file-lock-job-create-lock3                          Complete   1/1           66s        3h33m
readiness-file-owner-uid-job-readwritemany-uid                Complete   1/1           8s         3h38m
readiness-file-uid-job-readwritemany-uid                      Complete   1/1           6s         3h38m
readiness-mount-job-readwritemany                             Complete   1/1           5s         3h50m
readiness-mount-job-readwriteonce                             Complete   1/1           14s        3h50m
readiness-producer-job-readwritemany-parallel                 Complete   1/1           11s        3h47m
readiness-producer-job-readwritemany-parallel-multi           Complete   5/5           94s        3h42m
readiness-producer-job-readwritemany-parallel-multi-no-wait   Complete   5/5           94s        3h41m
readiness-producer-job-readwritemany-sequential               Complete   1/1           10s        3h48m
readiness-producer-job-readwriteonce-parallel                 Complete   1/1           14s        3h48m
readiness-producer-job-readwriteonce-parallel-multi           Complete   5/5           20s        3h44m
readiness-producer-job-readwriteonce-parallel-multi-no-wait   Complete   5/5           16s        3h44m
readiness-producer-job-readwriteonce-sequential               Complete   1/1           23s        3h49m
readiness-producer-subpath-job-readwritemany-sequential       Complete   1/1           10s        3h35m
readiness-read-file-job-readwritemany-gid                     Complete   1/1           5s         3h38m
readiness-read-file-job-readwritemany-sg                      Complete   1/1           14s        3h37m
readiness-read-file-job-readwriteonce-gid                     Complete   1/1           8s         3h36m
readiness-read-file-job-readwriteonce-sg                      Complete   1/1           6s         3h35m

Local storage-validation playbook container:
CONTAINER ID  IMAGE                                     COMMAND     CREATED      STATUS      PORTS       NAMES
3b5fc76bbddd  icr.io/cpopen/cpd/k8s-storage-test:5.1.0              4 hours ago  Up 4 hours              k8s-storage-val


Deleting storage-validation cluster resources from namespace: ibm-stor-val
job.batch "readiness-consumer--subpath-job-readwritemany-sequential" deleted
job.batch "readiness-consumer-job-readwritemany-parallel-final" deleted
job.batch "readiness-consumer-job-readwritemany-parallel-multi" deleted
job.batch "readiness-consumer-job-readwritemany-parallel-nocheck" deleted
job.batch "readiness-consumer-job-readwritemany-sequential" deleted
job.batch "readiness-consumer-job-readwriteonce-parallel-final" deleted
job.batch "readiness-consumer-job-readwriteonce-parallel-multi" deleted
job.batch "readiness-consumer-job-readwriteonce-parallel-nocheck" deleted
job.batch "readiness-consumer-job-readwriteonce-sequential" deleted
job.batch "readiness-create-file-job-readwritemany-gid" deleted
job.batch "readiness-create-file-job-readwritemany-sg" deleted
job.batch "readiness-create-file-job-readwriteonce-gid" deleted
job.batch "readiness-create-file-job-readwriteonce-sg" deleted
job.batch "readiness-edit-file-job-readwritemany-gid" deleted
job.batch "readiness-edit-file-job-readwritemany-sg" deleted
job.batch "readiness-edit-file-job-readwriteonce-gid" deleted
job.batch "readiness-edit-file-job-readwriteonce-sg" deleted
job.batch "readiness-file-lock-fcntl-job-create-lock" deleted
job.batch "readiness-file-lock-fcntl-job-create-lock2" deleted
job.batch "readiness-file-lock-fcntl-job-create-lock3" deleted
job.batch "readiness-file-lock-job-create-lock" deleted
job.batch "readiness-file-lock-job-create-lock2" deleted
job.batch "readiness-file-lock-job-create-lock3" deleted
job.batch "readiness-file-owner-uid-job-readwritemany-uid" deleted
job.batch "readiness-file-uid-job-readwritemany-uid" deleted
job.batch "readiness-mount-job-readwritemany" deleted
job.batch "readiness-mount-job-readwriteonce" deleted
job.batch "readiness-producer-job-readwritemany-parallel" deleted
job.batch "readiness-producer-job-readwritemany-parallel-multi" deleted
job.batch "readiness-producer-job-readwritemany-parallel-multi-no-wait" deleted
job.batch "readiness-producer-job-readwritemany-sequential" deleted
job.batch "readiness-producer-job-readwriteonce-parallel" deleted
job.batch "readiness-producer-job-readwriteonce-parallel-multi" deleted
job.batch "readiness-producer-job-readwriteonce-parallel-multi-no-wait" deleted
job.batch "readiness-producer-job-readwriteonce-sequential" deleted
job.batch "readiness-producer-subpath-job-readwritemany-sequential" deleted
job.batch "readiness-read-file-job-readwritemany-gid" deleted
job.batch "readiness-read-file-job-readwritemany-sg" deleted
job.batch "readiness-read-file-job-readwriteonce-gid" deleted
job.batch "readiness-read-file-job-readwriteonce-sg" deleted
configmap "consumer-cm" deleted
configmap "consumer-nocheck-cm" deleted
configmap "producer-cm" deleted
persistentvolumeclaim "readiness-readwritemany" deleted
persistentvolumeclaim "readiness-readwriteonce" deleted

Removing local storage-validation container:
k8s-storage-val

% ./cleanup.sh --namespace ibm-stor-val --command storage-validation

Listing storage-validation cluster resources in namespace: ibm-stor-val

CONFIGMAPS:
NAME                       DATA   AGE

PVCS:
No resources found in ibm-stor-val namespace.

JOBS:
No resources found in ibm-stor-val namespace.

Local storage-validation playbook container:
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES