Troubleshooting

This topic addresses some common problems.

No response from z/OSMF

In order for z/OS® Cloud Broker to create the catalog icons for the templates available in z/OSMF Cloud Provisioning & Management, the z/OS Cloud Broker deployment must be able to communicate with z/OSMF.

On the node where your z/OS Cloud Broker deployment is running, verify that the following command returns the expected output:

curl -k -X GET https://<user>:<pass>@<zosmf-ip>/zosmf/provisioning/rest/1.0/psc -H 'X-CSRF-ZOSMF-HEADER: zos-broker'

Force deployment on a specific node

In High Availability or Multi-Architecture OpenShift® Container Platform environments, you might want to force the z/OS Cloud Broker deployment to be on a specific OpenShift Container Platform node.

To do this, target the Deployment in the IBM Cloud Private web console and select Edit.

Look for the nodeSelector JSON block and add a label from the node you'd like it to be deployed on. In the example below, the addition of the role: master label forces the deployment on the IBM Cloud Private master node. By default, the z/OS Cloud Broker pods will prefer s390x worker nodes if they are available, and x86 worker nodes as a secondary.

"nodeSelector": {
  "beta.kubernetes.io/arch": "amd64",
  "role": "master"
},

Orphaned service instances or service bindings

Service instances or service bindings can become "orphaned" in OpenShift Container Platform under certain circumstances. To prevent this from happening, all service instances that are provisioned through the OpenShift Container Platform web console should always be de-provisioned through the OpenShift Container Platform web console as well.

If the z/OSMF templates were not changed, z/OSMF is online, and the service instance is still provisioned on z/OSMF, the current environment might be brought back into sync. Redeploying a matching z/OS Cloud Broker instance and ClusterServiceBroker resource should re-create all necessary registrations to allow the deprovisioning actions to complete.

However, there might be other scenarios in which this issue can occur if z/OS Cloud Broker fails to respond, or the communication to z/OSMF doesn't return as expected. This is common if the resource was deleted on z/OS because OpenShift Container Platform would not be informed of those actions.

If the resource exists on z/OS, opening an issue with a dump of the kubectl logs for the zos-broker pod will be helpful.

To delete the orphaned ServiceInstance or ServiceBinding, complete the following steps:

  1. Authenticate your oc client by using the login command from OpenShift Container Platform cluster.

  2. Attempt to delete the ServiceInstance or ServiceBinding with the oc command.

    oc get ServiceInstances
oc delete ServiceInstance <service-instance-name>

oc get ServiceBindings
oc delete ServiceBinding <service-binding-name>

    This will say it completed successfully, but issuing another oc get will show the resource is still defined in OpenShift Container Platform.

  3. Edit the resource and remove the finalizers block.

    oc edit ServiceInstance <service-instance-name>
oc edit ServiceBinding <service-binding-name>

    The finalizers: block looks similar to the following example:

    finalizers:
   - kubernetes-incubator/service-catalog

Save the edit, and you should see the service instance or service binding is removed from OpenShift Container Platform.

Collecting FFDC data

If you encounter a problem that is not addressed in this topic, you might have encountered a new problem. In order for us to fix the issue, you need to provide the logs from the OpenShift Container Platform or Cloud Foundry environment. Follow the processes below to gather z/OS Cloud Broker deployment logs.

Exporting deployment logs from OpenShift Container Platform

  1. From a terminal where the oc command is available, log in to the OpenShift Container Platform cluster.
  2. Switch to the project where the problem is occurring. For example:
    oc project <project name>
  3. List the pods for the deployment. For example:
    oc get pods
  4. Gather information for each pod with the following command.
    oc logs <pod_name> > zos-broker_<pod_name>.log
  5. After the information is collected for each pod, compress the created .log files with a zip utility and send us the compressed file.

Exporting deployment logs from Cloud Foundry

  1. From a terminal on the machine that is running the z/OS Cloud Broker Docker containers
  2. List the containers for the deployment. For example:
    docker ps -a
  3. Gather information for each container with the following command.
    docker logs <container_id> > <container_name>.log
  4. After the information is collected for each pod, compress the created .log files with a zip utility and send us the compressed file.

The ClusterServiceBroker resource fails to retrieve catalog response

The ClusterServiceBroker resource might initially fail to receive a Catalog response from z/OS Cloud Broker due to initialization being in progress. In that case, OpenShift Container Platform retries the Catalog request every 10 minutes. Services should appear in your OpenShift Container Platform Catalog as soon as you have one or more Services and Plans configured in the z/OS Cloud Broker Admin Dashboard.

Failure when deploying the z/OS Cloud Broker

You might encounter the following error when you create a template instance of the z/OS Cloud Broker.

Unable to recognize no matches for kind "ClusterServiceBroker" in version "servicecatalog.k8s.io/v1beta1"

This is because the Service Catalog extension not being enabled in your OpenShift Container Platform cluster. Follow the configuration documentation to enable the service-catalog extension.

Failure when viewing the OpenShift Container Platform Service Brokers

The following error indicates that you cannot view the Service Brokers under the menu Service Catalog/Broker Management.

clusterservicebrokers.servicecatalog.k8s.io is forbidden: User <user ID> cannot list resource "clusterservicebrokers" in API group "servicecatalog.k8s.io" at the cluster scope

This error occurs if a user is not properly configured as a cluster-admin. You can run the following command to grant the user the cluster-admin role.

oc adm policy add-cluster-role-to-user cluster-admin <user id>

For more information, see Authentication.