Troubleshooting

The ibm-cp4a-operator locates the Cloud Pak base images and has Ansible roles to handle the reconciliation logic and declare a set of playbook tasks for each component. The roles declare all the variables and defaults for how the role is executed.

The operator deployment creates two containers on your cluster, one for the operator and one for Ansible. By default Ansible uses /etc/ansible/hosts as its inventory file; the inventory represents the host machines that Ansible manages. You can create a different inventory file for each project that you have.

The following diagram shows how the operator watches for events, triggers an Ansible role when a custom resource changes, and then reconciles the resources for the deployed applications.

Getting the Ansible and operator logs

Note:

 For 20.0.1  and  For 20.0.2  The Ansible container shows the standard Ansible stdout logs. To see the logs of a container, run the following command.

kubectl logs deployment/ibm-cp4a-operator -c ansible > ansible.log

The operator logs contain much more information about the operator than Kubernetes does. To see the logs of the operator container, run the following command.

kubectl logs deployment/ibm-cp4a-operator -c operator > operator.log

If the operator log does not provide the level of detail that you need, you can gather more details by adding an annotation like the following example to your custom resource YAML:

metadata:
 ...
 annotations:
  "ansible.operator-sdk/verbosity": "3"
spec:

For verbosity value, the normal rules for Ansible verbosity apply, where higher values mean more output. Acceptable values range from 0 (only the most severe messages are output) to 7 (all debugging messages are output).

After you update the custom resource YAML, reapply the YAML for the changes to take effect.

Getting the runtime logs

 For 20.0.1  and  For 20.0.2  For runtime logs, go inside the pod that runs the Ansible container. The runner keeps information about the Ansible run in the container, which is located under /tmp/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>.

Copying the latest logs from the log volume

 New in 20.0.3  For troubleshooting purposes, you can also copy the logs from the log volume /logs/$operator_pod_name/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>/artifacts. The log contains information on the first 10 reconciles and the latest reconcile. The following commands get the operator pod name and make a copy of the logs to a local directory.

deployment_name=$(kubectl get icp4acluster | awk '{print $1}' | grep -v "NAME")
operator_pod_name=$(kubectl get pod|grep ibm-cp4a-operator | awk '{print $1}')
kubectl cp $operator_pod_name:/logs/$operator_pod_name/ansible-operator/runner/icp4a.ibm.com/v1/ICP4ACluster/<namespace>/$deployment_name/artifacts /<local_logpath>

Getting information about pending pods

If some pods are pending, choose one of the pods and run the following command to get more information.

kubectl describe pod <podname> 

Getting information about secrets

Kubernetes secrets are used extensively, so output about them might also be useful.

kubectl get secrets

Getting information about events

Kubernetes events are objects that provide more insight into what is happening inside a cluster, such as what decisions the scheduler makes or why some pods are evicted from a node. To get information about these events, run the following command.

kubectl get events > events.log

You can also add the verbose parameter to any kubectl command.

kubectl -v=9 get pods

Recreating the image pull secret

If your Docker registry secret expires, you can delete the secret and re-create it:

oc delete secret admin.registrykey -n <namespace>
oc create secret docker-registry admin.registrykey --docker-server=image-registry.openshift-image-registry.svc:5000 --docker-username=kubeadmin --docker-password=$(oc whoami -t)

Applying changes by restarting pods

In some cases, changes that you make in the custom resource YAML by using the operator or directly in the environment are not automatically propagated to all pods. For example, modifications to data source information or changes to Kubernetes secrets are not seen by running pods until the pods are restarted.

If changes applied by the operator or other modifications made in the environment do not provide the expected result, restart the pods by scaling the impacted deployments down to 0 then up to the desired number to have Kubernetes (OpenShift) terminate the existing pods and create new ones.