Troubleshooting
Problem
After creating a new App Host and completing the pairing, the App Host shows as paired but not running.
sudo kubectl get pods -A shows the 3 kube-system and new the App Host synchronizer and operator controller pods as running.
Symptom
App Host shows as paired but not running.
Cause
Adding a DNS Search Domain during deployment of the App Host .ova file is the cause.
Environment
CP4S 1.8 or Native IBM Security SOAR 42.x
App Host/Edgegateway 1.6 or 1.7
Diagnosing The Problem
App Host synchronizer log shows, cannot get cluster node metrics and "apphost.localdomain" not found
ControllersApi.getControllerJobs: Unable to connect to AppManager s48.cloudpaksecurity.com:443
2021-12-01 02:59:28.913 [pool-10-thread-7] WARN i.f.k.client.URLFromNodePortImpl - ControllersApi.getControllerCommands: Unable to connect to AppManager s48.cloudpaksecurity.com:443
ControllerHeartbeatSynchronization - Cannot get cluster node metrics
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.43.0.1/apis/metrics.k8s.io/v1beta1/nodes/apphost.localdomain. Message: nodemetrics.metrics.k8s.io "apphost.localdomain" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=metrics.k8s.io, kind=nodemetrics, name=apphost.localdomain, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=nodemetrics.metrics.k8s.io "apphost.localdomain" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).
Output of sudo kubectl cluster-info dump shows
E1203 18:50:45.806937 1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:apphost.localdomain: unable to fetch metrics from Kubelet apphost.localdomain (apphost.localdomain): Get https://apphost.localdomain:10250/stats/summary?only_cpu_and_memory=true: dial tcp 23.20.239.12:10250: i/o timeout
E1203 18:51:07.678188 1 reststorage.go:135] unable to fetch node metrics for node "apphost.localdomain": no metrics known for node
E1203 18:51:07.678712 1 reststorage.go:118] unable to fetch node metrics for node "apphost.localdomain": no metrics known for node "apphost.localdomain"
E1203 18:51:45.813998 1 manager.go:111] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:apphost.localdomain: unable to fetch metrics from Kubelet apphost.localdomain (apphost.localdomain): Get https://apphost.localdomain:10250/stats/summary?only_cpu_and_memory=true: dial tcp 23.20.239.12:10250: i/o timeout
E1203 18:51:51.867628 1 reststorage.go:135] unable to fetch node metrics for node "apphost.localdomain": no metrics known for node
==== END logs for container metrics-server of pod kube-system/metrics-server-7566d596c8-nfnw2 ====
==== START logs for container coredns of pod kube-system/coredns-7d7bd65d7b-k5mp5 ====
.:53
[INFO] plugin/reload: Running configuration MD5 = 4665410bf21c8b272fcfd562c482cb82
CoreDNS-1.6.9
linux/amd64, go1.14.1, 1766568
[ERROR] plugin/errors: 2 xxx.cloudpaksecurity.com.some_dns_search_domain.com. A: read udp 10.42.0.34:32937->9.9.9.x:53: i/o timeout
[ERROR] plugin/errors: 2 xxx.cloudpaksecurity.com.some_dns_search_domain.com. A: read udp 10.42.0.34:41192->9.9.9.x:53: i/o timeout
[ERROR] plugin/errors: 2 apphost.localdomain.some_dns_search_domain.com. A: read udp 10.42.0.34:52607->9.9.9.x:53: i/o timeout
==== END logs for container coredns of pod kube-system/coredns-7d7bd65d7b-k5mp5 ====
==== START logs for container local-path-provisioner of pod kube-system/local-path-provisioner-6d59f47c7-b5dd7 ====
ERROR: logging before flag.Parse: I1201 20:24:25.687945 1 leaderelection.go:187] attempting to acquire leader lease kube-system/rancher.io-local-path...
ERROR: logging before flag.Parse: I1201 20:24:43.082216 1 leaderelection.go:196] successfully acquired lease kube-system/rancher.io-local-path
ERROR: logging before flag.Parse: I1201 20:24:43.083545 1 controller.go:572] Starting provisioner controller rancher.io/local-path_local-path-provisioner-6d59f47c7-b5dd7_ad670eab-52e4-11ec-b510-26fcc3bc7e6d!
ERROR: logging before flag.Parse: I1201 20:24:43.083578 1 event.go:221] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"rancher.io-local-path", UID:"301f5b61-d28a-4d4e-a4af-c1fa87e0b219", APIVersion:"v1", ResourceVersion:"47014", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' local-path-provisioner-6d59f47c7-b5dd7_ad670eab-52e4-11ec-b510-26fcc3bc7e6d became leader
ERROR: logging before flag.Parse: I1201 20:24:43.183695 1 controller.go:621] Started provisioner controller rancher.io/local-path_local-path-provisioner-6d59f47c7-b5dd7_ad670eab-52e4-11ec-b510-26fcc3bc7e6d!
==== END logs for container local-path-provisioner of pod kube-system/local-path-provisioner-6d59f47c7-b5dd7 ====
Output of sudo kubectl top node shows
[root@apphost ~]# sudo kubectl top node
error: metrics not available yet
Normal output should show:
[root@apphost ~]# sudo kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
apphost.localdomain 142m 7% 3589Mi 45%
Resolving The Problem
Need to create/deploy a NEW AppHost .ova, leaving the DNS Search Domain blank.
Removing DNS Search Domain from nmtui does NOT solve the issue
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSIP9Q","label":"IBM Security SOAR"},"ARM Category":[{"code":"a8m0z0000001jTpAAI","label":"Integrations-\u003EAppHost"}],"ARM Case Number":"TS007686244","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
06 December 2021
UID
ibm16523770