Troubleshooting
Problem
Customer set the proxy configuration at App Host and is getting the following error on App Host:
"The App Host is paired but communication has not yet been successful."
Symptom
The appHostCoreDNS.txt shows:
Warning FailedCreatePodSandBox 30m (x922 over 3h50m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": proxyconnect tcp: EOF
Warning FailedCreatePodSandBox 4m59s (x116 over 30m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": proxyconnect tcp: EOF
Warning FailedCreatePodSandBox 4s (x2 over 17s) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": proxyconnect tcp: EOF
Cause
The App host can't reach out to the IBM official repository due to a local network issue.
Environment
Any versions
Diagnosing The Problem
Follow MustGather to collect App host logs and status.
Check the proxy settings via the commands:
manageAppHost showconfig
And
manageAppHost proxy --environment
It shows the following that is wrong
HTTP_PROXY=http://proxy.xx.com:8080
HTTPS_PROXY=https://proxy.xx.com:8080
HTTPS_PROXY=https://proxy.xx.com:8080
Resolving The Problem
The proxy setting needs to be:
HTTP_PROXY=http://proxy.xx.com:8080
HTTPS_PROXY=http://proxy.xx.com:8080
The steps are:
1. Clear the current wrong proxy setting:
manageAppHost proxy --clear
2. Reconfigure with the right settings in /etc/systemd/system/k3s.service.env file:
Here is an example:
HTTP_PROXY=http://proxy.xxx.com:8080
HTTPS_PROXY=http://proxy.xxx.com:8080
NO_PROXY=localhost,127.0.0.1,0.0.0.0,localaddress,10.0.0.0/8
3. Only applies if proxy server requires authentication. Update the system's cacerts to trust the CA certificate from the proxy itself by adding the certificate in PEM encoding to
/etc/pki/ca-trust/source/anchors
then run the following command:
sudo update-ca-trust
4. Restart the Kubernetes K3s.
sudo systemctl restart k3s
5. List all system containers to verify:
sudo kubectl get pods -n kube-system
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEGM63","label":"IBM Security QRadar SOAR on Cloud"},"ARM Category":[{"code":"a8m0z0000001jTpAAI","label":"Integrations-\u003EAppHost"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
24 August 2022
UID
ibm16613563