Troubleshooting
Problem
This technote address an issue that you may sometimes notice on the Nginx Ingress logs. An error "503 Service temporary Unavailable" is generated. See similar error below:
Symptom
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:48:27 [error] 23665#23665: *1269455996 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.253.60.45, server: smartit360.ngc02.group.echonet, request: "GET /api/widget/29/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9)) HTTP/1.1", upstream: "http://10.241.76.132:7500/api/widget/29/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9))", host: "smartit360.ngc02.group.echonet", referrer: "https://smartit360.ngc02.group.echonet/"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:48:27 [error] 23665#23665: *1269456001 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.253.60.45, server: smartit360.ngc02.group.echonet, request: "GET /api/widget/30/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9)) HTTP/1.1", upstream: "http://10.241.76.132:7500/api/widget/30/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9))", host: "smartit360.ngc02.group.echonet", referrer: "https://smartit360.ngc02.group.echonet/"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:48:27 [error] 23665#23665: *1269456009 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.253.60.45, server: smartit360.ngc02.group.echonet, request: "GET /api/widget/39/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9)) HTTP/1.1", upstream: "http://10.241.76.132:7500/api/widget/39/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9))", host: "smartit360.ngc02.group.echonet", referrer: "https://smartit360.ngc02.group.echonet/"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:48:27 [error] 23666#23666: *1269455995 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.253.60.45, server: smartit360.ngc02.group.echonet, request: "GET /api/widget/8/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9)) HTTP/1.1", upstream: "http://10.241.76.132:7500/api/widget/8/summary/undefined?ByDelphesTeam=1&Equ=71027&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:46:27%20GMT+0200%20(Paris,%20Madrid%20(heure%20d%E2%80%99%C3%A9t%C3%A9))", host: "smartit360.ngc02.group.echonet", referrer: "https://smartit360.ngc02.group.echonet/"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:48:33 [error] 23665#23665: *1269466634 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.253.60.45, server: smartit360.ngc02.group.echonet, request: "POST /api/widget/46/summary_post/undefined HTTP/1.1", upstream: "http://10.241.76.132:7500/api/widget/46/summary_post/undefined", host: "smartit360.ngc02.group.echonet", referrer: "https://smartit360.ngc02.group.echonet/"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:48:41 [error] 23666#23666: *1269458188 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 159.50.228.30, server: smartit360.ngc02.group.echonet, request: "GET /api/widget/8/summary/DP12E?ByDelphesTeam=1&Equ=71948&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:44:40%20GMT+0200 HTTP/1.1", upstream: "http://10.241.76.132:7500/api/widget/8/summary/DP12E?ByDelphesTeam=1&Equ=71948&Metier=ALL&dateNow=Tue%20Jul%2007%202020%2014:44:40%20GMT+0200", host: "smartit360.ngc02.group.echonet", referrer: "https://smartit360.ngc02.group.echonet/"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:50:14 [error] 23665#23665: *1269491093 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: 127.0.0.1, server: omc-odm.group.echonet, request: "GET / HTTP/1.1", upstream: "https://10.241.65.177:9443/", host: "omc-odm.group.echonet"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:50:14 [error] 23665#23665: *1269491139 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: 127.0.0.1, server: omc-odm.group.echonet, request: "GET / HTTP/1.1", upstream: "https://10.241.65.177:9443/", host: "omc-odm.group.echonet"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:50:15 [error] 23666#23666: *1269491247 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: 127.0.0.1, server: omc-odm.group.echonet, request: "GET / HTTP/1.1", upstream: "https://10.241.65.177:9443/", host: "omc-odm.group.echonet"
nginx-ingress-controller-sfjx2 nginx-ingress 2020/07/07 12:55:53 [error] 23665#23665: *1269538704 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 159.50.228.31, server: smartit360.ngc02.group.echonet, request: "POST /auth/logout HTTP/1.1", upstream: "http://10.241.76.132:7500/auth/logout", host: "smartit360.ngc02.group.echonet", referrer: "https://smartit360.ngc02.group.echonet/"
Environment
- Product Version: ICP 3.2.0
- Platform: Linux 64-Bit
- Operating System: Red Hat Enterprise Linux (RHEL) 7.6
- Service Type: BreakFix
- Virtualization Platform: VMWare
- High Availability (HA): Yes
- Problem Area: Networking
Diagnosing The Problem
From ingress nginx the connection to the URL was timed out.
- Please check which service is using that IP
10.241.xx.xxx.kubectl get svc --all-namespaces | grep 10.241.xx.xxx - Then check the pods of the service.
kubectl -n <your service namespace> get pods -l <selector in your service> -o wide. Compare the timestamp where the pod was created or restarted with the time when this timeout error happens (UTC 2020/07/07 12:48:27). - Describe the pod to see whether it was restarted (kubectl -n describe pods) or the container itself is actually not working. You can check the log of the pod.
- Please check log of kubelet (
/var/log/messages) and kube-proxy from the node where your pod was running.
Resolving The Problem
This issue seems to occus due to illumio deployment on ICP nodes.
To resolve this issue you will need to disable the illumio agent.
Document Location
Worldwide
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSBS6K","label":"IBM Cloud Private"},"ARM Category":[{"code":"a8m50000000Ck3xAAC","label":"IBM Cloud Private->Networking"}],"ARM Case Number":"TS003909326","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)","Line of Business":{"code":"LOB45","label":"Automation"}}]
Product Synonym
ICP;IBM Cloud Private; Cloud Private; Illumio
Was this topic helpful?
Document Information
Modified date:
24 July 2020
UID
ibm16252789