Sometimes network connections on your Kubernetes cluster can be very slow, and you need to understand where there are bottlenecks.
Since IBM Cloud Kubernetes Service is a managed service, debugging the network can be very difficult. In this post, we will show you how to debug network performance on your own by using open source tools.
iPerf3
iPerf3 is an open source software used frequently to measure the maximum bandwidth allowed for a network interface. This is not installed by default in every container image, so in order to install, you will need to define a container image by yourself.
For example:
FROM nginx
RUN apt-get update -y && apt-get install iperf3 -y
If you would like to use a predefined container image, you could use the iperf image from ClearLinux.
Once you have the container image, you have to deploy it into your IBM Cloud Kubernetes Service cluster. To do that, you can use a DaemonSet resource:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: iperf3
labels:
app: iperf3
spec:
selector:
matchLabels:
app: iperf3
template:
metadata:
labels:
app: iperf3
spec:
containers:
- name: iperf3
image: clearlinux/iperf:3
command: ['/bin/sh', '-c', 'sleep 1d']
ports:
- containerPort: 5201
As per DaemonSet definition, one replica for each node will be created.
Tests
Once the DaemonSet is ready, list all pods created:
kubectl get pods -o wide -l app=iperf3
The output should be something similar to this:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
iperf3-29dwc 1/1 Running 0 4m 172.30.24.203 10.187.165.223 <none>
iperf3-2xctw 1/1 Running 0 4m 172.30.20.85 10.94.110.40 <none>
iperf3-8zlhb 1/1 Running 0 4m 172.30.106.81 10.94.110.35 <none>
iperf3-hsv7g 1/1 Running 0 4m 172.30.133.17 10.185.69.203 <none>
iperf3-kx6h6 1/1 Running 0 4m 172.30.177.47 10.187.165.224 <none>
iperf3-wzkp5 1/1 Running 0 4m 172.30.32.22 10.185.69.204 <none>
Then, select one pod as master (which will receive all the packets that are sent) and the others as clients.
Opening a terminal, run the following on the master pod:
kubectl exec -it POD_NAME -- iperf3 -s -f M
Opening a separate terminal, run the following for each client pod:
kubectl exec -it POD_NAME -- iperf3 -c MASTER_POD_IP -f M
The result should be something similar to the following:
Connecting to host 172.30.70.199, port 5201
[ 5] local 172.30.35.143 port 34202 connected to 172.30.70.199 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 617 MBytes 617 MBytes/sec 41 1.43 MBytes
[ 5] 1.00-2.00 sec 714 MBytes 714 MBytes/sec 0 1.75 MBytes
[ 5] 2.00-3.00 sec 635 MBytes 635 MBytes/sec 74 1.12 MBytes
[ 5] 3.00-4.00 sec 669 MBytes 669 MBytes/sec 49 1.17 MBytes
[ 5] 4.00-5.00 sec 649 MBytes 648 MBytes/sec 340 908 KBytes
[ 5] 5.00-6.00 sec 641 MBytes 642 MBytes/sec 0 1.32 MBytes
[ 5] 6.00-7.00 sec 649 MBytes 649 MBytes/sec 0 1.63 MBytes
[ 5] 7.00-8.00 sec 670 MBytes 670 MBytes/sec 57 1.42 MBytes
[ 5] 8.00-9.00 sec 661 MBytes 661 MBytes/sec 11 1.26 MBytes
[ 5] 9.00-10.00 sec 634 MBytes 634 MBytes/sec 362 969 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.38 GBytes 654 MBytes/sec 934 sender
[ 5] 0.00-10.04 sec 6.38 GBytes 650 MBytes/sec receiver
iperf Done.
Conclusion
This is just one way to debug network performances in a Kubernetes cluster.
If the output of iPerf tests return low bitrates, there is something misconfigured on the cluster, and you should immediately ask to IBM Cloud support to have a look.
Engage the support via the IBM Cloud Support Center.