Troubleshooting Calico networks

Identifying and investigating Calico network issues.

Calico network issues might show up during or after IBM® Cloud Private installation. During installation, the installer runs checks to ensure seamless pod-to-pod connectivity in your cluster. However, if you still face issues, the following information might help you to identify the causes and resolve the issues.

Issues during installation of IBM Cloud Private

To avoid Calico network issues during installation, ensure that the following settings are correctly configured.

The calico_ipip_mode parameter must be set to Always if all the nodes in your cluster do not belong to the same subnet.

You must also set this parameter to Always if the nodes are deployed in cloud environments such as OpenStack, where source and destination checks prevent IP traffic from unknown IP address ranges. You must set this parameter even if all the nodes belong to the same subnet. This configuration enables encapsulation of pod to pod traffic over the underlying network infrastructure.
The calico_ip_autodetection_method parameter must be set so that Calico picks the correct interface on the node. If you have multiple interfaces, aliases, logical interfaces, bridge interfaces, or any other type of interfaces on the nodes, use either of the following settings to ensure that the auto-detect mechanism chooses the correct interface.
- calico_ip_autodetection_method: can-reach= >>> Note: This is the default setting.
- calico_ip_autodetection_method: interface=
The calico_tunnel_mtu parameter must be set based on the MTU of the interface that is configured to be used by Calico.

If calico_ipip_mode parameter is set to Always, 20 bytes are used for IP-IP tunnel header. You must set the calico_tunnel_mtu parameter to be at least 20 bytes less than the actual MTU of the interface.

If IPsec is enabled, 40 bytes are needed for the IPsec packet header. And, since you set calico_ipip_mode to Always when you enable IPsec, you also need the 20 bytes for the IP-IP tunnel header. Therefore, you must set the calico_tunnel_mtu parameter to be at least 60 bytes less than the actual MTU of the interface.
The network CIDR, existing host network, and the service cluster IP range must not be in conflict with each other.

Issues after installation of IBM Cloud Private

After your cluster is installed, you might see the IP connectivity issues across the pods. Service name resolution issues are a symptom of pods not being able to reach the DNS service, but are not always related to Calico networks.

In such situations, gather the following information from your cluster for troubleshooting. If you contact the support team for assistance, you can provide this information to the team.

Set up the Kubernetes CLI (kubectl). See Accessing your cluster from the Kubernetes CLI (kubectl).
Set up the calicoctl binary file that is available from the IBM Cloud Private installation media. See Installing the Calico CLI (calicoctl).
Get the list of nodes in your cluster.
```
kubectl get nodes -owide
```

Collect logs from the calico-node-* pod that runs on the node that is experiencing the mesh problem. For example, complete the following steps to get the logs from calico-node-amd64-48lf9 that runs on node 10.10.25.71.

Get a list of Calico pods.

kubectl get pods -o wide | grep calico-node

Following is a sample output:

calico-node-amd64-2cbjh                              2/2       Running   0          7h        10.10.25.70    10.10.25.70
calico-node-amd64-48lf9                              2/2       Running   0          7h        10.10.25.71    10.10.25.71
calico-node-amd64-75667                              2/2       Running   0          7h        10.10.25.7     10.10.25.7

Retrieve the logs from the calico-node container in the pod.
```
kubectl logs calico-node-amd64-48lf9 -c calico-node
```

Diagnose the problem.
1. Get routing table and interface details. Run these commands on all master nodes and on the nodes that have the pods that are experiencing connectivity issue.
  1. Get routing table details.
```
route -n
```
  2. Get interface details.
```
ifconfig -a
```
2. Get the Calico node list. Run the command on any master node.
```
calicoctl  get nodes
```
3. Get all the pods or end points that are on the Calico mesh. Run the command on any master node.
```
calicoctl  get workloadendpoints
```
4. Get Calico node status and diagnostics information. Run these commands on any master node and on the nodes that have the pods that are experiencing connectivity issues.
```
calicoctl node status
calicoctl node diags
```
5. Check the config.yaml and host files that are on your boot node.