Calico

Calico is an open source community project that provides networking for containers and virtual machines.

Calico is built on the third layer, also known as Layer 3 or the network layer, of the Open System Interconnection (OSI) model. Calico uses the Border Gateway Protocol (BGP) to build routing tables that facilitate communication among agent nodes. By using this protocol, Calico networks offer better performance and network isolation.

Calico implements the Kubernetes Container Network Interface (CNI) as a plug-in and provides agents for Kubernetes to provide networking for containers and pods.

Calico creates a flat Layer-3 network and assigns a fully routable IP address to every pod. It divides a large network CIDR (Classless Inter-Domain Routing) into smaller blocks of IP addresses and assigns one or more of these smaller blocks to nodes in the cluster. The division is done during IBM Cloud Private installation by using the network_cidr parameter in config.yaml in CIDR notation.

Calico by default creates a BGP mesh between all nodes of the cluster and broadcasts the routes for container networks to all worker nodes. Each node is configured to act as a Layer 3 gateway for the subnet. The subnet is assigned to the worker node and serves the connectivity to pod subnets that are hosted on the host. All nodes participate in the BGP mesh, which advertises all of the local routes that the worker nodes own to all other nodes. BGP peers that are external to the cluster can participate, but the size of the cluster affects how many BGP advertisements these external peers receive. Route reflectors might be required when the cluster scales pass a certain size.

For more informtion, see Configuring BGP Peers .

When routing pod traffic, Calico uses the system capabilities such as the node's local route tables and iptables. All pod traffic traverses iptables rules before they are routed to their destination.

Calico maintains its state by using an etcd key/value store. By default in IBM Cloud Private, Calico uses the same etcd key/value store as Kubernetes to store the policy and network configuration state.

Calico can be configured to allow pods to communicate with each other with or without IP-in-IP tunneling. IP-in-IP adds a header for all packets as part of the encapsulation, but allows containers to communicate on its overlay network through almost any non-overlapping underlay network.

In some environments where the underlay subnet address space is constrained and there is no access to add additional IP address pools, like on some public clouds, Calico can be a good fit. However, in environments that do not require an overlay, IP-in-IP tunneling must be disabled to remove the packet encapsulation resource use, and to allow any physical routing infrastructure to do packet inspection for compliance and audit. In such scenarios, the underlay network can be made aware of the additional pod subnets by adding the underlay network routers to the BGP mesh. For more information about a Calico network when nodes are on different network segments, see Calico network across different network segments.

Calico components

Calico has the following components:

1. calico/node agent
2. calico/cni
3. calico/kube-controller

To ensure that the nodes meet the Calico system requirements, review the information in Preparing the nodes.

calico/node agent

This entity consists of three components - felix, bird, and confd.

• The primary responsibility of felix is to program the host iptables and routes to provide the connectivity that you want to and from the pods on that host.
• bird is an open source BGP agent for Linux® that is used to exchange routing information between the hosts. The routes that are programmed by felix are picked up by bird and distributed among the cluster hosts.
• confd monitors the etcd data store for changes to the BGP configuration, such as IP Address Management (IPAM) information and autonomous system (AS) number. It also changes the bird configuration files and triggers bird to reload these files on each host. The calico/node agent creates veth-pairs to connect the pod network namespace with the host's default network namespace.

calico/cni

The CNI plug-in provides the IPAM functions by provisioning IP addresses for the pods that are hosted on the nodes.

calico/kube-controller

The calico/kube-controller watches Kubernetes NetworkPolicy objects and keeps the Calico data store in sync with the Kubernetes objects. The calico/node that is running on each node uses the information in the Calico etcd data store to program the local iptables.

calicoctl

calicoctl is a command-line tool that can be used to manage the Calico network and security policies and other Calico configurations. It communicates directly with etcd to manipulate the data store. It provides a number of resource management commands and can be used to troubleshoot Calico network issues. To set up your Calico CLI, see Installing the Calico CLI (calicoctl).

Calico network across different network segments

When nodes are on different network segments, they are connected by a router in the underlay and infrastructure network. The traffic between two nodes on different subnets traverses through the router, which is the gateway for the two subnets. If the router is not aware of the pod subnet, it cannot forward the packets between the hosts.

There are two ways to handle this scenario:

1. Calico can be configured to create IP-in-IP tunnel endpoints on each node for every subnet that is hosted on the node. Any packet that is originated by the pod and is egressing the node, is encapsulated with the IP-in-IP header and the node IP address is used as the source. This way, the infrastructure router does not see the pod IP addresses.

   The IP-in-IP tunneling brings in extra network throughput and latency due to additional packet processing at each endpoint to encapsulate and decapsulate packets. On bare metal, the resource use is not significant as certain network operations can be offloaded to the network interface cards. However, on virtual machines, the resource use can be significant and also affected by the number of CPU cores and network I/O technologies that are configured and used by the hypervisors. The additional packet encapsulation overhead can also be significant when smaller maximum transmission unit (MTU) sizes are used since it can introduce packet fragmentation. Jumbo frames must be enabled whenever possible.

2. The second option is to make the infrastructure router aware of the pod network. You can do so by enabling BGP on the router and adding the nodes in the cluster as BGP peers. These steps allow the router and the hosts to exchange the route information with each other. The size of the cluster in this scenario can come into play as in the BGP mesh. Every node in the cluster is a peer of the router after BGP is enabled on the router.

IBM Cloud Private configuration options for using Calico container networking

config.yaml:

network_type: calico
network_cidr: 10.1.0.0/16
calico_ipip_mode: Always
calico_tunnel_mtu: 1430
calico_ip_autodetection_method: can-reach={{ groups['master'][0] }}

• network_type: Choose calico to deploy Calico as the container network (default is calico )

• network_cidr: network_cidr: Choose a sufficiently large private IP address subnet, which is to be consumed by all the workloads that are planned to be hosted on the cluster. This IP address range must not conflict with the existing host network or with the service_cluster_ip_range. The number of IP addresses varies from version to version based on the number of components that are added to IBM Cloud Private and the number of components and features that you install. You can get the current total IP address count by listing all the pods that are installed.

  If you have N worker nodes and are targeting for P number of pods, then the safe IP address range is ( N x P ) + N. This range can tolerate N-1 node unavailability. The default value is 10.1.0.0/16, which is sufficient to host a large cluster.

• calico_ipip_mode: This option can be used to enable IP-in-IP tunneling. It must be enabled in environments that require an overlay network, for example, in environments where egress packets undergo a strict check for the source IP address against the host IP address, such as OpenStack. This option is also required to be enabled if the infrastructure routers cannot be configured to allow BGP routes to be exchanged.

  When this parameter is set to Always, it is necessary to allow ip protocol 4 through the infrastructure firewalls across all the nodes, for example, Security Groups in Openstack.

• calico_tunnel_mtu: This parameter sets the appropriate MTU on the tunnel device and on the pod interface. With this parameter, you can adjust the MTU according to the infrastructure network's path MTU (PMTU) to avoid packet fragmentation. Also, in certain cases, infrastructure firewalls are not programmed to allow ICMP PMTU error messages to be propagated. As a result, the L3 session does not learn the PMTU, which can lead to packet and connection loss.

  The default value is 1430, which is suitable for most scenarios and IaaS networks such as Openstack Neutron VxLAN networks. This value must be at least 20 bytes less than the interface MTU or PMTU to accommodate IP-in-IP header.

• calico_ip_autodetection_method: This parameter is used to select the interface on the node, and for pod to pod data communication across the nodes. Three methods are available to help select the data path for pods.

  • first-found

    This method uses the first valid IP address on a valid interface that is found first on the node.

  • interface=<interface name list>

    This parameter accepts a comma-separated list of regular expression names as value. It uses the first IP address that is found on the specified interface.

    Examples:

    calico_ip_autodetection_method: interface=eth0
    calico_ip_autodetection_method: interface=eth.*
    calico_ip_autodetection_method: interface=eth.*,ens.*
    

    Note: The network interface names cannot contain the following strings: docker.*, cbr.*, dummy.*, virbr.*, lxcbr.*, veth.*, lo, cali.*, tunl.*, or flannel.*.

  • can-reach=<remote IP address or host name>

    The can-reach method uses a node's local route table. It also uses the interface that is used to communicate to a specified destination address to determine the pod network interface. This parameter accepts a remote IP address or domain name as value.

    Some IP addresses are not recognized by Calico. Ensure that your interfaces do not have IP addresses in the following ranges:

    • 192.168.122.*: this IP address range is the default libvirt VM interface address range.

For more information about IBM Cloud Private deployment topologies that use Calico, see IBM Cloud Private deployment topologies.

Calico monitoring in Prometheus and Grafana

By default, the Prometheus component that is running on the management nodes scrapes the Calico node agent that is running in IBM Cloud Private for metrics. IBM Cloud Private includes a Grafana dashboard that displays cluster network metrics that are retrieved from Calico in graphic representation.

Additionally, alerts can be set by using these metrics that are collected from Felix.