Service discovery (kube-dns)

Kubernetes expects that a service is running within the pod network mesh that performs name resolution and acts as the primary name server within the cluster.

In IBM® Cloud Private, this is implemented by using CoreDNS which runs on the master nodes. CoreDNS resolves names for all services that are running in Kubernetes and forwarding name lookups against upstream name servers on behalf of containers. The DNS service itself runs as a ClusterIP service that is backed by one or more containers for high availability.

Kubernetes service names are resolved to ClusterIPs representing one or more pods that match a label selector. The cluster is assigned a cluster domain that is specified at installation time by using cluster_domain (this is cluster.local by default) to distinguish between names local to the cluster and external names.

Each Kubernetes cluster is logically separated into namespaces and each namespace acts as a subdomain for name resolution. Upon examining a container's /etc/resolv.conf, observe that the nameserver line points at an IP address internal to the cluster, and the search suffixes are generated in a particular order:

# cat /etc/resolv.conf 
nameserver <kube-dns ClusterIP>
search <namespace>.svc.<cluster_domain> svc.<cluster_domain> <cluster_domain> <additional ...>
options ndots:5

The <additional ...> is a list of search suffixes obtained from the worker node's /etc/resolv.conf file.

By default, a short host name like account-service has <namespace>.svc.<cluster_domain> appended to it so a pod that matches the label selector that is running in the same namespace as the running pod is selected. A pod can look up the ClusterIP of a pod in a different namespace by appending the namespace to the host name. For example, account-service.prod targets account-service that is running in the prod namespace, since the search suffix svc.<cluster_domain> is appended to the end.

Note the last line in /etc/resolv.conf, options ndots:5, which indicates to the container's system resolver that any host names being resolved that have fewer than five dots in the name must have the search domain suffixes appended to them. This can affect performance since lookups of external network resources with fewer than five dots in the name result in lookups for every entry in the search line. For example, a lookup of www.ibm.com results in lookups of www.ibm.com.<namespace>.svc.<cluster_domain>, www.ibm.com.svc.<cluster_domain>, www.ibm.com.<cluster_domain>, etc. before trying www.ibm.com. To resolve this issue, adding an additional . to the end of fully qualified domain names that are used in the application configuration prevents the system resolver from cycling through the list of suffixes in the name lookups (for example, www.ibm.com.).

Headless services

In some cases, it is desirable to not create a ClusterIP service at all; this can be achieved by specifying a service type of None. This creates A records for each pod that matches the label selector in the DNS, but no ClusterIP. This is typically used with StatefulSets where each of the pods needs to have resolvable names for communication between all pods in the set (for example, a clustered database like MongoDB). When a headless service is created, each pod's A record is in the format <pod-name>.<service-name>.<namespace>.svc.<cluster-domain>.

External services

It is possible to have Kubernetes proxy endpoints outside of the cluster. This can be accomplished by creating a Service resource with no label selector, and by either creating Endpoints resources manually that contain the IPs outside of the cluster to proxy, or by creating a Service resource with the type ExternalName containing an external DNS name. This action creates a CNAME record in the cluster's DNS. By using these functions, the cluster DNS can be used as service discovery for services both inside and outside of the cluster.