Customizing the cluster with the config.yaml file

The config.yaml file contains all the configuration settings that are needed to deploy your cluster.

From the config.yaml file, you can customize your installation by using various parameters.

The config.yaml also contains a list of the Docker images that are pulled from Docker Hub by the installer during the IBM® Cloud Private-CE (Community Edition) installation process. For an IBM Cloud Private-CE (Community Edition) installation, you can also store these installation images in a private image registry in place of pulling directly from Docker Hub. If images are stored in a private image registry, update the config.yaml file to point to the installation images in your private image registry. For an IBM Cloud Private installation, these installer Docker images are commented out since the images are available in the downloaded installer package.

Note: Before you update any section of the config.yaml file, review the inline comments in that section.

You can set or update the following parameters by modifying the config.yaml file.

  1. Open the /<installation_directory>/cluster/config.yaml file.

  2. Add or modify the parameters and values. The format for defining a parameter and values is <parameter_name>:<value>.

General settings

Table 1. General settings
Parameter Description Default value
offline_pkg_copy_path The directory to hold the temporary installation files during offline installation. This location must have at least 50 GB of available disk space.

If your /tmp directory has less than 50 GB of space, you must set this parameter to a location that has the available disk space requirement.

 /tmp
firewall_enabled If set to true, the installer keeps the local operating system firewall running and opens the ports that IBM Cloud Private needs on the firewall. false
wait_for_timeout This parameter specifies the default timeout value for operations. A setting of 3600 is ideal in most environments. 600
router_http_port The http port that is used by the IBM Cloud Private management ingress, which acts as a proxy for all IBM Cloud Private management services. 8080
router_https_port The https port that is used by the IBM Cloud Private management ingress, which acts as a proxy for all IBM Cloud Private management services. 8443
ingress_http_port The http port that is used by the NGINX ingress controller, which acts as a proxy for all user services. 80
ingress_https_port The https port that is used by the NGINX ingress controller, which acts as a proxy for all user services. 443
loopback_dns Set this parameter to true if the boot node uses a loopback IP, for example an IP address that starts with 127, as the DNS server. false
fips_enabled Set this parameter to true to enable Federal Information Processing Standard (FIPS) 140-2 compliance for IBM Cloud Private management ingress (management console), NGINX ingress controller (ingress service), image manager, and identity provider authorization configuration. false
ansible_python_interpreter Set this parameter to /usr/bin/python3 if you use python3 in your cluster nodes. /usr/bin/python
bootstrap_token_ttl Duration for which the bootstrap token is valid. "24h0m0s"
upload_chart_enabled Set this parameter to true to enable chart upload to Helm repository when the mgmt-repo management service is enabled. true

Kubernetes settings

Table 2. Kubernetes settings
Parameter Description Default value
kube_apiserver_extra_args Sets extra apiserver configurations for Kubernetes. Accepts a list of apiserver arguments that are provided in --key=value format. Note: When you set the AlwaysPullImages admission control plug-in, users must pull all the images from their own registry. None
kube_apiserver_secure_port Sets the Kubernetes apiserver secure port. 8001
kube_controller_manager_extra_args Sets extra controller configurations for Kubernetes. Accepts a list of controller arguments that are provided in --key=value format. None
kube_proxy_extra_args Sets extra proxy configurations for Kubernetes. Accepts a list of proxy arguments that are provided in --key=value format. None
kube_scheduler_extra_args Sets extra scheduler configurations for Kubernetes. Accepts a list of scheduler arguments that are provided in --key=value format. None
kubelet_extra_args Sets extra configurations for kubelet. Accepts a list of kubelet arguments that are provided in --key=value format. For example, to set the maximum number of pods that can run on a kubelet, set the following configuration:kubelet_extra_args: ["--max-pods=110"] None
auditlog_enabled Enables the Kubernetes audit log, which records the chronological sequence of activities by individual users, administrators, or other components of the system that modified the system. Set this parameter to true to enable the audit log. false
journal_path Sets the default path to store audit log data. Path to directory: /run/log/journal

Log settings

Table 3. Log settings
Parameter Description Values Default value
metrics_max_age Sets the maximum number of days to store system and application metrics. Metrics older than this specified number of days are removed. Metrics are removed at 23:59 on the specified day. Number of days 1
kibana_install IBM Cloud Private Version 3.2.0 deploys Kibana by default and integrates it with the IBM Cloud Private management console. If you do not want to install Kibana in your cluster, add kibana_install: false in the config.yaml file before you install your cluster. true or false true
logs_maxage Sets the maximum number of days to store logs in Elasticsearch. Number of days 1
docker_config: log-opts: Docker logs options.
max-size Set the maximum size of Docker log files. "100m"
max-file Set the maximum number of Docker log files. "10"
elasticsearch_storage_size Sets the storage size for retaining log data in Elasticsearch. "20Gi"

Network settings

Table 4. Network settings
Parameter Description Values Default value
network_type Type of network management in your cluster. calico or nsx-t calico
network_helm_chart_path Absolute path to the network helm chart. None None
calico_ipip_mode Allows Calico to be run on IP over IP mode. This setting is needed when worker nodes are in different subnetwork and BGP is not enabled in routers between the worker nodes. This setting is also needed in some cloud environment such as OpenStack, where virtual machines are not allowed to work as routers.
  • Always: IP in IP tunnel MeSH is created across cluster nodes for pod to pod communication.
  • Never: IP in IP tunnel MeSH is not created across cluster nodes for pod to pod communication.
  • CrossSubnet: IP in IP tunnel MeSH is created for the nodes that are on different subnets.
 Always
calico_tunnel_mtu The IPIP for Calico has a default MTU of 1430. If the main interface of your host has an MTU that is less than 1450, Calico IPIP has poor performance. Set the MTU such that the MTU of the host main interface minus the default MTU of the Calico IPIP tunnel is greater than or equal to 20. Positive integers 1430
network_cidr The IPv4 network to use for the entire network. This value must be in CIDR format. When you create a network_cidr, ensure that you select an IP range that does not conflict with the existing host network or with the service_cluster_ip_range. In most environments, you can use the default value.
Important: You cannot modify this parameter after installation since all the platform management services are using these IP ranges and the Kube-proxy is also aware of this network cidr.		 IP address in CIDR format 10.1.0.0/16
calico_ip_autodetection_method You can configure the Calico node to auto-detect the IP address that is used to route between nodes. You can use one of the following methods:
  • calico_ip_autodetection_method: first-found: This method uses the first valid IP address on a valid interface that is found first.
  • calico_ip_autodetection_method: interface: This parameter accepts a comma-separated list of regular expression names as value. It uses the first IP address that is found on the specified interface.
Examples:
  • calico_ip_autodetection_method: interface=eth0
  • calico_ip_autodetection_method: interface=eth.*
  • calico_ip_autodetection_method: interface=eth.*,ens.*
  • calico_ip_autodetection_method: can-reach=<remote IP address or host name>: The can-reach method uses your local routing to determine the IP address that is used to reach the specified destination. This parameter accepts a remote IP address or domain name as value.
    • Note:
  • In an environment that has multiple network interfaces (NICs), use the can-reach method to specify the network to be used for your workloads. In IBM Cloud Private, you can set calico_ip_autodetection_method: can-reach=<Master node IP address>.
  • The network interface names cannot contain the following strings: "docker.*", "cbr.*", "dummy.*", "virbr.*", "lxcbr.*", "veth.*", "lo", "cali.*", "tunl.*", or "flannel.*".
  • Some IPs are not recognized by Calico. Ensure that your interfaces do not have IPs in the following ranges:
    • 10.0.2.0/24 - this IP range is the default vagrant/virtualbox NAT interface address range.
    • 192.168.122.0/24 - this IP range is the default libvirt VM interface address range.
    		•
    • first-found

    • interface=INTERFACE-REGEX

    • can-reach=<remote IP address or domain name>

    		 can-reach={{ groups['master'][0] }}
    service_cluster_ip_range The Kubernetes service cluster IP range. This configuration allocates a block of IPs for services. When you create a service_cluster_ip_range, ensure that you select an IP range that does not conflict with the existing host network or with the network_cidr. service_cluster_ip_range is a virtual network. In most environments, you can keep the default value. IP address in CIDR format 10.0.0.0/16

    Cluster access settings

    Table 5. Cluster access settings
    Parameter Description Values Default value
    cluster_name The name of your cluster.

    In a multiple cluster environment, specify a distinct name for each cluster.

    cluster_name must consist of lowercase alphanumeric characters only.

    Note: Your active cluster_name only appears in the IBM Cloud Platform header after installation of IBM Multicloud Manager.

    The cluster_name parameter is used for constructing the region field in the resource Cloud Resource Names (CRNs). If you do not specify a cluster_name, the default value of mycluster is used in the CRN. The CRNs that are constructed with this information uniquely identify the resources and are used for administering and enforcing authorization. Do not change the value of this parameter after your cluster is installed as this change would break the authorization.

    		 NA mycluster
    cluster_CA_domain Specify the certificate authority (CA) domain to use in your cluster. NA {{ cluster_name }}.icp
    cluster_domain A Kubernetes internal DNS domain name. NA cluster.local
    cluster_lb_address In an environment that has multiple network interfaces (NICs), use cluster_lb_address to set a public or external IP address for the management services in your cluster. You can specify a fully qualified domain name instead of the IP address.
    This public address is assigned to the master node, used to access the console, and also used to configure kubectl.
    In an HA environment, cluster_lb_address masks the cluster_vip as the leading master IP.
    • IP address
    • Fully qualified domain name
    • OpenStack floating IP address
    		 None
    proxy_lb_address In an environment that has multiple network interfaces (NICs), use proxy_lb_address to set a public or external IP address that is to be used by the NodePort resource to allow external access to services. You can specify a fully qualified domain name instead of the IP address.
    • IP address
    • Fully qualified domain name
    • OpenStack floating IP address
    		 None

    Cluster settings

    Parameter Description Default Value

    Note: These configurations can be set for the supplied IBM Cloud Private Docker packages only. See IBM Cloud Private Docker packages. | single_cluster_mode: | If set to false, the installer deploys the IBM Multicloud Manager hub and the user can configure IBM Multicloud Manager features. All IBM Multicloud Manager components are installed and the default experience changes. For the management console, if the setting is true (default), only the Search feature is installed, which includes the console and API components.| true |

    Docker settings

    Note: These configurations can be set for the supplied IBM Cloud Private Docker packages only. See IBM Cloud Private Docker packages.

    Table 7. Docker settings
    Parameter Description Format Default Value
    docker_version Specify the version of Docker that you want to install.

    A package for the required version must be available in the /<installation_directory>/cluster/runtime-engine directory.

    		 string 18.06.2
    install_docker Allows the installer to automatically install Docker on your cluster nodes. true or false true
    docker_env Sets the environment for Docker. For example, you might configure an https_proxy location if Docker runs behind a firewall. The environment location is stored in the /etc/systemd/system/docker.service.d/docker-env-icp.conf file. "HTTP_PROXY=http://proxy-server:port/", "HTTPS_PROXY=http://proxy-server:port", "NO_PROXY=localhost,127.0.0.1,{{ cluster_CA_domain }}" None

    Proxy HA settings

    Table 8. Proxy HA settings
    Parameter Description Values Default
    proxy_vip_iface Sets the virtual IP interface for a proxy node HA environment. eth0 n/a
    proxy_vip Sets the virtual IP address for a proxy node HA environment. 127.0.1.1
    Do not specify the subnet in the IP address.    		 n/a
    vip_manager Specify the VIP manager to be used for master or proxy node in an HA environment. You can specify etcd or keepalived as the vip_manager. etcd

    Master HA settings

    Table 9. Master HA settings
    Parameter Description Values Default
    cluster_vip Sets the virtual IP address for IBM Cloud Private HA environment. 127.0.1.1
    Do not specify the subnet in the IP address.    		 n/a
    vip_iface Sets the virtual IP interface for IBM Cloud Private HA environment. eth0 n/a
    vip_manager Specify the VIP manager to be used for master or proxy node in an HA environment. You can specify etcd or keepalived as the vip_manager. etcd

    User settings

    Table 10. User settings
    Parameter Description Default
    ansible_user
    ansible_become
    ansible_become_password    		 IBM Cloud Private uses the ansible_user parameter value to access your cluster nodes during installation.
    If you use a non-administrator account that has sudo privileges to connect to a master or worker node, set the ansible_user to the user name and ansible_become to true.
    If you run sudo with a password, you must set the ansible_become_password parameter to the value of your non-root (sudo user) password. This variable is optional if you set NOPASSWD in the /etc/sudoers file.
    To configure these parameter values, see Configuring password authentication for cluster nodes.    		 None
    default_admin_user Sets a customized cluster administrator user name. If LDAP is enabled and you have an LDAP user with the user name admin, change the default_admin_user parameter value to something else to avoid a conflict of cluster administrator name and LDAP user name. admin
    default_admin_password This parameter sets the required default cluster administrator password. The password must satisfy all regular expressions that are specified in password_rules. If LDAP is enabled, this parameter sets the LDAP administrator password. n/a
    password_rules Defines a list of one or more password rules as regular expressions that the default_admin_password must pass. If not defined, the default_admin_password must meet the default password enforcement rule '^([a-zA-Z0-9\-]{32,})$': The password must be a minimum of 32 characters long and can contain only lower and uppercase alphanumeric characters and dashes. The rules must be defined in an array list. For more information and an example, see Customize your cluster. - '^([a-zA-Z0-9\-]{32,})$'

    GlusterFS settings

    Table 11. GlusterFS settings
    Parameter Description
    storage-glusterfs Provision storage on dedicated nodes.

    To set up GlusterFS, you must set several parameters. See Configuring GlusterFS during IBM Cloud Private installation.

    Cloud Provider settings

    Table 12. vSphere Cloud Provider settings
    Parameter Values Description
    cloud_provider vsphere

    vSphere Cloud Provider settings

    Sets up vSphere Cloud Provider.

    To set up vSphere Cloud Provider, you must set several parameters. See Configuring a vSphere Cloud Provider for more information.

    Note: If you want to configure a vSphere Cloud Provider after you install your IBM® Cloud Private cluster, ensure that you set kubelet_nodename: hostname in the <installation_directory>/cluster/config.yaml file during IBM® Cloud Private installation.
    aws

    AWS Cloud Provide settings

    Sets up AWS Cloud Provider.

    To set up AWS Cloud Provider, you need to set two parameters: cloud_provider to aws and kubelet_nodename to nodename.

    Ensure hostname and internal DNS name is consistent on AWS hosts.

    For more details about setting up IBM Cloud Private-CE (Community Edition) on an Amazon Web Services (AWS) cloud platform, see Run IBM Cloud Private on Amazon Web Services (AWS) cloud platform Opens in a new tab.
    azure

    Azure Cloud Provider settings

    Sets the cloud provider for Kubernetes integration to be Azure. To set up Azure Cloud Provider, you must set several additional parameters. See Configuring a Azure Cloud Provider for more details.

    Encrypting cluster data network traffic with IPsec

    Table 13. Encrypting cluster data network traffic with IPsec
    Parameter Description
    ipsec_mesh: Enables IPsec.

    To set up IPsec, you must set several parameters. See Encrypting cluster data network traffic with IPsec.

    Management service settings

    Parameter Description Default value
    management_services Use this parameter to enable or disable management services for your cluster. You can enable or disable services by changing the status of the service to enabled or disabled within the management_services section of the config.yaml file.

    For more information about management services, see Enabling and disabling IBM Cloud Private management services.

    		 Disabled
    • calico-route-reflector
    • platform-security-netpols
    • platform-pod-security
    • storage-glusterfs
    • storage-minio
    • istio
    • custom-metrics-adapter
    • vulnerability-advisor
    • node-problem-detector-draino
    • multicluster-endpoint
    • system-healthcheck-service
    		 Enabled
    • calico/nsx-t
    • kmsplugin
    • tiller
    • image-manager
    • kube-dns
    • cert-manager
    • monitoring-crd
    • nvidia-device-plugin
    • mongodb
    • metrics-server
    • nginx-ingress
    • service-catalog
    • platform-api
    • auth-idp
    • auth-apikeys
    • auth-pap
    • auth-pdp
    • icp-management-ingress
    • platform-ui
    • catalog-ui
    • security-onboarding
    • secret-watcher
    • oidcclient-watcher
    • metering
    • monitoring
    • helm-repo
    • mgmt-repo
    • helm-api
    • logging
    • image-security-enforcement
    • web-terminal
    • audit-logging
    • key-management
    • multicluster-hub

    Integrating VMware NSX-T 2.4 with IBM Cloud Private

    Table 15. Integrating VMware NSX-T 2.4 with IBM Cloud Private
    Parameter Description
    network_type: Set value to nsx-t.
    nsx_t: You must set several parameters to integrate VMware NSX-T 2.4 with IBM Cloud Private. See Integrating VMware NSX-T 2.4 with IBM Cloud Private.

    Environment isolation

    Table 16. Environment isolation
    Parameter Description
    isolated_namespaces: [] Use this parameter if you want to isolate a namespace. Add a namespace and allot a host group with dedicated worker nodes to the namespace. The syntax is [{namespace: <namespace_name>, hostgroup: <hostgroup_name>}]. For example, [{namespace: devops, hostgroup: dev}, {namespace: production, hostgroup: prod}]. The namespaces are created during IBM Cloud Private installation.
    isolated_proxies: [] Use this parameter if you want to allot proxy node host groups to a namespace. Add a namespace and allot a host group with dedicated proxy nodes to the namespace. The syntax is [{namespace: <namespace_name>, hostgroup: <hostgroup_name>, lb_address: <load_balancer_IP_address>}]. For example, [{namespace: devops, hostgroup: proxydev, lb_address: 172.68.20.11}, {namespace: production, hostgroup: proxyprod}]. If you are specifying a namespace that is not defined in the isolated_namespaces parameter, you must manually create the namespace after you install IBM Cloud Private.

    etcd settings

    Table 17. etcd settings
    Parameter Description Default value
    etcd_extra_args Sets extra etcd configurations. Accepts a list of arguments that are provided in --key=value format. ["--grpc-keepalive-timeout=0", "--grpc-keepalive-interval=0", "--snapshot-count=10000"]
    etcd_data_dir etcd data directory. For example, /var/lib/etcd. /var/lib/etcd
    etcd_wal_dir etcd wal directory. For example, /var/lib/etcd-wal. You could set the directory to a centralized remote log directory for persistent logging. The default value that etcd uses is --max-wals=5. /var/lib/etcd-wal

    Istio add-ons security settings

    Table 18. Istio add-ons security settings
    Parameter Description Default value
    istio_addon Configure Istio add-ons security settings.
    grafana Configure Grafana Istio add-on to view service mesh traffic data. Configure the user name and password to access the interface. None
    kiali Configure Kiali Istio add-on to view microservices data in the Istio service mesh. Configure the user name and password to access the interface. None

    NGINX ingress controller

    Table 19. NGINX ingress controller
    Parameter Description
    nginx-ingress:ingress Customize the NGINX ingress controller configuration. The ingress controller is enabled by default. For a list of configurable parameters, see Kubernetes nginx-ingress configuration options Opens in a new tab.
    config:
    disable-access-log Disables access log. Set the value to true or false.
    keep-alive-requests Sets the maximum number of requests that can be served through one keep-alive connection. For example, 10000.
    upstream-keepalive-connections Activates the cache for connections to upstream servers. The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed. For example, 64.
    worker-processes Sets the number of worker processes. For example, 2.
    extraArgs:
    publish-status-address Specify the address that the ingress controller must use. For example, "{{ proxy_external_address }}".
    enable-ssl-passthrough Send TLS connections directly to the pod instead of allowing NGINX to decrypt the communication. Set the value to true or false.

    Minio settings

    Table 20. Minio settings
    Parameter Description
    storage-minio To configure Minio, you must set several parameters. See, Configuring Minio during IBM Cloud Private installation.

    MongoDB security settings

    Table 21. MongoDB security settings
    Parameter Description Default value
    mongodb_credentials: Configure MongoDB access credentials.
      admin_user Configure administrator user name. admin
      admin_password Configure administrator password. If you do not specify a password, the installer generates a random string and sets that as the password. Randomly generated string

    Multicluster-endpoint settings

    Table 22. Multicluster-endpoint settings
    Parameter Description Default value
    multicluster-endpoint: Configure multicluster endpoint parameters.
      global: Configure multicluster global parameters.
        clusterName The displayed cluster name in the multicluster-hub. cluster_name parameter or local-cluster when you deploy on a multicluster-hub. "{{ cluster_name }}"
        clusterNamespace The namespace that the cluster will belong to in the multicluster-hub. cluster_name parameter or local-cluster when you deploy on a multicluster-hub. "{{ cluster_name }}"
      clusterLabels: Provide cluster labels.
        environment: Cluster environment label. Dev
        region: Region where the cluster is set up. US
        datacenter: Location of the datacenter where the cluster is set up. toronto
        owner: Owner of the cluster. marketing
      operator: Klusterlet operator configuration.
        bootstrapConfig: Information for registering with multicluster-hubs.
          hub0: Provide hub0 information.
            name: Provide hub0 name. hub0
            secret: Provide bootstrap secret that contains the kubeconfig for connecting to hub0. kube-system/klusterlet-bootstrap
          hub1: Provide hub1 information.
            name: Provide hub1 name. None
            secret: Provide bootstrap secret that contains the kubeconfig for connecting to hub1. None
      klusterlet: Provide Klusterlet configuration parameters.
        host: Provide host name for the Klusterlet service ingress or route. None
      prometheusIntegration:
        enabled: Enable or disable Prometheus. true
      policy:
        cemIntegration: Enable or disable CEM integration. false
      topology:
        enabled: Enable or disable topology. true
      serviceRegistry: The possible values for service registry plug-ins are kube-service, kube-ingress and istio.
        enabled: Enable or disable service registry. false
        dnsSuffix: Provide DNS suffix. mcm.svc
        plugins: Provide plug-ins. kube-service

    Note: For more information on service registry settings, see Working with IBM Multicloud Manager service discovery.

    Multicluster-hub settings

    Table 23. Multicluster-hub settings
    Parameter Description Default value
    multicluster-hub: Configure multicluster hub parameters.
      etcd: Configure etcd parameters for multicluster hub.
        persistence: Enable or disable the persistence volume for etcd. false
        localPath: The local directory for persistence volume in all management nodes for etcd. /var/lib/etcd-mcm