Upgrade Docker version on the Information Server Microservices tier

Steps

$ cd /opt/IBM/UGinstall/ugdockerfiles/
$ ansible -o -m setup -a 'filter=ansible_nodename' all
deployment_coordinator | SUCCESS => {"ansible_facts": {"ansible_nodename": "mltest-rhel-1.novalocal", ...
master-1 | SUCCESS => {"ansible_facts": {"ansible_nodename": "mltest-rhel-1.novalocal", ...
worker-1 | SUCCESS => {"ansible_facts": {"ansible_nodename": "mltest-rhel-2.novalocal", ...
worker-2 | SUCCESS => {"ansible_facts": {"ansible_nodename": "mltest-rhel-3.novalocal", ...

$ cd /opt/IBM/UGinstall/ugdockerfiles/
$ ./run_playbook.sh playbooks/upgrade_docker.yaml --limit master-1

1. On the master node, drain the chosen node by providing the discovered node name to the "kubectl drain" command. This stops all the application containers running on that node:

   $ kubectl drain mltest-rhel-2.novalocal --delete-local-data --ignore-daemonsets
   node/mltest-rhel-2.novalocal cordoned
   evicting pod "kafka-0"
   evicting pod "audit-trail-service-5c6bc7c677-lg8zk"
   evicting pod "cassandra-0"
   ...
   ...
   pod/ingress-nginx-controller-794bb7f58b-sppxz evicted
   pod/omag-6db4894cb-vffrz evicted
   pod/shop4info-event-consumer-88d76474b-v2mkr evicted
   node/mltest-rhel-2.novalocal evicted

2. On the node being upgraded, stop kubelet service. Note: use of sudo may be omitted if logged-in as root:

   $ sudo systemctl stop kubelet.service

3. On the node being upgraded, uninstall current Docker version:

   $ sudo yum remove docker-ce

4. On the master node, run Docker setup playbook, limiting it to the Ansible alias of the node being upgraded. When prompted, verify that the list of hosts is expected (should only specify a single node) and confirm with "yes". The playbook will continue and should report no failed tasks at the end:

   $ ./run_playbook.sh playbooks/platform/kubernetes/setup_docker.yaml --limit worker-1
   [INFO]  Console log output file: /opt/IBM/UGinstall/ugdockerfiles/logs/setup_docker_2020_05_19_13_50_32.log
   [INFO]  Checking for Ansible...
   [INFO]  Ansible version is 2.9.5
   [PASS]  Ansible version is supported
   [INFO]  Ansible uses python2
   [INFO]  Checking for required python2 libraries...
       OK: Module netaddr was found
       OK: Module dns was found
   [PASS]  Required python2 libraries are installed
   
   
   [INFO]  Checking hosts connectivity...
   master-1 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "ping": "dest=localhost"}
   deployment_coordinator | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "ping": "dest=localhost"}
   worker-1 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "ping": "dest=localhost"}
   worker-2 | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "ping": "dest=localhost"}
   
   
   Do you want to proceed? (yes/no): yes
   
   
   ...
   ...
   
   
   RUNNING HANDLER [com/ibm/ugi/kubeplatform/docker : Restart Docker service] ****************************************************************************************************************************************
   Tuesday 19 May 2020  15:52:13 +0200 (0:00:00.411)       0:01:25.395 *********** 
   changed: [worker-1]
   
   
   PLAY RECAP ********************************************************************************************************************************************************************************************************
   worker-1                   : ok=10   changed=4    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
   
   
   ...
   ...
5. On the node being upgraded, verify that the Docker version is now correct:

   ​$ sudo docker --version
   Docker version 19.03.5, build 633a0ea

6. When upgrading from releases before 11.7.1.1, on the node being upgraded, reconfigure kubelet service to use systemd driver for cgroups. To do this, edit the file /var/lib/kubelet/kubeadm-flags.env and remove the --cgroup-driver=cgroupfs option. Then, edit the file /var/lib/kubelet/config.yaml and set "cgroupDriver" property to "systemd".
7. When the node being upgraded is the master node, start the container image registry:

   $ sudo docker start registry
   registry

8. Finally, start the kubelet service on the node being upgraded:

   $ sudo systemctl start kubelet.service

9. Back on the master node, wait for the node being upgraded to become "Ready", by taking a look at the output of the "kubectl get nodes" command:

   $ kubectl get nodes
   NAME                      STATUS                      ROLES    AGE   VERSION
   mltest-rhel-1.novalocal   Ready                       master   42d   v1.17.4
   mltest-rhel-2.novalocal   Ready,SchedulingDisabled    <none>   42d   v1.17.4
   mltest-rhel-3.novalocal   Ready                       <none>   42d   v1.17.4

10. Finally, uncordon the node being upgraded:

    $ kubectl uncordon mltest-rhel-2.novalocal
    node/mltest-rhel-2.novalocal uncordoned