How to disable hyper-threading in IBM Cloud — best practices for HPC workloads.

Some high-performance computing (HPC) applications in EDA, weather, oil and natural gas and simulations in various domains benefit from running with hyper-threading (HT) disabled in the guest operating system. MPI application experts typically have handy tools to bind their MPI tasks to primary threads and leave the secondary threads idle. Software licensing issues and interference from OS threads that run these secondary threads cause issues for HPC applications, so some HPC experts prefer to disable hyper-threading altogether inside the guest virtual machines.

We have written a detailed document on how to disable HT through two different methods:  

1. A user-level script that one could run inside the virtual machine
2. Commands to disable at the boot time

The goal of this blog post is to give a short summary of these steps.

How to check if hyper-threading (HT) is on or off

Simply run the lscpu command and notice the output:

 [root@seelam-q-test ~]# lscpu
...
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
...

Two threads per core indicate that the HT is on. If it is off, you will have one thread per core. 

How to turn hyper-threading off and on — the dynamic way

You can disable HT by simply writing a zero into the online field of the sys fs location for a given CPU. To turn off HT of core 0, the command would be: echo 0 > /sys/devices/system/cpu/cpu1/online

This disables HT on core 0 but the other cores are still running in HT mode. You can repeat the above process for all cores or use the handy script provided in the documentation.

Turning on HT requires writing a “1” into the `online` field. 

This gives full control to the end user as to when to turn on and off the HT mode and which cores to turn on and off.

How to turn hyper-threading off — the static way

In situations where you know up front that these nodes are used for HPC applications exclusively or they are created and destroyed dynamically, and the end user should not be bothered with such changes, HT can be disabled at boot time by posting the script below in the cloud-init User Data field:

#cloud-config
bootcmd:
 - for cpunum in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | cut -s -d- -f2- | tr ',' '\n' | sort -un); do echo 0 > /sys/devices/system/cpu/cpu$cpunum/online; done

Conclusion

In this post, we showed how to disable hyper-threading in IBM Cloud virtual servers either by the end users or by the sys admins. Please refer to IBM Docs for detailed instructions. Try your HPC workloads on IBM Cloud directly or by using the HPC service and give us your feedback.