Configuring LSF to run Singularity jobs

Configure a Singularity application profile or queue in LSF to run Singularity container jobs.

Before you begin

  • Singularity, Version 2.2, or later, must be installed on an LSF server host.
  • Add the path to the Singularity executable file to the $PATH environment variable before submitting jobs, and ensure that it is accessible on each execution host.
  • The required Singularity images that you intend to use must be prepared before running LSF jobs in containers. Ensure that these images are accessible to each execution host.
  • If you use mounted file systems for LSF, other software installations, or for input/output directories, you must also bind required directories into containers to ensure that LSF works properly. To bind directories for Singularity containers, add mount hostfs = yes to the singularity.conf file. This parameter definition binds all mounted file systems to the container.
  • To run jobs with bsub -L login_shell or where the execution user is changed by user mapping, the execution user must be able to execute the singularity command under its $PATH environment variable. You can install the singularity binary file in the /usr/bin directory or add a symbolic link to the installation path of singularity.

About this task

You cannot run pre-execution and post-execution scripts in container jobs. The following are workarounds for specific pre-execution and post-execution operations:
  • To prepare data for the container as a pre-execution or post-execution operation, put this data into a directory that is mounted to a job container.
  • To customize the internal job container, you can customize the starter scripts to prepare the appropriate environment.

Procedure

Edit the lsb.applications or lsb.queues file and define the CONTAINER parameter for the application profile or queue to run Singularity container jobs.

If this parameter is specified in both files, the parameter value in the lsb.applications file overrides the value in the lsb.queues file.

CONTAINER=singularity[image(image_name) options(container_options)]

image
Required. This keyword specifies the Singularity image name that is used in running jobs.
options
Optional. This keyword specifies the Singularity job run options, which are passed to the job container with the singularity exec command in LSF.

To enable a pre-execution script to run, specify an at sign (@) and a full file path to the script, which the execution host must be able to access. Before the container job runs, LSF runs this script with LSF administrator privileges. While the script is running, the jobs' environment variables are passed to the script. When the script finishes running, the output is used as container startup options. The script must provide this output on one line.

Note:
  • Run singularity exec --help in the command line to view the options that the singularity command supports.
  • Before you specify the Singularity job run options, make sure that these options work with the singularity exec command in the command line.
  • The $LD_LIBRARY_PATH directory is cleaned according to the setuid bit that Singularity uses to work. Therefore, for programs that depend on $LD_LIBRARY_PATH to work (such as openmpi), ensure that you add $LD_LIBRARY_PATH to the ld.so.conf file and run ldconfig.
For example, to specify the ubuntu.img image for use with Singularity container jobs without specifying any optional keywords, 
Begin Application
NAME = singapp
CONTAINER = singularity[image(/file/path/ubuntu.img)]
DESCRIPTION = Singularity User Service
End Application
To specify the pre-execution script /share/usr/sing-options.sh, which generates the container startup options,
Begin Application
NAME = singappoptions
CONTAINER = singularity[image(/file/path/ubuntu.img) options(@/share/usr/sing-options.sh)]
DESCRIPTION = Singularity User Service with pre-execution script for options
End Application

For more details, refer to the CONTAINER parameter in the lsb.applications file or lsb.queues file.