Preparing LSF to run Docker jobs

Prepare LSF to run jobs in Docker containers.

Before you begin

The Docker Engine, Version 1.12, or later, must be installed on an LSF server host. The Docker daemon must be started on this host and can successfully start containers.

Procedure

  1. Edit the lsf.conf file and configure the following parameters:
    LSF_PROCESS_TRACKING=Y
    LSF_LINUX_CGROUP_ACCT=Y
    LSB_RESOURCE_ENFORCE="cpu memory"
  2. Edit the lsf.shared file and configure a new Boolean resource named docker.
    ...
    Begin Resource
    RESOURCENAME  TYPE    INTERVAL INCREASING CONSUMABLE DESCRIPTION  # Keywords
    ...
       docker     Boolean ()       ()         ()         (Docker container)
    ...
    End Resource 
  3. Edit the lsf.cluster file and attach the docker Boolean resource to the LSF server host that is running Docker Engine.

    This configuration enables LSF to automatically dispatch Docker jobs to the LSF server host that is running Docker Engine.

    ...
    Begin Host
        HOSTNAME  model  type  server  r1m  mem  swp  RESOURCES
        ...
        host1     !      !     1       3.5  ()   ()   (docker)
        ...
    End Host
  4. Check if /usr/bin/python is linked to the proper Python interpreter.

    In some versions of Linux (such as RHEL 8.x), there is no /usr/bin/python executable file on the host.

    Since the execution driver uses /usr/bin/python to execute the Python script, you must create a link from /usr/bin/python linking to the available Python executable file (both python2.x and python3.x are available).

    For example, to link to python3,
    $ ln -s /usr/bin/python3 /usr/bin/python