Span string

A span string specifies the locality of a parallel job. If span is omitted, LSF allocates the required processors for the job from the available set of processors.

Syntax

The span string supports the following syntax:

span[hosts=1]
Indicates that all the processors allocated to this job must be on the same host.
span[block=value]
For parallel jobs, LSF will allocate slots to the job based on block size. LSF tries to pack as many blocks on one host as possible, then goes to next one. Each host is only checked once.
span[ptile=value]
Indicates the number of processors on each host that should be allocated to the job, where value is one of the following:
  • Default ptile value, specified by n processors. In the following example, the job requests 4 processors on each available host, regardless of how many processors the host has:
    span[ptile=4]
  • Predefined ptile value, specified by ’!’. The following example uses the predefined maximum job slot limit lsb.hosts (MXJ per host type/model) as its value:
    span[ptile='!']
    Tip: If the host type/model does not define MXJ, the span[ptile='!'] value is ignored.
    Restriction: Under bash 3.0, the exclamation mark (!) is not interpreted correctly by the shell. To use predefined ptile value (ptile='!'), use the +H option to disable '!' style history substitution in bash (sh +H).
  • Predefined ptile value with optional multiple ptile values, per host type or host model:
    • For host type, you must specify same[type] in the resource requirement. In the following example, the job requests 8 processors on a host of type HP, and 2 processors on a host of type LINUX, and the predefined maximum job slot limit in lsb.hosts (MXJ) for other host types:
      span[ptile='!',HP:8,LINUX:2] same[type]
    • For host model, you must specify same[model] in the resource requirement. In the following example, the job requests 4 processors on hosts of model PC1133, and 2 processors on hosts of model PC233, and the predefined maximum job slot limit in lsb.hosts (MXJ) for other host models:
      span[ptile='!',PC1133:4,PC233:2] same[model]
span[stripe]
For parallel jobs, LSF stripes the tasks of the job across the free resources of the candidate hosts.

For example, if you submit a job that requests four tasks with the following command:

bsub -n 4 -R "span[stripe]" ./a.out

The task placement depends on the free resources:

  • If there is one candidate host, that host has four tasks (4).
  • If there are two candidate hosts, each host has two tasks (2,2).
  • If there are three candidate hosts, one host has two tasks, and the two other hosts have one task each (2,1,1).
  • If there are four candidate hosts, each host has one task (1,1,1,1).
span[stripe=max_tasks]
For parallel jobs, LSF stripes the tasks of the job across the free resources of the candidate hosts up to the specified maximum number of tasks on each host.
span[hosts=-1]

Disables span setting in the queue. LSF allocates the required processors for the job from the available set of processors.

Examples

The following examples are for jobs that are submitted in a cluster with the following idle hosts:

bhosts
HOST_NAME          STATUS       JL/U    MAX  NJOBS    RUN  SSUSP  USUSP    RSV
host1              ok              -     32      0      0      0      0      0
host2              ok              -     32      0      0      0      0      0
host3              ok              -     32      0      0      0      0      0
host4              ok              -     32      0      0      0      0      0
host5              ok              -     32      0      0      0      0      0
  • Submit a job with striping:
    bsub -n 32 -R "span[stripe]" myjob

    View the task distribution of a job with striping:

    bjobs
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
118     userA   RUN   normal     hostA       7*host1     myjob      Nov 29 14:27
                                             7*host2
                                             6*host3
                                             6*host4
                                             6*host5

    Tasks are distributed evenly to the hosts, and any additional tasks are distributed to the first hosts. There are 7 tasks on host1 and host2, and 6 tasks on host3, host4, and host5.

  • Submit a job with ptile:
    bsub -n 32 -R "span[ptile=8]" myjob

    View the task distribution of a job with ptile:

    bjobs
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
119     userA   RUN   normal     hostA       8*host1     myjob      Nov 29 14:29
                                             8*host2
                                             8*host3
                                             8*host4

    Tasks are distributed with exactly 8 tasks per host, except the last host might have less tasks than the ptile value if there are fewer total tasks to distribute.

  • Submit a job based on block size:
    bsub -n 32 -R "span[block=8]" myjob

    View the task distribution of a job based on block size:

    bjobs
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
120     userA   RUN   normal     hostA       32*host1    myjob      Nov 29 14:32

    Tasks are packed in blocks of 8.

  • Submit a job to a single host:
    bsub -n 32 -R "span[hosts=1]" myjob

    View the task distribution of a job that is submitted to a single host:

    bjobs
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
121     userA   RUN   normal     hostA       32*host1    myjob      Nov 29 14:34

    Tasks are placed into a single host (host1).

Resizable jobs

For resource requirements with span[hosts=1], a resize request is limited to slots on the first-execution host of the job. This behavior eliminates the ambiguities that arise when the span expression is modified from the time that the job was originally dispatched.

For span[ptile=n], the job will be allocated exactly n slots on some number of hosts, and a number between 1 and n slots (inclusive) on one host. This is true even if a range of slots is requested. For example, for the following job submission:
bsub -n "1,20" -R "span[ptile=2]" sleep 10000

This special span behavior does not only apply to resize requests. It applies to resizable jobs only when the original allocation is made, and in making additional resize allocations.

If every host has only a single slot available, the job is allocated one slot.

Resize requests with partially filled hosts are handled so that LSF does not choose any slots on hosts already occupied by the job. For example, it is common to use the ptile feature with span[ptile=1] to schedule exclusive jobs.

For a resizable job (auto-resizable or otherwise) with a range of slots requested and span[ptile=n], whenever the job is allocated slots, it will receive either of the following:
  • The maximum number of slots requested, comprising n slots on each of a number of hosts, and between 0 and n-1 (inclusive) slots on one host
  • n slots on each of a number of hosts, summing to some value less than the maximum
For example, if a job requests between 1 and 14 additional slots, and span[ptile=4] is part of the job resource requirement string, when additional slots are allocated to the job, the job receives either of the following:
  • 14 slots, with 2 slots on one host and 4 slots on each of 3 hosts
  • 4, 8 or 12 slots, such that 4 slots are allocated per host of the allocation
Note: Resizable jobs cannot have compound resource requirements.

Example

When running a parallel exclusive job, it is often desirable to specify span[ptile=1] so that the job is allocated at most one slot on each host. For an autoresizable job, new slots are allocated on hosts not already used by the job. The following job submission specifies this resource request:
bsub -x -ar -app <appplicaion_file> -n "1,100" -R "span[ptile=1]" myjob

When additional slots are allocated to a running job, the slots will be on new hosts, not already occupied by the job.

Block scheduling

For applications that are not especially sensitive to network latency, or where you prefer to get throughput, you can allocate slots for a parallel job with a specific block size. The applications specified by the job may be running as threaded processes on groups of n cores, but using MPI applications or other socket connections between blocks. LSF will allocate slots to the job based on block size. LSF tries to pack as many blocks on one host as possible, then goes to next one. Each host is only checked once. It does not matter which host contains the slot blocks. The job can start as soon as any previous job is complete.

In the illustration below, for example, each color represents a different job. There are four 16 way jobs:

For bsub -n 16 and block=4, only 4 x 4 slot blocks are necessary. It does not matter which host contains the slot blocks. The job can start as soon as any previous job is complete.

This packing policy is supported by the keyword block (“span[block=value]”) in the span section of the resource requirement string. “span[block=value]” can also be configured in the RES_REQ parameter in lsb.queues and lsb.applications.

When a block size is specified for a job, LSF allocates only a multiple of the block size for the job. For example, for jobs with block size = 4:

  • bsub -n 2,13: 4, 8 or 12 slots are allocated to the job (in blocks of size 4).
  • bsub -n 5: The job is rejected.
  • bsub -n 9,10: The job is rejected.
  • bsub -n 2,3: The job is rejected.
  • bsub -n 12: The job is accept, and allocates 3 blocks of size 4.
  • bsub -n 2: The job is rejected.
  • bsub -n 3: The job is rejected.

The minimum value in -n min,max is silently changed to a multiple of the block. For example:

bsub -n 2,8 -R span[block=4] sleep 1d

is changed to:

bsub -n 4,8 -R span[block=4] sleep 1d

LSF tries to pack as many blocks in to one host as possible, then goes to the next host. For example, assume host1 has 8 slots, and host2 has 8 slots, and host3 also has 8 slots, where 2 slots of each host are consumed by other jobs. For a job with -n 9 "span[block=3]”, the allocation will be:

  • host1: 6 slots
  • host2: 3 slots

The following is an example of how you can display hosts with their static and dynamic resource information, specify a block size and resource requirements for a job, and see the output:

bhosts
HOST_NAME  STATUS   JL/U   MAX  NJOBS   RUN  SSUSP  USUSP    RSV 
hostA      ok        -     8      0     0      0      0      0
hostB      ok        -     8      0     0      0      0      0
hostC      ok        -     8      0     0      0      0      0
hostD      unavail   -     1      0     0      0      0      0
hostE      ok        -     4      0     0      0      0      0
hostF      ok        -     4      0     0      0      0      0
 
bsub -n 24 -R "order[slots] span[block=4]" sleep 1d
Job <418> is submitted to default queue <normal>.
 
bjobs
JOBID  USER  STAT  QUEUE  FROM_HOST  EXEC_HOST  JOB_NAME  SUBMIT_TIME
418    user1 RUN   normal hostE      8*hostC    sleep 1d  Sep  4 21:36
                                     8*hostB    sleep 1d  Sep  4 21:36
                                     8*hostA    sleep 1d  Sep  4 21:36
 
bhosts
HOST_NAME  STATUS    JL/U   MAX  NJOBS   RUN  SSUSP  USUSP   RSV
hostA      closed     -     8      8     8      0      0     0
hostB      closed     -     8      8     8      0      0     0
hostC      closed     -     8      8     8      0      0     0
hostD      unavail    -     1      0     0      0      0     0
hostE      ok         -     4      0     0      0      0     0
hostF      ok         -     4      0     0      0      0     0

The following are some additional examples of how you can use “span[block=value]” when submitting a job with resource requirements:

  • To specify a predefined block value, per host type or host model, using !:

    bsub -n "2,10" –R "span[block='!'] same[type]" myjob

  • To specify a predefined block value with optional multiple block values, per host type or host model:

    bsub -n "2,10" –R “span[block='!',HP:8,SGI:8,LINUX:2] same[type]" myjob

    If the host type/model does not define MXJ, the default predefined block value is 1.

“span[block=value]” can be displayed by bjobs -l, bhist -l, bqueues -l, bapp -l and bacct -l.

When using the block scheduling feature, note the following:

  • For Queue Host Limit (HOSTLIMIT_PER_JOB), mbatchd will not reject a job with block=x because the exact number of allocated hosts can only be obtained during scheduling.
  • “span[block=value]” and “span[ptile=value]” cannot be specified at the same time. “span[block=value]” and “span[host=value]” also cannot be specified at the same time because span cannot accept more than one criteria and multiple -R does not support multiple span definitions.
  • For the LSF multicluster capability, when using the job forwarding model, job with block=x cannot be forwarded to a remote cluster which has a version prior to 9.1.2. When using the leasing model, job with block=x cannot be allocated to hosts leased from a remote cluster with a version prior to 9.1.2.