Usage string

This string defines the expected resource usage of the job. It is used to specify resource reservations for jobs, or for mapping jobs on to hosts and adjusting the load when running interactive jobs.

By default, no resources are reserved.

LSF rejects resource requirement strings where an rusage section contains a non-consumable resource.

Multi-phase resources

Multiple phases within the rusage string allow different time periods to have different memory requirements (load index mem). The duration of all except the last phase must be specified, while decay rates are all optional and are assumed to be 0 if omitted. If the optional final duration is left blank, the final resource requirement applies until the job is finished.

Multi-phase resource reservations cannot include increasing resources, but can specify constant or decreasing resource reservations over multiple periods of time.

Resource reservation limits

Resource requirement reservation limits can be set using the parameter RESRSV_LIMIT in lsb.queues. Queue-level RES_REQ rusage values (set in lsb.queues) must be in the range set by RESRSV_LIMIT, or the queue-level RES_REQ is ignored. Merged RES_REQ rusage values from the job and application levels must be in the range of RESRSV_LIMIT, or the job is rejected.

When both the RES_REQ and RESRSV_LIMIT are set in lsb.queues for a consumable resource, the queue-level RES_REQ no longer acts as a hard limit for the merged RES_REQ rusage values from the job and application levels. In this case only the limits set by RESRSV_LIMIT must be satisfied, and the queue-level RES_REQ acts as a default value.

Batch jobs

The resource usage (rusage) section can be specified at the job level, with the queue configuration parameter RES_REQ, or with the application profile parameter RES_REQ.

Basic syntax

rusage[usage_string [, usage_string][|| usage_string] ...]

where usage_string is:

load_index=value [:load_index=value]... [:duration=minutes[m]
 | :duration=hoursh | :duration=secondss [:decay=0 | :decay=1]]

Note: The default unit for duration is "minutes". To use hours or seconds as the unit, append "h" or "s" to the duration value. For example, duration=30 means 30 minutes, as does duration=30m explicitly. Accordingly, duration=30h means 30 hours, and duration=30s means 30 seconds.

The keyword threshold in the rusage section lets you specify a threshold at which the consumed resource must be before an allocation should be made. If the threshold is not satisfied for every host in the cluster, the job becomes pending.

To specify a threshold in the command line, use bsub -R to attach a threshold to a resource in the rusage section. For example:

bsub -R "rusage[bwidth=1:threshold=5]" sleep 100

You can use bmod -R to change the content of the rusage section. For example:

bmod -R "rusage[bwidth=1:threshold=7]" <job ID>

To specify a threshold in the configuration file, Use RES_REQ to attach a threshold to a resource in lsb.queues. For example:

RES_REQ = rusage[bwidth=1:threshold=5]

You can use RES_REQ to attach a threshold to a resource in lsb.applications. For example:

RES_REQ = rusage[bwidth=1:threshold=5]

Multi-phase memory syntax

rusage[multi_usage_string [, usage_string]...]

where multi_usage_string is:

mem=(v1 [v2 … vn]):[duration=(t1 [t2 … tm])][:decay=(d1 [d2... dk])]

for m = n|n-1. For a single phase (n=1), duration is not required.

if k > m, dm+1 to dk will be ignored; if k < m, dk+1 =.. = dm = 0.

usage_string is the same as the basic syntax, for any load_index other than mem.

Multi-phase syntax can be used with a single phase memory resource requirement as well as for multiple phases.

For multi-phase slot-based resource reservation, use with RESOURCE_RESERVE_PER_TASK=Y in lsb.params.

Multi-phase resource reservations cannot increase over time. A job submission with increasing resource reservations from one phase to the next will be rejected. For example:

bsub -R"rusage[mem=(200 300):duration=(2 3)]" myjob

specifies an increasing memory reservation from 200 MB to 300 MB. This job will be rejected.

Tip: When a multi-phase mem resource requirement is being used, duration can be specified separately for single-phase resources.

Load index

Internal and external load indices are considered in the resource usage string. The resource value represents the initial reserved amount of the resource.

Duration

The duration is the time period within which the specified resources should be reserved. Specify a duration equal to or greater than the ELIM updating interval.

If the value is followed by the letter s, m, or h, the specified time is measured in seconds, minutes, or hours respectively.
By default, duration is specified in minutes.
For example, the following specify a duration of 1 hour for multi-phase syntax:
- duration=(60)
- duration=(1h)
- duration=(3600s)
For example, the following specify a duration of 1 hour for single-phase syntax:
- duration=60
- duration=1h
- duration=3600s
  Tip: Duration is not supported for static shared resources. If the shared resource is defined in an lsb.resources Limit section, then duration is not applied.

Decay

The decay value indicates how the reserved amount should decrease over the duration.

A value of 1 indicates that system should linearly decrease the amount reserved over the duration.
A value of 0 causes the total amount to be reserved for the entire duration.

Values other than 0 or 1 are unsupported, and are taken as the default value of 0. If duration is not specified, decay value is ignored.

Tip: Decay is not supported for static shared resources. If the shared resource is defined in an lsb.resources Limit section, then decay is not applied.

Default

If a resource or its value is not specified, the default is not to reserve that resource. If duration is not specified, the default is to reserve the total amount for the lifetime of the job. (The default decay value is 0.)

Example

rusage[mem=50:duration=100:decay=1]

This example indicates that 50 MB memory should be reserved for the job. As the job runs, the amount reserved will decrease at approximately 0.5 MB per minute until the 100 minutes is up.

Resource reservation method

Specify the resource reservation method in the resource usage string by using the /job, /host, or /task keyword after the numeric value. The resource reservation method specified in the resource string overrides the global setting that is specified in the ReservationUsage section of the lsb.resources file. You can only specify resource reservation methods for consumable resources. Specify the resource reservation methods as follows:

value/task
Specifies per-task reservation of the specified resource. This is the equivalent of specifying PER_TASK for the METHOD parameter in the ReservationUsage section of the lsb.resources file.
value/job
Specifies per-job reservation of the specified resource. This is the equivalent of specifying PER_JOB for the METHOD parameter in the ReservationUsage section of the lsb.resources file.
value/host
Specifies per-host reservation of the specified resource. This is the equivalent of specifying PER_HOST for the METHOD parameter in the ReservationUsage section of the lsb.resources file.

Basic syntax:

resource_name=value/method:duration=value:decay=value

For example,

rusage[mem=10/host:duration=10:decay=0]

Multi-phase memory syntax:

resource_name=(value ...)/method:duration=(value ...):decay=value

For example,

rusage[mem=(50 20)/task:duration=(10 5):decay=0]

How simple queue-level and job-level rusage sections are resolved

Job-level rusage overrides the queue level specification:

For internal load indices (r15s, r1m, r15m, ut, pg, io, ls, it, tmp, swp, and mem), the job-level value cannot be larger than the queue-level value (unless the limit parameter RESRSV_LIMIT is being used as a maximum instead of the queue-level value).
For external load indices, the job-level rusage can be larger than the queue-level requirements.
For duration, the job-level value of internal and external load indices cannot be larger than the queue-level value.
For multi-phase simple rusage sections:
- For internal load indices (r15s, r1m, r15m, ut, pg, io, ls, it, tmp, swp, and mem), the first phase of the job-level value cannot be larger than the first phase of the queue-level value (unless the limit parameter RESRSV_LIMIT is being used as a maximum instead of the queue-level value).
- For duration and decay, if either job-level or queue-level is multi-phase, the job-level value will take precedence.

How simple queue-level and job-level rusage sections are merged

When both job-level and queue-level rusage sections are defined, the rusage section defined for the job overrides the rusage section defined in the queue. The two rusage definitions are merged, with the job-level rusage taking precedence. For example:

Example 1

Given a RES_REQ definition in a queue:

RES_REQ = rusage[mem=200:lic=1] ...

and job submission:

bsub -R "rusage[mem=100]" ...

The resulting requirement for the job is

rusage[mem=100:lic=1]

where mem=100 specified by the job overrides mem=200 specified by the queue. However, lic=1 from queue is kept, since job does not specify it.

Example 2

For the following queue-level RES_REQ (decay and duration defined):

RES_REQ = rusage[mem=200:duration=20:decay=1] ...

and job submission (no decay or duration):

bsub -R "rusage[mem=100]" ...

The resulting requirement for the job is:

rusage[mem=100:duration=20:decay=1]

Queue-level duration and decay are merged with the job-level specification, and mem=100 for the job overrides mem=200 specified by the queue. However, duration=20 and decay=1 from queue are kept, since job does not specify them.

rusage in application profiles

See Resource requirements for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.

How simple queue-level rusage sections are merged with compound rusage sections

When simple queue-level and compound application-level or job-level rusage sections are defined, the two rusage definitions are merged. If a job-level resource requirement (simple or compound) is defined, the application level is ignored and the job-level and queue-level sections merge. If no job-level resource requirement is defined, the application-level and queue-level merge.

When a compound resource requirement merges with a simple resource requirement from the queue-level, the behavior depends on whether the queue-level requirements are job-based or not.

Example 1

Job-based simple queue-level requirements apply to the first term of the merged compound requirements. For example:

Given a RES_REQ definition for a queue which refers to a job-based resource:

RES_REQ = rusage[lic=1] ...

and job submission resource requirement:

bsub -R "2*{rusage[mem=100] ...} + 4*{[mem=200:duration=20:decay=1] ...}"

The resulting requirement for the job is

bsub -R "2*{rusage[mem=100:lic=1] ...} + 4*{rusage[mem=200:duration=20:decay=1] ...}"

The job-based resource lic=1 from queue is added to the first term only, since it is job-based and wasn’t included the job-level requirement.

Example 2

Host-based or slot-based simple queue-level requirements apply to all terms of the merged compound requirements. For example:

For the following queue-level RES_REQ which does not include job-based resources:

RES_REQ = rusage[mem=200:duration=20:decay=1] ...

and job submission:

bsub -R "2*{rusage[mem=100] ...} + 4*{rusage[lic=1] ...}"

The resulting requirement for the job is:

2*{rusage[mem=100:duration=20:decay=1] ...} + 4*{rusage[lic=1:mem=200:duration=20:decay=1] ...}

Where duration=20 and decay=1 from queue are kept, since job does not specify them in any term. In the first term mem=100 from the job is kept; in the second term mem=200 from the queue is used since it wasn’t specified by the job resource requirement.

Specify multiple -R options

bsub accepts multiple -R options for the rusage section.

Restriction: Compound resource requirements do not support multiple -R options. Multi-phase rusage strings do not support multiple -R options.

You can specify multiple resource requirement strings instead of using the && operator. For example:

bsub -R "rusage[mem=100]" -R "rusage[tmp=50:duration=60]"

LSF merges the multiple -R options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.

The number of -R option sections is unlimited.

Comma-separated multiple resource requirements within one rusage string is supported. For example:

bsub -R "rusage[mem=20]"-R "rusage[mem=10||mem=10]" myjob

A given load index cannot appear more than once in the resource usage string.

Specify alternative usage strings

If you use more than one version of an application, you can specify the version you prefer to use together with a legacy version you can use if the preferred version is not available. Use the OR (||) expression to separate the different usage strings that define your alternative resources.

Job-level resource requirement specifications that use the || operator are merged with other rusage requirements defined at the application and queue levels.

Note: Alternative rusage strings cannot be submitted with compound resource requirements.

How LSF merges rusage strings that contain the || operator

The following examples show how LSF merges job-level and queue-level rusage strings that contain the || operator.

Queue level RES_REQ=rusage…	Job level bsub -R "rusage …	Resulting rusage string
[mem=200:duration=180]	`[w1=1 \|\| w2=1 \|\| w3=1]"`	`[w1=1, mem=200:duration=180 \|\| w2=1, mem=200:duration=180 \|\| w3=1, mem=200:duration=180]`
[w1=1 \|\| w2=1 \|\| w3=1]	`[mem=200:duration=180]"`	`[mem=200:duration=180, w1=1 \|\| mem=200:duration=180, w2=1 \|\| mem=200:duration=180, w3=1]`

Note:

Alternative rusage strings cannot be submitted with compound resource requirements.

Non-batch environments

Resource reservation is only available for batch jobs. If you run jobs using only LSF Base, such as through lsrun, LIM uses resource usage to determine the placement of jobs. Resource usage requests are used to temporarily increase the load so that a host is not overloaded. When LIM makes a placement advice, external load indices are not considered in the resource usage string. In this case, the syntax of the resource usage string is

res[=value]:res[=value]: ... :res[=value]

res is one of the resources whose value is returned by the lsload command.

rusage[r1m=0.5:mem=20:swp=40]

The preceding example indicates that the task is expected to increase the 1-minute run queue length by 0.5, consume 20 MB of memory and 40 MB of swap space.

If no value is specified, the task is assumed to be intensive in using that resource. In this case no more than one task will be assigned to a host regardless of how many CPUs it has.

The default resource usage for a task is r15s=1.0:r1m=1.0:r15m=1.0. This indicates a CPU-intensive task which consumes few other resources.

Resizable jobs

Unlike the other components of a resource requirement string that only pertain to adding additional slots to a running job, rusage resource requirement strings affect the resource usage when slots are removed from the job as well.

When adding or removing slots from a running job:

The amount of slot-based resources added to or removed from the job allocation is proportional to the change in the number of slots
The amount of job-based resources is not affected by a change in the number of slots
The amount of each host-based resource is proportional to the change in the number of hosts

When using multi-phase resource reservation, the job allocation is based on the phase of the resource reservation.

Note: Resizable jobs cannot have compound resource requirements.

Duration and decay of rusage

Duration and decay of resource usage and the || operator affect resource allocation.

Duration or decay of a resource in the rusage expression is ignored when scheduling the job for the additional slots.

If a job has the following rusage string: rusage[mem=100:duration=300], the resize request of one additional slot is scheduled on a host only if there are 100 units of memory available on that host. In this case, mem is a slot-based resource (RESOURCE_RESERVE_PER_TASK=Y in lsb.params).

Once the resize operation is done, if the job has been running less than 300 seconds then additional memory will be reserved only until the job has run for 300 seconds. If the job has been running for more than 300 seconds when the job is resized, no additional memory is reserved. The behavior is similar for decay.

The || operator lets you specify multiple alternative rusage strings, one of which is used when dispatching the job. You cannot use bmod to change rusage to a new one with a || operator after the job has been dispatched

For job resize, when the || operator is used, the resize request uses the rusage expression that was originally used to dispatch the job. If the rusage expression has been modified since the job started, the resize request is scheduled using the new single rusage expression.

Example 1

You want to run an autoresizable job such that every slot occupied by the job reserves 100 MB of swap space. In this case, swp is a slot-based resource (RESOURCE_RESERVE_PER_TASK=Y in lsb.params). Each additional slot that is allocated to the job should reserve additional swap space. The following job submission specifies this resource request:

bsub -ar -app <appplicaion_file> -n "1,100" -R "rusage[swp=100]" myjob

Similarly, if you want to release some of the slots from a running job, resources that are reserved by the job are decreased appropriately. For example, for the following job submission:

bsub -ar -app <appplicaion_file> -n 100 -R "rusage[swp=50]" myjob 
Job <123> is submitted to default queue.

you can run bresize release to release all the slots from the job on one host:

bresize release "hostA" 123

The swap space used by the job is reduced by the number of slots used on hostA times 50 MB.

Example 2

You have a choice between two versions of an application, each version having different memory and swap space requirements on hosts. If you submit an autoresizable job with the || operator, once the job is started using one version of an application, slots added to a job during a resize operation reserve resources depending on which version of the application was originally run. For example, for the following job submission:

bsub -n "1,100" -ar -R "rusage[mem=20:app_lic_v201=1 || mem=20:swp=50:app_lic_v15=1]" myjob

If the job starts with app_lic_v15, each additional slot added in a resize operation reserves 20 MB of memory and 50 MB of swap space.