bmod

Modifies job submission options of a job.

Synopsis

bmod [bsub_options] [job_ID | "job_ID[index]"]

bmod [-g job_group_name | -gn] [job_ID]

bmod [-hostfile file_path | -hostfilen]

bmod [-sla service_class_name | -slan] [job_ID]

bmod [-aps "system=value" | "admin=value" | -apsn] [job_ID]

bmod [-h | -V]

Option list

[-ar | -arn]

[-B | -Bn]

[-hl | -hln]

[-N | -Ne | -Nn]

[-r | -rn]

[-ul | -uln]

[-x | -xn]

[-a "esub_application [([argument[,argument...]])]..."]

[-app application_profile_name | -appn]

[-aps "system=value" | "admin=value" | -apsn]

[-b begin_time | -bn]

[-C core_limit | -Cn]

[-c [hour:]minute[/host_name | /host_model] | -cn]

[-clusters "all [~cluster_name] ... | cluster_name[+[pref_level]] ... [others[+[pref_level]]]" | -clustern]

[-cwd "current_working_directory" | -cwdn]

[-D data_limit | -Dn]

[-data "[host_name:]abs_file_path [[host_name:]abs_file_path …]" [-datachk] … | -data "tag:tag_name [tag:tag_name ...]" ... | -datan]

[-E "pre_exec_command [argument ...]" | -En]

[-Ep "post_exec_command [argument ...]" | -Epn]

[-e err_file | -en]

[-eptl [hour:]minute | -eptln]

[-eo err_file | -en]

[-ext[sched] "external_scheduler_options"]

[-F file_limit | -Fn]

[-f "local_file op [remote_file]" ... | -fn]

[-freq numberUnit | -freqn]

[-G user_group | -Gn]

[-g job_group_name | -gn]

[-i input_file | -in | -is input_file | -isn]

[-J job_name | -J "%job_limit" | -Jn]

[-Jd "job_description" | -Jdn]

[-jobaff "[# | ! | ~]attribute_name ..." | "attribute([# | ! | ~]attribute_name ...)" | "[# | ! | ~]samehost(job_id)" | "[# | ! | ~]samecu(job_id)" | -jobaffn]

[-k "checkpoint_dir [init=initial_checkpoint_period] [checkpoint_period]" | -kn]

[-L login_shell | -Ln]

[-Lp ls_project_name | -Lpn]

[-M mem_limit[!] | -Mn]

[-mig migration_threshold | -mign]

[-n min_tasks[, max_tasks] | -nn]

[-network network_res_req | -networkn]

[-notify "[exit ] [done ] [start ] [suspend]" | -notifyn]

[-o out_file | -on]

[-oo out_file | -oon]

[-outdir output_directory | -outdirn]

[-P project_name | -Pn]

[-p process_limit | -pn]

[-ptl [hour:]minute | -ptln]

[-Q "[exit_code ...] [EXCLUDE(exit_code ...)]"]

[-q "queue_name ..." | -qn]

[-R "res_req" [-R "res_req" ...] | -Rn]

[-rcacct "rc_account_name" | -rcacctn]

[-rnc resize_notification_cmd | -rncn]

[-S stack_limit | -Sn]

[-s signal | -sn]

[-sla service_class_name | -slan]

[-smt smt_value | -smtn]

[-sp priority | -spn]

[-stage " [storage= min_size [ , max_size ] ] [:in=path_to_stage_in_script ] [:out=path_to_stage_out_script ] " | -stagen]

[-T thread_limit | -Tn]

[-t term_time | -tn]

[-ti | -tin]

[-U reservation_ID |-Un]

[-u mail_user | -un]

[-v swap_limit | -vn]

[-W [hour:]minute[/host_name | /host_model] | -Wn]

[-We [hour:]minute[/host_name | /host_model] | -Wep [value] | -We+ [hour:]minute | -Wen]

[-w "dependency_expression" | -wn]

[-wa "[signal | command | CHKPNT]" | -wan]

[-wt "job_warning_time" | -wtn]

[-Z "new_command" | -Zs "new_command" | -Zsn]

[job_ID | "job_ID[index]"]

Description

Modifies the options of a previously submitted job, including forwarded jobs in an LSF multicluster capability environment. See the bsub command reference for complete descriptions of job submission options you can modify with bmod.

Only the owner of the job, the user group administrator (for jobs that are associated with a user group), or LSF administrators can modify the options of a job.

All options that are specified at submission time can be changed. The value for each option can be overridden with a new value by specifying the option as in bsub. To cancel an option or reset it to its default value, use the option string followed by "n". Do not specify an option value when you reset an option.

The -i, -in, and -Z options have counterparts that support spooling of input and job command files (-is, -isn, -Zs, and -Zsn). Options that are related to file names and job spooling directories support paths that contain up to 4094 characters for UNIX, or up to 255 characters for Windows.

Options that are related to command names can contain up to 4094 characters for UNIX, or up to 255 characters for Windows. Options that are related to job names can contain up to 4094 characters.

You can modify all options of a pending job, even if the corresponding bsub command option was not specified.

Modifying options for a forwarded pending job are different from modifying the options of a pending job:

You cannot modify the following options: -m, -q, -sla, -w.
For the following options, your modifications take effect only on the submission cluster and not on the execution cluster: -aps, -g, -k.
For the -J option, if you are changing only the job array limit, the option takes effect only on the submission cluster and not on the execution cluster.
When you apply a cancellation option (such as the -appn option to cancel a previous -app specification), the default value for that attribute depends on the local cluster configuration.

Like bsub, the bmod command calls the parent esub (mesub). The parent esub runs any mandatory esub executable files that are configured by an LSF administrator, and any executable file named esub (without .application in the file name) if it exists in the LSF_SERVERDIR directory. Only esub executable files that are invoked by the bsub command can change the job environment on the submission host. An esub invoked by the bmod command cannot change the job environment. Arguments for esub executable files can also be modified.

The -b option modifies the job begin time. If the year field is specified and the specified time is in the past, the start time condition is considered reached and LSF dispatches the job if slots are available.

The -t option modifies job termination time. If the year field is specified and the specified time is in the past, the job modification request is rejected.

Use the -clusters option to modify cluster names for LSF multicluster capability jobs. The bmod -clusters remote_clusters command can modify pending jobs only on the submission cluster.

The -cwd option specifies the current working directory (CWD) for a job. The -cwd option works only if the job is in pending state. The path can be absolute or relative to the submission directory. If the submission directory does not exist in the execution host, it tries the logical home directory. If that fails, the tmp directory is used for the CWD.

The path can include the same dynamic patterns as described for the bsub -cwd command.

The -cwdn option resets the value for the -cwd option to the submission directory from the bsub command.

If the job is submitted with the -app option but without the -cwd option, and the LSB_JOB_CWD parameter is not defined, the CWD defined in the specified application profile is used. If CWD is not defined in the application profile, the DEFAULT_JOB_CWD value is used. If neither of the two parameters is defined, the submission directory is used for CWD.

The -hl option enables per-job host-based memory and swap limit enforcement on hosts that support Linux cgroups. The -hln option disables host-based memory and swap limit enforcement. The -hl and -hln options apply only to pending jobs. The LSB_RESOURCE_ENFORCE="memory" parameter must be specified in the lsf.conf file for host-based memory and swap limit enforcement with the -hl option to take effect. If no memory or swap limit is specified for the job, or the LSB_RESOURCE_ENFORCE="memory" parameter is not specified, a host-based memory limit is not set for the job.

The -Epn option cancels the setting of job-level post-execution commands. The job-level post-execution commands do not run. Application-level post-execution commands run if they exist.

If a default user group is configured (with the DEFAULT_USER_GROUP parameter in the lsb.params file), the bmod -Gn command moves the job to the default user group. If the job is already attached to the default user group, the bmod -Gn command has no effect on that job. A job moved to a user group where it cannot run (without shares in a specified fair share queue, for example) is transferred to the default user group where the job can run.

The -gpu option specifies properties of GPU resources required by the job. For more information on this option see the bsub -gpu command.

For resizable jobs, the bmod -R "rusage[mem | swp]" command affects the resize allocation request only if the job is not dispatched.

The -M option takes effect only if the job can requeue and when it is run again.

The -m option modifies the first execution host list. When used with a compound resource requirement, the first host that is allocated must satisfy the first simple resource requirement string in the compound resource requirement.

When modifying the notification request using the -notify option, the changed record is also logged to the JOB_MODIFY2 event. The -notify option does not work with job arrays or job array elements.

The -notifyn option does not work with job array elements.

The -outdir option creates the output directory while the job is in pending state. This option supports the dynamic patterns for the output directory. For example, if user1 runs the command bmod -outdir "/scratch/joboutdir/%U/%J_%I" myjob, the system creates the directory /scratch/joboutdir /user1/jobid_0 for the job output directory.

The -outdirn option resets the output directory value to the DEFAULT_JOB_OUTDIR parameter, if it is defined, or sets the output directory to the submission directory where the original bsub command ran. The output directory can be modified only while the job is in pending state.

The -Q option does not affect running jobs. For rerunnable and requeue jobs, the -Q option affects the next run.

The -q option resubmits the job to a new queue, as if it was a new submission. By default, LSF dispatches jobs in a queue in order of arrival, so the modified job goes to the last position of the new queue, no matter what its position was in the original queue.

The -rn option resets the rerunnable job setting that is specified by the bsub -rn or bsub -r option. The application profile and queue level rerunnable job setting if any is used. The bmod -rn command does not disable or override job rerun if the job was submitted to a rerunnable queue or application profile with job rerun configured. bmod -rn command is different from the bsub -rn command, which does override the application profile and queue level rerunnable job setting.

The -ti option enables immediate automatic orphan job termination at the job level. The -ti and -tin options are command flags, not suboptions, so you do not need to respecify the original dependency expression from the -w option that is submitted with the bsub command. You can use either the bmod -w [-ti | -tin] or bmod -ti | -tin command. The -tin option cancels the -ti suboption of a submitted dependent job, in which case the cluster-level configuration takes precedence.

The -uln option sets the user shell limits for pending jobs to their default values. The -uln option is not supported on Windows.

-Wen cancels the estimated job run time. The runtime estimate does not take effect for the job.

Modifying running jobs

By default, you can modify resource requirements for running jobs (-R "res_req" except -R "cu[cu_string]") and the estimated running time for running or suspended jobs (-We, -We+, -Wep). To modify more job options for running jobs, define LSB_MOD_ALL_JOBS=Y in lsf.conf.

Note: Using the bmod -R command with a running job that uses a compound or alternative resource requirement as the effective resource requirement is not permitted. Changing the compound or alternative resource requirement of a running job is also rejected.

When the LSB_MOD_ALL_JOBS=Y parameter is set, only some bsub options can be modified for running jobs. You cannot make any other modifications after a job is dispatched. You can use bmod to modify the following options for running jobs:

CPU limit (-c [hour:]minute[/host_name | /host_model])
Memory limit (-M mem_limit)
Rerunnable jobs (-r | -rn)
Resource requirements (-R "res_req" except -R "cu[cu_string]")
Run limit (-W run_limit[/host_name | /host_model])

Note: You can modify the run limit for pending jobs as well.
Swap limit (-v swap_limit)
Standard output (stdout) file name up to 4094 characters for UNIX and Linux or 255 characters for Windows (-o output_file)
Standard error (stderr) file name up to 4094 characters for UNIX and Linux or 255 characters for Windows (-e error_file)
Overwrite standard output (stdout) file name up to 4094 characters for UNIX and Linux or 255 characters for Windows (-oo output_file)
Overwrite standard error (stderr) file name up to 4094 characters for UNIX and Linux or 255 characters for Windows (-eo error_file)

For remote running jobs, you can modify only the following attributes:

CPU limit ([-c cpu_limit[/host_spec] | -cn])
Memory limit ([-M mem_limit | -Mn])
Rerunnable attribute ([-r | -rn])
Run limit ([-W [hour:]minute[/host_name | /host_model] | -Wn])
Swap limit ([-v swap_limit | -vn])
Standard output/error ([-o out_file | -on] [-oo out_file | -oon] [-e err_file | -en][-eo err_file | -en])

Modified resource usage limits cannot exceed limits that are defined in the queue.

To modify the CPU limit or the memory limit of running jobs, the parameters LSB_JOB_CPULIMIT=Y and LSB_JOB_MEMLIMIT=Y must be defined in the lsf.conf file.

By default, options for the following resource usage limits are specified in KB:

Core limit (-C)
Memory limit (-M)
Stack limit (-S)
Swap limit (-v)

Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a different unit for the limit (MB, GB, TB, PB, EB, or ZB).

Modifying resource requirements

The -R option of bmod completely replaces any previous resource requirement specification. It does not add the modification to the existing specification. For example, if you submit a job with

bsub -R "rusage[res1=1]"

Then modify it with

bmod -R "rusage[res2=1]"

The new resource usage requirement for the job is [res2=1], not [res1=1; res2=1].

bmod does not support the OR (||) operator on the -R option.

bmod does not support multiple -R option strings for multi-phase rusage resource requirements.

Modified rusage consumable resource requirements for pending jobs must satisfy any limits set by the parameter RESRSV_LIMIT in the lsb.queues file. For running jobs, the maximums set by the RESRSV_LIMIT parameter must be satisfied but the modified rusage values can be lower than the minimum values.

Changes to multi-phase rusage strings on running jobs such as bmod -R "rusage[mem=(mem1 mem2):duration=(dur1 dur2)]" take effect immediately, and change the remainder of the current phase.

For example, a job is submitted with the following resource requirements:

bsub -R "rusage[mem=(500 300 200):duration=(20 30):decay=(1 0)]" myjob

And after 15 minutes of run time, the following modification is issued:

bmod -R "rusage[mem=(400 300):duration=(20 10):decay=(1 0)]" job_ID

The result is the following rusage string:

rusage[mem=(400 300):duration=(20 10):decay=(1 0)]

The running job will reserve (400-((400-300)*15/20)))=325 MB memory with decay for the next (20-15)=5 minutes of run time. The second phase then starts, reserving 300 MB of memory for the next 10 minutes with no decay, and end up with no memory that is reserved for the rest of the run time.

After 25 minutes of run time, another modification is issued:

bmod -R "rusage[mem=(200 100):duration=(20 10):decay=(1 0)]" job_ID

The job reserves 100 MB of memory with no decay for the next 5 minutes of runtime, followed by no reserved memory for the remainder of the job.

To remove all of the string input that is specified by using the bsub command, use the -Rn option.

For started jobs, bmod -R modifies the effective resource requirements for the job, along with the job resource usage. The effective resource requirement string for scheduled jobs represents the resource requirement that is used by the scheduler to make a dispatch decision. bmod -R updates the rusage part in the resource requirements, and keeps the other sections as they were in the original resource requirements. The rusage always represents the job runtime allocation, and is modified along with the job resource usage. For running jobs, you cannot change resource requirements to any of the following resource requirements:

Compound
Alternative
rusage siblings
Compound to simple

Modifying the estimated run time of jobs

The following options modify the estimated run time of a job:

-We [hour:]minute[/host_name | /host_model]

Sets an estimated run time. Specifying a host or host model normalizes the time with the CPU factor (time/CPU factor) of the host or model.

-We+ [hour:]minute]

Sets an estimated run time that is the value you specify added to the accumulated run time. For example, if you specify -We+ 30 and the job runs for 60 minutes, the new estimated run time is now 90 minutes.

Specifying a host or host model normalizes the time with the CPU factor (time/CPU factor) of the host or model.

-Wep [value]

Sets an estimated run time that is the percentage of job completion that you specify added to the accumulated run time. For example, if you specify -Wep+ 25 (meaning that the job is 25% complete) and the job runs for 60 minutes, the new estimated run time is now 240 minutes.

The range of valid values is greater than 0 and less than or equal to 100. Two digits after decimal are supported.

Specifying a host or host model normalizes the time with the CPU factor of the host or model (time/CPU factor).

Modifying job groups

Use the -g option of bmod and specify a job group path to move a job or a job array from one job group to another. For example, the following command moves job 105 to job group /risk_group/portfolio2/monthly:

bmod -g /risk_group/portfolio2/monthly 105

Like bsub -g, if the job group does not exist, LSF creates it.

The bmod -g option cannot be combined with other bmod options. It can operate only on pending jobs. It cannot operate on running or finished jobs.

You can modify your own job groups and job groups that other users create under your job groups. LSF administrators can modify job groups of all users.

You cannot move job array elements from one job group to another, only entire job arrays. If any job array elements in a job array are running, you cannot move the job array to another group. A job array can belong only to one job group at a time.

You cannot modify the job group of a job that is attached to a service class. Job groups cannot be used with resource-based SLAs that have guarantee goals.

If you want to specify array dependency by array name, set the JOB_DEP_LAST_SUB parameter in the lsb.params file. If this parameter is not set, the job is rejected if one of your previous arrays has the same name but a different index.

Modifying jobs in service classes

The -sla option modifies a job by attaching it to the specified service class. The -slan option detaches the specified job from a service class. If the service class does not exist, the job is not modified. For example, the following command attaches job 2307 to the service class Duncan:

bmod -sla Duncan 2307

The following command detaches job 2307 from the service class Duncan. If a default SLA is configured in lsb.params, the job is moved to the default service class.

bmod -slan 2307

You cannot do the following job modifications:

Use the -sla option with other bmod options
Modify the service class of job that is already attached to a job group. (Time-based SLAs only.) Use the bsla command to display the configuration properties of service classes that are configured in the lsb.serviceclasses file, and dynamic information about the state of each service class.
Modify a job such that it no longer satisfies the assigned guarantee SLA. Jobs auto-attached to guarantee SLAs reattach to another SLA as required, but jobs that are submitted with an SLA specified must continue to satisfy the SLA access restrictions.

If a default SLA is configured with the ENABLE_EGO_DEFAULT_SLA parameter in the lsb.params file, the bmod -slan option moves the job to the default SLA. If the job is already attached to the default SLA, the bmod -slan option has no effect on that job.

Modifying jobs associated with application profiles

The -app option modifies a job by associating it to the specified application profile. The -appn option dissociates the specified job from its application profile. If the application profile does not exist, the job is not modified.

You can modify the application profile only for pending jobs. For example, the following command associates job 2308 with the application profile fluent:

bmod -app fluent 2308

The following command dissociates job 2308 from the service class fluent:

bmod -appn 2308

Use bapp to display the properties of application profiles that are configured in LSB_CONFDIR/cluster_name/configdir/lsb.applications.

Modifying absolute priority scheduling options

Administrators can use bmod -aps to adjust the APS value for pending jobs. bmod -apsn cancels previous bmod -aps settings. You cannot combine bmod -aps with other bmod options.

You can change the APS value only for pending resizable jobs.

-aps "system=value" job_ID: Set a static nonzero APS value of a pending job. Setting a system APS value overrides any calculated APS value for the job. The system APS value cannot be applied to running jobs.
-aps "admin=value" job_ID: Set a nonzero ADMIN factor value for a pending job. The ADMIN factor adjusts the calculated APS value higher or lower. A negative admin value lowers the calculated APS value, and a positive value raises the calculated APS value relative to other pending jobs in the APS queue.
You cannot configure APS weight, limit, or grace period for the ADMIN factor. The ADMIN factor takes effect as soon as it is set.
-apsn: Use the bmod -apsn option to cancel previous bmod -aps settings. You cannot apply the bmod -apsn option to running jobs in an APS queue. An error is issued if the job has no system APS priority or ADMIN factor set.

Modifying resizable jobs

Use the -rnc and -ar options to modify the autoresizable attribute or resize notification command for resizable jobs. You can modify the autoresizable attribute only for pending jobs (PSUSP or PEND). You can modify the resize notification command only for unfinished jobs (not DONE or EXIT jobs).

-rnc resize_notification_cmd: Specify the name of an executable file to be invoked on the first execution host when the job allocation is modified (both shrink and grow). The bmod -rnc option overrides any notification command that is specified in the application profile.
-rncn: Remove the notification command setting from the job.
-ar: Specify that the job is autoresizable.
-arn: Remove job-level autoresizable attribute from the job.

Modifying network scheduling options for PE jobs

The -network option modifies the network scheduling options for IBM Parallel Environment (PE) jobs. The -networkn option removes any network scheduling options for the PE job.

You cannot modify the network scheduling options for running jobs, even if LSB_MOD_ALL_JOBS=y.

Modifying memory rusage for affinity jobs

When you use the bmod command to modify memory rusage of a running job with an affinity resource request, host-level available memory and available memory in NUMA nodes might be inconsistent when you use bhosts -l -a. Inconsistencies happen because the modified resource requirement takes effect in the next scheduling cycle for affinity scheduling, but it takes effect immediately at the host level. The bmod command updates only resource usage that LSF accounts; it has no effect on the running jobs. For memory binding, when a process is bound to some NUMA node, LSF limits which NUMA node the process gets physical memory from. LSF does not ask the operating system to reserve any physical memory for the process.

Modifying jobs with a user-specified host file

Use the -hostfile option to modify a pending job with a user-specified host file.

bmod -hostfile "host_alloc_file" ./a.out <job_id>

A user-specified host file contains specific hosts and slots that a user wants to use for a job. For example, if you know what the best host allocation for a job is based on factors such as network connection status, you can choose to submit a job with a user specified host file. The user specified host file specifies the order in which to launch tasks, ranking the slots specified in the file. The resulting rank file is also made available to other applications (such as MPI).

Important:

The -hostfile cannot be used with either the –n or –m option.
The -hostfile option cannot be combined with –R or compound res_req.
Do not use a user specified host file if you enabled task geometry because it can cause conflicts and jobs might fail.
If resources are not available at the time that a task is ready, use advance reservation instead of a user-specified host file, to ensure that reserved slots are available and to guarantee that a job runs smoothly.

Any user can create a user specified host file. It must be accessible by the user from the submission host. It lists one host per line in the following format:

# This is a user specified host file
<host_name1>   [<# slots>]
<host_name2>   [<# slots>]
<host_name1>   [<# slots>]
<host_name2>   [<# slots>]
<host_name3>   [<# slots>]
<host_name4>   [<# slots>]

The following rules apply to the user specified host file:

Specifying the number of slots for a host is optional. If no slot number is indicated, the default is 1.
A host name can be either a host in a local cluster or a host leased-in from a remote cluster (host_name@cluster_name).
A user specified host file must contain hosts from the same cluster only.
A host name can be entered with or without the domain name.

Host names can be used multiple times, and the order of the hosts represents the placement of tasks.

#first three tasks
host01                      3
#fourth tasks
host02
#next three tasks
host03                      3

Start comments with the # character.

The user specified host file is deleted along with other job-related files when a job is cleaned.

To remove a user-specified host file that is specified for a pending job, use the -hostfilen option:

bmod -hostfilen <job_id>

Modifying job data requirements

The -data option modifies the data staging requirements for a pending job that is submitted with the bsub -data option. If file transfers for previously specified files are still in progress, they are not stopped. Only new transfers are initiated for the new data management requirement as needed. Use -datan to cancel the data staging requirement for the job.

Modifying the data requirements of a job replaces the entire previous data requirement string in the -data option with the new one. When you modify or add a part of the data requirement string, you must specify the entire data requirement string in bmod -data with the modifications.

The sanity check for the existence of files or folders and whether the user can access them, discovery of the size and modification time of the files or folders, and generation of the hash from the bmod command occurs in the transfer job. This equalizes modification performance between jobs with and without data requirements. The -datachk option can perform full checking for jobs with a data requirement. The -datachk option can be specified only with the -data command. If the data requirement is for a tag, this option has no effect.

You must have read access to the specified file to modify the data requirements for the job.

You cannot use bmod -data or bmod -datan to modify the data management requirements of a running or finished job, or for a job that is already forwarded to or from another cluster.

For example, if a job that was originally submitted with the following command:

bsub -data /proj/user1/dataset_A_%I -J A[1-10] myjob.sh

And you modify the job with the following command:

bmod -data /proj/user1/dataset_B_%I "A[1]"

Then all the paths /proj/user1/dataset_B_1, /proj/user1/dataset_B_2 ... /proj/user1/dataset_B_10 must exist. If the bmod command succeeds, then A[1] still has all 10 data files in its data requirement.

Options

job_ID | "job_ID[index]": Modifies jobs with the specified job ID.
Modifies job array elements that are specified by "job_ID[index]".
-h: Prints command usage to stderr and exits.
-V: Prints LSF release version to stderr and exits.

Limitations

If you do not specify -e or -eo before the job is dispatched, you cannot modify the name of job error file for a running job. Modifying the job output options of remote running jobs is not supported.