Global fair share dynamic user priority

When a local fair share policy schedules jobs, the dynamic user priority calculation also considers resource consumption of users in remote clusters. This prevents fair share from accruing locally to each cluster and ensures fair usage across clusters.

For a global fair share participating queue, remote fair share load will be a factor in dynamic user priority for the share account in the queue. If a share account uses many resources on other clusters, its dynamic user priority will be lower compared with its competitors that use fewer resources on other clusters. Dynamic user priority is calculated by global resource usage to achieve scheduling fairness among clusters.

How LSF calculates dynamic priority for global fair share

For a global fair share participating queue, the formula for calculating dynamic user priority of a share account is as follows:
dynamic priority = number_shares /
(cpu_time * CPU_TIME_FACTOR
+ (historical_run_time + run_time) * RUN_TIME_FACTOR
+ (committed_run_time - run_time) * COMMITTED_RUN_TIME_FACTOR
+ (1 + job_slots) * RUN_JOB_FACTOR
+ (1 + fwd_job_slots) * FWD_JOB_FACTOR
+ fair share_adjustment * fair share_ADJUSTMENT_FACTOR
+ remote_fair share_load)
+ ((historical_gpu_run_time + gpu_run_time) * ngpus_physical) * GPU_RUN_TIME_FACTOR
where:
remote_fair share_load = cpu_time_remote * CPU_TIME_FACTOR
+ (historical_run_time_remote + run_time_remote) * RUN_TIME_FACTOR
+ (committed_run_time_remote - run_time_remote) * COMMITTED_RUN_TIME_FACTOR
+ job_slots_remote * RUN_JOB_FACTOR
+ (1 + fwd_job_slots_remote) * FWD_JOB_FACTOR
+ fair share_adjustment_remote * fair share_ADJUSTMENT_FACTOR
+ ((historical_gpu_run_time_remote + gpu_run_time_remote) * ngpus_physical) * GPU_RUN_TIME_FACTOR

Whether or not ENABLE_HIST_RUN_TIME is set for a global fair share queue, the historical run time for share accounts in the global fair share queue is reported to GPD. When GPD receives historical run time from one cluster, it broadcasts the historical run time to other clusters. The local configuration determines whether the remote historical run time received from GPD is used in the calculation for fair share scheduling priority for the queue. That is, if ENABLE_HIST_RUN_TIME is set in the local cluster, the remote historical run time is used in the calculation for fair share scheduling priority for the queue.

As with local fair share, you can give additional weight to the various factors in the priority calculation by setting the following parameters for the queue in lsb.queues or for the cluster in lsb.params:

  • CPU_TIME_FACTOR
  • RUN_TIME_FACTOR
  • ENABLE_HIST_RUN_TIME
  • COMMITTED_RUN_TIME_FACTOR
  • NUM_RESERVE_JOBS
  • NUM_START_JOBS
  • SHARE_ADJUSTMENT

When the queue value is not defined, the cluster-wide value from lsb.params is used.