Moab Administrator's Guide - Resource Usage Limits
Moab Workload Manager®

10.4   Resource Usage Limits

  • 10.4.1 Configuring Actions
  • 10.4.2 Specifying Hard and Soft Policy Violations
  • 10.4.3 Constraining Walltime Usage
   Resource usage limits constrain the amount of resources a given job may consume. These limits are generally proportional to the resources requested and may include walltime, any standard resource, or any specified generic resource. The parameter RESOURCELIMITPOLICY controls which resources are limited, what limit policy is enforced per resource, and what actions the scheduler should take in the event of a policy violation.

10.4.1 Configuring Actions

   The RESOURCELIMITPOLICY parameter accepts a number of policies, resources, and actions using the format and values defined below.

NOTE: If walltime is the resource to be limited, be sure that the resource manager is configured to not interfere if a job surpasses its given walltime. For TORQUE, this is done by using $ignwalltime in the configuration on each mom node.

Format

RESOURCELIMITPOLICY <RESOURCE>:[<SPOLICY>,]<HPOLICY>:[<SACTION>,]<HACTION>[:[<SVIOLATIONTIME>,]<HVIOLATIONTIME>]...

ResourceDescription
CPUTIMEmaximum total job proc-seconds used by any single job (allows scheduler enforcement of cpulimit)
DISKlocal disk space (in MB) used by any single job task
JOBMEMmaximum real memory/RAM (in MB) used by any single job
JOBPROCmaximum processor load associated with any single job
MEMmaximum real memory/RAM (in MB) used by any single job task
MINJOBPROCminimum processor load associated with any single job (action taken if job is using 5% or less of potential CPU usage)
NETWORKmaximum network load associated with any single job task
PROCmaximum processor load associated with any single job task
SWAPmaximum virtual memory/SWAP (in MB) used by any single job task
WALLTIMErequested job walltime

PolicyDescription
ALWAYStake action whenever a violation is detected
EXTENDEDVIOLATIONtake action only if a violation is detected and persists for greater than the specified time limit
BLOCKEDWORKLOADONLYtake action only if a violation is detected and the constrained resource is required by another job

ActionDescription
CANCELterminate the job
CHECKPOINTcheckpoint and terminate job
MIGRATErequeue the job and require a different set of hosts for execution
NOTIFYnotify admins and job owner regarding violation
REQUEUEterminate and requeue the job
SUSPENDsuspend the job and leave it suspended for an amount of time defined by the X parameter

Example - Notify and then cancel job if requested memory is exceeded

moab.cfg
# if job exceeds memory usage, immediately notify owner
# if job exceeds memory usage for more than 5 minutes, cancel the job

RESOURCELIMITPOLICY MEM:ALWAYS,EXTENDEDVIOLATION:NOTIFY,CANCEL:00:05:00

Example - Checkpoint job on walltime violations

moab.cfg
# if job exceeds requested walltime, checkpoint job
RESOURCELIMITPOLICY WALLTIME:ALWAYS:CHECKPOINT

# when checkpointing, send term signal, followed by kill 1 minute later
RMCFG[base] TYPE=PBS CHECKPOINTTIMEOUT=00:01:00 CHECKPOINTSIG=SIGTERM

Example - Cancel jobs that use 5% or less of potential CPU usage for more than 5 minutes

moab.cfg
RESOURCELIMITPOLICY MINJOBPROC:EXTENDEDVIOLATION:CANCEL:5:00

Example - Migrating a job when it blocks other workload

moab.cfg
RESOURCELIMITPOLICY JOBPROC:BLOCKEDWORKLOADONLY:MIGRATE

10.4.2 Specifying Hard and Soft Policy Violations

   Moab is able to perform different actions for both hard and soft policy violations.  In most resource management systems, a mechanism does not exist to allow the user to specify both hard and soft limits.  To address this, Moab provides the RESOURCELIMITMULTIPLIER parameter which allows per partition and per resource multiplier factors to be specified to generate the actual hard and soft limits to be used.  If the factor is less than one, the soft limit will be lower than the specified value and an Moab action will be taken before the specified limit is reached.  If the factor is greater than one, the hard limit will be set higher than the specified limit allowing a buffer space before the hard limit action is taken.

  In the example below, job owners will be notified by email when their memory reaches 100% of the target and the job will be canceled if it reaches 125% of the target.  For wallclock usage, the job will be requeued when it reaches 90% of the specified limit and will be checkpointed when it reaches the full limit.

moab.cfg
RESOURCELIMITPOLICY       MEM:ALWAYS:NOTIFY,CANCEL
RESOURCELIMITPOLICY       WALLTIME:ALWAYS:REQUEUE,CHECKPOINT

RESOURCELIMITMULTIPLIER   MEM:1.25,WALLTIME:0.9

10.4.3 Constraining Walltime Usage

   While Moab constrains walltime using the parameter RESOURCELIMITPOLICY like other resources, it also allows walltime exception policies which are not available with other resources.  In particular, Moab allows jobs to exceed the requested wallclock limit by an amount specified on a global basis using the JOBMAXOVERRUN parameter or on a per credential basis using the OVERRUN attribute of the *CFG credential parameters.

moab.cfg
JOBMAXOVERRUN    00:10:00
CLASSCFG[debug]  overrun=00:00:30  

# send USR1 signal 5 minutes before job is to be terminated
CLASSCFG[viz]    overrun=00:30:00  PRETERMINATIONSIGNAL=13@00:05:00

See Also
© 2001-2008 Cluster Resources, Incorporated