[Moabusers] soft policy violation scheduling

[Douglas Wightman]

There are some problems associated with prioritizing jobs based on
SPVIOLATIONWEIGHT.  The first is that any time a job action is
performed, meaning anytime Moab starts, stops, suspends, resumes a job
the priority for all other jobs would have to be recalculated and the
entire queue would have to be reordered.  Moab does not currently do
this for obvious optimization reasons.  Also, any time a job action
occurred all existing idle job reservations would have to be reevaluated
as they may now be invalid.  

Currently, jobs can be prioritied based on SPVIOLATIONWEIGHT but
intra-iteration violations will not be detected.  This can result in
inaccurate priority scheduling within the same iteration.

Some of the possible ways of doing what you want would involve a
combination of the following parameters:

JOBMAXSTARTPERITERATION 1

http://www.clusterresources.com/products/mwm/docs/a.fparameters.shtml#jobmaxstartperiteration

RESERVATIONPOLICY NEVER

http://www.clusterresources.com/products/mwm/docs/a.fparameters.shtml#reservationpolicy

And a new parameter to turn off the soft-policy scheduling cycle.
Timeframes for feature requests such as this will be dependent on the
type of contract you currently have with Cluster Resources.

The new development is relatively minor so we would be interested in
working with you on these requests.

- Douglas

On Tue, 2007-07-10 at 18:27 -0400, Michael Santos wrote:
> Hi,
> 
> 	According to section 8.2.2 of the Moab Admin guide, when scheduling  
> jobs Moab first considers jobs that meet soft policy constraints, and  
> then those that meet hard policy constraints.  My reading is that  
> even if a job that violates soft policy constraints has a higher  
> priority than a job that doesn't violate any constraints, the job  
> that doesn't violate any constraints gets scheduled first.
> 
> 	What I would really like would be to disable this feature, and have  
> jobs scheduled based only on priority (assuming they meet hard policy  
> constraints).  I would like to use SPVIOLATIONWEIGHT to penalize the  
> priority of jobs that violate soft policy constraints.  The result  
> should be that sometimes jobs that violate soft policy constraints  
> get scheduled ahead of those that do not, depending on the interplay  
> between other priority components and the value of SPVIOLATIONWEIGHT.
> 
> 	I would greatly appreciate any suggestions on how I might accomplish  
> this.  If you are interested in a concrete example, I have included  
> one below.  We are running
> moab server version 5.0.0p4 (snap NA) (rev. 6448M)
> 
> 	I have a related simple question: how do I specify a finite soft  
> limit, infinite hard limit combination?  I settled on, e.g.,  
> MAXPROC=24,-1, after trying MAXPROC=24,0 with unexpected results.  I  
> couldn't find any documentation on this.
> 
> 	Thank you.
> 
> Best regards,
> Michael Santos
> Space Telescope Science Institute
> 
> Concrete example:
> 
> moab.cfg (relevant part):
> 
> QUEUETIMEWEIGHT     10
> 
> CREDWEIGHT              1
> QOSCFG[normal]        PRIORITY=0
> QOSCFG[special]        PRIORITY=100000 MAXPROC=1,-1
> 
> SERVICEWEIGHT           1
> SPVIOLATIONWEIGHT       -100000
> 
> Situation:
> 
> 2 proc cluster.  A QoS: special job is running indefinitely on proc  
> 1.  Proc 2 is running a job that finishes.  There are two queued 1- 
> processor jobs.  They are
> 
> Job A: QoS: special, queuetime: 10 min
> Job B: QoS: normal, queuetime: 1 min
> 
> If Job A were also QoS normal, Job A would execute now.  But since  
> Job A violates QoS special's soft policy processor limit, it is  
> blocked, and Job B executes.  This is despite the fact that, as I  
> understand it, the priorities are:
> 
> Priority(A) = 100 (queuetime) + 100000 (QoS) + -100000 (SP violation)  
> = 100
> Priority(B) = 10 (queuetime)
> 
> Here's what I would like: As long as QoS special isn't saturating its  
> resource limits, I want its jobs at the top of the priority list.   
> But if QoS special is saturating its resource limits, I want its jobs  
> to be prioritized based on their times in queue, not automatically  
> shoved to the bottom of the priority list.  A job submitted to QoS  
> special should never execute later than it would have had it been  
> submitted to QoS normal instead.
> 
> Any advice on how to implement this would be greatly appreciated.
> 
> _______________________________________________
> moabusers mailing list
> moabusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/moabusers