[Moabusers] soft policy violation scheduling

[Michael Santos]

Hi,

	According to section 8.2.2 of the Moab Admin guide, when scheduling  
jobs Moab first considers jobs that meet soft policy constraints, and  
then those that meet hard policy constraints.  My reading is that  
even if a job that violates soft policy constraints has a higher  
priority than a job that doesn't violate any constraints, the job  
that doesn't violate any constraints gets scheduled first.

	What I would really like would be to disable this feature, and have  
jobs scheduled based only on priority (assuming they meet hard policy  
constraints).  I would like to use SPVIOLATIONWEIGHT to penalize the  
priority of jobs that violate soft policy constraints.  The result  
should be that sometimes jobs that violate soft policy constraints  
get scheduled ahead of those that do not, depending on the interplay  
between other priority components and the value of SPVIOLATIONWEIGHT.

	I would greatly appreciate any suggestions on how I might accomplish  
this.  If you are interested in a concrete example, I have included  
one below.  We are running
moab server version 5.0.0p4 (snap NA) (rev. 6448M)

	I have a related simple question: how do I specify a finite soft  
limit, infinite hard limit combination?  I settled on, e.g.,  
MAXPROC=24,-1, after trying MAXPROC=24,0 with unexpected results.  I  
couldn't find any documentation on this.

	Thank you.

Best regards,
Michael Santos
Space Telescope Science Institute

Concrete example:

moab.cfg (relevant part):

QUEUETIMEWEIGHT     10

CREDWEIGHT              1
QOSCFG[normal]        PRIORITY=0
QOSCFG[special]        PRIORITY=100000 MAXPROC=1,-1

SERVICEWEIGHT           1
SPVIOLATIONWEIGHT       -100000

Situation:

2 proc cluster.  A QoS: special job is running indefinitely on proc  
1.  Proc 2 is running a job that finishes.  There are two queued 1- 
processor jobs.  They are

Job A: QoS: special, queuetime: 10 min
Job B: QoS: normal, queuetime: 1 min

If Job A were also QoS normal, Job A would execute now.  But since  
Job A violates QoS special's soft policy processor limit, it is  
blocked, and Job B executes.  This is despite the fact that, as I  
understand it, the priorities are:

Priority(A) = 100 (queuetime) + 100000 (QoS) + -100000 (SP violation)  
= 100
Priority(B) = 10 (queuetime)

Here's what I would like: As long as QoS special isn't saturating its  
resource limits, I want its jobs at the top of the priority list.   
But if QoS special is saturating its resource limits, I want its jobs  
to be prioritized based on their times in queue, not automatically  
shoved to the bottom of the priority list.  A job submitted to QoS  
special should never execute later than it would have had it been  
submitted to QoS normal instead.

Any advice on how to implement this would be greatly appreciated.