[Moabusers] job templates and flags

[Troy Baer]

I'm trying to get get jobs that request a certain node feature
("perfmon") to have dedicated access to the nodes they're allocated, and
it seems job templates are the way to accomplish that.  However, I can't
seem to quite make it work.  (This is in Moab 5.1.0p9, BTW.)

Here's my job template setup for these jobs:

# Force perfmon jobs to have the nodes to themselves
JOBCFG[perfmon.min]     REQFEATURES=perfmon
JOBCFG[perfmon.set]     FLAGS=BENCHMARK,DEDICATED
JOBMATCHCFG[perfmon]    JMIN=perfmon.min JSET=perfmon.set

What happens when I submit a job that matches this template is that it
gets scheduled on a node with other jobs most of the time.  It seems
like the DEDICATED flag on the template is ignored, and it's not clear
what (if anything) the BENCHMARK flag does:

$ qsub -I -l nodes=1:ppn=1:perfmon
qsub: waiting for job 104089.opt-batch.osc.edu to start
[...job lands on node with other jobs...]

# checkjob 104089
job 104089
[...]
Job Templates: perfmon
[...]
Flags:          BENCHMARK,SUSPENDABLE,PREEMPTEE,INTERACTIVE
Attr:           PREEMPTEE,INTERACTIVE,checkpoint
[...]

So it looks like the DEDICATED flag isn't available to job templates,
but in principle I can always used a QOS level to do that:

QOSCFG[nodeexclusive]   QFLAGS=DEDICATED
JOBCFG[perfmon.set]     QOS=nodeexclusive

However, when I do that, Moab kicks my jobs out:

troy at opt-login01:~$ qsub -I -l nodes=1:ppn=1:perfmon
qsub: waiting for job 104119.opt-batch.osc.edu to start
qsub: PBS: MOAB_INFO:  interactive job can never run - job cannot use
requested QOS 'nodeexclusive'

What am I missing here?

(Also, is there a way to have either a job template or a QOS level
*remove* a flag from a job?  I'd love to have this perfmon template be
able to remove the SUSPENDABLE and PREEMPTEE flags from these jobs...)

Thanks in advance,
	--Troy
-- 
Troy Baer, Supercomputing Support Engineer
Ohio Supercomputer Center
Phone:  614-292-9701
http://www.osc.edu/supercomputing/