[torquedev] routing queues
siegert at sfu.ca
Wed Aug 4 19:58:58 MDT 2010
On Wed, Jul 28, 2010 at 07:21:01PM -0700, Martin Siegert wrote:
> Can somebody tell me which routine in the code is responsible for
> assigning the correct queue?
I believe I now understand at least somewhat how routing is implemented.
Consider the following setup:
set queue default queue_type = Route
set queue default route_destinations = qs
set queue default route_destinations += q1
set queue qs queue_type = Execution
set queue qs resources_min.nodes = 1:ppn=2
set queue qs resources_min.procs = 2
set queue q1 queue_type = Execution
set queue q1 resources_max.nodect = 1
set queue q1 resources_max.nodes = 1:ppn=1
set queue q1 resources_max.procs = 1
Submit a job with -l nodes=1:ppn=1.
In order to route the job into the correct queue the routine comp_resc2
in attr_fn_resc.c is called. This routine is supposed to check whether
the min requirements of the queue are satisfied. When this is checked for
the queue "qs" in the case of *wiresc->rs_defin->rs_name being "nodes" the
rs_comp routine at line 605 (probably also at line 550) basically runs a
strcmp("1:ppn=2", "1:ppn=1") which returns 1 and consequently comp_resc_gt
is increased by 1, hence the check comp_resc_gt > 0 at svr_jobfunc.c,
line 1466 is true and chk_resc_limits returns with PBSE_EXCQRESC indicating
that the min requirements are indeed violated for queue qs. Everything
Submit a job with -l nodes=1:ppn=10.
Everything remains the same! The point being that rs_comp again runs a
strcmp, now with arguments "1:ppn=2", "1:ppn=10", which also returns 1
indicating again that the min requirements are violated for queue qs.
This cannot be correct.
Now swap order of the route destinations:
set queue default route_destinations = q1
set queue default route_destinations += qs
and submit a job with -l nodes=1:ppn=2.
This time the critical code is in the routine chk_svr_resc_limit (in
svr_jobfunc.c) which is called from chk_resc_limits (svr_jobfunc.c,
line 1476). When the nodes resource is checked, i.e., LimitName at line
1023 svr_jobfunc.c is set to "nodes", the comparison between job request
and queue settings are made in lines 1050 - 1058 of svr_jobfunc.c:
if ((isdigit(*(jbrc_nodes->rs_value.at_val.at_str))) &&
job_nodes = atoi(jbrc_nodes->rs_value.at_val.at_str);
queue_nodes = atoi(qurc->rs_value.at_val.at_str);
if (queue_nodes < job_nodes)
and jbrc_nodes->rs_value.at_val.at_str is "1:ppn=2" and
qurc->rs_value.at_val.at_str is "1:ppn=1". Hence both
*(qurc->rs_value.at_val.at_str) are 1 and both job_nodes and queue_nodes
are set to 1. Consequently no violation of the max. queue limit is
detected. In fact, the whole :ppn=x part is ignored.
In conclusion, routing based on "nodes" requests is not working.
The question is how should it be working? E.g., what is "larger":
nodes=1:ppn=2 or nodes=2:ppn=1? This does not really make sense.
IMHO the only sensible comparision for requests of the form
nodes=x:ppn=y is to compare x*y with queue limits. And such a method
could then be expanded to include procs as well: there should be an
attribute, say procct, that gets set to x*y + z, where z comes from
procs=z requests; procct can be calculated for the queue min and max
limits and compared with the job request.
More information about the torquedev