I have tested the CLASSCFG[<queuename>] ...<div><br></div><div><i>CLASSCFG[team1] MAXNODE=3</i><br><br>And restarted maui to make the config take effect.</div><div>But when I submit 3 jobs each apply for 2 nodes, the three jobs are all running.</div>
<div><i><br></i></div><div><div><i>1665.vkvm161057 STDIN shubao.sx 0 <span style="background-color:rgb(204,0,0)">R </span>team1 </i></div><div><i>1666.vkvm161057 STDIN shubao.sx 0 <span style="background-color:rgb(204,0,0)">R </span> eam1 </i></div>
<div><i>1667.vkvm161057 STDIN shubao.sx 0 <span style="background-color:rgb(204,0,0)">R </span> eam1 </i></div><div><br></div><div>I think maybe I should have a look at the torque's source code.</div>
<br><div class="gmail_quote">On Sun, Jan 27, 2013 at 10:49 PM, Burkhard Bunk <span dir="ltr"><<a href="mailto:bunk@physik.hu-berlin.de" target="_blank">bunk@physik.hu-berlin.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
the "nodes" resource has always been string-valued, and this cannot<br>
be changed without breaking current installations. (The docs must be in<br>
error at this point.)<br>
Setting a default is ok, as in<br>
<br>
set queue defq resources_default.nodes = 1:ppn=4<br>
<br>
but "resources_min.nodes" and "resources_max.nodes" are considered invalid.<br>
<br>
As we are on it:<br>
The interpretation of "resources_max.nodect" seems to be tricky.<br>
With torque alone (FIFO scheduler), it used to act on a per-queue level,<br>
but when I introduced Maui, the interpretation changed to a per-job limitation. I found this with torque-2.5.x and maui-3.3, no idea whether<br>
it's intentional or a bug.<br>
A more precise handling of limits should be possible with Maui's<br>
<br>
CLASSCFG[<queuename>] ...<br>
<br>
and a (space separated) list of settings e.g. for<br>
<br>
MAXJOB MAXJOBPERUSER<br>
MAXNODE MAXNODEPERUSER<br>
MAXPROC MAXPROCPERUSER<br>
<br>
(see Manual, part 6.2.1), but I haven't tested this so far.<div class="im HOEnZb"><br>
<br>
Regards,<br>
Burkhard Bunk.<br>
------------------------------<u></u>------------------------------<u></u>----------<br>
<a href="mailto:bunk@physik.hu-berlin.de" target="_blank">bunk@physik.hu-berlin.de</a> Physics Institute, Humboldt University<br>
fax: <a href="tel:%2B%2B49-30%202093%207628" value="+493020937628" target="_blank">++49-30 2093 7628</a> Newtonstr. 15<br>
phone: <a href="tel:%2B%2B49-30%202093%207980" value="+493020937980" target="_blank">++49-30 2093 7980</a> 12489 Berlin, Germany<br>
------------------------------<u></u>------------------------------<u></u>----------<br>
<br></div><div class="HOEnZb"><div class="h5">
On Sun, 27 Jan 2013, shixing wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
When I set the queue both with nodect and nodes like this:create queue team1<br>
set queue team1 queue_type = Execution<br>
set queue team1 resources_max.nodect = 2<br>
set queue team1 resources_max.nodes = 2<br>
set queue team1 keep_completed = 2592000<br>
set queue team1 enabled = True<br>
set queue team1 started = True<br>
<br>
I can also submit large jobs and all the nodes used by the running jobs exceed<br>
the resources_max.nodect or resources_max.nodes (here are both 2).<br>
I submit the jobs 4 times like this:<br>
echo "sleep 100" | qsub -l nodes=1:ppn=4 -q team1<br>
<br>
And qstat shows the command like this:<br>
1661.vkvm161057 STDIN shubao.sx 0 R team1 <br>
<br>
1662.vkvm161057 STDIN shubao.sx 0 R team1 <br>
<br>
1663.vkvm161057 STDIN shubao.sx 0 R team1 <br>
<br>
1664.vkvm161057 STDIN shubao.sx 0 R team1 <br>
<br>
I use torque 4.1.3 and maui 3.3.1.<br>
<br>
On Sun, Jan 27, 2013 at 3:11 AM, Gustavo Correa <<a href="mailto:gus@ldeo.columbia.edu" target="_blank">gus@ldeo.columbia.edu</a>> wrote:<br>
I misundertood Shixing's original question.<br>
I though he wanted to prevent each job to exceed a certain number<br>
of nodes,<br>
but what he wants to apply the limit to the sum of all jobs in the<br>
queue, correct?<br>
<br>
Burkhard is right.<br>
"nodect" intent seems to be to limit the number of nodes used by<br>
all jobs in a specific queue,<br>
whereas AFAIK "nodes" limits the number of nodes each job can<br>
request when submitted to<br>
a specific queue, right?<br>
<br>
Things may have changed in recent versions, but "nodes" , with the<br>
interpretation above, works for me in Torque 2.4.11, with Maui<br>
3.2.6p21.<br>
<br>
However, "nodect", with the interpretation above, doesn't work for<br>
me,<br>
as Shixing also noted, even if I add "ppn=8" to my qsub command,<br>
to request all processors in my nodes, and try to exhaust the<br>
available resources and hit the nodect limit.<br>
Maybe there is a way to implement what Shixing wants in Maui?<br>
<br>
Quoting the Torque Admin Guide, section 4.1.1, "Queues<br>
attributes":<br>
<br>
"nodes integer Specifies the number of nodes "<br>
[Note, integer, not a string. Not in this context at least.]<br>
<br>
"nodect integer Sets the number of nodes available. By<br>
default, TORQUE will set the number of nodes available to the<br>
number of nodes listed in the $TORQUE_HOME/server_priv/nodes file.<br>
nodect can be set to be greater than or less than that number.<br>
Generally, it is used to set the node count higher than the number<br>
of physical nodes in the cluster."<br>
<br>
<a href="http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/4-serverPolicies/queue" target="_blank">http://docs.adaptivecomputing.<u></u>com/torque/4-1-3/help.htm#<u></u>topics/4-serverPolicies/queue</a><br>
Attributes.htm<br>
<br>
Admittedly, the Guide wording is not very clear.<br>
It could include "on a per queue basis", "on a per job basis", or<br>
something the like,<br>
to clarify the context.<br>
The final sentence in "nodect" sounds a bit awkward.<br>
Does it work to set the node count *smaller* than the number<br>
of physical nodes?<br>
Does this depend on the scheduler configuration? [pbs_sched,<br>
Maui, Moab]<br>
<br>
Somebody from Adaptive could clarify.<br>
<br>
Gus Correa<br>
<br>
On Jan 26, 2013, at 8:31 AM, Burkhard Bunk wrote:<br>
<br>
> Hi,<br>
><br>
> I would support that.<br>
> The correct form, IMHO, is<br>
><br>
> set queue myqueue resources_max.nodect = 10<br>
><br>
> Regards,<br>
> Burkhard Bunk.<br>
><br>
------------------------------<u></u>------------------------------<u></u>----------<br>
> <a href="mailto:bunk@physik.hu-berlin.de" target="_blank">bunk@physik.hu-berlin.de</a> Physics Institute, Humboldt<br>
University<br>
> fax: <a href="tel:%2B%2B49-30%202093%207628" value="+493020937628" target="_blank">++49-30 2093 7628</a> Newtonstr. 15<br>
> phone: <a href="tel:%2B%2B49-30%202093%207980" value="+493020937980" target="_blank">++49-30 2093 7980</a> 12489 Berlin, Germany<br>
><br>
------------------------------<u></u>------------------------------<u></u>----------<br>
><br>
> On Sat, 26 Jan 2013, André Gemünd wrote:<br>
><br>
>> Hi,<br>
>> I'm a bit surprised by that. Since when is nodes an integer in<br>
Torque? It used to be a string in earlier versions.<br>
>><br>
>> Greetings<br>
>> André<br>
>><br>
>> ----- Ursprüngliche Mail -----<br>
>>> Hi Shixing<br>
>>> Have you tried:<br>
>>> set queue myqueue resources_max.nodes = 10<br>
>>> More info:<br>
>>><a href="http://docs.adaptivecomputing.com/torque/4-1-3/help.htm#topics/4-serverPolicies/queue" target="_blank">http://docs.<u></u>adaptivecomputing.com/torque/<u></u>4-1-3/help.htm#topics/4-<u></u>serverPolicies/queue</a><br>
Attributes.htm<br>
>>> under "Assigning queue resource limits".<br>
>>> I hope this helps,<br>
>>> Gus<br>
>>> On 01/25/2013 05:44 AM, André Gemünd wrote:<br>
>>> > I don't think that setting resources_available.nodect has<br>
any<br>
>>> > effect.<br>
>>> > Have you tried resources_max.nodect?<br>
>>> ><br>
>>> > Greetings<br>
>>> ><br>
>>> > ----- Ursprüngliche Mail -----<br>
>>> >><br>
>>> >> Hi, all:<br>
>>> >> Recently I have set up a cluster with 200 nodes. And this<br>
cluster<br>
>>> >> is<br>
>>> >> designed for serving several team. And I want split the<br>
nodes to<br>
>>> >> some queues.<br>
>>> >> I have set the queue attr like this:<br>
>>> >> set queue team1 resources_available.nodect = 3<br>
>>> >><br>
>>> >><br>
>>> >> But when I submit a job applying nodes>3, it will also run<br>
>>> >> successfully. The submit command is :<br>
>>> >> echo "sleep 100" | qsub -l nodes=4 -q team1<br>
>>> >><br>
>>> >><br>
>>> >> So how can I set the max nodes for the queues?<br>
>>> >> --<br>
>>> >> Best wishes!<br>
>>> >> My Friend~<br>
>>> >> ______________________________<u></u>_________________<br>
>>> >> torqueusers mailing list<br>
>>> >> <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
>>> >> <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/<u></u>mailman/listinfo/torqueusers</a><br>
>>> >><br>
>>> ><br>
>>> ______________________________<u></u>_________________<br>
>>> torqueusers mailing list<br>
>>> <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
>>> <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/<u></u>mailman/listinfo/torqueusers</a><br>
>><br>
>> --<br>
>> André Gemünd<br>
>> Fraunhofer-Institute for Algorithms and Scientific Computing<br>
>> <a href="mailto:andre.gemuend@scai.fraunhofer.de" target="_blank">andre.gemuend@scai.fraunhofer.<u></u>de</a><br>
>> Tel: <a href="tel:%2B49%202241%2014-2193" value="+492241142193" target="_blank">+49 2241 14-2193</a><br>
>> /C=DE/O=Fraunhofer/OU=SCAI/OU=<u></u>People/CN=Andre Gemuend<br>
>> ______________________________<u></u>_________________<br>
>> torqueusers mailing list<br>
>> <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
>> <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/<u></u>mailman/listinfo/torqueusers</a><br>
> ______________________________<u></u>_________________<br>
> torqueusers mailing list<br>
> <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
> <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/<u></u>mailman/listinfo/torqueusers</a><br>
<br>
______________________________<u></u>_________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/<u></u>mailman/listinfo/torqueusers</a><br>
<br>
<br>
<br>
<br>
--<br>
Best wishes!<br>
My Friend~<br>
<br>
</blockquote>
</div></div><br>_______________________________________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Best wishes!<br>My Friend~
</div>