<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
        {font-family:Wingdings;
        panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p
        {mso-style-priority:99;
        margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {mso-style-priority:34;
        margin-top:0in;
        margin-right:0in;
        margin-bottom:0in;
        margin-left:.5in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
span.EmailStyle19
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
 /* List Definitions */
 @list l0
        {mso-list-id:950014513;
        mso-list-type:hybrid;
        mso-list-template-ids:-800534558 -1083902446 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
        {mso-level-start-at:2;
        mso-level-number-format:bullet;
        mso-level-text:-;
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        margin-left:20.25pt;
        text-indent:-.25in;
        font-family:"Calibri","sans-serif";
        mso-fareast-font-family:Calibri;
        mso-bidi-font-family:"Times New Roman";}
ol
        {margin-bottom:0in;}
ul
        {margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext="edit">
  <o:idmap v:ext="edit" data="1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=EN-US link=blue vlink=purple>

<div class=WordSection1>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Chris,<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; vmem should &nbsp;always be greater than pmem. I think
that Torque corrects <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>this should it be specified.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>In the csh, the limit command will show&nbsp; you the resource
limits in a job.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; If you submit several jobs like:<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>echo 'csh -c limit' | qsub -l vmem=18GB,pmem=14GB,mem=2GB,nodes=1:ppn=1<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>you can see the effects of combinations of vmem, pmem and mem<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>For me, running Torque version 2.3.6:<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>it seems as though <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp; mem= does not have an effect on the csh limits,<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; datasize and memoryuse are &nbsp;set to pmem<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>and<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; &nbsp;vmemoryuse is set to max(vmem , pmem)<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; I use a web form where users request for cpus, walltime
time and memory per process.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The result is a Torque script with the &nbsp;#PBS &#8211;l &nbsp;line
to use.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;1) For the cluster that uses OpenMPI, I calculate:<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp;&nbsp; cput=#cpus*walltime<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>and<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp;&nbsp; &nbsp;vmem=ppn*pmem. &nbsp;&nbsp;<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp; If this exceeds the memory on a node, I calculate
how many processes can fit<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>on a single node and use that for ppn, and adjust the number of
nodes accordingly.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp; I do this because it seems as though OpenMPI reports
back the sum of the vmem <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>for all OpenMPI processes on a node.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;2)&nbsp; &nbsp;For the cluster that uses MPICH, I set
vmem=pmem.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; since it seems that MPICH does not report back<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp; I&#8217;ve found that there is no such thing as a
small amount of swapping.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Further, <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp;&nbsp; if a node is shared, the swapping is unfair
to other users on the node,<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; as it make their programs swap also.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;&nbsp;&nbsp; if no nodes are shared, the single user of
the node should just get all<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;the memory on the node.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;Users do sometimes over-allocate arrays, but then the
space is just allocated in virtual space<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>and never used.&nbsp; <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; You can try out the web form for the cluster using OpenMPI
at:<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><a
href="http://hpcgroup.public.iastate.edu/HPC/lightningsmp/lightningsmp_script_writer.html">http://hpcgroup.public.iastate.edu/HPC/lightningsmp/lightningsmp_script_writer.html</a><o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp; I can&#8217;t say that it is perfect, but it gets the job
done.<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoListParagraph style='margin-left:20.25pt;text-indent:-.25in;
mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#1F497D'><span style='mso-list:Ignore'>-<span
style='font:7.0pt "Times New Roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><![endif]><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Jim<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;James Coyle, PhD<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;High Performance Computing Group&nbsp;&nbsp;&nbsp;&nbsp; <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;115 Durham
Center&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;Iowa State
Univ.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&nbsp;Ames, Iowa
50011&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; web:
http://www.public.iastate.edu/~jjc<o:p></o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>

<div>

<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>

<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
torqueusers-bounces@supercluster.org
[mailto:torqueusers-bounces@supercluster.org] <b>On Behalf Of </b>Rushton
Martin<br>
<b>Sent:</b> Tuesday, August 10, 2010 8:46 AM<br>
<b>To:</b> Chris Vaughan; Torque Users Mailing List<br>
<b>Subject:</b> Re: [torqueusers] Limiting max vmem+pmem<o:p></o:p></span></p>

</div>

</div>

<p class=MsoNormal><o:p>&nbsp;</o:p></p>

<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Are you sure you are asking the right question?&nbsp; The physical
memory used by a process (its working set) is mapped to its virtual memory, and
is therefore a part of it.</span><o:p></o:p></p>

<div>

<p class=MsoNormal>&nbsp;<o:p></o:p></p>

</div>

<div>

<p><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:navy'>Martin Rushton<br>
</span></b><i><span style='font-size:8.0pt;font-family:"Arial","sans-serif";
color:navy'>Weapons Technologies<br>
Tel: 01959 514777, Mobile: 07939 219057<br>
email: jmrushton@QinetiQ.com</span></i><span style='font-size:8.0pt;font-family:
"Arial","sans-serif";color:navy'><br>
<a href="http://www.qinetiq.com/">www.QinetiQ.com</a><br>
<i>QinetiQ - Delivering customer-focused solutions</i> <o:p></o:p></span></p>

<p><i><span style='font-size:8.0pt;font-family:"Arial","sans-serif";color:#339966'>Please
consider the environment before printing this email.</span></i><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";color:#339966'> </span><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";color:navy'><o:p></o:p></span></p>

</div>

<div>

<p class=MsoNormal>&nbsp;<o:p></o:p></p>

</div>

<blockquote style='margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'>

<p class=MsoNormal><o:p>&nbsp;</o:p></p>

<div class=MsoNormal align=center style='text-align:center'>

<hr size=2 width="100%" align=center>

</div>

<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> torqueusers-bounces@supercluster.org
[mailto:torqueusers-bounces@supercluster.org] <b>On Behalf Of </b>Chris Vaughan<br>
<b>Sent:</b> 10 August 2010 14:24<br>
<b>To:</b> torqueusers<br>
<b>Subject:</b> [torqueusers] Limiting max vmem+pmem</span><o:p></o:p></p>

<div>

<p class=MsoNormal><span style='color:black'>All, <o:p></o:p></span></p>

<div>

<p class=MsoNormal><span style='color:black'><br>
<br>
<o:p></o:p></span></p>

</div>

<div>

<p class=MsoNormal><span style='color:black'>Is there a way to limit the total
combined size of pmem and vmem within TORQUE? &nbsp;The user is comfortable
with some swapping but would like to limit the combined amount of vmem+pmem
without setting two different individual limits for both.<o:p></o:p></span></p>

</div>

<div>

<p class=MsoNormal><span style='color:black'><br>
<br>
<o:p></o:p></span></p>

</div>

<div>

<p class=MsoNormal><span style='color:black'>Thanks,<o:p></o:p></span></p>

</div>

<div>

<p class=MsoNormal><span style='color:black'><br>
<br>
<o:p></o:p></span></p>

</div>

<div>

<p class=MsoNormal><span style='color:black'>-- <br>
<br>
<o:p></o:p></span></p>

<div>

<div>

<div>

<p><strong><span style='color:#1F497D'>Chris Vaughan</span></strong><span
style='color:#A6A6A6'>&nbsp; |&nbsp; Technical Consultant - EMEA</span><span
style='color:black'><o:p></o:p></span></p>

<p><span style='color:#A6A6A6'>3000 Cathedral Hill | Guildford GU2 7YB&nbsp;
|&nbsp; United Kingdom</span><span style='color:black'><o:p></o:p></span></p>

<p><span style='color:black'>&nbsp;<o:p></o:p></span></p>

<p><span style='color:black'><img border=0 id="_x0000_i1026"
src="http://www.adaptivecomputing.com/sig.png"><o:p></o:p></span></p>

</div>

</div>

</div>

<p class=MsoNormal><span style='color:black'><br>
<br>
<o:p></o:p></span></p>

</div>

</div>

</blockquote>

<p><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>This email
and any attachments to it may be confidential and are intended solely for the
use of the individual to whom it is addressed. If you are not the intended
recipient of this email, you must neither take any action based upon its
contents, nor copy or show it to anyone. Please contact the sender if you
believe you have received this email in error. QinetiQ may monitor email
traffic data and also the content of email for the purposes of security.
QinetiQ Limited (Registered in England &amp; Wales: Company Number: 3796233)
Registered office: 85 Buckingham Gate, London SW1E 6PD </span><a
href="http://www.QinetiQ.com"><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>http://www.qinetiq.com</span></a><o:p></o:p></p>

</div>

</div>

</body>

</html>