<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:10.0pt;
font-family:"Times New Roman","serif";}
span.emailstyle17
{mso-style-name:emailstyle17;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.emailstyle18
{mso-style-name:emailstyle18;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:2129665061;
mso-list-type:hybrid;
mso-list-template-ids:108185630 945743166 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:7200;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:20.25pt;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D"> Huh?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> I assume this response for some other message.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:20.25pt;text-indent:-.25in;mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Jim C.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> torqueusers-bounces@supercluster.org [mailto:torqueusers-bounces@supercluster.org]
<b>On Behalf Of </b>Brian Haymore<br>
<b>Sent:</b> Friday, December 14, 2012 12:56 PM<br>
<b>To:</b> Torque Users Mailing List<br>
<b>Subject:</b> Re: [torqueusers] How to run mpirun of intel on torque<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">There are no quotas present on the scratch file systems. So for you to be getting a quota limited message tells me that somehow the tool you're using is trying to write something to your user home directory. You should look to clean up some of your home directory to bring yourself below your quota limit .<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">--<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">Brian D Haymore<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">University of Utah<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">Center for High Performance Computing<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">155 South 1452 East, RM 405<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">Salt Lake City, Utah 84112<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">Phone: 801-558-1150, Fax: 801-585-5366<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black"><a href="http://www.map.utah.edu/umaplink/0019.html">http://www.map.utah.edu/umaplink/0019.html</a><o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black">"Coyle, James J [ITACD]" <<a href="mailto:jjc@iastate.edu">jjc@iastate.edu</a>> wrote:<o:p></o:p></span></pre>
<pre><span style="font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></pre>
<div>
<div>
<p class="MsoNormal"><span style="color:#1F497D"> I have a script that I run from /var/spool/mom_priv/prologue whenever a josb that</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">requires all the cores on the node. This script kills any user level processes that are not
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">the job owner or a torque manager.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> This gets rid of any effect of users who should not be logged on to the compute node, that</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">is who have just ssh’d in rather than using qsub –I</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> I clean out scratch space on exit with epilogue if it is a dedicated job.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> You could couple this with techniques to prevent ssh logins durin the job</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">(For jobs which dedicate the node, have prologue disallow logins from other users,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">and restore normal function on exit.)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:torqueusers-bounces@supercluster.org">torqueusers-bounces@supercluster.org</a> [<a href="mailto:torqueusers-bounces@supercluster.org">mailto:torqueusers-bounces@supercluster.org</a>]
<b>On Behalf Of </b>David Roman<br>
<b>Sent:</b> Friday, December 14, 2012 6:53 AM<br>
<b>To:</b> 'torqueusers@supercluster.org'<br>
<b>Subject:</b> [torqueusers] How to run mpirun of intel on torque</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="FR">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR"> </span><o:p></o:p></p>
<p class="MsoNormal">I am sorry, but my english is really sad. I thank you for your patience for that.<o:p></o:p></p>
<p class="MsoNormal">I installed TORQUE 4.1.0, with MAU 3.3.1. I compiled OPENMPI with option --with-tm=/usr/local/torque.<o:p></o:p></p>
<p class="MsoNormal">I disallowed ssh connections for my users on executable nodes. In /etc/ssh/sshd_config I set
<o:p></o:p></p>
<p class="MsoNormal">AllowGroups root admin<o:p></o:p></p>
<p class="MsoNormal">This works fine.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">But now, I install MPI Intel Librarie and Ifort.<o:p></o:p></p>
<p class="MsoNormal">When I open a interactive pbs session:<o:p></o:p></p>
<p class="MsoNormal">qsub -I -l nodes=2:ppn=8<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I am connected on a node.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I run mpi job <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">mpirun -genv I_MPI_FABRICS_LIST tmi ./my_program<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">But it can not start, because it cannot connect on the other node.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">If I append users in AllowGroups of /etc/ssh/sshd_config it works.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">But if I do this, all users can connect on executable nodes, without use torque, and this is bad.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">How can I do to disallow ssh connection without torque or make mpirun of intel works like openmpi, without ssh connection allowed for users ?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span lang="FR">Thank you</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR">David</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>