<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:Tahoma;
        panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
span.EmailStyle17
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">We use environment modules too. I have folks put those in their scripts.<o:p></o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">#PBS -l procs=10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">module compile/pgi mpi/openmpi<o:p></o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">mpirun myprog<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">Now when they say my program is not running, I can easily replicate their environment because it is all in the script AND they don&#8217;t shoot themselves in the
 foot if they are submitting a job from within a job.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">What could work if you are using -V is something like:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">for val in $(env|awk -F= '/^PBS/{print $1}'); do
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">echo $val=${!val}&gt;/tmp/$PBS_JOBID.sh;
<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">unset $val;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">done<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">qsub -V somescript.sh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">source /tmp/$PBS_JOBID.sh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">rm /tmp/$PBS_JOBID.sh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">Brian Andrus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">ITACS/Research Computing<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">Naval Postgraduate School<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">Monterey, California<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D">voice: 831-656-6238<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,&quot;sans-serif&quot;;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;">From:</span></b><span style="font-size:10.0pt;font-family:&quot;Tahoma&quot;,&quot;sans-serif&quot;"> torqueusers-bounces@supercluster.org [mailto:torqueusers-bounces@supercluster.org]
<b>On Behalf Of </b>Ken Nielson<br>
<b>Sent:</b> Wednesday, January 23, 2013 3:11 PM<br>
<b>To:</b> Torque Users Mailing List<br>
<b>Subject:</b> Re: [torqueusers] PBS environmental variables and -V<o:p></o:p></span></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Matt,<br>
<br>
I guess you would call it a bug. I think it is simply years of maintenance without knowledge of the original intent. Rick made a ticket for this and we will see what it currently does and try to figure out what it should do and then let everyone know.<br>
<br>
Ken<o:p></o:p></p>
<div>
<p class="MsoNormal">On Wed, Jan 23, 2013 at 3:30 PM, Matthew Britt &lt;<a href="mailto:msbritt@umich.edu" target="_blank">msbritt@umich.edu</a>&gt; wrote:<o:p></o:p></p>
<p class="MsoNormal">As a counter argument, the value of PBS_NODEFILE isn't getting set to the same value for the second job. &nbsp; Like PBS_NODEFILE, I would expect the PBS_O variables get set correctly for the submitted job and not the environment passed into
 it. &nbsp;It isn't consistent w/ PBS_O variables either, as PBS_O_QUEUE is overwritten w/ the correct value (I exported a different value into PBS_O_QUEUE to check).<br>
<br>
fwiw, we're using Environmental Modules, so we set several variables for software packages, like license servers, process launchers (like hydra), etc. &nbsp;The users might not be aware that these variables are necessary, so we have the users load appropriate module(s)
 and submit with -V.<br>
<br>
If the PBS_O_ variables are treated differently than PBS_ variables, that's fine; I was curious if it was by design or was a bug.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
&nbsp;- Matt<br>
<br>
--------------------------------------------<br>
Matthew Britt<br>
CAEN HPC Group - College of Engineering<br>
<a href="mailto:msbritt@umich.edu">msbritt@umich.edu</a><br>
<br>
<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">On Jan 23, 2013, at 2:54 PM, &quot;Andrus, Brian Contractor&quot; &lt;<a href="mailto:bdandrus@nps.edu">bdandrus@nps.edu</a>&gt; wrote:<br>
<br>
&gt; Seems to me that would be by design and you need to be aware of it.<br>
&gt; -V basically just takes the output of 'env' and sets anything that is set.<br>
&gt; Since you already have PBS_* variables, I would expect them to be set already.<br>
&gt;<br>
&gt; This could be handled by pbs_mom if it were to first sent the -V stuff and then set the PBS_* stuff.<br>
&gt; You can do this by iterating through and 'unset' all the PBS_* variables before doing the qsub.<br>
&gt;<br>
&gt; I tend to rend this moot by highly discouraging the use of -V<br>
&gt; It's use makes it difficult to troubleshoot when folks use things like &quot;./a.out&quot; to run their programs.<br>
&gt;<br>
&gt;<br>
&gt; Brian Andrus<br>
&gt; ITACS/Research Computing<br>
&gt; Naval Postgraduate School<br>
&gt; Monterey, California<br>
&gt; voice: <a href="tel:831-656-6238">831-656-6238</a><br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;&gt; -----Original Message-----<br>
&gt;&gt; From: <a href="mailto:torqueusers-bounces@supercluster.org">torqueusers-bounces@supercluster.org</a> [mailto:<a href="mailto:torqueusers-">torqueusers-</a><br>
&gt;&gt; <a href="mailto:bounces@supercluster.org">bounces@supercluster.org</a>] On Behalf Of Matthew Britt<br>
&gt;&gt; Sent: Tuesday, January 22, 2013 11:22 AM<br>
&gt;&gt; To: Torque Users Mailing List<br>
&gt;&gt; Subject: [torqueusers] PBS environmental variables and -V<br>
&gt;&gt;<br>
&gt;&gt; Hello everyone. &nbsp; What is the expected behavior in precedence between<br>
&gt;&gt; PBS environmental variables and -V when a job is submitted &nbsp;from an<br>
&gt;&gt; interactive job. &nbsp;With torque 4.1.4 (and possibly earlier), the variables from<br>
&gt;&gt; the environment of the interactive shell are passed along w/ the newly<br>
&gt;&gt; submitted job (either interactive or batch). &nbsp; We've seen both PBS_O_HOST<br>
&gt;&gt; and PBS_O_WORKDIR be set to values of the first job rather than the<br>
&gt;&gt; attributes of the second job.<br>
&gt;&gt;<br>
&gt;&gt; As an example:<br>
&gt;&gt;<br>
&gt;&gt; [msbritt@nyx ~]$ cd bin<br>
&gt;&gt; [msbritt@nyx bin]$ pwd<br>
&gt;&gt; /home/msbritt/bin<br>
&gt;&gt; [msbritt@nyx bin]$ qsub -I -l nodes=1,walltime=5:00 -q flux -A msbritt_flux -<br>
&gt;&gt; V<br>
&gt;&gt; qsub: waiting for job <a href="http://9445802.nyx.engin.umich.edu" target="_blank">
9445802.nyx.engin.umich.edu</a> to start<br>
&gt;&gt; qsub: job <a href="http://9445802.nyx.engin.umich.edu" target="_blank">9445802.nyx.engin.umich.edu</a> ready<br>
&gt;&gt;<br>
&gt;&gt; [msbritt@nyx5515 ~]$ echo $PBS_O_WORKDIR<br>
&gt;&gt; /home/msbritt/bin<br>
&gt;&gt; [msbritt@nyx5515 ~]$ echo $PBS_O_HOST<br>
&gt;&gt; <a href="http://nyx.engin.umich.edu" target="_blank">nyx.engin.umich.edu</a><br>
&gt;&gt; [msbritt@nyx5515 ~]$ pwd<br>
&gt;&gt; /home/msbritt<br>
&gt;&gt;<br>
&gt;&gt; [msbritt@nyx5515 ~]$ qsub -I -l nodes=1,walltime=5:00 -q flux -A<br>
&gt;&gt; msbritt_flux -V<br>
&gt;&gt; qsub: waiting for job <a href="http://9445813.nyx.engin.umich.edu" target="_blank">
9445813.nyx.engin.umich.edu</a> to start<br>
&gt;&gt; qsub: job <a href="http://9445813.nyx.engin.umich.edu" target="_blank">9445813.nyx.engin.umich.edu</a> ready<br>
&gt;&gt;<br>
&gt;&gt; [msbritt@nyx5623 ~]$ pwd<br>
&gt;&gt; /home/msbritt<br>
&gt;&gt; [msbritt@nyx5623 ~]$ echo $PBS_O_WORKDIR<br>
&gt;&gt; /home/msbritt/bin &nbsp; &nbsp;(arguably should be /home/msbritt)<br>
&gt;&gt; [msbritt@nyx5623 ~]$ echo $PBS_O_HOST<br>
&gt;&gt; <a href="http://nyx.engin.umich.edu" target="_blank">nyx.engin.umich.edu</a> &nbsp;(arguably should be nyx5515)<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; Should -V not read the PBS_O_* variables on job submission or at least be<br>
&gt;&gt; overwritten and correctly set in the next job, or should -V trump ?<br>
&gt;&gt;<br>
&gt;&gt; Thanks,<br>
&gt;&gt; &nbsp; - Matt<br>
&gt;&gt;<br>
&gt;&gt; --------------------------------------------<br>
&gt;&gt; Matthew Britt<br>
&gt;&gt; CAEN HPC Group - College of Engineering<br>
&gt;&gt; <a href="mailto:msbritt@umich.edu">msbritt@umich.edu</a><br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; _______________________________________________<br>
&gt;&gt; torqueusers mailing list<br>
&gt;&gt; <a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
&gt;&gt; <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">
http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
&gt; _______________________________________________<br>
&gt; torqueusers mailing list<br>
&gt; <a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
&gt; <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">
http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
<br>
_______________________________________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
</body>
</html>