Thx for your comments . <div><br></div><div>Troy, torque run my script with  your modifications  , now in the output files ( mpidata.$PBS_JOBID.$FILE ) i have the following error  : </div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><b>mpiexec: Warning: task 0 died with signal 11 (Segmentation fault).</b></div></div><div><div><b>mpiexec: Warning: tasks 1-11 died with signal 15 (Terminated).</b></div></div></blockquote><div><br></div><div>The log of my nodes : </div>
<div><br></div><div><div><span class="Apple-style-span" style="font-size: x-small;"><b>cat /var/spool/torque/mom_logs/20100929 | grep 1040.master</b></span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:17;0001;   pbs_mom;Job;TMomFinalizeJob3;job 1040.master started, pid = 29017</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:17;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 2, sid 29065, cmd /bin/sh</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:17;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 3, sid 29066, cmd /bin/sh</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:17;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 4, sid 29067, cmd /bin/sh</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:17;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 5, sid 29068, cmd /bin/sh</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:18;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 2 terminated, sid=29065</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:23;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 3 signal 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:23;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 29066 task 3 gracefully with sig 15</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:28;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=29066/state=Z) with sig 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:28;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 4 signal 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:28;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 29067 task 4 gracefully with sig 15</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:33;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=29067/state=Z) with sig 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:33;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 5 signal 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:33;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 29068 task 5 gracefully with sig 15</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:38;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=29068/state=Z) with sig 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:38;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 3 terminated, sid=29066</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:38;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 4 terminated, sid=29067</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:38;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 5 terminated, sid=29068</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:03:02;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 29018 task 1 gracefully with sig 15</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:03:02;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 1 terminated, sid=29017</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:03:02;0008;   pbs_mom;Job;1040.master;job was terminated</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:03:02;0080;   pbs_mom;Job;1040.master;obit sent to server</span></div><div><br></div><div><br></div><div><br></div><div><div><span class="Apple-style-span" style="font-size: x-small;"><b>[mpiX@quad4 ~]$ cat /var/spool/torque/mom_logs/20100929 | grep 1040.master</b></span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:07;0008;   pbs_mom;Job;1040.master;JOIN JOB as node 1</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 6, sid 9232, cmd /bin/sh</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 7, sid 9233, cmd /bin/sh</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 8, sid 9234, cmd /bin/sh</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 9, sid 9235, cmd /bin/sh</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 10, sid 9236, cmd /bin/sh</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 11, sid 9237, cmd /bin/sh</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 12, sid 9238, cmd /bin/sh</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:08;0008;   pbs_mom;Job;1040.master;start_process: task started, tid 13, sid 9239, cmd /bin/sh</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:14;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 6 signal 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:14;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9232 task 6 gracefully with sig 15</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:19;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9232/state=Z) with sig 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:19;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 7 signal 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:19;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9233 task 7 gracefully with sig 15</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:23;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9233/state=Z) with sig 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:23;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 8 signal 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:23;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9234 task 8 gracefully with sig 15</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:28;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9234/state=Z) with sig 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:28;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 9 signal 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:28;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9235 task 9 gracefully with sig 15</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:33;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9235/state=Z) with sig 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:33;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 10 signal 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:33;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9236 task 10 gracefully with sig 15</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:38;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9236/state=Z) with sig 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:38;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 11 signal 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:38;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9237 task 11 gracefully with sig 15</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:42;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9237/state=Z) with sig 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:42;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 12 signal 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:42;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9238 task 12 gracefully with sig 15</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:47;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9238/state=Z) with sig 9</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:47;0008;   pbs_mom;Job;1040.master;im_request: SIGNAL_TASK 1040.master from node 0 task 13 signal 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:47;0008;   pbs_mom;Job;1040.master;kill_task: killing pid 9239 task 13 gracefully with sig 15</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0008;   pbs_mom;Job;1040.master;kill_task: not killing process (pid=9239/state=Z) with sig 9</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 6 terminated, sid=9232</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 7 terminated, sid=9233</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 8 terminated, sid=9234</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 9 terminated, sid=9235</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 10 terminated, sid=9236</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 11 terminated, sid=9237</span></div><div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 12 terminated, sid=9238</span></div>
<div><span class="Apple-style-span" style="font-size: x-small;">09/29/2010 18:02:52;0080;   pbs_mom;Job;1040.master;scan_for_terminated: job 1040.master task 13 terminated, sid=9239</span></div></div><div><br></div><div><br>
</div><div><br></div><div class="gmail_quote">On Thu, Sep 30, 2010 at 8:26 AM, Glen Beane <span dir="ltr">&lt;<a href="mailto:glen.beane@gmail.com">glen.beane@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5">On Wed, Sep 29, 2010 at 3:42 PM, Troy Baer &lt;<a href="mailto:tbaer@utk.edu">tbaer@utk.edu</a>&gt; wrote:<br>
&gt; On Wed, 2010-09-29 at 14:13 -0500, Abraham Zamudio wrote:<br>
&gt;&gt; I have a mpich2 program , This program takes one ( argv[1] ) argument<br>
&gt;&gt; (  ./program    file_to_analyze ) .<br>
&gt;&gt;<br>
&gt;&gt; I send him to the queue  of torque<br>
&gt;<br>
&gt;&gt; #####################<br>
&gt;&gt; #### run_all_files.sh ####<br>
&gt;&gt; #####################<br>
&gt;&gt; $FOLDER = /path/to/files<br>
&gt;&gt; for i in $(ls $FOLDER ); do<br>
&gt;&gt;     qsub cola.qsub $i<br>
&gt;&gt; done<br>
&gt;&gt; #####################<br>
&gt;<br>
&gt;&gt; #################<br>
&gt;&gt; #### cola.qsub ####<br>
&gt;&gt; #################<br>
&gt;&gt; #PBS -S /bin/bash<br>
&gt;&gt; #PBS -N proof<br>
&gt;&gt; #PBS -q queue_2<br>
&gt;&gt; #PBS -l nodes=Four_processors:ppn=4+Eight_processors:ppn=8<br>
&gt;&gt; #PBS -j oe<br>
&gt;&gt; #PBS -o cola.$PBS_JOBID.$1<br>
&gt;&gt;<br>
&gt;&gt; mpiexec /PATH/TO/MPI_SOFTWARE/program   $1<br>
&gt;&gt; #################<br>
&gt;<br>
&gt; That&#39;s not how qsub processes its command line arguments.  Setting an<br>
&gt; environment variable that gets propagated into the jobs using the -v<br>
&gt; flag to qsub might work, though:<br>
&gt;<br>
&gt; ########################<br>
&gt; ### run_all_files.sh ###<br>
&gt; ########################<br>
&gt; $FOLDER = /path/to/files<br>
&gt; for i in $(ls $FOLDER )<br>
&gt; do<br>
&gt;    qsub -v FILE=$i cola.qsub<br>
&gt; done<br>
&gt;<br>
&gt; #################<br>
&gt; ### cola.qsub ###<br>
&gt; #################<br>
&gt; #PBS -S /bin/bash<br>
&gt; #PBS -N proof<br>
&gt; #PBS -q queue_2<br>
&gt; #PBS -l nodes=Four_processors:ppn=4+Eight_processors:ppn=8<br>
&gt; #PBS -j oe<br>
&gt; #PBS -o cola.$PBS_JOBID.$FILE<br>
&gt; mpiexec /PATH/TO/MPI_SOFTWARE/program $FILE<br>
&gt;<br>
&gt; Do environment variable macro substitutions work in the arguments to the<br>
&gt; -e and -o flags?  (I was under the impression that they didn&#39;t.)<br>
<br>
</div></div>torque will use wordexp to expand shell variables in the -o and -e<br>
arguments, so your example should work provided wordexp was found by<br>
./configure<br>
<div><div></div><div class="h5">_______________________________________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Abraham Zamudio Ch.<br><br>
</div>