<div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial"><DIV>I've just installed torque and maui on a HP blade system. we have 16 nodes, each has 2 xeon e5620 processors. Both serial and parallel jobs within a single node can be successfully submited and run perfectly. However, if I set </DIV>
<DIV> </DIV>
<DIV><EM> #PBS -l nodes=X(X larger than 1):ppn </EM></DIV>
<DIV><EM></EM> </DIV>
<DIV>I can see the job in R status with qstat command, but it is not running acctually. After canceling the job, I get following error message:</DIV>
<DIV> </DIV>
<DIV><EM> [mpiexec@node1] HYD_pmcd_pmiserv_send_signal (./pm/mpiserv/mpiserv_cb.c:184): assert (!closed) failed</EM></DIV>
<DIV><EM> [mpiexec@node1] ui_cmd_cb (./pm/pmiserv/pmiserv_pmci.c:74): unable to send SIGUSR1 downstream</EM></DIV>
<DIV><EM> [mpiexec@node1] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status</EM></DIV>
<DIV><EM> [mpiexec@node1] HYD_pmci_wait_for_completion (./pm/pmserv/pmiserv_pmci.c:179): error waiting for event</EM></DIV>
<DIV><EM> [mpiexec@node1] main (./ui/mpich/mpiexec.c:397): process manager error waiting for completion</EM></DIV>
<DIV><EM> </EM></DIV>
<DIV> I've also found that the $PBS_NODEFILE(e.g. JOBID.node1 file in /var/spool/torque/aux ) exists only on the first node among the nodes assign for this jobs.</DIV>
<DIV> Further more, If I replace the $PBS_NODEFILE with a local file containing computing nodes in PBS script, it works well and job can be run on all the nodes assigned:</DIV>
<DIV><EM>#!/bin/sh</EM></DIV>
<DIV><EM>#PBS -N name</EM></DIV>
<DIV><EM>#PBS -e errorfile</EM></DIV>
<DIV><EM>#PBS -o outfile</EM></DIV>
<DIV><EM>#PBS -q test</EM></DIV>
<DIV><EM>#PBS -l nodes=2</EM></DIV>
<DIV><EM>cd $work_dir</EM></DIV>
<DIV><EM>#mpiexec -f $PBS_NODEFILE ./executables....</EM></DIV>
<DIV><EM>mpiexec -f hosts ./executables....</EM></DIV>
<DIV><EM></EM> </DIV>
<DIV>hosts file:</DIV>
<DIV>node1</DIV>
<DIV>node1</DIV>
<DIV>node1</DIV>
<DIV>node1</DIV>
<DIV>node2</DIV>
<DIV>node2</DIV>
<DIV>node2</DIV>
<DIV>node2 </DIV>
<DIV><EM></EM> </DIV>
<DIV>Interestingly, if I add :ppn=4 after the #PBS -l nodes=2, i.e. </DIV>
<DIV> #PBS -l nodes=2:ppn=4</DIV>
<DIV>The PBS script fails again even if I use local host file.</DIV>
<DIV> </DIV>
<DIV>Can anyone help me?</DIV>
<DIV><EM></EM> </DIV>
<DIV><EM></EM> </DIV>
<DIV><EM></EM> </DIV>
<DIV><EM></EM> </DIV>
<DIV><EM></EM> </DIV>
<DIV><EM></EM> </DIV>
<DIV><EM></EM> </DIV>
<DIV><EM></EM> </DIV>
<DIV> </DIV></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>