<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html; charset=ISO-8859-1"
 http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
On 09/28/2010 08:57 AM, Abraham Zamudio wrote:
<blockquote
 cite="mid:AANLkTi=t1V88vq3BVBJqeOMsT0QjfgWfnN7cMhvOt1QD@mail.gmail.com"
 type="cite">
  <meta http-equiv="content-type"
 content="text/html; charset=ISO-8859-1">
  <font class="Apple-style-span" color="#eeffe2"
 face="arial, sans-serif"><span class="Apple-style-span"
 style="border-collapse: collapse;"><font class="Apple-style-span"
 color="#000000"><span class="Apple-style-span"
 style="font-size: large;">Hi everybody ,&nbsp;</span></font></span></font>
  <div><font class="Apple-style-span" color="#eeffe2"
 face="arial, sans-serif"><span class="Apple-style-span"
 style="border-collapse: collapse;"><font class="Apple-style-span"
 color="#000000"><span class="Apple-style-span"
 style="font-size: large;"><br>
  </span></font></span></font></div>
  <div><font class="Apple-style-span" color="#eeffe2"
 face="arial, sans-serif" size="4"><span class="Apple-style-span"
 style="border-collapse: collapse; font-size: 15px;"><font
 class="Apple-style-span" color="#000000"><span class="Apple-style-span"
 style="border-collapse: separate; font-size: 13.3333px; color: rgb(136, 136, 136);">
  <div id="gt-res-content" class="almost_half_cell"
 style="padding-top: 9px; padding-right: 16px;">
  <div dir="ltr" style=""><span id="result_box" class="short_text"
 style="display: block;"><span title=""><span class="Apple-style-span"
 style="background-color: rgb(255, 255, 255);"><font
 class="Apple-style-span" color="#000000"><span class="Apple-style-span"
 style="font-size: large;">I have a problem with one of my nodes :&nbsp;</span></font></span></span></span><span
 id="result_box" class="short_text" style="display: block;"><span
 title=""><span class="Apple-style-span"
 style="background-color: rgb(255, 255, 255);"><font
 class="Apple-style-span" color="#000000"><span class="Apple-style-span"
 style="font-size: large;"><br>
  </span></font></span></span></span><span id="result_box"
 class="short_text" style="display: block;"><span title=""><span
 class="Apple-style-span" style="background-color: rgb(255, 255, 255);"><font
 class="Apple-style-span" color="#000000"><span class="Apple-style-span"
 style="font-size: large;"><br>
  </span></font></span></span></span><span id="result_box"
 class="short_text" style="display: block;"><span title=""><span
 class="Apple-style-span" style="background-color: rgb(255, 255, 255);"><font
 class="Apple-style-span" color="#000000"><span class="Apple-style-span"
 style="font-size: large;"><span id="result_box" class="short_text"
 style="display: block;"><span class="Apple-style-span"
 style="font-size: x-small;"><b>[mpiX@quad2 ~]$ cat
/var/spool/torque/mom_logs/20100928 | grep 46.master</b></span></span><span
 id="result_box" class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0008; &nbsp; pbs_mom;Job;46.master;JOIN JOB as node 1</span></span><span
 id="result_box" class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0001; &nbsp; pbs_mom;Job;46.master;task not started, '/bin/sh',
stdio setup failed (see syslog)</span></span><span id="result_box"
 class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0008; &nbsp; pbs_mom;Job;46.master;ERROR: &nbsp; &nbsp;received request
'SPAWN_TASK' from <a moz-do-not-send="true"
 href="http://10.10.10.3:1023">10.10.10.3:1023</a> for job '46.master'
(cannot start task)</span></span><span id="result_box"
 class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0001; &nbsp; pbs_mom;Job;46.master;task not started, '/bin/sh',
stdio setup failed (see syslog)</span></span><span id="result_box"
 class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0008; &nbsp; pbs_mom;Job;46.master;ERROR: &nbsp; &nbsp;received request
'SPAWN_TASK' from <a moz-do-not-send="true"
 href="http://10.10.10.3:1023">10.10.10.3:1023</a> for job '46.master'
(cannot start task)</span></span><span id="result_box"
 class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0001; &nbsp; pbs_mom;Job;46.master;task not started, '/bin/sh',
stdio setup failed (see syslog)</span></span><span id="result_box"
 class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0008; &nbsp; pbs_mom;Job;46.master;ERROR: &nbsp; &nbsp;received request
'SPAWN_TASK' from <a moz-do-not-send="true"
 href="http://10.10.10.3:1023">10.10.10.3:1023</a> for job '46.master'
(cannot start task)</span></span><span id="result_box"
 class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0001; &nbsp; pbs_mom;Job;46.master;task not started, '/bin/sh',
stdio setup failed (see syslog)</span></span><span id="result_box"
 class="short_text" style="display: block;"><span
 class="Apple-style-span" style="font-size: x-small;">09/28/2010
09:29:29;0008; &nbsp; pbs_mom;Job;46.master;ERROR: &nbsp; &nbsp;received request
'SPAWN_TASK' from <a moz-do-not-send="true"
 href="http://10.10.10.3:1023">10.10.10.3:1023</a> for job '46.master'
(cannot start task)</span></span>
  <div><br>
  </div>
  <div>The status of job is active&nbsp;</div>
  <div><br>
  </div>
  <div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><b>[mpiX@master
mpi_fitting]$ showq</b></span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">ACTIVE
JOBS--------------------</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">JOBNAME
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;USERNAME &nbsp; &nbsp; &nbsp;STATE &nbsp;PROC &nbsp; REMAINING &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;STARTTIME</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">46 &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; mpiX &nbsp; &nbsp;Running &nbsp; &nbsp;12 &nbsp; &nbsp;00:35:52 &nbsp;Tue Sep 28 09:32:56</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">&nbsp;&nbsp; &nbsp;
1 Active Job &nbsp; &nbsp; &nbsp; 12 of &nbsp; 12 Processors Active (100.00%)</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">&nbsp;&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2 of &nbsp; &nbsp;2 Nodes Active &nbsp; &nbsp; &nbsp;(100.00%)</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">IDLE
JOBS----------------------</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">JOBNAME
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;USERNAME &nbsp; &nbsp; &nbsp;STATE &nbsp;PROC &nbsp; &nbsp; WCLIMIT &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;QUEUETIME</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">0
Idle Jobs</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">BLOCKED
JOBS----------------</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">JOBNAME
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;USERNAME &nbsp; &nbsp; &nbsp;STATE &nbsp;PROC &nbsp; &nbsp; WCLIMIT &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;QUEUETIME</span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;"><br>
  </span></div>
  <div><span class="Apple-style-span" style="font-size: x-small;">Total
Jobs: 1 &nbsp; Active Jobs: 1 &nbsp; Idle Jobs: 0 &nbsp; Blocked Jobs: 0</span></div>
  <div><br>
  </div>
  <div>The same software (mpich2+gsl) run on a single node of 8
cores,&nbsp;This problem occurs when two nodes use .&nbsp;</div>
  <span class="Apple-style-span" style="color: rgb(136, 136, 136);">
  <div id="gt-res-tools" class="g-section"
 style="width: 686px; vertical-align: top; display: inline-block; margin-top: 8px;"></div>
  </span></div>
  <div><br>
  </div>
  </span></font></span></span></span></div>
  </div>
  <div id="gt-res-tools" class="g-section"
 style="width: 686px; vertical-align: top; display: inline-block; margin-top: 8px;">
  <div id="gt-res-listen" tabindex="0" class="gt-icon-c"
 style="color: rgb(17, 17, 204); text-decoration: none; cursor: pointer; float: left; margin-right: 1em; outline-style: none;"></div>
  </div>
  </span></font></span></font></div>
  <div><font class="Apple-style-span" color="#eeffe2"
 face="arial, sans-serif" size="4"><span class="Apple-style-span"
 style="border-collapse: collapse; font-size: 15px;"><font
 class="Apple-style-span" color="#000000"><br>
  </font></span></font></div>
  <div><font class="Apple-style-span" color="#eeffe2"
 face="arial, sans-serif" size="4"><span class="Apple-style-span"
 style="border-collapse: collapse; font-size: 15px;"><font
 class="Apple-style-span" color="#000000"><br>
  </font></span></font>-- <br>
Abraham Zamudio Ch.<br>
  <br>
  </div>
  <pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
torqueusers mailing list
<a class="moz-txt-link-abbreviated" href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.supercluster.org/mailman/listinfo/torqueusers">http://www.supercluster.org/mailman/listinfo/torqueusers</a>
  </pre>
</blockquote>
What does qstat show? Did you look at syslog?<br>
<br>
Ken Nielson<br>
Adaptive Computing<br>
</body>
</html>