<div dir="ltr">Do your systems have multiple interfaces.If so then check if same network is used for communication.<br><br><div class="gmail_quote">On Fri, Aug 29, 2008 at 6:06 AM, scoggins <span dir="ltr"><<a href="mailto:jscoggins@lbl.gov">jscoggins@lbl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Torque 2.1.3 problem:<br>
<br>
I am getting the following error message when I qsub a job:<br>
<br>
Message[0] job cannot be started on RM sched-00 - cannot set hostlist: cannot set job '98.sched-00 ' attr 'Resource_List:neednodes' to 'n0000.ikea:ppn=4+n0001.ikea:ppn=4' (rc: 15070 'Server could not connect to MOM')<br>
<br>
<br>
I can not figure out why.<br>
<br>
I ran pbs_iff -t n0000.ikea 15002 and I get the following error:<br>
...<br>
<br>
<br>
poll([{fd=3, events=POLLIN|POLLHUP, revents=POLLIN}], 1, 20000) = 1<br>
fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)<br>
read(3, "+2+15+15005+0+72+41Unknown reque"..., 262144) = 60<br>
write(2, "pbs_iff: Unknown request MSG=can"..., 51pbs_iff: Unknown request MSG=cannot decode message<br>
) = 51<br>
exit_group(1) = ?<br>
<br>
<br>
<br>
PBS commands output:<br>
<br>
pbsnodes -a n0000.ikea<br>
<br>
n0000.ikea<br>
state = free<br>
np = 8<br>
properties = ikea,quadcore<br>
ntype = cluster<br>
status = opsys=linux,uname=Linux n0000.ikea 2.6.18-92.1.10.el5 #1 SMP Tue Aug 5 07:42:41 EDT 2008 x86_64,sessions=? 0,nsessions=? 0,nusers=0,idletime=784246,totmem=48453372kb,availmem=48342900kb,physmem=16443868kb,ncpus=8,loadave=0.00,netload=98910831,state=free,jobs=,varattr=,rectime=1219968506<br>
<br>
<br>
momctl -h n0000.ikea -d 9<br>
<br>
Host: n0000.ikea/n0000.ikea Version: 2.3.1 PID: 27784<br>
Server[0]: sched-00 (<a href="http://10.0.0.30:15001" target="_blank">10.0.0.30:15001</a>)<br>
Init Msgs Received: 2 hellos/2 cluster-addrs<br>
Init Msgs Sent: 3 hellos<br>
Last Msg From Server: 2620 seconds (CLUSTER_ADDRS)<br>
Last Msg To Server: 10 seconds<br>
HomeDirectory: /var/spool/torque/ikea/n0000/mom_priv<br>
stdout/stderr spool directory: '/var/spool/torque/ikea/n0000/spool/' (3472979 blocks available)<br>
NOTE: syslog enabled<br>
HomeDirectory: /var/spool/torque/ikea/n0000/mom_priv<br>
MOM active: 6142 seconds<br>
Server Update Interval: 45 seconds<br>
LogLevel: 0 (use SIGUSR1/SIGUSR2 to adjust)<br>
Communication Model: RPP<br>
MemLocked: TRUE (mlock)<br>
TCP Timeout: 20 seconds<br>
Prolog: /var/spool/torque/ikea/n0000/mom_priv/prologue (disabled)<br>
Alarm Time: 0 of 10 seconds<br>
Trusted Client List: <a href="http://10.0.2.9" target="_blank">10.0.2.9</a>,<a href="http://10.0.2.7" target="_blank">10.0.2.7</a>,<a href="http://10.0.2.6" target="_blank">10.0.2.6</a>,<a href="http://10.0.2.5" target="_blank">10.0.2.5</a>,<a href="http://10.0.2.4" target="_blank">10.0.2.4</a>,<a href="http://10.0.2.3" target="_blank">10.0.2.3</a>,<a href="http://10.0.2.2" target="_blank">10.0.2.2</a>,<a href="http://10.0.2.1" target="_blank">10.0.2.1</a>,<a href="http://10.0.2.0" target="_blank">10.0.2.0</a>,<a href="http://10.0.0.30" target="_blank">10.0.0.30</a>,<a href="http://10.0.7.18" target="_blank">10.0.7.18</a>,<a href="http://10.0.7.17" target="_blank">10.0.7.17</a>,<a href="http://10.0.7.16" target="_blank">10.0.7.16</a>,<a href="http://10.0.7.15" target="_blank">10.0.7.15</a>,<a href="http://10.0.7.14" target="_blank">10.0.7.14</a>,<a href="http://10.0.7.13" target="_blank">10.0.7.13</a>,<a href="http://10.0.7.12" target="_blank">10.0.7.12</a>,<a href="http://10.0.7.11" target="_blank">10.0.7.11</a>,<a href="http://10.0.7.10" target="_blank">10.0.7.10</a>,<a href="http://10.0.7.9" target="_blank">10.0.7.9</a>,<a href="http://10.0.7.8" target="_blank">10.0.7.8</a>,<a href="http://10.0.7.7" target="_blank">10.0.7.7</a>,<a href="http://10.0.7.6" target="_blank">10.0.7.6</a>,<a href="http://10.0.7.5" target="_blank">10.0.7.5</a>,<a href="http://10.0.7.4" target="_blank">10.0.7.4</a>,<a href="http://10.0.7.3" target="_blank">10.0.7.3</a>,<a href="http://10.0.7.2" target="_blank">10.0.7.2</a>,<a href="http://10.0.7.1" target="_blank">10.0.7.1</a>,<a href="http://10.0.7.0" target="_blank">10.0.7.0</a>,<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><br>
Copy Command: /usr/bin/scp -rpB<br>
NOTE: no local jobs detected<br>
<br>
diagnostics complete<br>
<br>
Here is what the server_logs are saying:<br>
<br>
08/28/2008 17:33:09;0001;PBS_Server;Req;;Server could not connect to MOM<br>
08/28/2008 17:33:09;0080;PBS_Server;Req;req_reject;Reject reply code=15070(Server could not connect to MOM), aux=0, type=ModifyJob, from root@sched-00<br>
08/28/2008 17:33:09;0008;PBS_Server;Job;101.sched-00;Job Modified at request of root@sched-00<br>
<br>
<br>
<br>
Jobs stay queued and checkjob shows:<br>
<br>
BLOCK MSG: job hold active - Batch (recorded at last scheduling iteration)<br>
Message[0] job cannot be started on RM <a href="http://sched-00.scs.lbl.gov" target="_blank">sched-00.scs.lbl.gov</a> - cannot set hostlist: cannot set job '101.sched-00' attr 'Resource_List:neednodes' to 'n0000.ikea:ppn=4+n0001.ikea:ppn=4' (rc: 15070 'Server could not connect to MOM')<br>
<br>
Message[1] cannot start job on reserved resources - job cannot be started on RM sched-00 - cannot set hostlist: cannot set job '101.sched-00' attr 'Resource_List:neednodes' to 'n0000.ikea:ppn=4+n0001.ikea:ppn=4' (rc: 15070 'Server could not connect to MOM')<br>
<br>
Any help would be much appreciated.<br>
<br>
Thanks<br>
<br>
Jackie<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Regards--<br>Rishi Pathak<br>Pune-Maharastra<br>
</div>