<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
There are multiple interfaces but only one is connected to the network.<div><br></div><div>Thanks</div><div><br></div><div>Jackie</div><div><br></div><div><br></div><div><div><div>On Aug 29, 2008, at 1:29 AM, rishi pathak wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">Do your systems have multiple interfaces.If so then check if same network is used for communication.<br><br><div class="gmail_quote">On Fri, Aug 29, 2008 at 6:06 AM, scoggins <span dir="ltr">&lt;<a href="mailto:jscoggins@lbl.gov">jscoggins@lbl.gov</a>></span> wrote:<br> <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Torque 2.1.3 problem:<br> <br> I am getting the following error message when I qsub a job:<br> <br> Message[0] job cannot be started on RM sched-00  - cannot set hostlist: cannot set job '98.sched-00 ' attr 'Resource_List:neednodes' to 'n0000.ikea:ppn=4+n0001.ikea:ppn=4' (rc: 15070 'Server could not connect to MOM')<br> <br> <br> I can not figure out why.<br> <br> I ran pbs_iff -t n0000.ikea 15002 and I get the following error:<br> ...<br> <br> <br> poll([{fd=3, events=POLLIN|POLLHUP, revents=POLLIN}], 1, 20000) = 1<br> fcntl(3, F_GETFL)                       = 0x802 (flags O_RDWR|O_NONBLOCK)<br> read(3, "+2+15+15005+0+72+41Unknown reque"..., 262144) = 60<br> write(2, "pbs_iff: Unknown request MSG=can"..., 51pbs_iff: Unknown request MSG=cannot decode message<br> ) = 51<br> exit_group(1)                           = ?<br> <br> <br> <br> PBS commands output:<br> <br> pbsnodes -a n0000.ikea<br> <br> n0000.ikea<br>     state = free<br>     np = 8<br>     properties = ikea,quadcore<br>     ntype = cluster<br>     status = opsys=linux,uname=Linux n0000.ikea 2.6.18-92.1.10.el5 #1 SMP Tue Aug 5 07:42:41 EDT 2008 x86_64,sessions=? 0,nsessions=? 0,nusers=0,idletime=784246,totmem=48453372kb,availmem=48342900kb,physmem=16443868kb,ncpus=8,loadave=0.00,netload=98910831,state=free,jobs=,varattr=,rectime=1219968506<br> <br> <br> momctl -h n0000.ikea -d 9<br> <br> Host: n0000.ikea/n0000.ikea   Version: 2.3.1   PID: 27784<br> Server[0]: sched-00 (<a href="http://10.0.0.30:15001" target="_blank">10.0.0.30:15001</a>)<br>  Init Msgs Received:     2 hellos/2 cluster-addrs<br>  Init Msgs Sent:         3 hellos<br>  Last Msg From Server:   2620 seconds (CLUSTER_ADDRS)<br>  Last Msg To Server:     10 seconds<br> HomeDirectory:          /var/spool/torque/ikea/n0000/mom_priv<br> stdout/stderr spool directory: '/var/spool/torque/ikea/n0000/spool/' (3472979 blocks available)<br> NOTE:  syslog enabled<br> HomeDirectory:          /var/spool/torque/ikea/n0000/mom_priv<br> MOM active:             6142 seconds<br> Server Update Interval: 45 seconds<br> LogLevel:               0 (use SIGUSR1/SIGUSR2 to adjust)<br> Communication Model:    RPP<br> MemLocked:              TRUE  (mlock)<br> TCP Timeout:            20 seconds<br> Prolog:                 /var/spool/torque/ikea/n0000/mom_priv/prologue (disabled)<br> Alarm Time:             0 of 10 seconds<br> Trusted Client List:    <a href="http://10.0.2.9" target="_blank">10.0.2.9</a>,<a href="http://10.0.2.7" target="_blank">10.0.2.7</a>,<a href="http://10.0.2.6" target="_blank">10.0.2.6</a>,<a href="http://10.0.2.5" target="_blank">10.0.2.5</a>,<a href="http://10.0.2.4" target="_blank">10.0.2.4</a>,<a href="http://10.0.2.3" target="_blank">10.0.2.3</a>,<a href="http://10.0.2.2" target="_blank">10.0.2.2</a>,<a href="http://10.0.2.1" target="_blank">10.0.2.1</a>,<a href="http://10.0.2.0" target="_blank">10.0.2.0</a>,<a href="http://10.0.0.30" target="_blank">10.0.0.30</a>,<a href="http://10.0.7.18" target="_blank">10.0.7.18</a>,<a href="http://10.0.7.17" target="_blank">10.0.7.17</a>,<a href="http://10.0.7.16" target="_blank">10.0.7.16</a>,<a href="http://10.0.7.15" target="_blank">10.0.7.15</a>,<a href="http://10.0.7.14" target="_blank">10.0.7.14</a>,<a href="http://10.0.7.13" target="_blank">10.0.7.13</a>,<a href="http://10.0.7.12" target="_blank">10.0.7.12</a>,<a href="http://10.0.7.11" target="_blank">10.0.7.11</a>,<a href="http://10.0.7.10" target="_blank">10.0.7.10</a>,<a href="http://10.0.7.9" target="_blank">10.0.7.9</a>,<a href="http://10.0.7.8" target="_blank">10.0.7.8</a>,<a href="http://10.0.7.7" target="_blank">10.0.7.7</a>,<a href="http://10.0.7.6" target="_blank">10.0.7.6</a>,<a href="http://10.0.7.5" target="_blank">10.0.7.5</a>,<a href="http://10.0.7.4" target="_blank">10.0.7.4</a>,<a href="http://10.0.7.3" target="_blank">10.0.7.3</a>,<a href="http://10.0.7.2" target="_blank">10.0.7.2</a>,<a href="http://10.0.7.1" target="_blank">10.0.7.1</a>,<a href="http://10.0.7.0" target="_blank">10.0.7.0</a>,<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><br> Copy Command:           /usr/bin/scp -rpB<br> NOTE:  no local jobs detected<br> <br> diagnostics complete<br> <br> Here is what the server_logs are saying:<br> <br> 08/28/2008 17:33:09;0001;PBS_Server;Req;;Server could not connect to MOM<br> 08/28/2008 17:33:09;0080;PBS_Server;Req;req_reject;Reject reply code=15070(Server could not connect to MOM), aux=0, type=ModifyJob, from root@sched-00<br> 08/28/2008 17:33:09;0008;PBS_Server;Job;101.sched-00;Job Modified at request of root@sched-00<br> <br> <br> <br> Jobs stay queued and checkjob shows:<br> <br> BLOCK MSG: job hold active - Batch (recorded at last scheduling iteration)<br> Message[0] job cannot be started on RM <a href="http://sched-00.scs.lbl.gov" target="_blank">sched-00.scs.lbl.gov</a> - cannot set hostlist: cannot set job '101.sched-00' attr 'Resource_List:neednodes' to 'n0000.ikea:ppn=4+n0001.ikea:ppn=4' (rc: 15070 'Server could not connect to MOM')<br> <br> Message[1] cannot start job on reserved resources - job cannot be started on RM sched-00 - cannot set hostlist: cannot set job '101.sched-00' attr 'Resource_List:neednodes' to 'n0000.ikea:ppn=4+n0001.ikea:ppn=4' (rc: 15070 'Server could not connect to MOM')<br> <br> Any help would be much appreciated.<br> <br> Thanks<br> <br> Jackie<br> <br> <br> <br> <br> <br> _______________________________________________<br> torqueusers mailing list<br> <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br> <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br> </blockquote></div><br><br clear="all"><br>-- <br>Regards--<br>Rishi Pathak<br>Pune-Maharastra<br> </div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">_______________________________________________</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">torqueusers mailing list</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a href="http://www.supercluster.org/mailman/listinfo/torqueusers">http://www.supercluster.org/mailman/listinfo/torqueusers</a></div> </blockquote></div><br></div></body></html>