<html><body><div style="color:#000; background-color:#fff; font-family:arial, helvetica, sans-serif;font-size:10pt"><div><span>Excuse me, let me correct something.</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span>As root there is no problem. There were some jobs on the removed node, so by running "qdel -p" they are now removed from qstat.</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span>Problem is, I, as a user can not run qstat</span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color:
 transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span>mahmood@orca:~$ qstat<br>pbs_iff: cannot read reply from pbs_server<br>No Permission.<br>qstat: cannot connect to server hpclab.orca (errno=15007) Unauthorized Request<br></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><br><span></span></div><div style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: arial,helvetica,sans-serif; background-color: transparent; font-style: normal;"><span>Before removing the node, I was able to run the command.
 <br></span></div><div>&nbsp;</div><div><div><font style="BACKGROUND-COLOR:#ffffff;" color="#0080ff" face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(0, 0, 0);">Regards,</span><br style="color:rgb(0, 0, 0);"><span style="color:rgb(0, 0, 0);">Mahmood</span><b><br></b></font></div></div><div><br></div>  <div style="font-family: arial, helvetica, sans-serif; font-size: 10pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1">  <b><span style="font-weight:bold;">From:</span></b> Mahmood Naderan &lt;nt_mahmood@yahoo.com&gt;<br> <b><span style="font-weight: bold;">To:</span></b> torque cluster &lt;torqueusers@supercluster.org&gt; <br> <b><span style="font-weight: bold;">Sent:</span></b> Monday, February 25, 2013 4:12 PM<br> <b><span style="font-weight: bold;">Subject:</span></b> [torqueusers] pbs_iff: cannot read reply from pbs_server<br> </font>
 </div> <br>
<div id="yiv1247819471"><div><div style="color:#000;background-color:#fff;font-family:arial, helvetica, sans-serif;font-size:10pt;"><div><br></div><div><span>Hi</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>I removed one node from /var/spool/pbs/server_priv/nodes, then I ran the following command on the server</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>schedctl -k<br></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>qterm</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>pbs_server</span></div><div
 style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>maui</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br><span></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>Now, as root, I am not able to delete jobs<br></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span><br>root@orca:/home/mahmood# qdel 93098 93111 93406<br>pbs_iff: cannot read reply from pbs_server<br>No Permission.<br>qdel: cannot connect to server hpclab.orca (errno=15007) Unauthorized Request<br>qdel: Server could not connect to MOM 93111.hpclab.orca<br><br></span></div><div
 style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;">Indeed the pbs_server process is running</div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;">root@orca:/home/mahmood# ps aux | grep pbs_server<br>root&nbsp;&nbsp;&nbsp;&nbsp; 16737&nbsp; 0.0&nbsp; 0.0&nbsp; 42604&nbsp; 4100 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; S&nbsp;&nbsp;&nbsp; <span class="yiv1247819471yshortcuts" id="yiv1247819471lw_1361795998_0">15:31</span>&nbsp;&nbsp; 0:00 pbs_server<br>root&nbsp;&nbsp;&nbsp;&nbsp;
 21969&nbsp; 0.0&nbsp; 0.0&nbsp;&nbsp; 9384&nbsp;&nbsp; 880
 pts/1&nbsp;&nbsp;&nbsp; S+&nbsp;&nbsp; <span class="yiv1247819471yshortcuts" id="yiv1247819471lw_1361795998_1">15:49</span>&nbsp;&nbsp;
 0:00 grep --color=auto pbs_server<br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;">Also <span>the server log shows nothing (as far as I understand)</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span><br></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>02/25/2013 15:31:17;0086;PBS_Server;Svr;PBS_Server;Shutdown request from root@hpclab.orca<br>02/25/2013 15:31:17;0086;PBS_Server;Svr;PBS_Server;Starting to shutdown the server, type is Quick<br>02/25/2013
 15:31:21;0002;PBS_Server;Svr;PBS_Server;Server shutdown completed<br>02/25/2013 15:31:21;0002;PBS_Server;Svr;Log;Log closed<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;Log;Log opened<br>02/25/2013 15:31:47;0006;PBS_Server;Svr;PBS_Server;Server hpclab.orca started, initialization type = 1<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;Act;Account file /var/spool/pbs/server_priv/accounting/20130225 opened<br>02/25/2013 15:31:47;0040;PBS_Server;Req;setup_nodes;setup_nodes()<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue orcaq<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue medium<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue small<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue very_small<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue big<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;PBS_Server;Expected 5, recovered 5 queues<br>02/25/2013
 15:31:47;0100;PBS_Server;Job;93098.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93098.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93111.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93111.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93406.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93406.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93523.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93523.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93524.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013
 15:31:47;0086;PBS_Server;Job;93524.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93536.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93536.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93536.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93605.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93605.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93607.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93607.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93608.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013
 15:31:47;0086;PBS_Server;Job;93608.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93609.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93609.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93612.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93612.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93613.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93613.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;PBS_Server;Expected 12, recovered 12 jobs<br>02/25/2013 15:31:47;0006;PBS_Server;Svr;PBS_Server;Using ports Server:15001&nbsp; Scheduler:15004&nbsp; MOM:15002 (server:
 'hpclab.orca')<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;daemonize_server;INFO:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parent is exiting<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;daemonize_server;INFO:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parent is exiting<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;daemonize_server;INFO:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; child process in background<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;PBS_Server;Server Ready, pid = 16737, loglevel=0<br>02/25/2013 15:31:47;0004;PBS_Server;Svr;WARNING;ALERT: unable to contact node orca<br>02/25/2013 15:31:52;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 3.0.0, loglevel = 0<br>02/25/2013 15:36:52;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 3.0.0, loglevel = 0<br>02/25/2013 15:41:53;0040;PBS_Server;Svr;hpclab.orca;Scheduler was sent the command scheduler_first<br>02/25/2013 15:41:53;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 3.0.0, loglevel = 0<br>02/25/2013
 15:41:53;0080;PBS_Server;Req;dis_request_read;req header bad, dis error 7 (Premature end of message), type=Connect<br>02/25/2013
 15:41:53;0080;PBS_Server;Req;req_reject;Reject reply code=15058(Bad DIS
 based Request Protocol MSG=cannot decode message), aux=0, type=Connect,
 from @<br>02/25/2013 15:41:53;0002;PBS_Server;Req;dis_reply_write;DIS reply failure, -1<br></span></div><br><div>&nbsp;</div><div><div><font style="BACKGROUND-COLOR:#ffffff;" color="#0080ff" face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(0, 0, 0);">Regards,</span><br style="color:rgb(0, 0, 0);"><span style="color:rgb(0, 0, 0);">Mahmood</span><b><br></b></font></div></div></div></div></div><br>_______________________________________________<br>torqueusers mailing list<br><a ymailto="mailto:torqueusers@supercluster.org" href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br><a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br><br><br> </div> </div>  </div></body></html>