<html><body><div style="color:#000; background-color:#fff; font-family:arial, helvetica, sans-serif;font-size:10pt"><div><br></div><div><span>Hi</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>I removed one node from /var/spool/pbs/server_priv/nodes, then I ran the following command on the server</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>schedctl -k<br></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>qterm</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>pbs_server</span></div><div style="
color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>maui</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br><span></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>Now, as root, I am not able to delete jobs<br></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span><br>root@orca:/home/mahmood# qdel 93098 93111 93406<br>pbs_iff: cannot read reply from pbs_server<br>No Permission.<br>qdel: cannot connect to server hpclab.orca (errno=15007) Unauthorized Request<br>qdel: Server could not connect to MOM 93111.hpclab.orca<br><br></span></div><div
 style="color:rgb(0, 0, 0);font-size:13.3333px;
font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;">Indeed the pbs_server process is running</div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;">root@orca:/home/mahmood# ps aux | grep pbs_server<br>root&nbsp;&nbsp;&nbsp;&nbsp; 16737&nbsp; 0.0&nbsp; 0.0&nbsp; 42604&nbsp; 4100 ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; S&nbsp;&nbsp;&nbsp; <span class="yshortcuts" id="lw_1361795998_0">15:31</span>&nbsp;&nbsp; 0:00 pbs_server<br>root&nbsp;&nbsp;&nbsp;&nbsp; 21969&nbsp; 0.0&nbsp; 0.0&nbsp;&nbsp; 9384&nbsp;&nbsp; 880
 pts/1&nbsp;&nbsp;&nbsp; S+&nbsp;&nbsp; <span class="yshortcuts" id="lw_1361795998_1">15:49</span>&nbsp;&nbsp;
 0:00 grep --color=auto pbs_server<br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><br></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;">Also <span>the server log shows nothing (as far as I understand)</span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span><br></span></div><div style="color:rgb(0, 0, 0);font-size:13.3333px;font-family:arial, helvetica, sans-serif;background-color:transparent;font-style:normal;"><span>02/25/2013 15:31:17;0086;PBS_Server;Svr;PBS_Server;Shutdown request from root@hpclab.orca<br>02/25/2013 15:31:17;0086;PBS_Server;Svr;PBS_Server;Starting to shutdown the server, type is Quick<br>02/25/2013
 15:31:21;0002;PBS_Server;Svr;PBS_Server;Server shutdown completed<br>02/25/2013 15:31:21;0002;PBS_Server;Svr;Log;Log closed<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;Log;Log opened<br>02/25/2013 15:31:47;0006;PBS_Server;Svr;PBS_Server;Server hpclab.orca started, initialization type = 1<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;Act;Account file /var/spool/pbs/server_priv/accounting/20130225 opened<br>02/25/2013 15:31:47;0040;PBS_Server;Req;setup_nodes;setup_nodes()<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue orcaq<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue medium<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue small<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue very_small<br>02/25/2013 15:31:47;0086;PBS_Server;Svr;PBS_Server;Recovered queue big<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;PBS_Server;Expected 5, recovered 5 queues<br>02/25/2013
 15:31:47;0100;PBS_Server;Job;93098.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93098.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93111.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93111.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93406.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93406.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93523.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93523.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93524.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013
 15:31:47;0086;PBS_Server;Job;93524.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93536.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93536.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93536.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93605.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93605.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93607.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93607.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93608.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013
 15:31:47;0086;PBS_Server;Job;93608.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93609.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93609.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93612.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93612.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0100;PBS_Server;Job;93613.hpclab.orca;enqueuing into orcaq, state 4 hop 1<br>02/25/2013 15:31:47;0086;PBS_Server;Job;93613.hpclab.orca;Requeueing job, substate: 42 Requeued in queue: orcaq<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;PBS_Server;Expected 12, recovered 12 jobs<br>02/25/2013 15:31:47;0006;PBS_Server;Svr;PBS_Server;Using ports Server:15001&nbsp; Scheduler:15004&nbsp; MOM:15002 (server:
 'hpclab.orca')<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;daemonize_server;INFO:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parent is exiting<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;daemonize_server;INFO:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parent is exiting<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;daemonize_server;INFO:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; child process in background<br>02/25/2013 15:31:47;0002;PBS_Server;Svr;PBS_Server;Server Ready, pid = 16737, loglevel=0<br>02/25/2013 15:31:47;0004;PBS_Server;Svr;WARNING;ALERT: unable to contact node orca<br>02/25/2013 15:31:52;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 3.0.0, loglevel = 0<br>02/25/2013 15:36:52;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 3.0.0, loglevel = 0<br>02/25/2013 15:41:53;0040;PBS_Server;Svr;hpclab.orca;Scheduler was sent the command scheduler_first<br>02/25/2013 15:41:53;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 3.0.0, loglevel = 0<br>02/25/2013
 15:41:53;0080;PBS_Server;Req;dis_request_read;req header bad, dis error 7 (Premature end of message), type=Connect<br>02/25/2013
 15:41:53;0080;PBS_Server;Req;req_reject;Reject reply code=15058(Bad DIS
 based Request Protocol MSG=cannot decode message), aux=0, type=Connect,
 from @<br>02/25/2013 15:41:53;0002;PBS_Server;Req;dis_reply_write;DIS reply failure, -1<br></span></div><br><div>&nbsp;</div><div><div><font style="BACKGROUND-COLOR:#ffffff;" color="#0080ff" face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(0, 0, 0);">Regards,</span><br style="color:rgb(0, 0, 0);"><span style="color:rgb(0, 0, 0);">Mahmood</span><b><br></b></font></div></div></div></body></html>