John,<div><br></div><div>Thanks for that.  Unfortunately, that probably means I need to move back to version 3, since I can&#39;t use 4.1.  Does anyone have any idea if the hyphen-in-hostname problem is going to be fixed any time soon in 4.1?</div>

<div><br></div><div>Thanks,</div><div><br></div><div>Mike</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 2, 2012 at 6:36 PM, John Hanks <span dir="ltr">&lt;<a href="mailto:john.hanks@usu.edu" target="_blank">john.hanks@usu.edu</a>&gt;</span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Mike,<div class="gmail_extra"><br>FWIW I can confirm seeing the same behavior with pbs_server in 4.0.2. Although in our case it was loss of the same config settings every time and it was simple enough to just keep a file with the missing stuff handy to feed to qmgr after a pbs_server restart. Since we don&#39;t have the hyphen problem we&#39;re now running pbs_server from 4.1.3 (4.1.2 up until a few days ago) and pbs_mom from 4.0.2 and so far haven&#39;t seen this issue again. I moved away from 4.0.2 after a series of pbs_server crashes all on the same day for no reason that was obvious to me. <br>


<br>jbh<div><div class="h5"><br><br><div class="gmail_quote">On Fri, Nov 2, 2012 at 5:22 PM, Mike Dacre <span dir="ltr">&lt;<a href="mailto:mike.dacre@gmail.com" target="_blank">mike.dacre@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">




<div>
<div><span style="color:rgb(34,34,34);font-family:arial,sans-serif">Hi Everyone,</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">I am having a major issue I can&#39;t figure out.  When I start pbs_server I get the following error:</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">



<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">PBS_Server: LOG_ERROR::get_parent_and_</span><u></u><span style="color:rgb(34,34,34);font-family:arial,sans-serif">child,
 Cannot find closing tag</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">PBS_Server: LOG_ERROR::svr_recov_xml, Error creating attribute resources_assigned</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">I also find that and changes I make with qmgr are undone when I restart pbs_server and also pbs_server crashes when my users are using it.  There is nothing in the log, even at log level
 7, it just dies.  It seems like the server can&#39;t write to the torque home directory (/var/spool/torque).  When I start over with pbs_server -t create, the error goes away for a while.  Then after some number of restarts, the error is back.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">



<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">This is the third time this has happened, before this the queue at least restarted successfully.  This time, one of my queues just disappeared, and all of the jobs associated with it were
 deleted when the server was restarted.  This is a MAJOR problem, as it represents hours of lost time for my users.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">Part of the qmgr config disappeared.  Not all of it, just the default queue that was being used, and some of my changes to the server config.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">



<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">You can look at the attached log.  It is only log level 0, but you can see close to the top where I restarted the server and then all of this mayhem happened.  I should note that I made
 no changes to the server config before this restart.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">I am using torque 4.0.2 (I can&#39;t use 4.1.2 because I have a hyphen in my hostname which totally throws it for a loop, and jobs just don&#39;t run) with maui 3.3.1.  It was compiled with the
 following options:</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">./configure --enable-blcr --enable-docs --enable-syslog</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">The permissions of /var/spool/torque:</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-xr-x   13  root root 4.0K Oct 24 17:01 .</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-xr-x.  17  root root 4.0K Oct 23 19:20 ..</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-xr-x     2  root root 4.0K Oct 24 10:13 aux</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxrwxrwt   2  root root 4.0K Oct 23 19:20 checkpoint</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-xr-x     2  root root 4.0K Oct 23 19:20 job_logs</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-xr-x     2  root root 4.0K Oct 30 00:01 mom_logs</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-x--x     3  root root 4.0K Oct 23 19:23 mom_priv</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">-rw-r--r--        1  root root   66  Oct 23 21:07 pbs_environment</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-xr-x     2  root root 4.0K Oct 23 19:24 sched_logs</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-x---      3  root root 4.0K Oct 23 21:07 sched_priv</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-xr-x     2  root root 4.0K Oct 30 00:00 server_logs</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">-rw-r--r--        1  root root   14  Oct 23 21:07 server_name</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxr-x---    13  root root 4.0K Oct 30 20:05 server_priv</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxrwxrwt   2  root root 4.0K Oct 24 10:13 spool</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">drwxrwxrwt   2  root root 4.0K Oct 23 19:20 undelivered</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">output of qmgr -c &#39;p s&#39;:</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif"># Create queues and set their attributes.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif"># Create and define queue default</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">create queue default</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default queue_type = Execution</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default Priority = 0</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default resources_max.neednodes = slave</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default resources_default.neednodes = slave</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default resources_default.nice = 0</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default resources_available.ncpus = 160</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default resources_available.neednodes = slave</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default resources_available.nodes = 20</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default max_user_run = 100</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default enabled = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue default started = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif"># Create and define queue long</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">create queue long</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long queue_type = Execution</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long Priority = -10</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long max_running = 140</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_max.mem = 32gb</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_max.ncpus = 128</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_max.neednodes = slave</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_max.nodes = 16</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_min.cput = 02:00:01</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_default.mem = 2gb</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_default.neednodes = slave</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_default.nice = 15</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_available.mem = 600gb</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_available.ncpus = 128</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_available.neednodes = slave</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long resources_available.nodes = 16</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long enabled = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue long started = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif"># Create and define queue high_priority</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">create queue high_priority</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue high_priority queue_type = Execution</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue high_priority Priority = 10000</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue high_priority resources_max.walltime = 56:00:00</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue high_priority resources_default.nice = -10</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue high_priority resources_default.walltime = 48:00:00</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue high_priority enabled = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set queue high_priority started = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif"># Set server attributes.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">#</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server acl_hosts = fraser-server</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server default_queue = default</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server log_events = 511</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server mail_from = adm</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server query_other_jobs = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server resources_available.mem = 625gb</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server resources_default.mem = 4gb</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server scheduler_iteration = 600</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server node_check_rate = 150</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server tcp_timeout = 300</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server job_stat_rate = 45</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server poll_jobs = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server mom_job_sync = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server allow_node_submit = True</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server next_job_number = 3301</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif">set server moab_array_compatible = True</span><span><font color="#888888"><br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-family:arial,sans-serif">
<font color="#222222" face="arial, sans-serif">-Mike</font> </font></span></div>
<div></div>
</div>

</blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
<br></blockquote></div><br></div>