<div>I'm happy to say that the problem is solved!</div>
<div>Indeed the core of the problem was that certain jobs, that were submitted by one of our research groups, created a higher load average than they should have. For example, in a node that has 4 CPUs, two jobs running on it made it's load average go to 3.8 or even higher. That resulted in PBS indicating that this particular node is already busy, and that it cannot accept more jobs. </div>
<div>Therefor, we could not use many free CPUs, because some nodes were reported as 'busy' while they had free CPUs doing nothing at the moment.</div>
<div> </div>
<div>The problem was solved by increasing the ideal_load and max_load values on each node. The '...has more processors utilized than dedicated' error that appeared while using the diagnose -n command, disappeared. Therefor, the problem was resolved.</div>
<div> </div>
<div>Special thanks for Jan Ploski for his tremendous help and his involvement in analyzing this problem. Thanks Jan!</div>
<div><br>Itay M.<br></div>
<div class="gmail_quote">On Sun, Feb 24, 2008 at 5:57 AM, Chris Samuel <<a href="mailto:csamuel@vpac.org" target="_blank">csamuel@vpac.org</a>> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div><br>----- "Itay M" <<a href="mailto:itaym.tau@gmail.com" target="_blank">itaym.tau@gmail.com</a>> wrote:<br><br>> I'm not sure if they are multithreaded (needs further checking with<br>> the developers) - but you're right. The load should be no more than 2<br>
> for 2 jobs, but infact its >2 . The jobs are C++ compiled with g++<br>> compiler. Maybe a compilation switch will help with reducing the load<br>> average to 1 per job?<br><br></div>There are other ways to get a high load average on<br>
a node, for instance I/O intensive jobs that cause<br>other processes to block in the D state waiting for<br>the disks will do it.<br><br>cheers,<br>Chris<br><font color="#888888">--<br>Christopher Samuel - (03) 9925 4751 - Systems Manager<br>
The Victorian Partnership for Advanced Computing<br> P.O. Box 201, Carlton South, VIC 3053, Australia<br>VPAC is a not-for-profit Registered Research Agency<br></font></blockquote></div><br>