Bugfix: Re: [Mauiusers] Bug in memory limit enforcement after maui restart

[Martin Kleinschmidt]

a while ago, I sent a question about a bug in maui. my original post can
be found below as a fullquote - hope that's OK...

yesterday I found a bit time to dig into the maui sources.
The problem ist that in iteration 1, the total amount of memory used by
a job is compared to the amount of memory dedicated to a SINGLE
task of the job. In all following iterations, the total amount of memory
used is divided by the number of tasks before comparison.
So only in iteration 1, jobs are wrongly cancelled.

I did not have time enough to search for the reason leading to this
error, but found a simnple workaround:

in src/moab/MSys.c replace

  MLimitEnforceAll(&MPar[0]);

 by

  if ((MSched.Iteration != 0) && (MSched.Reload == TRUE)) {

      MLimitEnforceAll(&MPar[0]);

  }

hoping this might help someone...


   ...martin


On Do, 20 Dez 2007, Martin Kleinschmidt wrote:

>there seems to be a bug in the memory limit enforcement procedure. I
>have been testing for a while now, why sometimes jobs die when
>restarting maui.
>
>my example job was submitted (via torque) with
>-l nodes=1:4
>-l mem=5000mb
>
>it runs without problem, but when restarting maui it is killed and by
>setting the loglevel to 255 I finally found:
>
>12/20 15:31:06 INFO:     job 3369 exceeds requested memory limit (3658 >
>1250)
>12/20 15:31:06 MSysRegEvent(JOBRESVIOLATION:  job '3369' in state
>'Running' has exceeded MEM resource limit (3658 > 1250) (action CANCEL
>will be taken)  job start time: Thu Dec 20 15:29:34
>,0,0,1)
>
>so the total memory usage is roported to be 3658 out of 5000 mb (which
>agrees with what it is really using) , but this value is then compared
>to 1250 which is the limit per task (5000/4=1250).
>This leads to a cencellation of the job.
>
>The maui version is maui-3.2.6p19
>
>   ...martin
>
>
>ENFORCERESOURCELIMITS ON
>RESOURCELIMITPOLICY[0] MEM:ALWAYS:CANCEL