[Mauiusers] configuring disk space

[Kevin Van Workum]

Dave,

Yes, that is what I want. Thanks.

Also, if TORQUE is trying to set the file ulimit, then the following
in pbs_mom.8 is rather misleading and should probably be updated.

       size[fs=<FS>]
              Specifies  that  the  available and configured disk space in the
              <FS> filesystem is to be reported to the pbs_server  and  sched-
              uler.   NOTE:  To request disk space on a per job basis, specify
              the file resource as in 'qsub -l nodes=1,file=1000kb'  For exam-
              ple,   the   available   and   configured   disk  space  in  the
              /localscratch filesystem will be reported:

              size[fs=/localscratch]

Kevin

On 2/5/07, Dave Jackson <jacksond at clusterresources.com> wrote:
> Kevin,
>
>   The failure in TORQUE is occurring because TORQUE is trying to set the
> file ulimit.  My guess is that this is not what you want.  You want Maui
> to schedule diskspace as a consumable resource but not enforce anything
> at the OS level?  Is this correct?  Moab supports a 'ddisk' (dedicated
> disk) RM extension which is a disk constraint enforced by the scheduler
> as a consumable resource but not enforced via ulimits, ie,
>
> > qsub -l ddisk=4096mb,walltime=1000 trestjob.cmd
>
>   If this is what you want, I think we could roll that into Maui without
> any trouble.
>
> Dave
>
>
>
> On Mon, 2007-02-05 at 15:33 -0700, Dave Jackson wrote:
> > Kevin,
> >
> >   I would guess you are requesting this correctly with qsub.  I am not
> > certain why TORQUE is rejecting this but it looks like Maui is doing the
> > right thing.  (Also, the mom 'size' option was correct, my fingers went
> > on auto-pilot in writing the previous email)
> >
> >   When I test this under TORQUE 2.2.0, my job fails with the following
> > in the job's stderr file
> >
> > --------
> > getsize() failed for file in mom_set_limits
> > --------
> >
> >   It looks like we need to move this discussion over to torqueusers.  Do
> > you want to re-post this?  I will get started looking at what we can do
> > to make getsize working on larger values.
> >
> > Dave
> >
> >
> > On Mon, 2007-02-05 at 17:17 -0500, Kevin Van Workum wrote:
> > > I tried using:
> > > NODECFG[amd24] CFGDISK=10000
> > >
> > > checknode amd24 returned:
> > > Configured Resources: PROCS: 1  MEM: 503M  SWAP: 1484M  DISK: 10000M
> > > Utilized   Resources: DISK: 9999M
> > > Dedicated  Resources: [NONE]
> > >
> > > But my job doesn't start if I request more than 1mb, e.g. 'qsub -l
> > > file=2mb'. So it looks like maui thinks there is only 1mb of disk
> > > available.
> > >
> > > Also, I couldn't find any 'file' option for mom's config, but I did
> > > find the 'size' option. So I tried 'size[fs=/tmp]'.
> > >
> > > In this case checknode amd24 returned:
> > > Configured Resources: PROCS: 1  MEM: 503M  SWAP: 1483M  DISK: 12G
> > > Utilized   Resources: DISK: 1855M
> > > Dedicated  Resources: [NONE]
> > >
> > > Which is close to the correct disk space for /tmp. 'df -h /tmp' gives:
> > > [root at amd24 ~]# df /tmp -h
> > > Filesystem                                    Size  Used Avail Use% Mounted on
> > > /dev/mapper/VolGroup00-LogVol00 13G  1.9G 10G  16%    /
> > >
> > > But I found that if I request file=4096mb or more, the job fails.
> > > mom's log says:
> > > "pbs_mom;Job;TMomFinalizeJob3;job not started, Failure job exec
> > > failure, after files staged, no retry"
> > > and pbs_server says:
> > > "PBS_Server: stream_eof, connection to amd24 is bad, remote service
> > > may be down, message may be corrupt, or connection may have been
> > > dropped remotely (End of File).  setting node state to down"
> > >
> > > Requesting less than file=4096mb works fine.
> > >
> > > Maybe I'm requesting the required disk space incorrectly on qsub's command line.
> > >
> > > Kevin
> > >
> > > On 2/5/07, Dave Jackson <jacksond at clusterresources.com> wrote:
> > > > Kevin,
> > > >
> > > >   I think you can indicate the amount of diskspace available using the
> > > > TORQUE 'file' option in mom config.  Otherwise, you should be able to
> > > > populate this info directly via Maui using 'NODECFG[X] CFGDISK=<VAL>'
> > > > where val is specified in MB.  If this does not work, let me know and we
> > > > will get it fixed.
> > > >
> > > > Dave
> > > >
> > > > On Mon, 2007-02-05 at 16:16 -0500, Kevin Van Workum wrote:
> > > > > My nodes have various amounts of local scratch disk space. How do I
> > > > > tell maui how much local scratch disk space each node has, and how do
> > > > > I request a certain amount of disk space on each node for a particular
> > > > > job?
> > > > >
> > > > > I'm testing with maui-3.2.6p19 and torque-2.1.6.
> > > > >
> > > >
> > > >
> > >
> > >
> >
> > _______________________________________________
> > mauiusers mailing list
> > mauiusers at supercluster.org
> > http://www.supercluster.org/mailman/listinfo/mauiusers
>
>


-- 
Kevin Van Workum, Ph.D.
Vice President
Senior System Administrator
www.clusterondemand.com
ONLINE COMPUTER CLUSTERS