[torquedev] 2.3.0 cpuset fixes

[Chad Vizino]

I misdirected the message below to torquedev-request.  Sorry about that.

Perhaps the fixes below could be considered for the next 2.3.0 snap if 
they haven't been repaired already.

   -Chad

-------- Original Message --------
Subject: Re: [torqueusers] ncpus=? 0
Date: Thu, 27 Mar 2008 08:20:52 -0400
From: Chad Vizino <vizino at psc.edu>
To: siri <didier.siri at univ-provence.fr>
CC: torquedev-request at supercluster.org, torqueusers at supercluster.org
References: <F020B8F2-B338-45A4-8555-AD3E4FBF705A at univ-provence.fr>

Greetings,

We have been using 2.3.0 (the release, not a snap) for a week or so on
our Altix systems (one with 144 cpus, the other with 768) with cpusets
enabled.  There are a few bugs that you should be aware of:

1) ncpus not obtained and showing ncpus=? in pbsnodes output.

Fix: src/resmom/linux/mom_mach.c
at lines 3333-3338, delete if condition around fscanf.

2) cpuset handling has a logic error in displaying clean up messages.
There's a file descriptor leak in cpuset cleanup and left unchecked
pbs_mom will hit its file descriptor limit and stop working.  Also,
cpuset cleanups are slow (N seconds, where N is the number of cpus in
the cpuset) due to an unnecessary sleep.  Finally, depending on how big
your machine is, you may need to increase an array in the cpuset
creation routine.

Fix: src/resmom/linux/cpuset.c
at line 67, remove "!" before cpuset_delete(childpath)
at line 84, add "fclose(fd);"
at line 85, delete "sleep(1);"
at line 92, add "closedir(dir);"
at line 222, array size for cpusbuf[] may not be big enough (depends on
how many cpus you have (need about 4 chars per cpu to be safe))

Not a bug, but depending on the size of your machine, you may need to
increase the number of file descriptors per process in
setup_program_environment() in src/resmom/mom_mach.c at line 6271.  We
added this and upped limit in /etc/security/limits.conf:

   /* temporary hack to work around 1024 limit */
   getrlimit(RLIMIT_NOFILE,   &rlimit);
   if (rlimit.rlim_cur < 4096 || rlimit.rlim_max < 4096) {
           rlimit.rlim_cur = 4096;
           rlimit.rlim_max = 4096;
           setrlimit(RLIMIT_NOFILE,   &rlimit);
   }


In our server "nodes" file we have:

host np=N

(note no ":ts" after host).  We choose to make N 4 cpus smaller than the
physical number of cpus on the system since our boot cpuset is 4 cpus
and 1 memory node.

When submitting a job, use "-l nodes=1:ppn=4" for example.  cpusets are
not constructed when using "-l ncpus=...".

Hope this helps a little.  We're still playing with
exlusive/non-exclusive cpuset settings and limiting the job to specific
memory nodes to see how things work.  I'd be interested in your experiences.

Regards,

Chad Vizino
Pittsburgh Supercomputing Center
Pittsburgh, PA

siri wrote:
> Hi,
> 
> When I run 'pbsnodes -a' I get:
> hamilton
>     state = free
>     np = 20
>     ntype = time-shared
>     status = arch=ia64,opsys=SLES10,uname=Linux hamilton
> 2.6.16.54-0.2.5-default #1 SMP Mon Jan 21 13:29:51 UTC 2008
> ia64,sessions=10261 13317
> 17005,nsessions=3,nusers=3,idletime=2148,totmem=30467392kb,availmem=24140416kb,physmem=21763424kb,ncpus=? 
> 0,loadave=7.06,netload=55930301,state=free,jobs=,varattr=,rectime=1206551197
> 
> As you can see, I get a strange output for ncpus...
> However, the file server_priv/nodes contains "hamilton:ts np=20" and I
> expected to find ncpus=20 ...
> 
> By the way, torque2.3.0 has been compiled on an altix box with cpuset
> enable.
> 
> Another question: shall I use "-l nodes=1:ppn=4" or "-l ncpus=4" in
> order to run a job on 4 cpus ? The first case results in "qsub: Job
> exceeds queue resource limits MSG=cannot locate feasible nodes" while in
> the second case, the job runs fine, but only using 1 cpu !
> 
> Any idea ?
> 
> 
> 
> Didier SIRI, Professeur
> 
> Equipe "Chimie Théorique"
> 
> Laboratoire Chimie Provence
> 
> UMR6264-CNRS
> 
> Université de Provence
> 
> Case 521-Faculté de Saint-Jérôme
> 
> Av. Esc. Normandie Niemen
> 
> 13397 MARSEILLE Cedex 20
> 
> Tel : (33)-4-91-28-28-41
> 
> Fax : (33)-4-91-28-87-58
> 
> Port : 06-78-45-96-91
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
_______________________________________________
torqueusers mailing list
torqueusers at supercluster.org
http://www.supercluster.org/mailman/listinfo/torqueusers