<div dir="ltr">I cherry-picked the commit into 4.1-dev. I only had to help it along a little. <br><br>8b0db74<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Dec 10, 2013 at 2:02 PM, Steven Lo <span dir="ltr">&lt;<a href="mailto:slo@cacr.caltech.edu" target="_blank">slo@cacr.caltech.edu</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <div><br>
      HI David,<br>
      <br>
      Thanks for the info.  Very helpful.<br>
      <br>
      It would be nice if we can get a fix for the 4.1 branch since it
      would provide a quick fix<br>
      for our cluster (some jobs are failing due to lack of memory). 
      That will also give us<br>
      a little bit of breathing room for the upgrade.  I presume the fix
      will affect pbs_mom<br>
      binary only?<br>
      <br>
      Again.  Thanks much for your help.<span class="HOEnZb"><font color="#888888"><br>
      <br>
      Steven.</font></span><div><div class="h5"><br>
      <br>
      <br>
      On 12/10/2013 12:42 PM, David Beer wrote:<br>
    </div></div></div><div><div class="h5">
    <blockquote type="cite">
      <div dir="ltr">Oh, and to answer your other question it hasn&#39;t
        been checked in to the 4.1-dev branch. If you need that let us
        know and we can probably port it.</div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">
          On Tue, Dec 10, 2013 at 1:41 PM, David Beer <span dir="ltr">&lt;<a href="mailto:dbeer@adaptivecomputing.com" target="_blank">dbeer@adaptivecomputing.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">Steven,
              <div><br>
              </div>
              <div>4.2.7 is due out in mid-January. To clarify, &quot;master&quot;
                in git is the equivalent of trunk in subversion or
                mercurial. When new major releases are made, such as
                4.5.0, they are branched from master.</div>
            </div>
            <div>
              <div>
                <div class="gmail_extra"><br>
                  <br>
                  <div class="gmail_quote">On Tue, Dec 10, 2013 at 11:36
                    AM, Steven Lo <span dir="ltr">&lt;<a href="mailto:slo@cacr.caltech.edu" target="_blank">slo@cacr.caltech.edu</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div bgcolor="#FFFFFF" text="#000000">
                        <div><br>
                          Hi Ken,<br>
                          <br>
                          Not sure what &quot;master&quot; means.  Does it include
                          4.1.x?<br>
                          <br>
                          If there is no fix for 4.1.x, our only option
                          is to upgrade to 4.2.x or 4.5.0.<br>
                          <br>
                          BTW, do you know when 4.2.7 will be out
                          officially?<br>
                          <br>
                          Thanks.<span><font color="#888888"><br>
                              <br>
                              Steven.</font></span>
                          <div>
                            <div><br>
                              <br>
                              <br>
                              On 12/10/2013 10:27 AM, Ken Nielson wrote:<br>
                            </div>
                          </div>
                        </div>
                        <div>
                          <div>
                            <blockquote type="cite">
                              <div dir="ltr">
                                <div>
                                  <div>
                                    <div>
                                      <div>
                                        <div>
                                          <div>Steven,<br>
                                            <br>
                                          </div>
                                          We found the problem and have
                                          fixed it. You will find the
                                          fix in Github for 4.2-dev,
                                          4.5.0 and master.<br>
                                          <br>
                                        </div>
                                        The fix was made on December 6
                                        by me (knielson). The
                                        description is &quot;Fixed a problem
                                        where the stack size was not
                                        getting set&quot;<br>
                                        <br>
                                      </div>
                                      The commit hashes are as follows:<br>
                                      <br>
                                    </div>
                                    master    
                                    9c584930a20cd6d99b1690b9fd82747d794e021b<br>
                                  </div>
                                  4.2-dev   
                                  f78ae1d290dd9279af8b4f20a74e188c61ed22d3<br>
                                  4.5.0       
                                  12c8e5b6deeb8ea4af6f46767b21eca846663a42<br>
                                  <br>
                                </div>
                                You can create a patch from these if you
                                need.<br>
                                <br>
                                Let me know if you need anything else.<br>
                                <br>
                                Regards<br>
                                <div>
                                  <div>
                                    <div>
                                      <div>
                                        <div><br>
                                          <br>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                              <div class="gmail_extra"><br>
                                <br>
                                <div class="gmail_quote">On Mon, Dec 9,
                                  2013 at 12:50 PM, Steven Lo <span dir="ltr">&lt;<a href="mailto:slo@cacr.caltech.edu" target="_blank">slo@cacr.caltech.edu</a>&gt;</span>
                                  wrote:<br>
                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                    <div bgcolor="#FFFFFF" text="#000000">
                                      <div><br>
                                        We are using libc 2.5.<br>
                                        <br>
                                        The stack size for our host is
                                        set to 1600000 KB.  If there are
                                        2 threads then pbs_mom will use
                                        ~3GB then?<br>
                                        <br>
                                        stack size              (kbytes,
                                        -s) 1600000<br>
                                        <br>
                                        <br>
                                        Is there a workaround to this
                                        problem?  If not, upgrade to
                                        4.2.6.1 or 4.2.7 is recommended?<br>
                                        <br>
                                        Thanks.<span><font color="#888888"><br>
                                            <br>
                                            Steven.</font></span>
                                        <div>
                                          <div><br>
                                            <br>
                                            <br>
                                            On 12/06/2013 10:55 AM, Ken
                                            Nielson wrote:<br>
                                          </div>
                                        </div>
                                      </div>
                                      <div>
                                        <div>
                                          <blockquote type="cite">
                                            <div dir="ltr">
                                              <div>My apologies.  I have
                                                been doing more testing
                                                and it appears libc 2.18
                                                still has the bug.  I
                                                will be reporting this
                                                the GNU and hopefully
                                                they will have a fix
                                                soon.<br>
                                                <br>
                                              </div>
                                              Again my apologies for
                                              jumping the gun.<br>
                                            </div>
                                            <div class="gmail_extra"><br>
                                              <br>
                                              <div class="gmail_quote">On
                                                Fri, Dec 6, 2013 at
                                                11:23 AM, Ken Nielson <span dir="ltr">&lt;<a href="mailto:knielson@adaptivecomputing.com" target="_blank">knielson@adaptivecomputing.com</a>&gt;</span>
                                                wrote:<br>
                                                <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                  <div dir="ltr">
                                                    <div>
                                                      <div>
                                                        <div>It is nice
                                                          to have
                                                          confirmation
                                                          on what we
                                                          found about
                                                          the
                                                          pthread_attr_setstacksize.
                                                          As you said,
                                                          the default
                                                          stack size of
                                                          a new thread
                                                          is the value
                                                          of ulimit -s *
                                                          1000.
                                                          Currently the
                                                          TORQUE code
                                                          tries to make
                                                          a minimum
                                                          stack size but
                                                          it does not
                                                          check for a
                                                          maximum. So if
                                                          you have your
                                                          stack size set
                                                          to 300,000 you
                                                          will get
                                                          300,000,000
                                                          bytes
                                                          allocated for
                                                          each thread.
                                                          On the mom
                                                          there are two
                                                          threads so you
                                                          get 600 Mb
                                                          allocated. On
                                                          the server it
                                                          gets huge. <br>
                                                          <br>
                                                          The bug in
                                                          libc is that
                                                          the
                                                          pthread_attr_setstacksize
                                                          is ignored. My
                                                          development
                                                          box is running
                                                          libc version
                                                          2.15 and the
                                                          bug is still
                                                          there.
                                                          However, I
                                                          installed libc
                                                          2.18 on a
                                                          CentOS 6 box
                                                          and the bug
                                                          appears to be
                                                          fixed.  I am
                                                          going to
                                                          modify the MOM
                                                          code to also
                                                          set a maximum
                                                          stack size.<br>
                                                        </div>
                                                        <br>
                                                      </div>
                                                      The TORQUE fix
                                                      will show up in
                                                      4.2.7. <br>
                                                      <br>
                                                    </div>
                                                    Regards<br>
                                                  </div>
                                                  <div class="gmail_extra">
                                                    <div>
                                                      <div><br>
                                                        <br>
                                                        <div class="gmail_quote">On
                                                          Fri, Dec 6,
                                                          2013 at 10:52
                                                          AM, Steven Lo
                                                          <span dir="ltr">&lt;<a href="mailto:slo@cacr.caltech.edu" target="_blank">slo@cacr.caltech.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          <div><br>
                                                          Hi David,<br>
                                                          <br>
                                                          The nodes
                                                          which we
                                                          observed are
                                                          running the
                                                          following
                                                          version:<br>
                                                          <br>
                                                          -bash-3.2# ldd
                                                          /opt/torque/sbin/pbs_mom
                                                          | grep libc.so<br>
                                                              libc.so.6
                                                          =&gt;
                                                          /lib64/libc.so.6
(0x00002b18eae2a000)<br>
                                                          <br>
                                                          -bash-3.2# ldd
                                                          --version<br>
                                                          ldd (GNU libc)
                                                          2.5<br>
                                                          <br>
                                                          <br>
                                                          -bash-3.2#
                                                          qstat
                                                          --version<br>
                                                          Version:
                                                          4.1.5.1<br>
                                                          Revision: <br>
                                                          <br>
                                                          -bash-3.2#
                                                          uname -a<br>
                                                          Linux
                                                          zwicky005
                                                          2.6.18-308.1.1.el5
                                                          #1 SMP Fri Feb
                                                          17 16:51:01
                                                          EST 2012
                                                          x86_64 x86_64
                                                          x86_64
                                                          GNU/Linux<br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          We see that
                                                          it&#39;s using ~3G
                                                          of memory:<br>
                                                          <br>
                                                          -bash-3.2# top
                                                          -p 16695<br>
                                                          <br>
                                                          top - 09:46:45
                                                          up 81 days, 
                                                          1:01,  1
                                                          user,  load
                                                          average: 9.19,
                                                          9.17, 9.11<br>
                                                          Tasks:   1
                                                          total,   0
                                                          running,   1
                                                          sleeping,   0
                                                          stopped,   0
                                                          zombie<br>
                                                          Cpu(s):
                                                          74.6%us, 
                                                          0.7%sy, 
                                                          0.0%ni,
                                                          24.6%id, 
                                                          0.0%wa, 
                                                          0.0%hi, 
                                                          0.0%si, 
                                                          0.0%st<br>
                                                          Mem: 
                                                          24675856k
                                                          total,
                                                          24286304k
                                                          used,  
                                                          389552k
                                                          free,  
                                                          497860k
                                                          buffers<br>
                                                          Swap:
                                                          49150856k
                                                          total, 
                                                          4750564k used,
                                                          44400292k
                                                          free,
                                                          10798448k
                                                          cached<br>
                                                          <br>
                                                            PID
                                                          USER      PR 
                                                          NI  VIRT  RES 
                                                          SHR S %CPU
                                                          %MEM    TIME+ 
                                                          COMMAND           
                                                          <br>
                                                          16695
                                                          root      15  
                                                          0 3195m 3.1g
                                                          7052 S  0.3
                                                          13.1  77:50.71
                                                          pbs_mom         
                                                          <br>
                                                          <br>
                                                          <br>
                                                          We came across
                                                          this posting
                                                          and not sure
                                                          if this is
                                                          relevant:<br>
                                                          <br>
                                                          <a href="http://comments.gmane.org/gmane.comp.clustering.torque.user/13557" target="_blank">http://comments.gmane.org/gmane.comp.clustering.torque.user/13557</a><br>

                                                          <br>
                                                          <br>
                                                          Thanks for
                                                          looking into
                                                          this.<br>
                                                          <br>
                                                          Steven.<br>
                                                          <br>
                                                          <br>
                                                          On 12/06/2013
                                                          09:04 AM,
                                                          David Beer
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">The
                                                          issue is that
                                                          in some
                                                          versions of
                                                          libc, the
                                                          pthread stack
                                                          size will
                                                          default to
                                                          1000 * &lt;the
                                                          value set in
                                                          ulimit -s&gt;,
                                                          even though
                                                          TORQUE
                                                          specifies what
                                                          stack size
                                                          each thread
                                                          should have. I
                                                          will work to
                                                          get a list of
                                                          the versions
                                                          of libc that
                                                          have this bug.
                                                          Ken is the one
                                                          that
                                                          discovered
                                                          this defect,
                                                          so I&#39;ll ask
                                                          him for the
                                                          info or ask
                                                          him to post
                                                          the info.</div>
                                                          <div class="gmail_extra"><br>
                                                          <br>
                                                          <div class="gmail_quote">On

                                                          Fri, Dec 6,
                                                          2013 at 9:02
                                                          AM, Gus Correa
                                                          <span dir="ltr">&lt;<a href="mailto:gus@ldeo.columbia.edu" target="_blank">gus@ldeo.columbia.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">David<br>
                                                          <br>
                                                          For the
                                                          benefit of all
                                                          Torque users,<br>
                                                          could you
                                                          please
                                                          disclose all
                                                          combinations
                                                          of libc
                                                          versions<br>
                                                          and Torque
                                                          versions that
                                                          have this
                                                          problem?<br>
                                                          <br>
                                                          Thank you,<br>
                                                          Gus Correa<br>
                                                          <div><br>
                                                          On 12/05/2013
                                                          08:52 PM,
                                                          David Beer
                                                          wrote:<br>
                                                          &gt; Steven,<br>
                                                          &gt;<br>
                                                          &gt; What OS
                                                          and version of
                                                          the pthread
                                                          library (libc)
                                                          do you have?
                                                          We know<br>
                                                          &gt; of a
                                                          rather large
                                                          memory leak
                                                          related to
                                                          different
                                                          versions these
                                                          libraries.<br>
                                                          &gt;<br>
                                                          &gt;<br>
                                                          &gt; On Thu,
                                                          Dec 5, 2013 at
                                                          12:01 PM,
                                                          Steven Lo &lt;<a href="mailto:slo@cacr.caltech.edu" target="_blank">slo@cacr.caltech.edu</a><br>
                                                          </div>
                                                          <div>&gt;
                                                          &lt;mailto:<a href="mailto:slo@cacr.caltech.edu" target="_blank">slo@cacr.caltech.edu</a>&gt;&gt;



                                                          wrote:<br>
                                                          &gt;<br>
                                                          &gt;<br>
                                                          &gt;     Hi,<br>
                                                          &gt;<br>
                                                          &gt;     We&#39;ve
                                                          discovered
                                                          that pbs_mom
                                                          on most nodes
                                                          are using over
                                                          3GB of<br>
                                                          &gt;    
                                                          memory.<br>
                                                          &gt;     Is
                                                          there a known
                                                          memory leak
                                                          issue for
                                                          version
                                                          4.1.5.1?  If
                                                          so, is there<br>
                                                          &gt;     a
                                                          patch for<br>
                                                          &gt;     it or
                                                          we have to
                                                          upgrade to
                                                          other version
                                                          like 4.1.7 or
                                                          4.2.6.1?<br>
                                                          &gt;<br>
                                                          &gt;    
                                                          Thanks in
                                                          advance for
                                                          your
                                                          suggestion.<br>
                                                          &gt;<br>
                                                          &gt;    
                                                          Steven.<br>
                                                          &gt;<br>
                                                          &gt;    
                                                          _______________________________________________<br>
                                                          &gt;    
                                                          torqueusers
                                                          mailing list<br>
                                                          </div>
                                                          &gt;     <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a>
                                                          &lt;mailto:<a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a>&gt;<br>
                                                          <div>
                                                          <div>&gt;    
                                                          <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                                                          &gt;<br>
                                                          &gt;<br>
                                                          &gt;<br>
                                                          &gt;<br>
                                                          &gt; --<br>
                                                          &gt; David
                                                          Beer | Senior
                                                          Software
                                                          Engineer<br>
                                                          &gt; Adaptive
                                                          Computing<br>
                                                          &gt;<br>
                                                          &gt;<br>
                                                          &gt;
                                                          _______________________________________________<br>
                                                          &gt;
                                                          torqueusers
                                                          mailing list<br>
                                                          &gt; <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
                                                          &gt; <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                                                          <br>
_______________________________________________<br>
                                                          torqueusers
                                                          mailing list<br>
                                                          <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
                                                          <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          <br clear="all">
                                                          <span><font color="#888888">
                                                          <div><br>
                                                          </div>
                                                          -- <br>
                                                          <div>David
                                                          Beer | Senior
                                                          Software
                                                          Engineer</div>
                                                          <div>Adaptive
                                                          Computing</div>
                                                          </font></span></div>
                                                          <span><font color="#888888">
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          <pre>_______________________________________________
torqueusers mailing list
<a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a>
</pre>
                                                          </font></span></blockquote>
                                                          <br>
                                                          </div>
                                                          <br>
_______________________________________________<br>
                                                          torqueusers
                                                          mailing list<br>
                                                          <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
                                                          <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                                                          <br>
                                                          </blockquote>
                                                        </div>
                                                        <br>
                                                        <br clear="all">
                                                        <br>
                                                        -- <br>
                                                      </div>
                                                    </div>
                                                    <span><font color="#888888">Ken
                                                        Nielson<br>
                                                        <a href="tel:%2B1%20801.717.3700" value="+18017173700" target="_blank">+1
                                                          801.717.3700</a>
                                                        office <a href="tel:%2B1%20801.717.3738" value="+18017173738" target="_blank">+1
                                                          801.717.3738</a>
                                                        fax<br>
                                                        1712 S. East Bay
                                                        Blvd, Suite 300 
                                                        Provo, UT  84606<br>
                                                        <a href="http://www.adaptivecomputing.com" target="_blank">www.adaptivecomputing.com</a><br>
                                                        <br>
                                                      </font></span></div>
                                                </blockquote>
                                              </div>
                                              <br>
                                              <br clear="all">
                                              <br>
                                              -- <br>
                                              Ken Nielson<br>
                                              <a href="tel:%2B1%20801.717.3700" value="+18017173700" target="_blank">+1
                                                801.717.3700</a> office
                                              <a href="tel:%2B1%20801.717.3738" value="+18017173738" target="_blank">+1
                                                801.717.3738</a> fax<br>
                                              1712 S. East Bay Blvd,
                                              Suite 300  Provo, UT 
                                              84606<br>
                                              <a href="http://www.adaptivecomputing.com" target="_blank">www.adaptivecomputing.com</a><br>
                                              <br>
                                            </div>
                                            <br>
                                            <fieldset></fieldset>
                                            <br>
                                            <pre>_______________________________________________
torqueusers mailing list
<a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a>
</pre>
                                          </blockquote>
                                          <br>
                                        </div>
                                      </div>
                                    </div>
                                    <br>
_______________________________________________<br>
                                    torqueusers mailing list<br>
                                    <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
                                    <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                                    <br>
                                  </blockquote>
                                </div>
                                <br>
                                <br clear="all">
                                <br>
                                -- <br>
                                Ken Nielson<br>
                                <a href="tel:%2B1%20801.717.3700" value="+18017173700" target="_blank">+1
                                  801.717.3700</a> office <a href="tel:%2B1%20801.717.3738" value="+18017173738" target="_blank">+1
                                  801.717.3738</a> fax<br>
                                1712 S. East Bay Blvd, Suite 300  Provo,
                                UT  84606<br>
                                <a href="http://www.adaptivecomputing.com" target="_blank">www.adaptivecomputing.com</a><br>
                                <br>
                              </div>
                              <br>
                              <fieldset></fieldset>
                              <br>
                              <pre>_______________________________________________
torqueusers mailing list
<a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a>
</pre>
                            </blockquote>
                            <br>
                          </div>
                        </div>
                      </div>
                      <br>
                      _______________________________________________<br>
                      torqueusers mailing list<br>
                      <a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a><br>
                      <a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                      <br>
                    </blockquote>
                  </div>
                  <br>
                  <br clear="all">
                  <div><br>
                  </div>
                  -- <br>
                  <div>David Beer | Senior Software Engineer</div>
                  <div>Adaptive Computing</div>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div>David Beer | Senior Software Engineer</div>
        <div>Adaptive Computing</div>
      </div>
      <br>
      <fieldset></fieldset>
      <br>
      <pre>_______________________________________________
torqueusers mailing list
<a href="mailto:torqueusers@supercluster.org" target="_blank">torqueusers@supercluster.org</a>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a>
</pre>
    </blockquote>
    <br>
  </div></div></div>

<br>_______________________________________________<br>
torqueusers mailing list<br>
<a href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
<a href="http://www.supercluster.org/mailman/listinfo/torqueusers" target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div>David Beer | Senior Software Engineer</div><div>Adaptive Computing</div>
</div>