<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix"><br>
      Hi David,<br>
      <br>
      The nodes which we observed are running the following version:<br>
      <br>
      -bash-3.2# ldd /opt/torque/sbin/pbs_mom | grep libc.so<br>
      &nbsp;&nbsp;&nbsp; libc.so.6 =&gt; /lib64/libc.so.6 (0x00002b18eae2a000)<br>
      <br>
      -bash-3.2# ldd --version<br>
      ldd (GNU libc) 2.5<br>
      <br>
      <br>
      -bash-3.2# qstat --version<br>
      Version: 4.1.5.1<br>
      Revision: <br>
      <br>
      -bash-3.2# uname -a<br>
      Linux zwicky005 2.6.18-308.1.1.el5 #1 SMP Fri Feb 17 16:51:01 EST
      2012 x86_64 x86_64 x86_64 GNU/Linux<br>
      <br>
      <br>
      <br>
      We see that it's using ~3G of memory:<br>
      <br>
      -bash-3.2# top -p 16695<br>
      <br>
      top - 09:46:45 up 81 days,&nbsp; 1:01,&nbsp; 1 user,&nbsp; load average: 9.19,
      9.17, 9.11<br>
      Tasks:&nbsp;&nbsp; 1 total,&nbsp;&nbsp; 0 running,&nbsp;&nbsp; 1 sleeping,&nbsp;&nbsp; 0 stopped,&nbsp;&nbsp; 0
      zombie<br>
      Cpu(s): 74.6%us,&nbsp; 0.7%sy,&nbsp; 0.0%ni, 24.6%id,&nbsp; 0.0%wa,&nbsp; 0.0%hi,&nbsp;
      0.0%si,&nbsp; 0.0%st<br>
      Mem:&nbsp; 24675856k total, 24286304k used,&nbsp;&nbsp; 389552k free,&nbsp;&nbsp; 497860k
      buffers<br>
      Swap: 49150856k total,&nbsp; 4750564k used, 44400292k free, 10798448k
      cached<br>
      <br>
      &nbsp; PID USER&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PR&nbsp; NI&nbsp; VIRT&nbsp; RES&nbsp; SHR S %CPU %MEM&nbsp;&nbsp;&nbsp; TIME+&nbsp;
      COMMAND&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <br>
      16695 root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 15&nbsp;&nbsp; 0 3195m 3.1g 7052 S&nbsp; 0.3 13.1&nbsp; 77:50.71
      pbs_mom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <br>
      <br>
      <br>
      We came across this posting and not sure if this is relevant:<br>
      <br>
      <a class="moz-txt-link-freetext" href="http://comments.gmane.org/gmane.comp.clustering.torque.user/13557">http://comments.gmane.org/gmane.comp.clustering.torque.user/13557</a><br>
      <br>
      <br>
      Thanks for looking into this.<br>
      <br>
      Steven.<br>
      <br>
      <br>
      On 12/06/2013 09:04 AM, David Beer wrote:<br>
    </div>
    <blockquote
cite="mid:CAFUQeZ0jo91ON-8Tjc5UVh7LuAxRvJ2knNXV=901N7ROFpvFgg@mail.gmail.com"
      type="cite">
      <div dir="ltr">The issue is that in some versions of libc, the
        pthread stack size will default to 1000 * &lt;the value set in
        ulimit -s&gt;, even though TORQUE specifies what stack size each
        thread should have. I will work to get a list of the versions of
        libc that have this bug. Ken is the one that discovered this
        defect, so I'll ask him for the info or ask him to post the
        info.</div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">On Fri, Dec 6, 2013 at 9:02 AM, Gus
          Correa <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:gus@ldeo.columbia.edu" target="_blank">gus@ldeo.columbia.edu</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">David<br>
            <br>
            For the benefit of all Torque users,<br>
            could you please disclose all combinations of libc versions<br>
            and Torque versions that have this problem?<br>
            <br>
            Thank you,<br>
            Gus Correa<br>
            <div class="im"><br>
              On 12/05/2013 08:52 PM, David Beer wrote:<br>
              &gt; Steven,<br>
              &gt;<br>
              &gt; What OS and version of the pthread library (libc) do
              you have? We know<br>
              &gt; of a rather large memory leak related to different
              versions these libraries.<br>
              &gt;<br>
              &gt;<br>
              &gt; On Thu, Dec 5, 2013 at 12:01 PM, Steven Lo &lt;<a
                moz-do-not-send="true"
                href="mailto:slo@cacr.caltech.edu">slo@cacr.caltech.edu</a><br>
            </div>
            <div class="im">&gt; &lt;mailto:<a moz-do-not-send="true"
                href="mailto:slo@cacr.caltech.edu">slo@cacr.caltech.edu</a>&gt;&gt;
              wrote:<br>
              &gt;<br>
              &gt;<br>
              &gt; &nbsp; &nbsp; Hi,<br>
              &gt;<br>
              &gt; &nbsp; &nbsp; We've discovered that pbs_mom on most nodes are
              using over 3GB of<br>
              &gt; &nbsp; &nbsp; memory.<br>
              &gt; &nbsp; &nbsp; Is there a known memory leak issue for version
              4.1.5.1? &nbsp;If so, is there<br>
              &gt; &nbsp; &nbsp; a patch for<br>
              &gt; &nbsp; &nbsp; it or we have to upgrade to other version like
              4.1.7 or 4.2.6.1?<br>
              &gt;<br>
              &gt; &nbsp; &nbsp; Thanks in advance for your suggestion.<br>
              &gt;<br>
              &gt; &nbsp; &nbsp; Steven.<br>
              &gt;<br>
              &gt; &nbsp; &nbsp; _______________________________________________<br>
              &gt; &nbsp; &nbsp; torqueusers mailing list<br>
            </div>
            &gt; &nbsp; &nbsp; <a moz-do-not-send="true"
              href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a>
            &lt;mailto:<a moz-do-not-send="true"
              href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a>&gt;<br>
            <div class="HOEnZb">
              <div class="h5">&gt; &nbsp; &nbsp; <a moz-do-not-send="true"
                  href="http://www.supercluster.org/mailman/listinfo/torqueusers"
                  target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                &gt;<br>
                &gt;<br>
                &gt;<br>
                &gt;<br>
                &gt; --<br>
                &gt; David Beer | Senior Software Engineer<br>
                &gt; Adaptive Computing<br>
                &gt;<br>
                &gt;<br>
                &gt; _______________________________________________<br>
                &gt; torqueusers mailing list<br>
                &gt; <a moz-do-not-send="true"
                  href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
                &gt; <a moz-do-not-send="true"
                  href="http://www.supercluster.org/mailman/listinfo/torqueusers"
                  target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
                <br>
                _______________________________________________<br>
                torqueusers mailing list<br>
                <a moz-do-not-send="true"
                  href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a><br>
                <a moz-do-not-send="true"
                  href="http://www.supercluster.org/mailman/listinfo/torqueusers"
                  target="_blank">http://www.supercluster.org/mailman/listinfo/torqueusers</a><br>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div>David Beer | Senior Software Engineer</div>
        <div>Adaptive Computing</div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
torqueusers mailing list
<a class="moz-txt-link-abbreviated" href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.supercluster.org/mailman/listinfo/torqueusers">http://www.supercluster.org/mailman/listinfo/torqueusers</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>