<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Read requests should be small compared
to the bytes returned, so 1MB of read requests may trigger
considerably more return data traffic. <br>
<br>
Jeff Anderson-Lee<br>
<br>
On 11/8/2012 12:58 PM, Mike Dacre wrote:<br>
</div>
<blockquote cite="mid:F10998BFD824451C899EF7AF549CC6E9@gmail.com"
type="cite">
<div> Hi Craig, </div>
<div><br>
</div>
<div>Yes you are right, it is outbound only. What I have been
trying to do is limit the traffic from the client side. The
settings below are from the individual slave nodes. It is very
difficult to quiet the system as it is being actively used,
however I have tried to apply the setting below with very
restrictive limits - around 1MB/s per node - without seeing any
discernible slowdown of traffic. I also tried from the server
side just to see if it would slow down reads (which are outbound
from the server's perspective), also without any effect.</div>
<div><br>
</div>
<div>Is there any chance the problem is that the NFS is async? I
know that is a problem for cgroups, but I didn't think it would
be a problem here because tc/iptables filtering/mangling is at
the packet level.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Mike</div>
<p style="color: #A0A0A8;">On Thursday, November 8, 2012 at 12:34
PM, Craig Tierney wrote:</p>
<blockquote type="cite"
style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span>
<div>
<div>
<div>Mike,</div>
<div><br>
</div>
<div>I can't follow all of your settings because I am not
an expert in the</div>
<div>tc command, however I have played with it. What I
remember is that</div>
<div>traffic shaping only is outbound. I would quiet the
system and run</div>
<div>some tests where you do some writes from multiple
nodes, then reads.</div>
<div>Do they seem throttled at all? Is the performance
difference in each</div>
<div>direction?</div>
<div><br>
</div>
<div>Craig</div>
<div><br>
</div>
<div>On Thu, Nov 8, 2012 at 12:44 PM, Mike Dacre <<a
moz-do-not-send="true"
href="mailto:mike.dacre@gmail.com">mike.dacre@gmail.com</a>>
wrote:</div>
<blockquote type="cite">
<div>
<div>Hi Everyone,</div>
<div><br>
</div>
<div>I am having an unusual problem: I have an
Infiniband network connecting my</div>
<div>nodes and I have a RAID array mounted over NFS on
every node. What is</div>
<div>happening is that the nodes are reading/writing
too fast to the NFS mount,</div>
<div>and the IO of the array is being maxed out, which
results in terrible</div>
<div>performance for interactive commands (e.g. ls). I
have tried traffic</div>
<div>shaping with iptables and tc on both the server
and slave nodes with no</div>
<div>success at all. I am not even certain those
commands are working properly</div>
<div>on an IPoIB NIC (ib0).</div>
<div><br>
</div>
<div>The TC command I am trying is:</div>
<div><br>
</div>
<div> $TC qdisc add dev ib0 root handle 1:0 htb</div>
<div> $TC class add dev ib0 parent 1:0 classid 1:1 htb
rate 50mbps ceil 50mbps</div>
<div> $TC class add dev ib0 parent 1:1 classid 1:2 htb
rate 10mbps ceil 20mbps</div>
<div> $TC qdisc add dev ib0 parent 1:2 sfq</div>
<div> $TC filter add dev ib0 parent 1:0 protocol ip
u32 match ip sport 2049</div>
<div>0xffff flowid 1:2</div>
<div> $TC filter add dev ib0 parent 1:0 protocol ip
u32 match ip dport 2049</div>
<div>0xffff flowid 1:2</div>
<div><br>
</div>
<div>or</div>
<div><br>
</div>
<div> $TC qdisc add dev ib0 root handle 1:0 htb</div>
<div> $TC class add dev ib0 parent 1:0 classid 1:1 htb
rate 50mbps ceil 50mbps</div>
<div> $TC class add dev ib0 parent 1:1 classid 1:2 htb
rate 10mbps ceil 20mbps</div>
<div> $TC qdisc add dev ib0 parent 1:2 sfq</div>
<div> $TC filter add dev ib0 parent 1:0 protocol ip
prio 1 handle 6 fw flowid</div>
<div>1:2</div>
<div><br>
</div>
<div>With the following iptables:</div>
<div><br>
</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
tcp -m multiport --sport</div>
<div>2049 -j MARK --set-mark 6</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
tcp -m multiport --sport</div>
<div>2049 -j RETURN</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
udp -m multiport --sport</div>
<div>2049 -j MARK --set-mark 6</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
udp -m multiport --sport</div>
<div>2049 -j RETURN</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
tcp -m multiport --dport</div>
<div>2049 -j MARK --set-mark 6</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
tcp -m multiport --dport</div>
<div>2049 -j RETURN</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
udp -m multiport --dport</div>
<div>2049 -j MARK --set-mark 6</div>
<div>/sbin/iptables -A POSTROUTING -t mangle -o ib0 -p
udp -m multiport --dport</div>
<div>2049 -j RETURN</div>
<div><br>
</div>
<div>I don't want to go back to ethernet NFS and only
use Infiniband for MPI</div>
<div>because I want to maximize IO when there is a
single node doing all the IO,</div>
<div>and ethernet is just too slow for that.</div>
<div><br>
</div>
<div>Any thoughts?</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Mike</div>
<div><br>
</div>
<div>_______________________________________________</div>
<div>torqueusers mailing list</div>
<div><a moz-do-not-send="true"
href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a></div>
<div><a moz-do-not-send="true"
href="http://www.supercluster.org/mailman/listinfo/torqueusers">http://www.supercluster.org/mailman/listinfo/torqueusers</a></div>
</div>
</blockquote>
<div>_______________________________________________</div>
<div>torqueusers mailing list</div>
<div><a moz-do-not-send="true"
href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a></div>
<div><a moz-do-not-send="true"
href="http://www.supercluster.org/mailman/listinfo/torqueusers">http://www.supercluster.org/mailman/listinfo/torqueusers</a></div>
</div>
</div>
</span> </blockquote>
<div> <br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
torqueusers mailing list
<a class="moz-txt-link-abbreviated" href="mailto:torqueusers@supercluster.org">torqueusers@supercluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.supercluster.org/mailman/listinfo/torqueusers">http://www.supercluster.org/mailman/listinfo/torqueusers</a>
</pre>
</blockquote>
<br>
</body>
</html>