10 gigabit Ethernet - Add on question

[David_Kewley at Dell.com]

 > -----Original Message-----
> From: linux-poweredge-bounces at dell.com 
> [mailto:linux-poweredge-bounces at dell.com] On Behalf Of Matthias Saou
> Sent: Tuesday, January 08, 2008 12:01 AM
> To: linux-poweredge-Lists
> Subject: Re: 10 gigabit Ethernet - Add on question
> 
> Kuba Ober wrote :
> 
> > On Monday 31 December 2007, Robert D. Holtz - Lists wrote:
> > > How do you topple the memory bus/cup/peripheral bus restraints to
even push
> > > past ~2gbps per host, in a practical host environment?
> > 
> > 1gbps is AFAIK less than 100 megabytes/s if you factor in overheads.
> > 
> > So 10 gbps is say 800 megabytes/s, practically, at a high link
utilization. It 
> > should be relatively easy to achieve that on modern hardware. The
memory 
> > bandwidths are a few times beyond that, same goes for DMA
bandwidths. With 
> > current PCI Express-supporting chipsets, if you do memory-to-network

> > transfers (just for benchmarking), the other bus devices won't even
notice, 
> > and all the CPU will see is less leftover memory bandwidth.
> > 
> > One DDR2 PC-3200 memory stick will happily transfer way more than
1GB/s to 4 
> > PCIe lanes, on relatively standard hardware. I tried.
> 
> This is true for "raw" network traffic, because limits quickly depend
> on what and how you transfer. My testings on some brand new dual quad
> core PE1950 III servers (E5410 @ 2.33GHz w/ 6MB cache) show for
> instance that an rsync/ssh transfer from one to the other maxes out at
> 50MB/s because that corresponds to one ssh process taking 100% of one
> of the 8 cores. This is one of those cases where pure MHz power
> would be more useful than having so many cores :-)

Makes sense. :) The 50 MB/s figure is interesting to know.  I wonder how
it'd do if you used a different ssh cipher?

On the other side of the issue, when the traffic is simple (doesn't
inherently require lots of CPU), you can do better than Kuba Ober
suggests.  I routinely see around 120 MB/s (pretty much sustained) of
web traffic from a single PE 2950 or 1850 with GigE.

Considering 10G, Myricom reports their NICs and drivers can do as much
as 99% of peak theoretical bandwidth, again in simple and well-tuned
tests, using either 10G Ethernet or 10G Myrinet MX protocol:
http://www.myri.com/scs/performance/.

David

David Kewley
Dell Infrastructure Consulting Services
Resident Consultant
Cell: 602-460-7617
David_Kewley at Dell.com

I speak only for myself, not for Dell.

Dell Services: http://www.dell.com/services/
How am I doing? Email my manager Daniel_ONeil at Dell.com with any
feedback.