10 gigabit Ethernet - Add on question

[Peter Grandi]

[ ... on 10gb/s performance ... ]

>> My testings on some brand new dual quad core PE1950 III
>> servers (E5410 @ 2.33GHz w/ 6MB cache) show for instance that
>> an rsync/ssh transfer from one to the other maxes out at
>> 50MB/s because that corresponds to one ssh process taking
>> 100% of one of the 8 cores.

The quickest cipher is RC4 ('arcfour'), then Blowfish and AES
(which are broadly equivalent but depends on the CPU which one
is slightly faster).

Between two old PCs (source Athlon 2GHZ, sink Athlon XP 1.6GHz)
I get 33MB/s with RC4 and 26MB/s with Blowfish or AES using this
test:

  dd bs=8k count=10000 if=/dev/zero | ssh -o Ciphers=arcfour HOST dd bs=8k of=/dev/null

Between two Dell 2950 with Xeon 5160s I get 67MB/s with RC4 and
indeed around 54MB/s with Blowfish and 47MB/s with AES.

Anyhow let's say that SSH is not the best protocol for data
transfer, just perhaps the most convenient.

> Has anyone done any type of testing with real
> life data loads, [ ... ]

Yes, the people doing CERN Grid stuff for example. Of course
conditions matter greatly, in particular jumbo frames, and it
also varies quite a bit by card, and motherboard chipset.

  http://WWW.Myrinet.com/scs/READMES/README.myri10ge-linux

Also some manufacturers have published more or less credible
results, for example one of the more reasonable manufacturers
have published these fairly credible numbers:

  http://WWW.Myrinet.com/scs/performance/

Very plausible (particularly as they are not from Myrinet)
numbers for file service over 10gb/s:

  http://Wiki.Lustre.org/index.php?title=Myri-10G_Ethernet

around 500MB/s reading or writing per client node, on links
capable of around 1100MB/s measured with 'netperf'.

> Dell box that can achieve realistic throughput even getting
> close to the 10gbps rate?

The Myrinet tests on different chipsets and motherboards are
especially interesting as they also list some Dell systems:

  http://WWW.Myrinet.com/scs/performance/PCIe_motherboards/

> At 10gpbs the TCP transport itself begins to show that
> familiar saw tooth pattern and never hits the limit.

That depends a bit on parameters of course. For example the
window size matters a lot, and depending on RTT several MBs are
necessary. CPU speed matters, and host bus matters enormously
(PCIe is much better).