Bad Infiniband latency with SC1435

[Ferdinand Geier]

Hi all,

we are working on a new cluster installation using Dell servers SC1435 and 
Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20). We've 
installed the OFED 1.3.1 package on a SLES10, SP2, (Kernel is 
2.6.16.60-0.33-smp).

Using the ib_*_lat utility, we get the following latencies (in microseconds):

mode          typical best    worst
rdma          3.78    3.74    28.48
read          26.38   10.73   67.07
send          14.77   5.18    71.02
write         3.90    3.75    27.12

Also the IMB gives pingpong latencies of about 9usecs. All this is very 
disappointing.

I guess a problem within the PCI configuration. Are there any tunable 
parameters? Any other hints?

Thanks,
Ferdinand


--
Ferdinand Geier
ParTec Cluster Competence Center GmbH
Munich, Germany