[Beowulf] Network considerations for new generation cheap beowulfcluster

[Mark Hahn]

> I agree that all of the options (Infiniband, Myrinet, and 10 Gb Ethernet) are 
> too expensive.

I'm curious what kinds of costs you're seeing (per-port) for each of these.

> I have been looking into the low latency 10 Gb Ethernet cards 
> from NetEffect, which use the iWARP specifications to provide low latency. I

why do you think iWARP is necessary to provide low latency?

> haven't done any testing, yet, but the numbers that they are releasing show 
> them competitive with Infiniband/Myrinet as the number of processes increase.

do you mean as you increase number of processes on a single node (that is,
sharing a single interconnect port), or number of processes in the whole 
job or cluster?

> Plus, I expect 10 Gb switches to rapidly drop in price. I believe that the

I hope for that as well, but am not sanguine.  expensive optic tranceivers 
preclude commoditization of small (~20 pt) switches, and I've heard people 
say bad things about the practicality of mass-produced/cheap 10G-baseT.
(mainly complaining about complexity and power requirements.)


> and post some numbers. Here is a link to some of the numbers that NetEffect 
> is publishing:
>
> http://www.hpcwire.com/hpc/716435.html

no usable latency numbers there.  if you squint, it looks like they're
claiming latency of around 7 us, which is _not_ competitive with even 
myri 2G (nor recent IB nor myri 10G.)

>> the cheapest cable i see is 1 meter and $70

nothing wrong with $70 cables - you need to quote the whole per-port price,
including nic, cable and switch port.  it looks to me as if Myri 10G is 
around $1500/port; I've never had a good read on IB prices (deconvolved 
from vendor/discount pricing issues.)

>> Cheapest card i see is $715

nothing wrong with $715, even if the all-in per-port price is $1500 - 
it just means you won't be using $1000 desktop-spec nodes.  that's OK,
since if you're worried about ~3 us latency and 1GB bandwidth, you
should also be using multiple cores, ECC memory, and probably a few GB/node,
and therefore can easily amortize $715/node.

>> So the node price starts at $765, which is already way way more than the 
>> total price of 1 node.

only if you're looking at extremely low-end nodes.  for such nodes,
the only viable option is zero-cost Gb nics, of course, and mass-market 
switches (ie, not high-end chassis switches, etc).

5 years ago, the low-end approach was 100bT; now its 1000bT.  the prime 
target for that approach (serial or EP) has simply gotten broader;
I don't see this as anything to complain about.  for "real" parallel,
you have to pay for the network you need.  there as well, you now get 
more for your money, no complaints.  complaining that you can't get 1 us, 
1GBps interconnect for $50/port is just silliness.