[Beowulf] Building a new cluster - seeking some advice

[Mark Hahn]

> 1. I'd like to go diskless.  I've never done this before (the other two
> clusters are...diskful?).  I've used Fedora on both of the previous
> clusters.  Is this a good choice for diskless?  Any advice on where to
> start with diskless or operating system choices?

I prefer diskless installs:
 	- NFS root: fast, can be RO, no significant server load.
 	- node-specific files on tmpfs: hardly any - pidfiles mostly.
 	- local disk for swap, /tmp: disks are cheap and fast, why not?

such an approach is really nicely scalable and very pleasant to 
maintain.  a diskful cluster, by comparison, is often annoying:
disk failures actually matter, and it's not that hard for nodes 
to get out of sync.  systemimager does a good job of reimaging nodes,
but it's still not quite as "liberating" as just resetting a node,
knowing it's ephemeral...

> 2. Given my budget (about 20K), I plan on going with GigE on about 24
> nodes.  Am I right in thinking that faster network interconnects are
> just too expensive for this budget?

Greg's right: buy the right interconnect, not just the cheapest.

> 3. I'll be spending most of my cluster's time diagonalizing large
> matrices.  I plan on using ScaLAPACK eventually; currently I just use
> LAPACK/ATLAS and do individual matrices on each node.  The only thing

my experience with scalapack and diagonalization is with monster-sized 
sparse matrices, which seem to be fairly latency-sensitive.  if your 
workload is anything like that, gigabit isn't going to scale well,
at least with a conventional mpi+tcp stack.  (I'm looking forward to 
the OpenMX stack for this reason.)

> 	* Intel Core 2 Duo E6850 Conroe 3.0GHz ($280)
> 	* 8 GB (4 X 2 GB) DDR2 800 (~$200)

did you consider AMD?  "large matrices" makes me think of memory balance
(bandwidth per flop), where AMD normally leads Intel.

>  The motherboard does NOT have integrated video.  Will I need video
> output?  Can you even build a node without it?

this is a bios issue: will the board boot without a video card?
I guess you can try configuring it with the card, then remove the card
and see if it still boots.  I would make sure you can't get integrated
video - these days, such boards are often cheaper.

> motherboards with adequate support for 8GB memory and 1333 FSB don't
> have video.

I would also consider AMD, which has lots of integrated-video options.

> seems like a waste.  From reading around, it seems like there is no
> advantage really to DDR3 memory...is that right?  Any advice on the

power savings, probably some headroom in clock, but it's really at
the early-adopter stage, I think.

regards, mark hahn.