[Beowulf] Which distro for the cluster?

[Andrew M.A. Cater]

On Fri, Dec 29, 2006 at 09:39:59AM +1100, Chris Samuel wrote:
> On Friday 29 December 2006 04:24, Robert G. Brown wrote:
> 
> > I'd be interested in comments to the contrary, but I suspect that Gentoo
> > is pretty close to the worst possible choice for a cluster base. Maybe
> > slackware is worse, I don't know.
> 
> But think of the speed you could emerge applications with a large cluster, 
> distcc and ccache! :-)
> 
> Then add on the hours of fun trying to track down a problem that's unique to 
> your cluster due to combinations of compiler quirks, library versions, kernel 
> bugs and application odditites..
> 

This is a valid point. If you are not a professional sysadmin / don't 
have one of those around, you don't want to spend time needlessly doing 
hacker/geek/sysadmin type wizardry - you need to get on with your 
research.

Most of the academics and bright people on this list have become Beowulf 
experts and admins. by default - no one else has been there to do it for 
them - but that's not originally their main area of expertise. 

Pick a distribution that you know that provides the maximum ease of 
maintenance with the maximum number of useful applications already 
packaged / readily available / easily ported. This will depend on your 
problem set: simulating nuclear explosions/weather storm cells/crashing 
cars or are you sequencing genomes/calculating pi/drawing ray traced 
images?
 
> > I personally would suggest that you go with one of the mainstream,
> > reasonably well supported, package based distributions.  Centos, FC, RH,
> > SuSE, Debian/Ubuntu.
> 
> I'd have to agree there.
> 

Red Hat Enterprise based solutions don't cut it on the application 
front / packaged libraries in my (very limited) experience. The 
upgrade/maintenance path is not good - it's easier to start from scratch 
and reinstall than to move from one major release to another.

Fedora Core - you _have_ to be joking :) Lots of applications - but 
little more than 6 - 12 months of support.

SuSE is better than RH in some respects, worse in others. OpenSuSE - you 
may be on your own. SLED 10.2 may have licence costs?

Debian (and to a lesser extent Ubuntu) has the largest set of 
pre-packaged "stuff" for specialist maths that I know of and has 
reasonable general purpose tools.

> > I myself favor RH derived, rpm-based, 
> > yum-supported distros that can be installed by PXE/DHCP, kickstart, yum
> > from a repository server.  Installation of such a cluster on diskful
> > systems proceeds as follows:
> 

If I read the original post correctly, you're talking of an initial 8 
nodes or so and a head node. Prototype it - grab a couple of desktop 
machines from somewhere, a switch and some cat 5. Set up three machines:
one head and two nodes. Work your way through a toy problem. Do this for 
Warewulf/Rocks/Oscar or whatever - it will give you a feel for something 
of the complexity you'll get and the likely issues you'll face.

> What I'd really like is for a kickstart compatible Debian/Ubuntu (but with 
> mixed 64/32 bit support for AMD64 systems). I know the Ubuntu folks started 
> on this [1], but I don't think they managed to get very far. 
> 

dpkg --get-selections >> tempfile ; pxe boot for new node ; scp tempfile 
root at newnode ; ssh newnode; dpkg --set-selections < /root/tempfile ; 
apt-get update ; apt-get dselect-upgrade

goes a long way :)

> The sad fact of the matter is that often it's the ISV's and cluster management 
> tools that determine what choice of distro you have. :-(
> 

HP and IBM are distro neutral - they'll install / support whatever you 
ask them to (and pay them for).

> Yes, I know all about LSB but there are a grand total of 0 applications 
> certified for the current version (3.x) [2] and a grand total of 1 certified 
> application (though on 3 platforms) [3] over the total life of the LSB 
> standards. [4]
> 
> To paraphrase the Blues Brothers:
> 
>   ISV Vendor: Oh we got both kinds of Linux here, RHEL and SLES!
> 
> Bah humbug. :-)
> 
> Chris
>