[Beowulf] debugging

[Mark Hahn]

>> Does anyone have any advice? I am open to try out other things as well if 
>> possible. I am just starting to learn debugger techniques for a parallel 
>> program.

we bought Allinea DDT, which is pretty good.  I think it's a good
illustration that the parallel part of debugging doesn't have to be 
hard.  getting parallel logic is still nontrivial, as often serial is.

> Core dumps are your friends (though most linux distros turn them off by

while post-mortem approaches are quite valuable (though I probably use 
strace/ltrace more frequently), they do not compare to being able to 
walk your program through its paces, monitoring variables along the way,
even messing with the parallelism.

> default now).  That and compiling your code with symbols, and debugging

it is unfortunate that compiling with debugging will normally disable 
a good number of valuable optimizations.  I emphasize to users that 
production code (ie, which consumes, say >1k cpu hours) needs to be 
maximally tweaked.  anything else is stealing from other users.