From andrew at moonet.co.uk Thu Nov 1 02:43:38 2007 From: andrew at moonet.co.uk (andrew holway) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. Message-ID: Hi all, I'm trying to find out about the effects of virtualisation on high performance interconnects. Effects on latency and bandwidth. If you have anything please pass it on. Thanks Andy From csamuel at vpac.org Thu Nov 1 03:42:41 2007 From: csamuel at vpac.org (Chris Samuel) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: References: Message-ID: <200711012142.41757.csamuel@vpac.org> On Thu, 1 Nov 2007, andrew holway wrote: > I'm trying to find out about the effects of virtualisation on high > performance interconnects. Effects on latency and bandwidth. Google is your friend.. :-) There is an IBM presentation from the 2006 Xen conference on virtualising InfiniBand networks, including measurements and comparisons with native results. http://www.xen.org/files/xs0106_virtualizing_infiniband.pdf VMware results might be a bit harder to find given their license restrictions on benchmarking. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From patrick at myri.com Thu Nov 1 07:02:58 2007 From: patrick at myri.com (Patrick Geoffray) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: References: Message-ID: <4729DC92.6040603@myri.com> Hi Andrew, andrew holway wrote: > I'm trying to find out about the effects of virtualisation on high > performance interconnects. Effects on latency and bandwidth. Virtualization has virtually (pun intended) no effect on OS-bypass (user-level talking directly to the virtualized hardware) operations. However, it does for all non-OS-bypass operations. The most important of these is memory registration, because it's not a one-time cost as the Ohio State papers have always portrayed it. For applications that do not effectively reuse buffers, or when you cannot hijack the native malloc/munmap/sbrk symbols, or when you cannot guarantee that the memory is going to stay around (OS buffer cache for example), the memory registration (and deregistration !) is going to happen in the critical path. If the memory registration is ~150us for 1 page, then adding 50us of hypervisor/IOMMU overhead seems relatively cheap. If the memory registration is ~2us for 1 page, then it's bad. To give you an idea of the impact on latency/bandwidth, the communication time (DMA + wire + DMA) for a 4K page is in the order of 10us at 10Gb/s. Patrick From hahn at mcmaster.ca Thu Nov 1 08:15:06 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: <4729DC92.6040603@myri.com> References: <4729DC92.6040603@myri.com> Message-ID: > However, it does for all non-OS-bypass operations. The most important of > these is memory registration, because it's not a one-time cost as the Ohio > State papers have always portrayed it. For applications that do not it's also worth pointing out that virtualization introduces some amount of jitter to a parallel application's timing. this will not matter at all if the app is fairly small and/or loosely-coupled. but for nontrivial sized jobs that message frequently or in large collectives... From deadline at eadline.org Thu Nov 1 08:34:34 2007 From: deadline at eadline.org (Douglas Eadline) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] LECCIBG 2007 Message-ID: <1889.64.65.216.177.1193931274.squirrel@mail.eadline.org> For those attending SC07, I am pleased to announce the LECCIBG 1.0 .The good news is I had nothing to do with the organization, so it should be successful. BTW, the 1.0 implies the previous years were in beta phase. All the details are on the web page: http://www.clustermonkey.net//content/view/164/57/ We will be having a pool tournament with a prize! All the details are on the web, plus there is even a invitation: http://clustermonkey.net/download/LECCIBG/LECCIBG-07-TourneyPostcard.pdf See you there! -- Doug From rgb at phy.duke.edu Thu Nov 1 10:00:00 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: References: <4729DC92.6040603@myri.com> Message-ID: On Thu, 1 Nov 2007, Mark Hahn wrote: >> However, it does for all non-OS-bypass operations. The most important of >> these is memory registration, because it's not a one-time cost as the Ohio >> State papers have always portrayed it. For applications that do not > > it's also worth pointing out that virtualization introduces some amount of > jitter to a parallel application's timing. this will not > matter at all if the app is fairly small and/or loosely-coupled. > but for nontrivial sized jobs that message frequently or in large > collectives... This is absolutely true, and is easily visible -- I rather mean audible -- if you try to play music under a VM. It "works" in the sense that WMP under XP-Pro will dutifully play mp3s or a cast of some sort, but the jitter is clearly audible as a mix of "noise" from very short holes inserted into the play interspersed with gaps when you lose enough cycles to actually break into the decoding process. I'm very enthusiastic about VMs for use both in desktop or server environments (where the advantages are profound and can be very cost-beneficial) and in certain clustering environments or problems, but even if you view the application space for the computer as being "flat" in some sense, there are just more processes competing for cycles when you have two fat OS layers running instead of on thin one alongside your application. A lot of bookkeeping gets done twice, two schedulers are running (one slicing a slice of the other), they both have various running services independently, and there is also VMware or whatever eating cycles of its own managing the interface. The NUMBER of burned cycles may not be all that significant, if the host and guest are both mostly idle except for the primary running process, but the timing becomes far noisier. I don't know how many of these problems are really "fundamental" though. I wish I knew more about the hardware support for VMs. All I know is that current chips have it, but I don't know what it consists of. Does anybody? Want to share? rgb > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone(cell): 1-919-280-8443 Web: http://www.phy.duke.edu/~rgb Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From lindahl at pbm.com Thu Nov 1 11:50:43 2007 From: lindahl at pbm.com (Greg Lindahl) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: <4729DC92.6040603@myri.com> References: <4729DC92.6040603@myri.com> Message-ID: <20071101185037.GA8707@bx9.net> On Thu, Nov 01, 2007 at 10:02:58AM -0400, Patrick Geoffray wrote: > Virtualization has virtually (pun intended) no effect on OS-bypass > (user-level talking directly to the virtualized hardware) operations. If implemented that way. I would bet that not all OS-bypass hardware with virtual drivers is implemented that way. For interfaces like Myrinet and InfiniPath InfiniBand, it's easy to do the right thing. For the Mellanox IB chip, it's hard, and lots of actions require coordination. You're just using rose-colored glases because of where you work ;-) -- greg From Shainer at mellanox.com Thu Nov 1 13:11:08 2007 From: Shainer at mellanox.com (Gilad Shainer) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: <200711012142.41757.csamuel@vpac.org> Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784FCB8F62@mtiexch01.mti.com> > On Thu, 1 Nov 2007, andrew holway wrote: > > > I'm trying to find out about the effects of virtualisation on high > > performance interconnects. Effects on latency and bandwidth. > > Google is your friend.. :-) > > There is an IBM presentation from the 2006 Xen conference on > virtualising InfiniBand networks, including measurements and > comparisons with native results. > > http://www.xen.org/files/xs0106_virtualizing_infiniband.pdf > > VMware results might be a bit harder to find given their > license restrictions on benchmarking. > When you have a VMM between the virtual machine and physical interconnect, there are more layers of SW and therefore decrease in IO performance. This is regardless if you are using OS bypass or not, RDMA or send/receive etc. Good way to solve this is to use IO that offloads the VMM, and that move back the "network" jobs from the VMM back to the network. IO channels with ability to do the protection as part of the adapter, eliminate the overhead that the VMM creates and get you the same performance of a native OS. Gilad. From dnlombar at ichips.intel.com Fri Nov 2 12:18:27 2007 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: References: <4729DC92.6040603@myri.com> Message-ID: <20071102191827.GA8587@nlxdcldnl2.cl.intel.com> On Thu, Nov 01, 2007 at 01:00:00PM -0400, Robert G. Brown wrote: > > I don't know how many of these problems are really "fundamental" though. > I wish I knew more about the hardware support for VMs. All I know is > that current chips have it, but I don't know what it consists of. Does > anybody? Want to share? I'm not quite sure you're looking for this level of detail, but the Intel docs are here: Volume 3B: System Programming Guide specifcally deals with virtualization. -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From rgb at phy.duke.edu Fri Nov 2 12:18:11 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: <20071102191827.GA8587@nlxdcldnl2.cl.intel.com> References: <4729DC92.6040603@myri.com> <20071102191827.GA8587@nlxdcldnl2.cl.intel.com> Message-ID: On Fri, 2 Nov 2007, Lombard, David N wrote: > On Thu, Nov 01, 2007 at 01:00:00PM -0400, Robert G. Brown wrote: >> >> I don't know how many of these problems are really "fundamental" though. >> I wish I knew more about the hardware support for VMs. All I know is >> that current chips have it, but I don't know what it consists of. Does >> anybody? Want to share? > > I'm not quite sure you're looking for this level of detail, but the Intel > docs are here: > > > Volume 3B: System Programming Guide specifcally deals with virtualization. I was kinda hoping for the reader's digest version, yeah, but I'll try to find time to take a "figure it all out peek" sometime in the next century...;-) rgb -- Robert G. Brown Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone(cell): 1-919-280-8443 Web: http://www.phy.duke.edu/~rgb Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From charliep at cs.earlham.edu Fri Nov 2 12:24:16 2007 From: charliep at cs.earlham.edu (Charlie Peck) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] MPI for Python? Message-ID: <556717FF-64CF-403D-8AC6-689588FE0B9C@cs.earlham.edu> We'd like to start using MPI with Python. We've found a number of different bindings, mympi and pympi seem to be the most commonly used but there are others as well. We're not Python experts and were wondering what others might suggest is the "best" MPI binding to use with Python. thanks, charlie From lindahl at pbm.com Fri Nov 2 15:33:13 2007 From: lindahl at pbm.com (Greg Lindahl) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784FCB8F62@mtiexch01.mti.com> References: <200711012142.41757.csamuel@vpac.org> <9FA59C95FFCBB34EA5E42C1A8573784FCB8F62@mtiexch01.mti.com> Message-ID: <20071102223310.GB11052@bx9.net> On Thu, Nov 01, 2007 at 01:11:08PM -0700, Gilad Shainer wrote: > Good way to solve this is to use IO that offloads > the VMM, and that move back the "network" jobs from the VMM back to the > network. IO channels with ability to do the protection as part of the > adapter, eliminate the overhead that the VMM creates and get you the > same performance of a native OS. Gilad has a hammer -- and every problem is a nail. -- greg From lrea at cct.lsu.edu Fri Nov 2 12:05:16 2007 From: lrea at cct.lsu.edu (Laurie Rea) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] The Center for Computation & Technology at Louisiana State University Message-ID: <472B74EC.5020603@cct.lsu.edu> The National Science Foundation awarded the Louisiana Optical Network Initiative, or LONI, a prestigious grant for $2.2 million that allows Louisiana to join the TeraGrid, a backbone of national cyberinfrastructure. TeraGrid is a nationwide, NSF-funded research infrastructure that incorporates high-performance computing resources at nine sites across the country. The award will fund initiatives for Louisiana researchers to develop broader use of LONI resources as well as underwrite the operations costs associated with joining TeraGrid. As a member of TeraGrid, LONI will contribute one half of its centerpiece supercomputer, Queen Bee?s, computational cycles to the TeraGrid community. LONI researchers will in turn be able to utilize the TeraGrid?s national resources. This will give Louisiana researchers better opportunities to collaborate on nationwide scientific research projects and give LONI experience supporting a national user community. High Performance Computing (HPC) which provides support for LONI consists of Louisiana State University?s Center for Computation & Technology (CCT) and Information Technology Services (ITS). Due to this prestigious two year grant award, HPC will be *_hiring ?so, please visit www.cct.lsu.edu for more information on the positions that are available? or ?for the following positions:_* *IT Analyst 3 - (HPC Enablement) **IT Analyst 1,2,3 or Consultant- (Storage Specialist)** IT Analyst 2,3 or Consultant - (User Services/Two positions anticipated) **IT Analyst 2,3 or Consultant - (System Administrator/Two positions anticipated) **IT Analyst 3 or Consultant - (CTSS/Scripting Aministrator) *http://www.cct.lsu.edu/home* * -- Laurie K. Rea Human Resource Coordinator Center for Computation & Technology Louisiana State University 216 Johnston Hall Baton Rouge, LA 70803 jobs@cct.lsu.edu From lindahl at pbm.com Fri Nov 2 16:50:00 2007 From: lindahl at pbm.com (Greg Lindahl) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Free money from Seagate Message-ID: <20071102234959.GA27977@bx9.net> http://www.harddrive-settlement.com/notice.htm If you bought a drive between March 22, 2001 and Sep 6, 2007, and you can find the receipt and the serial #, you can get 5% of your money back. I figure at least some of you big cluster weenies have a few drives and some paperwork handy... -- greg From bernard at vanhpc.org Fri Nov 2 17:16:13 2007 From: bernard at vanhpc.org (Bernard Li) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Free money from Seagate In-Reply-To: <20071102234959.GA27977@bx9.net> References: <20071102234959.GA27977@bx9.net> Message-ID: On 11/2/07, Greg Lindahl wrote: > http://www.harddrive-settlement.com/notice.htm > > If you bought a drive between March 22, 2001 and Sep 6, 2007, and you > can find the receipt and the serial #, you can get 5% of your money > back. Actually, according to the online claim form, the drive has to be: (1) your purchase occurred before January 1, 2006 inorder to be eligible for cash -- otherwise you just get backup software from them. Here's the direct link: http://www.harddrive-settlement.com/secure/fileNew.aspx Cheers, Bernard From jaime.perea at gmail.com Tue Nov 6 00:43:25 2007 From: jaime.perea at gmail.com (Jaime Perea) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] MPI for Python? In-Reply-To: <556717FF-64CF-403D-8AC6-689588FE0B9C@cs.earlham.edu> References: <556717FF-64CF-403D-8AC6-689588FE0B9C@cs.earlham.edu> Message-ID: <200711060943.25615.Jaime.Perea@gmail.com> El Viernes, 2 de Noviembre de 2007, Charlie Peck escribi?: > We'd like to start using MPI with Python. We've found a number of > different bindings, mympi and pympi seem to be the most commonly used > but there are others as well. We're not Python experts and were > wondering what others might suggest is the "best" MPI binding to use > with Python. > > thanks, > charlie > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf On my experience, it depends on your needs. Sometimes in order to handle all the data structures that python allows, a pickle is done before actually sending the message, which obviously increases the latency. If you want to go faster you should avoid this and send/receive plain arrays. pypar implements this. I don't know if pypar is the only wrapper which allows for this but it works quite well. On the other hand it doesn't implement all the mpi calls. -- Jaime D. Perea Duarte. Linux registered user #10472 Dep. Astrofisica Extragalactica. Instituto de Astrofisica de Andalucia (CSIC) Apdo. 3004, 18080 Granada, Spain. From patrick.ohly at intel.com Tue Nov 6 00:58:58 2007 From: patrick.ohly at intel.com (Patrick Ohly) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Virtualisation and high performance interconnects. In-Reply-To: References: <4729DC92.6040603@myri.com> <20071102191827.GA8587@nlxdcldnl2.cl.intel.com> Message-ID: <1194339538.9038.38.camel@ecld0pohly> On Fri, 2007-11-02 at 15:18 -0400, Robert G. Brown wrote: > On Fri, 2 Nov 2007, Lombard, David N wrote: > > > On Thu, Nov 01, 2007 at 01:00:00PM -0400, Robert G. Brown wrote: > >> > >> I don't know how many of these problems are really "fundamental" though. > >> I wish I knew more about the hardware support for VMs. All I know is > >> that current chips have it, but I don't know what it consists of. Does > >> anybody? Want to share? > > > > I'm not quite sure you're looking for this level of detail, but the Intel > > docs are here: > > > > > > Volume 3B: System Programming Guide specifcally deals with virtualization. > > I was kinda hoping for the reader's digest version, yeah, but I'll try > to find time to take a "figure it all out peek" sometime in the next > century...;-) Perhaps you'll find these articles more approachable: Extending Xen with Intel? Virtualization Technology http://download.intel.com/technology/itj/2006/v10i3/v10-i3-art03.pdf Intel? Virtualization Technology Specification for the IA-32 Intel? Architecture http://www.cs.utah.edu/classes/cs7940-010-rajeev/spr06/papers/vm.pdf There is also a recent paper about IO and virtual machines: Intel? Virtualization Technology for Directed I/O http://download.intel.com/technology/computing/vptech/Intel(r)_VT_for_Direct_IO.pdf -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter. The email footer below is automatically added to comply with company policy; for this email the "intended recipient(s)" are all human and non-human inhabitants of planet Earth. --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. From andrew at moonet.co.uk Wed Nov 7 08:16:35 2007 From: andrew at moonet.co.uk (andrew holway) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] SPECint_base2000 Message-ID: Does anyone have the SPECint_base2000 figures for the new intel harpertowns? Will they perform much better then Clovertown in this. Thanks Andy From tom.elken at qlogic.com Wed Nov 7 10:45:11 2007 From: tom.elken at qlogic.com (Tom Elken) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] SPECint_base2000 In-Reply-To: References: Message-ID: <6DB5B58A8E5AB846A7B3B3BFF1B4315A016EBE32@AVEXCH1.qlogic.org> > -----Original Message----- > From: beowulf-bounces@beowulf.org > [mailto:beowulf-bounces@beowulf.org] On Behalf Of andrew holway > Sent: Wednesday, November 07, 2007 8:17 AM > To: Beowulf List > Subject: [Beowulf] SPECint_base2000 > > Does anyone have the SPECint_base2000 figures for the new > intel harpertowns? There are no published #s at www.spec.org on SPECint_base2006 -- SPECint_base2000 is obsolete, so you won't see any results posted on that metric. According to Wikipedia, these will be named Xeon E54xx and Xeon X54xx. At SPEC, there are no CPU2006 results with 'E54' or 'X54' in the CPU name. Wikipedia says that the launch will be on 11/11, so you should see SPECint_base2006 results soon. > > Will they perform much better then Clovertown in this. An article at Anandtech on database performance http://www.anandtech.com/IT/showdoc.aspx?i=3099 suggests the increase in performance will be small -- more when you include performance per watt. On SPECint, the larger L2 cache (6MB vs. 4MB per 2 cores) will probably help some, but not as much as for SPECfp. -Tom > > Thanks > > Andy > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org To change your > subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From midair77 at gmail.com Wed Nov 7 11:01:43 2007 From: midair77 at gmail.com (Steven Truong) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Need recommendation for a new 512 core Linux cluster Message-ID: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> Hi, all. I would like to know for that many cores, what kind of file system should we go with? Currently we have a couple of clusters with around 100 cores and NFS seems to be ok but not great. We definitely need to put in place a parallel file system for this new cluster and I do not know which one I should go with? Lustre, GFS, PVFS2 or what else? Could you share your experiences regarding this aspect? I also would like to know how many head nodes should I need to manage jobs and queues. And what else should I have to worry about? Thank you very much for sharing any experiences. From landman at scalableinformatics.com Wed Nov 7 16:22:40 2007 From: landman at scalableinformatics.com (Joe Landman) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Need recommendation for a new 512 core Linux cluster In-Reply-To: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> References: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> Message-ID: <473256D0.3060906@scalableinformatics.com> Steven Truong wrote: > Hi, all. I would like to know for that many cores, what kind of file > system should we go with? Currently we have a couple of clusters with > around 100 cores and NFS seems to be ok but not great. We definitely > need to put in place a parallel file system for this new cluster and I > do not know which one I should go with? Lustre, GFS, PVFS2 or what > else? Could you share your experiences regarding this aspect? Hi Steven: What is the nature of your IO? That is, are your jobs dominated by large sequential reads and writes, or are the nodes effectively reading/writing when they want (small, random-ish IO). Are your programs already set for parallel IO (MPI-IO), or is there a single node that handles most of your IO requests for your jobs We have looked at GFS recently for some of our storage cluster offerings, and while inexpensive, it appears to have some bottlenecks which render it less than ideal for HPC cluster storage. There are some papers on technologies to improve it: http://www.cse.ohio-state.edu/~liangs/paper/liang-icpp2006.pdf The general contenders could be Lustre, PVFS2, and a few others. As Lustre was just acquired by Sun, my concern would be continued Linux support going forward. Again, all of this depends upon your read/write patterns, and what you want to do with it (is this scratch/temp space, or "permanent" storage space)? > > I also would like to know how many head nodes should I need to manage > jobs and queues. And what else should I have to worry about? This depends upon usage patterns. How critical is it that your job scheduler stay up? How many users will be submitting jobs? Will they do so interactively, or via web tools? Which scheduler do you plan to deploy? Which OS? 512 cores would fit nicely in 32 nodes with quad socket quad core. Floor space shouldn't be an issue. Heat/power could be. > > Thank you very much for sharing any experiences. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From bill at cse.ucdavis.edu Wed Nov 7 16:29:52 2007 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Need recommendation for a new 512 core Linux cluster In-Reply-To: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> References: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> Message-ID: <47325880.7070001@cse.ucdavis.edu> Steven Truong wrote: > Hi, all. I would like to know for that many cores, what kind of file > system should we go with? That would depend on how you plan to use it, how big it is, how much money you have, local expertise, performance expectations, reliability expectations, expected parallelism, average filesize, locking requirements (posix? bitrange? relaxed? none), usage (mpi-io? database? multi-writer? append? pub/get?) and other variables. > Currently we have a couple of clusters with > around 100 cores and NFS seems to be ok but not great. Quantifying not great would be useful. Quantifying exactly what bandwidth or latency in your I/O system is supporting a workload you are working with is even better. Even something as crude as looking at the ifconfig packet counters can be useful. Many cluster packages (like rocks and others) support ganglia that allow for looking at things like network bandwidth used per hour, week, month. If those graphs happen to include your NAS nodes all the better. > We definitely > need to put in place a parallel file system for this new cluster and I > do not know which one I should go with? Lustre, GFS, PVFS2 or what > else? Could you share your experiences regarding this aspect? This has come up multiple times, I suggest looking at the beowulf archives. If you are writing your own applications I'd look at hadoop/googlefs if put/get kind of behavior looks like a good fit. http://www.dcache.org/ looks rather promising for some workloads as well and features (corrections welcome) nfs-4.1 protocol support for supporting clients that don't have to know the details of the distributed magic at work. In general I'd recommend figuring out your performance and reliability requirements FIRST, then decide if you really need parallel. It's not rocket science these days to put 16-48 disks in a single box, a few 600-800MB/sec raid controllers and a 10G uplink to a switch to allow reasonable performance for some uses. The biggest pitfalls I've seen with lustre and pvfs2 (corrections welcome) is they depend on the reliability of the hardware, so you end up either feeling the pain when a storage node dies, or hooking up pairs of storage nodes to one array and handling fallover (and hoping the array doesn't die). ceph.sourceforge.net looks interesting in that it doesn't depend on reliable storage nodes, but it doesn't look production ready quite yet (corrections welcome). > I also would like to know how many head nodes should I need to manage > jobs and queues. And what else should I have to worry about? I'd say 1 for the jobs and queue, 512 cores is only on the order of 64-128 nodes usually, not particularly hard to keep track of the queue unless you have some strange setup like 1M 1 second jobs you want to submit hourly. or something. Again more detail needed. > > Thank you very much for sharing any experiences. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at mcmaster.ca Wed Nov 7 17:18:08 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Need recommendation for a new 512 core Linux cluster In-Reply-To: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> References: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> Message-ID: > Hi, all. I would like to know for that many cores, what kind of file > system should we go with? what kind of workload? 512c, these days, could be 64 nodes or fewer. will you have a fast (ie, > Gb) interconnect available? > Currently we have a couple of clusters with > around 100 cores and NFS seems to be ok but not great. since nontrivial $ is involved, I'd recommend quantifying that a bit more. for instance, suspend all your jobs and run a pure bandwidth test on your NFS server(s). then resume all the jobs, but collect basic data on aggregate IO (log vmstat, /proc/partitions, sar, even tcpdump) over a day or two I'd say that achieving 2-500 MB/s aggregate to a single NFS server (assuming IB or 10G) is pretty easy today. is that enough? if it's not enough, can simply run 2-4 such servers? yes, having a single namespace is very convenient, but these days, disks are embarassingly large and cheap. even a very small, managable and cheap single server is ~10 TB. if your goals is a total of 40TB but any one user will only use ~2TB, then there's not much downside (and some upsides) to implementing 40 as 4x10TB. > We definitely > need to put in place a parallel file system for this new cluster and I > do not know which one I should go with? Lustre, GFS, PVFS2 or what > else? Could you share your experiences regarding this aspect? my experience with Lustre is via HP SFS, which means it's constrained by HP's particular hw and sw choices. we have 4x 70TB and 1x 200TB SFS system. they are now reasonably stable (but didn't start out that way), and we don't appear to have bandwidth or througput problems. (we have >2000 users, very heterogenous, spanning serial to big mpi.) our main SFS clusters have 12-24 OSS's (content servers), with SATA-based storage that's dual-path'ed. metadata performance is annoying to users, since any nontrivial directory takes noticably long to run ls in (users usually have ls='ls -FC', so listing means statting every entry.) and if you have admin-type needs, such as periodically profiling all files by user/size/hash, it's a real problem to traverse 10-20M used inodes at ~700 stats/second. > I also would like to know how many head nodes should I need to manage > jobs and queues. one. do you have tons of very short jobs, or a very inefficient scheduler? since a head node can be pretty cheap (2-4 GB, 2-4 cores), I'd probably throw a coupl extra in. you can burn an almost arbitrary amount of cycles doing monitoring/logging-type things, and it's very handy to have a node to failover scheduler/logging/monitoring-type services onto. I would definitely keep such admin nodes distinct from either fileserver or user-login nodes. regards, mark hahn. From hahn at mcmaster.ca Wed Nov 7 20:58:35 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 5 01:06:33 2008 Subject: [Beowulf] Need recommendation for a new 512 core Linux cluster In-Reply-To: <47325880.7070001@cse.ucdavis.edu> References: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> <47325880.7070001@cse.ucdavis.edu> Message-ID: > welcome) nfs-4.1 protocol support for supporting clients that don't have > to know the details of the distributed magic at work. so far, my understanding is that pnfs lets a naive client work transparently only by routing the IO through the metadata server. parallel-aware clients will distinguish between the MDS and content servers, and can carry on parallel IO direct to the latter. -mark From laytonjb at charter.net Fri Nov 9 04:58:47 2007 From: laytonjb at charter.net (Jeffrey B. Layton) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] Need recommendation for a new 512 core Linux cluster In-Reply-To: References: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> <47325880.7070001@cse.ucdavis.edu> Message-ID: <47345987.4000207@charter.net> Mark Hahn wrote: >> welcome) nfs-4.1 protocol support for supporting clients that don't have >> to know the details of the distributed magic at work. > > so far, my understanding is that pnfs lets a naive client work > transparently > only by routing the IO through the metadata server. parallel-aware > clients will distinguish between the MDS and content servers, and can > carry on parallel IO direct to the latter. I'm pretty sure that the pNFS requires all clients and drivers to allow parallel access from the clients to the storage outside of the metadata server. While I haven't read the specs, I'm pretty sure this is true. If the clients didn't allow parallel access to the storage outside of the metadata server then it's not NFS 4.1. It's just plain NFS 3 or 4. Jeff From peter.st.john at gmail.com Thu Nov 8 09:36:17 2007 From: peter.st.john at gmail.com (Peter St. John) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? Message-ID: Recently, probably you noticed, Walmart began selling a $200 linux PC. (Apparently the OS is just Ubuntu 7.10 with a small xindow manager instead of Gnome or KDE). Now Slashdot points to http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold separately for $60 ("development board"). It has 1.5GHz CPU, unpopulated memory (slots for 2GB), one 10/100 connection. Does this look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 of them as compute nodes per 8 port router, with a bigger head node for fileserving. (actually I'll use a spare room but you know what I mean). An arrangement like this might be faster RAM access per core, compared to multicore, since each core has no competition for is't own memory, right? Thanks, Peter From andrew at moonet.co.uk Thu Nov 8 09:52:40 2007 From: andrew at moonet.co.uk (andrew holway) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] encoding video cluster style. Message-ID: I have a friend who encodes huge amounts of video from DV to wmv. He wants to speed up the process so I suggested that he might want to parallelise. Is there anyone that can dig me out of this hole as suggest how I might go about doing this. there are open source codes, libavcodec that could maybe be parallelised . Possible? Ta Andy From landman at scalableinformatics.com Thu Nov 8 10:22:38 2007 From: landman at scalableinformatics.com (Joe Landman) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] encoding video cluster style. In-Reply-To: References: Message-ID: <473353EE.1030100@scalableinformatics.com> andrew holway wrote: > I have a friend who encodes huge amounts of video from DV to wmv. He > wants to speed up the process so I suggested that he might want to > parallelise. > > Is there anyone that can dig me out of this hole as suggest how I > might go about doing this. there are open source codes, libavcodec > that could maybe be parallelised . > Possible? I would think a coarse grain parallelization is what is indicated. This is to a degree, how rendering is done. Some number of frames at a time distributed to some number of nodes. Basically have converters running on N nodes, each getting start/stop frame indexes. From there, each does the conversion. A final process stitches the frames together. > > Ta > > Andy > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From rgb at phy.duke.edu Thu Nov 8 10:39:46 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: On Thu, 8 Nov 2007, Peter St. John wrote: > Recently, probably you noticed, Walmart began selling a $200 linux PC. > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager > instead of Gnome or KDE). Now Slashdot points to > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > separately for $60 ("development board"). It has 1.5GHz CPU, > unpopulated memory (slots for 2GB), one 10/100 connection. Does this > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > of them as compute nodes per 8 port router, with a bigger head node > for fileserving. (actually I'll use a spare room but you know what I > mean). An arrangement like this might be faster RAM access per core, > compared to multicore, since each core has no competition for is't own > memory, right? Well by now you surely have heard the YMMV litany enough times not to hear it again from me, but YMMV quite a bit here so let me indicate a few potential difficulties. a) For this money, I'm guessing the CPU is a 32 bit Celery, which has a very small L2. For some code this won't matter, but if you're worrying about multiple cores and a memory bottleneck, let me assure you the L2 bottleneck on a single 32-bit channel will likely be much worse. b) Amdahl's law rewards higher clock and fewer CPUs over lower clock and more CPUs almost (but not quite) without exception. I doubt that you are an exception. c) A 64-bit CPU has some superlinear speedup compared to a 32-bit CPU at constant clock, for memory bound code especially. 64-bit CPUs have much larger caches as well. This CAN work against you for very cache unfriendly code, but again in 99% of all applications it will work for you -- it is what a cache "does". d) A perfectly fair question is to what extent the memory bus is oversubscribed on a 64-bit dual core, say, a very cheap AMD-64 at roughly twice the clock, with more than twice the total memory bandwidth, and with two cores. This is the question that depends in detail on YOUR APPLICATION. Many applications are de facto CPU bound and you get clock speed scaling within a CPU family all the way down to small cache Celerons. Others are vehemently not. "YMMV", so you have to analyze YOUR application to figure out which it is, where the easiest way by far to find out is to just try it. Sounds like it will cost you somewhere between $100 and $200 to set up a minimal system -- cheap case/power, motherboard, memory, a borrowed video card. You can probably beg, borrow, or buy a dual core AMD at some middling low (but much higher!) clock for no more than $400. Run your presumably EP application on the one, and on the other two at a time. Buy lots of the winner, use the loser as a desktop or head node (even the Celery should be fine for that, especially on a 100 Mbps network). Now, I'm a gambling man (as you may not know) and I will bet you one bottle, can, or glass of ice-cold or cellar cool clean and refreshing or thick and chewy beer as the winner prefers, to be delivered at a mutually convenient time (such as both of us sitting side by side at in a venue that purveys said beverages), that the medium-low end AMD-64 kicks the ass of the maximally cheap Celery in price-performance on your application (where I have an unfair advantage in that I know something about your application, but I'd make the same bet if I didn't). To go into detail, I expect that at contant cost you'll end up with somewhere in the ballpark of 2-3x aggregate bogomips/$ from the AMD, that memory bottlenecks will eat up no more than a small part of it (I actually expect the AMDs to win here TOO because of the probably at least doubled total memory bandwidth and larger cache), that when you factor in a roughly 4x increase in required system volume and 3x increase in total power consumption required to run the same number of Celeries that will match the AMD, at a marginal cost of roughly $200/year in increased power costs and some increased investment of your "free" time to install and mange the extra systems... well, let's just say that I think that the Celeries will look ugly. And I'd expect similar savings from the lowball dual core Xeons, honesly -- system price around $350-500 stripped to match where you vary in this range to find the sweet spot in terms of total memory, processor clock, and other configuration details. Before you turn me down, note that this is a win-win bet for both of us, since the winner gets to buy the next round...;-) rgb > Thanks, > Peter > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone(cell): 1-919-280-8443 Web: http://www.phy.duke.edu/~rgb Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From diep at xs4all.nl Thu Nov 8 10:58:09 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: For a compute cluster wouldn't it be a thought to also consider the cost of 3 years of nonstop electricity for the amount of gflops it delivers? Vincent On Nov 8, 2007, at 6:36 PM, Peter St. John wrote: > Recently, probably you noticed, Walmart began selling a $200 linux PC. > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager > instead of Gnome or KDE). Now Slashdot points to > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > separately for $60 ("development board"). It has 1.5GHz CPU, > unpopulated memory (slots for 2GB), one 10/100 connection. Does this > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > of them as compute nodes per 8 port router, with a bigger head node > for fileserving. (actually I'll use a spare room but you know what I > mean). An arrangement like this might be faster RAM access per core, > compared to multicore, since each core has no competition for is't own > memory, right? > Thanks, > Peter > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From peter.st.john at gmail.com Thu Nov 8 11:02:41 2007 From: peter.st.john at gmail.com (Peter St. John) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: RGB, I look forward to buying you a beer :-) Hmmm, Dopple Bock. Unfortunately Christmas will be in CA this year, not NC, on account of my nephews new baby. (Of course it's a good thing in other ways :-) I'm thinking, it might be fun to have say 4 cheapo 1.5GHz nodes and say 1 3GHz with two cores and let them compete. My algorithm wants to referee. (My algorithms talk to me.) Your points about the L2 and the 32-bit-ness give me pause. I maybe don't want to saddle myself with 32 bit things that won't play well with near-future 64 bit things, but OTOH it's cheap to look see. Thanks, Peter On Nov 8, 2007 1:39 PM, Robert G. Brown wrote: > > On Thu, 8 Nov 2007, Peter St. John wrote: > > > Recently, probably you noticed, Walmart began selling a $200 linux PC. > > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager > > instead of Gnome or KDE). Now Slashdot points to > > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > > separately for $60 ("development board"). It has 1.5GHz CPU, > > unpopulated memory (slots for 2GB), one 10/100 connection. Does this > > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > > of them as compute nodes per 8 port router, with a bigger head node > > for fileserving. (actually I'll use a spare room but you know what I > > mean). An arrangement like this might be faster RAM access per core, > > compared to multicore, since each core has no competition for is't own > > memory, right? > > Well by now you surely have heard the YMMV litany enough times not to > hear it again from me, but YMMV quite a bit here so let me indicate a > few potential difficulties. > > a) For this money, I'm guessing the CPU is a 32 bit Celery, which has a > very small L2. For some code this won't matter, but if you're worrying > about multiple cores and a memory bottleneck, let me assure you the L2 > bottleneck on a single 32-bit channel will likely be much worse. > > b) Amdahl's law rewards higher clock and fewer CPUs over lower clock > and more CPUs almost (but not quite) without exception. I doubt that > you are an exception. > > c) A 64-bit CPU has some superlinear speedup compared to a 32-bit CPU > at constant clock, for memory bound code especially. 64-bit CPUs have > much larger caches as well. This CAN work against you for very cache > unfriendly code, but again in 99% of all applications it will work for > you -- it is what a cache "does". > > d) A perfectly fair question is to what extent the memory bus is > oversubscribed on a 64-bit dual core, say, a very cheap AMD-64 at > roughly twice the clock, with more than twice the total memory > bandwidth, and with two cores. This is the question that depends in > detail on YOUR APPLICATION. Many applications are de facto CPU bound > and you get clock speed scaling within a CPU family all the way down to > small cache Celerons. Others are vehemently not. "YMMV", so you have > to analyze YOUR application to figure out which it is, where the easiest > way by far to find out is to just try it. > > Sounds like it will cost you somewhere between $100 and $200 to set up a > minimal system -- cheap case/power, motherboard, memory, a borrowed > video card. You can probably beg, borrow, or buy a dual core AMD at > some middling low (but much higher!) clock for no more than $400. Run > your presumably EP application on the one, and on the other two at a > time. Buy lots of the winner, use the loser as a desktop or head node > (even the Celery should be fine for that, especially on a 100 Mbps > network). > > Now, I'm a gambling man (as you may not know) and I will bet you one > bottle, can, or glass of ice-cold or cellar cool clean and refreshing or > thick and chewy beer as the winner prefers, to be delivered at a > mutually convenient time (such as both of us sitting side by side at in > a venue that purveys said beverages), that the medium-low end AMD-64 > kicks the ass of the maximally cheap Celery in price-performance on your > application (where I have an unfair advantage in that I know something > about your application, but I'd make the same bet if I didn't). > > To go into detail, I expect that at contant cost you'll end up with > somewhere in the ballpark of 2-3x aggregate bogomips/$ from the AMD, > that memory bottlenecks will eat up no more than a small part of it (I > actually expect the AMDs to win here TOO because of the probably at > least doubled total memory bandwidth and larger cache), that when you > factor in a roughly 4x increase in required system volume and 3x > increase in total power consumption required to run the same number of > Celeries that will match the AMD, at a marginal cost of roughly > $200/year in increased power costs and some increased investment of your > "free" time to install and mange the extra systems... well, let's just > say that I think that the Celeries will look ugly. And I'd expect > similar savings from the lowball dual core Xeons, honesly -- system > price around $350-500 stripped to match where you vary in this range to > find the sweet spot in terms of total memory, processor clock, and other > configuration details. > > Before you turn me down, note that this is a win-win bet for both of us, > since the winner gets to buy the next round...;-) > > rgb > > > Thanks, > > Peter > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > > -- > Robert G. Brown > Duke University Dept. of Physics, Box 90305 > Durham, N.C. 27708-0305 > Phone(cell): 1-919-280-8443 > Web: http://www.phy.duke.edu/~rgb > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > From peter.st.john at gmail.com Thu Nov 8 11:04:31 2007 From: peter.st.john at gmail.com (Peter St. John) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: Vincent, That's tough for me to answer, presumably the 1.5 is cheaper per hertz in power than a 3 GHz, but because of the other issues it may not be cheaper in GFLOPS per power. No hablo EE. Peter On Nov 8, 2007 1:58 PM, Vincent Diepeveen wrote: > For a compute cluster wouldn't it be a thought to also consider the > cost of 3 years of nonstop electricity for the amount of gflops it > delivers? > > Vincent > > > On Nov 8, 2007, at 6:36 PM, Peter St. John wrote: > > > Recently, probably you noticed, Walmart began selling a $200 linux PC. > > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager > > instead of Gnome or KDE). Now Slashdot points to > > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > > separately for $60 ("development board"). It has 1.5GHz CPU, > > unpopulated memory (slots for 2GB), one 10/100 connection. Does this > > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > > of them as compute nodes per 8 port router, with a bigger head node > > for fileserving. (actually I'll use a spare room but you know what I > > mean). An arrangement like this might be faster RAM access per core, > > compared to multicore, since each core has no competition for is't own > > memory, right? > > Thanks, > > Peter > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > From eugen at leitl.org Thu Nov 8 11:24:22 2007 From: eugen at leitl.org (Eugen Leitl) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: <20071108192422.GY4005@leitl.org> On Thu, Nov 08, 2007 at 01:39:46PM -0500, Robert G. Brown wrote: > a) For this money, I'm guessing the CPU is a 32 bit Celery, which has a > very small L2. For some code this won't matter, but if you're worrying It's actually a VIA C7-D. Way worse than a Celery in absolute performance, but might be okayish to stellar in terms of Ops/Joule. Has a real RNG as well as AES and SHA-1/256 as well as a Montgomery multiplier. Sounds good for a firewall, except even here you'd want Intel NICs. -- Eugen* Leitl leitl http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE From gmpc at sanger.ac.uk Thu Nov 8 11:34:35 2007 From: gmpc at sanger.ac.uk (Guy Coates) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] encoding video cluster style. In-Reply-To: <473353EE.1030100@scalableinformatics.com> References: <473353EE.1030100@scalableinformatics.com> Message-ID: <473364CB.9090509@sanger.ac.uk> Joe Landman wrote: > andrew holway wrote: >> I have a friend who encodes huge amounts of video from DV to wmv. He >> wants to speed up the process so I suggested that he might want to >> parallelise. Take a look at DVD::RIP; it allows you to do parallel video transcoding. It come with a nice gui and a simple queuing system. It is actually a pretty good introduction to clustered computing. http://www.exit1.org/dvdrip/ Cheers, Guy -- Dr. Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK Tel: +44 (0)1223 834244 x 6925 Fax: +44 (0)1223 496802 -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From jerker at Update.UU.SE Wed Nov 7 23:01:12 2007 From: jerker at Update.UU.SE (Jerker Nyberg) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] Need recommendation for a new 512 core Linux cluster In-Reply-To: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> References: <28bb77d30711071101v436346ddg3f6295ec2b05f69b@mail.gmail.com> Message-ID: On Wed, 7 Nov 2007, Steven Truong wrote: > We definitely need to put in place a parallel file system for this new > cluster and I do not know which one I should go with? Lustre, GFS, > PVFS2 or what else? Could you share your experiences regarding this > aspect? Also check out GlusterFS at http://www.gluster.org/glusterfs.php. It is parallel and may stripe and replicate the files between volumes and servers. GPLv3. (I run it for testing on two small old clusters, if you have any questions let me know or ask at the gluster-mailinglist). --Jerker Nyberg, Uppsala, Sweden From James.P.Lux at jpl.nasa.gov Thu Nov 8 12:09:11 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: <6.2.3.4.2.20071108120804.02f12ed8@mail.jpl.nasa.gov> At 09:36 AM 11/8/2007, Peter St. John wrote: >Recently, probably you noticed, Walmart began selling a $200 linux PC. See, e.g., http://home.earthlink.net/~jimlux/beowulf/walmart.htm from 2002 >(Apparently the OS is just Ubuntu 7.10 with a small xindow manager >instead of Gnome or KDE). Now Slashdot points to >http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold >separately for $60 ("development board"). It has 1.5GHz CPU, >unpopulated memory (slots for 2GB), one 10/100 connection. Does this >look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 >of them as compute nodes per 8 port router, with a bigger head node >for fileserving. (actually I'll use a spare room but you know what I >mean). An arrangement like this might be faster RAM access per core, >compared to multicore, since each core has no competition for is't own >memory, right? >Thanks, >Peter >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Thu Nov 8 12:12:00 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: On Thu, 8 Nov 2007, Peter St. John wrote: > Vincent, > That's tough for me to answer, presumably the 1.5 is cheaper per hertz > in power than a 3 GHz, but because of the other issues it may not be > cheaper in GFLOPS per power. No hablo EE. As I said in my longer reply, total system power costs per IP or FLOP are probably going to be much higher for the smaller system. Desktop systems typically draw 80-120 watts per CPU almost independent of clock speed, varying according to load. I haven't measured the AMD-64 and don't know what CPU is involved in the WalMart box, but I'm guestimating 100W each, with the AMD as noted 3-4x faster, so there IS a considerable cost differential in the power per useable instruction. Unless, of course, I'm wrong. Kill-a-watt time. A measurement is worth a thousand estimates. rgb > Peter > > On Nov 8, 2007 1:58 PM, Vincent Diepeveen wrote: >> For a compute cluster wouldn't it be a thought to also consider the >> cost of 3 years of nonstop electricity for the amount of gflops it >> delivers? >> >> Vincent >> >> >> On Nov 8, 2007, at 6:36 PM, Peter St. John wrote: >> >>> Recently, probably you noticed, Walmart began selling a $200 linux PC. >>> (Apparently the OS is just Ubuntu 7.10 with a small xindow manager >>> instead of Gnome or KDE). Now Slashdot points to >>> http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold >>> separately for $60 ("development board"). It has 1.5GHz CPU, >>> unpopulated memory (slots for 2GB), one 10/100 connection. Does this >>> look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 >>> of them as compute nodes per 8 port router, with a bigger head node >>> for fileserving. (actually I'll use a spare room but you know what I >>> mean). An arrangement like this might be faster RAM access per core, >>> compared to multicore, since each core has no competition for is't own >>> memory, right? >>> Thanks, >>> Peter >> >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >> >> > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone(cell): 1-919-280-8443 Web: http://www.phy.duke.edu/~rgb Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From deadline at eadline.org Thu Nov 8 12:36:59 2007 From: deadline at eadline.org (Douglas Eadline) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> Peter, Having some experience with low cost hardware, If you are doing number crunching multi-core seems to provide the best bang for buck. The following is the HPL performance that you can get for $2500. The Kronos and Microwulf clusters are detailed on http://clustermonkey.net, Norbert is the subject of a November Linux Magazine article. CPU Cluster Name Clock Release HPL Processor Speed (MHz) Date Performance --------------------------------------------------------------------- Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) If you draw a line (3 points I know) you get to 80 GFLOPS by 2010. Actually with some tweaking I got Norbert up to 47.7 HPL GFLOPS. And, notice I qaulify the performance as "HPL GFLOPS" as YMMV. With really low cost systems one important aspect is the interconnect. The PCIe buses on low end motherboards allows one to use inexpensive PCIe (Intel) Ethernet cards vs 32 PCI. Some of the on-board GigE implementations are not very good. -- Doug > Recently, probably you noticed, Walmart began selling a $200 linux PC. > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager > instead of Gnome or KDE). Now Slashdot points to > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > separately for $60 ("development board"). It has 1.5GHz CPU, > unpopulated memory (slots for 2GB), one 10/100 connection. Does this > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > of them as compute nodes per 8 port router, with a bigger head node > for fileserving. (actually I'll use a spare room but you know what I > mean). An arrangement like this might be faster RAM access per core, > compared to multicore, since each core has no competition for is't own > memory, right? > Thanks, > Peter > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > !DSPAM:473349b8291457110611695! > -- Doug From mathog at caltech.edu Thu Nov 8 12:54:29 2007 From: mathog at caltech.edu (David Mathog) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? Message-ID: Eugen Leitl wrote: > It's actually a VIA C7-D. Way worse than a Celery in absolute performance, > but might be okayish to stellar in terms of Ops/Joule. To expand on the stellar part slightly, if the proposed cluster is to be powered on all the time, but won't be in use all the time, it's worth noting that the C7 has very low power modes which are still running, albeit very slowly. This would make such a cluster quite responsive in terms of starting jobs from the "just sitting around" state, without requiring any of the complications, overhead, and longish delays associated with returning from sleep modes. In theory the kernel should handle this power conservation for you automatically if the cpufreq modules are installed and configured properly. The power consumption when the system is sitting around would be minimal, probably just a few watts. This assumes that the board in question actually supports these low power modes. This is probably NOT a safe assumption for an el cheopo board. Regards, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From john.leidel at gmail.com Thu Nov 8 13:04:46 2007 From: john.leidel at gmail.com (John Leidel) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> References: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> Message-ID: <1194555886.4298.26.camel@e521.site> I have to second Douglas' motion on onboard ethernet devices. The latest nvidia and broadcom chipsets do not carry wonderful driver support quite yet from many open source distros. If one can afford it, an Intel PCIe GigE card is certainly worth it. On Thu, 2007-11-08 at 15:36 -0500, Douglas Eadline wrote: > Peter, > > Having some experience with low cost hardware, If you are > doing number crunching multi-core seems to provide the > best bang for buck. The following is the HPL performance that > you can get for $2500. The Kronos and Microwulf clusters > are detailed on http://clustermonkey.net, Norbert is the subject > of a November Linux Magazine article. > > CPU > Cluster Name Clock Release HPL > Processor Speed (MHz) Date Performance > --------------------------------------------------------------------- > Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) > Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) > Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) > > > If you draw a line (3 points I know) you get to 80 GFLOPS > by 2010. Actually with some tweaking I got Norbert > up to 47.7 HPL GFLOPS. And, notice I qaulify the performance > as "HPL GFLOPS" as YMMV. > > With really low cost systems one important aspect is the > interconnect. The PCIe buses on low end motherboards allows > one to use inexpensive PCIe (Intel) Ethernet cards vs > 32 PCI. Some of the on-board GigE implementations are > not very good. > > -- > Doug > > > > > > Recently, probably you noticed, Walmart began selling a $200 linux PC. > > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager > > instead of Gnome or KDE). Now Slashdot points to > > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > > separately for $60 ("development board"). It has 1.5GHz CPU, > > unpopulated memory (slots for 2GB), one 10/100 connection. Does this > > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > > of them as compute nodes per 8 port router, with a bigger head node > > for fileserving. (actually I'll use a spare room but you know what I > > mean). An arrangement like this might be faster RAM access per core, > > compared to multicore, since each core has no competition for is't own > > memory, right? > > Thanks, > > Peter > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > !DSPAM:473349b8291457110611695! > > > > From andrew at moonet.co.uk Thu Nov 8 13:03:45 2007 From: andrew at moonet.co.uk (andrew holway) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <6.2.3.4.2.20071108120804.02f12ed8@mail.jpl.nasa.gov> References: <6.2.3.4.2.20071108120804.02f12ed8@mail.jpl.nasa.gov> Message-ID: Im still not convinced, bang for buck your going to get more clustering this junk than buying commodity hardware. Benchmarks at the ready. Andy On 08/11/2007, Jim Lux wrote: > At 09:36 AM 11/8/2007, Peter St. John wrote: > >Recently, probably you noticed, Walmart began selling a $200 linux PC. > > See, e.g., http://home.earthlink.net/~jimlux/beowulf/walmart.htm from 2002 > > > > >(Apparently the OS is just Ubuntu 7.10 with a small xindow manager > >instead of Gnome or KDE). Now Slashdot points to > >http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > >separately for $60 ("development board"). It has 1.5GHz CPU, > >unpopulated memory (slots for 2GB), one 10/100 connection. Does this > >look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > >of them as compute nodes per 8 port router, with a bigger head node > >for fileserving. (actually I'll use a spare room but you know what I > >mean). An arrangement like this might be faster RAM access per core, > >compared to multicore, since each core has no competition for is't own > >memory, right? > >Thanks, > >Peter > >_______________________________________________ > >Beowulf mailing list, Beowulf@beowulf.org > >To change your subscription (digest mode or unsubscribe) visit > >http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From peter.st.john at gmail.com Thu Nov 8 13:04:19 2007 From: peter.st.john at gmail.com (Peter St. John) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> References: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> Message-ID: Doug, Thanks. I visited Clustermonkey before I came here. There's just too much good stuff. There was discussion previously, I think, maybe elsewhere, about the NIC card offloading some work from the CPU, as opposed to on-board ethernet...which is what you mean in the case of getting nics for the PCIe slot "vs 32 PCI"...oh you mean PCIe card as opposed to PCI card, not as opposed to the onboard connection. These walmart things have one connection on board, is that worth replacing with a card? Thanks, Peter On Nov 8, 2007 3:36 PM, Douglas Eadline wrote: > Peter, > > Having some experience with low cost hardware, If you are > doing number crunching multi-core seems to provide the > best bang for buck. The following is the HPL performance that > you can get for $2500. The Kronos and Microwulf clusters > are detailed on http://clustermonkey.net, Norbert is the subject > of a November Linux Magazine article. > > CPU > Cluster Name Clock Release HPL > Processor Speed (MHz) Date Performance > --------------------------------------------------------------------- > Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) > Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) > Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) > > > If you draw a line (3 points I know) you get to 80 GFLOPS > by 2010. Actually with some tweaking I got Norbert > up to 47.7 HPL GFLOPS. And, notice I qaulify the performance > as "HPL GFLOPS" as YMMV. > > With really low cost systems one important aspect is the > interconnect. The PCIe buses on low end motherboards allows > one to use inexpensive PCIe (Intel) Ethernet cards vs > 32 PCI. Some of the on-board GigE implementations are > not very good. > > -- > Doug > > > > > > > Recently, probably you noticed, Walmart began selling a $200 linux PC. > > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager > > instead of Gnome or KDE). Now Slashdot points to > > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold > > separately for $60 ("development board"). It has 1.5GHz CPU, > > unpopulated memory (slots for 2GB), one 10/100 connection. Does this > > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 > > of them as compute nodes per 8 port router, with a bigger head node > > for fileserving. (actually I'll use a spare room but you know what I > > mean). An arrangement like this might be faster RAM access per core, > > compared to multicore, since each core has no competition for is't own > > memory, right? > > Thanks, > > Peter > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > !DSPAM:473349b8291457110611695! > > > > > -- > Doug > From gerry.creager at tamu.edu Thu Nov 8 13:48:48 2007 From: gerry.creager at tamu.edu (Gerry Creager) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] I know you guys are out there Message-ID: <47338440.90707@tamu.edu> How many of our system integrators/vendors who lurk on the list (or post) are going to be in Reno next week? I need to start chatting, and getting some information. We're considering building out a high throughput cluster and need to discuss Intel/AMD options (with documentation/benchmark results), interfconnects, minimum RAM configs, power, thermal, etc. A lot of this has gone past the list in the past, both distant and recent, and I'm starting to get the cycles to dig into the archives, but I'd like to start talking 1-on-1 to get an idea of what we're spending what our options are, who I should (and shouldn't) be talking to, etc. Off-list responses probably are best... Thanks gerry -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From James.P.Lux at jpl.nasa.gov Thu Nov 8 14:35:12 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: <6.2.3.4.2.20071108143342.0316a798@mail.jpl.nasa.gov> At 10:39 AM 11/8/2007, Robert G. Brown wrote: >On Thu, 8 Nov 2007, Peter St. John wrote: > >>Recently, probably you noticed, Walmart began selling a $200 linux PC. >>(Apparently the OS is just Ubuntu 7.10 with a small xindow manager >>instead of Gnome or KDE). Now Slashdot points to >>http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold >>separately for $60 ("development board"). It has 1.5GHz CPU, >>unpopulated memory (slots for 2GB), one 10/100 connection. Does this >>look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 >>of them as compute nodes per 8 port router, with a bigger head node >>for fileserving. (actually I'll use a spare room but you know what I >>mean). An arrangement like this might be faster RAM access per core, >>compared to multicore, since each core has no competition for is't own >>memory, right? > >Well by now you surely have heard the YMMV litany enough times not to >hear it again from me, but YMMV quite a bit here so let me indicate a >few potential difficulties. > > a) For this money, I'm guessing the CPU is a 32 bit Celery, which has a >very small L2. More likely a Via CPU of some sort. Jim From James.P.Lux at jpl.nasa.gov Thu Nov 8 14:43:31 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: <6.2.3.4.2.20071108143609.03168cf0@mail.jpl.nasa.gov> At 11:04 AM 11/8/2007, Peter St. John wrote: >Vincent, >That's tough for me to answer, presumably the 1.5 is cheaper per hertz >in power than a 3 GHz, but because of the other issues it may not be >cheaper in GFLOPS per power. No hablo EE. >Peter > >On Nov 8, 2007 1:58 PM, Vincent Diepeveen wrote: > > For a compute cluster wouldn't it be a thought to also consider the > > cost of 3 years of nonstop electricity for the amount of gflops it > > delivers? > > In general, a N GHz processor will be poorer in a flops/Watt sense than a 2N GHz processor. The power draw is a combination of a fixed load plus a frequency dependent load, so for the SAME processor, running it at N/2 GHz consumes more than 50% of the power of running it at N GHz. If you go to a faster processor design, the frequency dependent load gets smaller (smaller feature sizes= smaller capacitance to charge and discharge on each transition). The core voltage is also usually smaller on higher speed processors, which also reduces the power dissipation (smaller number of joules to change the voltage from zero to one or vice versa). So, in general, a 2N GHz processor consumes less than twice the power of a N GHz processor. Complicating this all is: a) A significant fraction of the load in a PC is all the other stuff that's toggling back and forth, like memory address and data lines. This will be driven more by the FSB speed, which might be the same for the two processors. b) You may have a lower core voltage, but the regulator making that voltage may or may not be as efficient. c) Power supply efficiency can vary a LOT from model to model. All the way from 50% for a really crummy design to 95% for a good design. d) Faster processors aren't necessarily architecturally identical to slower processors. They may have different pipeline depths, different microcode, different ALU strategies, etc. It's not just a matter of shrinking the masks and turning up the clock. Jim From James.P.Lux at jpl.nasa.gov Thu Nov 8 14:52:56 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: <6.2.3.4.2.20071108120804.02f12ed8@mail.jpl.nasa.gov> Message-ID: <6.2.3.4.2.20071108144834.02f32470@mail.jpl.nasa.gov> At 01:03 PM 11/8/2007, andrew holway wrote: >Im still not convinced, bang for buck your going to get more >clustering this junk than buying commodity hardware. Benchmarks at the >ready. > >Andy No question there.. However, if you want a low capital investment toy cluster to learn on, the $200 node is quite attractive. For instance, say you had a class of 10 people wanting to learn about cluster design and admin. You could go and get a big cluster and install the necessary management tools to let them share the cluster, or use an already existing cluster. OTOH, you could also buy 10 $2K clusters, and let them have at it on their very own. They'll learn all the things about cluster admining, rebuilding images, etc. For this kind of thing, raw performance isn't important, number of nodes is, because there's a qualitative difference between running a standalone machine and running a group of N>3 machines that have to talk. Somewhere around 5-10 computers, brute force techniques to install software like sneakernet start to really break down. From diep at xs4all.nl Thu Nov 8 15:49:46 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> References: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> Message-ID: Can you actually show which software you run to get those gflops? Actual truth currently is that the quadcores are far superior because of the power draw in the long run for number crunching and a far bigger, though far from 2x faster processing speed than dual cores. for real low power number crunching of course you don't put in harddrives, that wastes energy for nothing as well as money. boot from usb obviously. Core2 is far superior, and barcelona core based chips still have to show up; AMD's oldie K8 is nowhere near the speed you need for number crunching with double precision floating points to core2. In all cases the power draw of those cpu's, regardless whether it's intel or AMD, is eating up way more watts than they quote on the internet for TDP's. Calculating with those tdp's is quite impossible. A quadcore 2.4Ghz machine, without harddrive, definitely when measured with a good multimeter and a reasonable power efficient power supply, was measured at around 170 watts. If you replace that cpu by some other cpu, you might perhaps get away with a tad less, but still close to that 170 watt. For number crunching for say a year, 170 watt hammers in bigtime into energy costs. So putting a dualcore chip inside is a ridicioulous thought in itself. On Nov 8, 2007, at 9:36 PM, Douglas Eadline wrote: > Peter, > > Having some experience with low cost hardware, If you are > doing number crunching multi-core seems to provide the > best bang for buck. The following is the HPL performance that > you can get for $2500. The Kronos and Microwulf clusters > are detailed on http://clustermonkey.net, Norbert is the subject > of a November Linux Magazine article. > > CPU > Cluster Name Clock Release HPL > Processor Speed (MHz) Date Performance > --------------------------------------------------------------------- > Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) > Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) > Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) > > > If you draw a line (3 points I know) you get to 80 GFLOPS > by 2010. Actually with some tweaking I got Norbert > up to 47.7 HPL GFLOPS. And, notice I qaulify the performance > as "HPL GFLOPS" as YMMV. > > With really low cost systems one important aspect is the > interconnect. The PCIe buses on low end motherboards allows > one to use inexpensive PCIe (Intel) Ethernet cards vs > 32 PCI. Some of the on-board GigE implementations are > not very good. > > -- > Doug > > > > >> Recently, probably you noticed, Walmart began selling a $200 linux >> PC. >> (Apparently the OS is just Ubuntu 7.10 with a small xindow manager >> instead of Gnome or KDE). Now Slashdot points to >> http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold >> separately for $60 ("development board"). It has 1.5GHz CPU, >> unpopulated memory (slots for 2GB), one 10/100 connection. Does this >> look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 >> of them as compute nodes per 8 port router, with a bigger head node >> for fileserving. (actually I'll use a spare room but you know what I >> mean). An arrangement like this might be faster RAM access per core, >> compared to multicore, since each core has no competition for is't >> own >> memory, right? >> Thanks, >> Peter >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> !DSPAM:473349b8291457110611695! >> > > > -- > Doug > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From diep at xs4all.nl Thu Nov 8 16:02:13 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: Message-ID: <7ABF4B5F-4310-48C6-A36A-53BD33692AFE@xs4all.nl> David, Building a $2500 cluster in order to NOT run software at it then nonstop beats the idea of building a cluster. The reason you build a cluster in the first place is to run software cheaper and faster than when you would run it on a single node. That assumes you actually RUN software and that you have a lack of processing power nonstop. So the machines are running all the time. Additionally it's a private cluster, not some government type thing. I tend ro remember the government model assumed in the end 70% usage effectively of processing power. That's not real true for private users of clusters. You really get far above 90% usage. So you can argue the idle states do matter in the end for energy costs, but you definitely can't assume it's idle majority of the time. Building a $2500 cluster in order to then not let it run day and night definitely is a thrown away $2500. Of course one should raise this amount of money a tad to include energy costs, or simply use the $2500 including energy costs for a year or 3, which seems to be the economic life cycle of a system, after which it draws too much power for its performance compared to the newer generation cpu's. Even if we would use the government model of 70% effective usage, then the C7 cpu's, arm boards, mips boards and all those 'cheapo' solutions always lose it to the power costs, so effectively no one who wants a lot of crunching power buys them because of that. Additionally they're real slow those cpu's compared to intels core2. On Nov 8, 2007, at 9:54 PM, David Mathog wrote: > Eugen Leitl wrote: >> It's actually a VIA C7-D. Way worse than a Celery in absolute >> performance, >> but might be okayish to stellar in terms of Ops/Joule. > > To expand on the stellar part slightly, if the proposed cluster > is to be powered on all the time, but won't be in use all the > time, it's worth noting that the C7 has very low power modes > which are still running, albeit very slowly. This would make such a > cluster quite responsive in terms of starting jobs from the "just > sitting around" state, without requiring any of the complications, > overhead, and longish delays associated with returning from > sleep modes. In theory the kernel should handle this power > conservation > for you automatically if the cpufreq modules are > installed and configured properly. The power consumption when > the system is sitting around would be minimal, probably just a few > watts. > > This assumes that the board in question actually supports these low > power modes. This is probably NOT a safe assumption for an el cheopo > board. > > Regards, > > David Mathog > mathog@caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From diep at xs4all.nl Thu Nov 8 16:12:13 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: <6.2.3.4.2.20071108120804.02f12ed8@mail.jpl.nasa.gov> Message-ID: You're right, a lot of those 'clusters' are just theoretic clusters. Those who are number crunching wouldn't ever think of buying something else than quad cores as of now. About some remark someone made that the c7 also supports encryption bla bla, you gotta be careful there. For fun i had installed bitlocker at a quadcore AMD numbercrunching machine, and the machine is broken now, just after a few months. Now of course i'm gonna get a lecture of RGB about why windoze sucks even more than its blowup rate indicates and he'd be right about it then, yet my important remark here is that dropping a few things like sha1 etc is total useless, as when number crunching then encryption is a thing you want to avoid at all costs. For ISPs who want to encrypt at tcp/ip level i bet we aren't gonna get any public info from them what their concerns is; i'd rather guess that they all go for intel mobile cpu's anyway, lower power states there definitely is important; it's only about power there. Vincent On Nov 8, 2007, at 10:03 PM, andrew holway wrote: > Im still not convinced, bang for buck your going to get more > clustering this junk than buying commodity hardware. Benchmarks at the > ready. > > Andy > > On 08/11/2007, Jim Lux wrote: >> At 09:36 AM 11/8/2007, Peter St. John wrote: >>> Recently, probably you noticed, Walmart began selling a $200 >>> linux PC. >> >> See, e.g., http://home.earthlink.net/~jimlux/beowulf/walmart.htm >> from 2002 >> >> >> >>> (Apparently the OS is just Ubuntu 7.10 with a small xindow manager >>> instead of Gnome or KDE). Now Slashdot points to >>> http://www.linuxdevices.com/news/NS5305482907.html, the MB being >>> sold >>> separately for $60 ("development board"). It has 1.5GHz CPU, >>> unpopulated memory (slots for 2GB), one 10/100 connection. Does this >>> look to y'all like fair FLOPS/$ for a kitchen project? I'm >>> thinking 6 >>> of them as compute nodes per 8 port router, with a bigger head node >>> for fileserving. (actually I'll use a spare room but you know what I >>> mean). An arrangement like this might be faster RAM access per core, >>> compared to multicore, since each core has no competition for >>> is't own >>> memory, right? >>> Thanks, >>> Peter >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From diep at xs4all.nl Thu Nov 8 16:22:28 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <6.2.3.4.2.20071108143609.03168cf0@mail.jpl.nasa.gov> References: <6.2.3.4.2.20071108143609.03168cf0@mail.jpl.nasa.gov> Message-ID: <45A38524-5B57-40D1-8057-A81D980EE7F8@xs4all.nl> Very good remark Jim, Another complicating factor is that for intel and amd there is definitely ready available libraries written in assembly that are lightning fast, whereas for C* type processors i can't remember there is much, as your "more Ghz into battle is better rule" always means that the C* chip is gonna be slower anyway, also in future incarnations over intel&amd, so why bother writing good libraries for that chip? Vincent On Nov 8, 2007, at 11:43 PM, Jim Lux wrote: > At 11:04 AM 11/8/2007, Peter St. John wrote: >> Vincent, >> That's tough for me to answer, presumably the 1.5 is cheaper per >> hertz >> in power than a 3 GHz, but because of the other issues it may not be >> cheaper in GFLOPS per power. No hablo EE. >> Peter >> >> On Nov 8, 2007 1:58 PM, Vincent Diepeveen wrote: >> > For a compute cluster wouldn't it be a thought to also consider the >> > cost of 3 years of nonstop electricity for the amount of gflops it >> > delivers? >> > > > > In general, a N GHz processor will be poorer in a flops/Watt sense > than a 2N GHz processor. > > The power draw is a combination of a fixed load plus a frequency > dependent load, so for the SAME processor, running it at N/2 GHz > consumes more than 50% of the power of running it at N GHz. > > If you go to a faster processor design, the frequency dependent > load gets smaller (smaller feature sizes= smaller capacitance to > charge and discharge on each transition). The core voltage is also > usually smaller on higher speed processors, which also reduces the > power dissipation (smaller number of joules to change the voltage > from zero to one or vice versa). So, in general, a 2N GHz > processor consumes less than twice the power of a N GHz processor. > > Complicating this all is: > a) A significant fraction of the load in a PC is all the other > stuff that's toggling back and forth, like memory address and data > lines. This will be driven more by the FSB speed, which might be > the same for the two processors. > b) You may have a lower core voltage, but the regulator making that > voltage may or may not be as efficient. > c) Power supply efficiency can vary a LOT from model to model. All > the way from 50% for a really crummy design to 95% for a good design. > d) Faster processors aren't necessarily architecturally identical > to slower processors. They may have different pipeline depths, > different microcode, different ALU strategies, etc. It's not just > a matter of shrinking the masks and turning up the clock. > > > Jim > > > From deadline at eadline.org Thu Nov 8 16:25:37 2007 From: deadline at eadline.org (Douglas Eadline) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> Message-ID: <56082.192.168.1.1.1194567937.squirrel@mail.eadline.org> > Can you actually show which software you run to get those gflops? As I said, HPL with Goto BLAS, pretty standard. > > Actual truth currently is that the quadcores are far superior because > of the power draw in the long run for number crunching and a far bigger, > though far from 2x faster processing speed than dual cores. > > for real low power number crunching of course you don't put in > harddrives, that wastes energy for nothing as well as money. > boot from usb obviously. Core2 is far superior, and barcelona core > based chips still have to show up; PXE boot is a bit more obvious (Warewulf). > > AMD's oldie K8 is nowhere near the speed you need for number > crunching with double precision floating points to core2. > > In all cases the power draw of those cpu's, regardless whether it's > intel or AMD, is eating up way more watts than they quote on the > internet for TDP's. > Calculating with those tdp's is quite impossible. > > A quadcore 2.4Ghz machine, without harddrive, definitely when > measured with a good multimeter and a reasonable power efficient > power supply, was measured at around 170 watts. > > If you replace that cpu by some other cpu, you might perhaps get away > with a tad less, but still close to that 170 watt. > > For number crunching for say a year, 170 watt hammers in bigtime into > energy costs. The Norbert cluster (4 nodes, 3 without hard drives) requires 250 Watts when Idle and 371-483 Watts when loaded running HPL. Measured at the wall socket with Kill-a-watt meter. Managing power is certainly possible when the three diskless nodes are not in use. There are two strategies 1) When not in use put the nodes in standby mode (draws about 5 Watts) and use Wake-on-LAN to start it up. Because it uses PXE boot and a RAM disk, it boots in about a minute. If you use a scheduler like SGE or Torque it should be possible to have the nodes started when needed. 2) The new tickles kernels may be able to drop the nodes in to power saving Intel C states (the lowest is about 1.5 watts) when not in use (although tickles kernels (2.6.22+) only work with 32 processors right now). The first methods is more general and the second is based on Intel processors (and is new and untested). Overall it should be possible to have a small cluster use about as much power as desktop when idle. > > So putting a dualcore chip inside is a ridicioulous thought in itself. $52 dollars (US) per GFLOPS and 11 Watts per GFLOPS speaks for itself -- Doug > > On Nov 8, 2007, at 9:36 PM, Douglas Eadline wrote: > >> Peter, >> >> Having some experience with low cost hardware, If you are >> doing number crunching multi-core seems to provide the >> best bang for buck. The following is the HPL performance that >> you can get for $2500. The Kronos and Microwulf clusters >> are detailed on http://clustermonkey.net, Norbert is the subject >> of a November Linux Magazine article. >> >> CPU >> Cluster Name Clock Release HPL >> Processor Speed (MHz) Date Performance >> --------------------------------------------------------------------- >> Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) >> Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) >> Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) >> >> >> If you draw a line (3 points I know) you get to 80 GFLOPS >> by 2010. Actually with some tweaking I got Norbert >> up to 47.7 HPL GFLOPS. And, notice I qaulify the performance >> as "HPL GFLOPS" as YMMV. >> >> With really low cost systems one important aspect is the >> interconnect. The PCIe buses on low end motherboards allows >> one to use inexpensive PCIe (Intel) Ethernet cards vs >> 32 PCI. Some of the on-board GigE implementations are >> not very good. >> >> -- >> Doug >> >> >> >> >>> Recently, probably you noticed, Walmart began selling a $200 linux >>> PC. >>> (Apparently the OS is just Ubuntu 7.10 with a small xindow manager >>> instead of Gnome or KDE). Now Slashdot points to >>> http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold >>> separately for $60 ("development board"). It has 1.5GHz CPU, >>> unpopulated memory (slots for 2GB), one 10/100 connection. Does this >>> look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 >>> of them as compute nodes per 8 port router, with a bigger head node >>> for fileserving. (actually I'll use a spare room but you know what I >>> mean). An arrangement like this might be faster RAM access per core, >>> compared to multicore, since each core has no competition for is't >>> own >>> memory, right? >>> Thanks, >>> Peter >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >>> >>> >> >> >> -- >> Doug >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > > !DSPAM:4733a0a048701735719037! > -- Doug From dag at sonsorol.org Thu Nov 8 16:27:53 2007 From: dag at sonsorol.org (Chris Dagdigian) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <7ABF4B5F-4310-48C6-A36A-53BD33692AFE@xs4all.nl> References: <7ABF4B5F-4310-48C6-A36A-53BD33692AFE@xs4all.nl> Message-ID: It is dangerous to project *your* particular use cases and workflows upon the community at large. Most of the clusters I end up building or working on (academic, government and corporate sites) are intended to support periodic spikes in computing demands. For university sites this could be the end of each semester as student projects become due and for research labs it may be for a 10 day period proceeding the submission of a major paper or grant applications. For EDA companies entire clusters may lay idle until some massive validation process needs to kick off. Clusters built to meet peak demand rarely hit 90% utilization (averaged over time) and often have lots of idle capacity sitting around waiting for a peak period to arrive. That is why we pay particular attention to things like Project Hedeby from Sun and the EGO stuff from Platform Computing along with the various homegrown based systems that people have built to power-on and power-off nodes (via IPMI) based on the length of the pending job list. In systems built for peak power and not constant throughput power control is a big deal and the eventual goal I'd love to see is more grid schedulers and resource brokers becoming hardware aware to the point of being able to power on and off nodes based on a given policy. It's coming and I've seen some neat homegrown solutions already. My $.02 as always -Chris On Nov 8, 2007, at 7:02 PM, Vincent Diepeveen wrote: > Building a $2500 cluster in order to NOT run software at it then > nonstop beats the idea of building a cluster. That assumes > you actually RUN software and that you have a lack of processing > power nonstop. So the machines are running all the time. > Additionally it's a private cluster, not some government type thing. > > I tend ro remember the government model assumed in the end 70% usage > effectively of processing power. That's not real true for private > users of clusters. You really get far above 90% usage. > > So you can argue the idle states do matter in the end for energy > costs, but you definitely can't assume it's idle majority of the > time. Building a $2500 cluster in order to then not let it run day > and night definitely is a thrown away $2500. From deadline at eadline.org Thu Nov 8 16:50:13 2007 From: deadline at eadline.org (Douglas Eadline) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> Message-ID: <41494.192.168.1.1.1194569413.squirrel@mail.eadline.org> > Doug, > Thanks. I visited Clustermonkey before I came here. There's just too > much good stuff. > There was discussion previously, I think, maybe elsewhere, about the > NIC card offloading some work from the CPU, as opposed to on-board > ethernet...which is what you mean in the case of getting nics for the > PCIe slot "vs 32 PCI"...oh you mean PCIe card as opposed to PCI card, > not as opposed to the onboard connection. These walmart things have > one connection on board, is that worth replacing with a card? Many of the new low cost motherboards use PCIe to connect onboard NICS. These should be able perform well but often don't due to driver issues or cheap chipsets. Some still use 32bit PCI interface as well. It all depends on the chipset. Most modern low-cost boards also have at least a 1X PCIe slot into which and Intel PT desktop NIC ($40) can be used. While these are not IB or Myrinet, they do perform well for a low cost part. TCP latency measured by Netpipe is usually less than 30 ?s (through a switch, 10.8 ?s has been reported if you use GAMMA). Look at the Netpipe data below to see how these NICs compare to 32 bit PCI NICs. http://basement-supercomputing.com/download/benchmarks/Intel-PT-PCIe/netpipe.Intel-PT-PCIe-throughput_vs_blocksize.png -- Doug > Thanks, > Peter > > On Nov 8, 2007 3:36 PM, Douglas Eadline wrote: >> Peter, >> >> Having some experience with low cost hardware, If you are >> doing number crunching multi-core seems to provide the >> best bang for buck. The following is the HPL performance that >> you can get for $2500. The Kronos and Microwulf clusters >> are detailed on http://clustermonkey.net, Norbert is the subject >> of a November Linux Magazine article. >> >> CPU >> Cluster Name Clock Release HPL >> Processor Speed (MHz) Date Performance >> --------------------------------------------------------------------- >> Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) >> Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) >> Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) >> >> >> If you draw a line (3 points I know) you get to 80 GFLOPS >> by 2010. Actually with some tweaking I got Norbert >> up to 47.7 HPL GFLOPS. And, notice I qaulify the performance >> as "HPL GFLOPS" as YMMV. >> >> With really low cost systems one important aspect is the >> interconnect. The PCIe buses on low end motherboards allows >> one to use inexpensive PCIe (Intel) Ethernet cards vs >> 32 PCI. Some of the on-board GigE implementations are >> not very good. >> >> -- >> Doug >> >> >> >> >> >> > Recently, probably you noticed, Walmart began selling a $200 linux PC. >> > (Apparently the OS is just Ubuntu 7.10 with a small xindow manager >> > instead of Gnome or KDE). Now Slashdot points to >> > http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold >> > separately for $60 ("development board"). It has 1.5GHz CPU, >> > unpopulated memory (slots for 2GB), one 10/100 connection. Does this >> > look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 >> > of them as compute nodes per 8 port router, with a bigger head node >> > for fileserving. (actually I'll use a spare room but you know what I >> > mean). An arrangement like this might be faster RAM access per core, >> > compared to multicore, since each core has no competition for is't own >> > memory, right? >> > Thanks, >> > Peter >> > _______________________________________________ >> > Beowulf mailing list, Beowulf@beowulf.org >> > To change your subscription (digest mode or unsubscribe) visit >> > http://www.beowulf.org/mailman/listinfo/beowulf >> > >> > >> > >> >> >> -- >> Doug >> > > !DSPAM:473379d814678362916074! > -- Doug From laytonjb at charter.net Fri Nov 9 17:21:25 2007 From: laytonjb at charter.net (Jeffrey B. Layton) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <6.2.3.4.2.20071108144834.02f32470@mail.jpl.nasa.gov> References: <6.2.3.4.2.20071108120804.02f12ed8@mail.jpl.nasa.gov> <6.2.3.4.2.20071108144834.02f32470@mail.jpl.nasa.gov> Message-ID: <47350795.9080400@charter.net> Jim Lux wrote: > At 01:03 PM 11/8/2007, andrew holway wrote: >> Im still not convinced, bang for buck your going to get more >> clustering this junk than buying commodity hardware. Benchmarks at the >> ready. >> >> Andy > > No question there.. > > However, if you want a low capital investment toy cluster to learn on, > the $200 node is quite attractive. OK, I'm coming out of the closet and admitting that I was seriously looking at these low-power motherboards with the VIA C7-D (links from www.thinkgos.com) "Hi, I'm Jeff and I'm a Cluster-Holic" My thought was to get a reasonably large ATX case with a good power supply and cram 4 of this little motherboards into the case. Having 4 of these systems with an Intel GigE NIC inside a case with a couple of 120mm fans kind of interested me. You make one of the boards the head node and connect some hard drives and CD to it along with using the on-board 10/100 for external connections. I was also going to put a 5-port GigE switch in the case also. But, as people have pointed out, this little board may not be worth it. While it is extremely low-power (20W max) is very alluring, I didn't realize it's only 32-bit. Shame about that. I still may get a board of the $200 Walmart box just to see how little power I can get before the performance is so bad I dump the box. Jeff From laytonjb at charter.net Fri Nov 9 17:33:53 2007 From: laytonjb at charter.net (Jeffrey B. Layton) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> References: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> Message-ID: <47350A81.70608@charter.net> I was going to hold this back for after Thanksgiving, but I've been working on a small project all year to track the costs of AMD and Intel systems. I chose 3 AMD CPU models (4200+, 5000+, and 6000+) and 3 Intel models (Cores 2: 2.33 GHz, 2.66 GHz and Quad-core 2.4 GHz). I then chose a couple of memory configurations to see the effects of memory fluctuations. The system configurations have everything - cases, DVD, GigE switches (I used 2 GigE switches - one for computation and one for storage), hard drives, cables, motherboards, etc. Only the head node has hard drives and the compute nodes are all diskless. The spreadsheet computes the total cost (including shipping) from the component costs for systems from 1-8 nodes and a 16 node system (I switched from 8-port GigE switches to 16-port GigE switches for this last configuration). I've been tracking the costs for these systems since about Feb. I've also added the theoretical system performance (GFLOPS) so I can compute $/GFLOPS as well as $/node, and $/core. I wish I had power estimates though. I've also been finding the $2,500 systems from this overall group. It's fun to look at how many nodes you can get for $2,500 as the year went along. I'm hoping to publish an article on ClusterMonkey later this year with the data and the plots. It's actually quite interesting. Think of it as finding the optimal system for $2,500. The most interesting thing is that the best $/GFLOPS is the Intel quad-core Q6600. But that shouldn't be too surprising since the Intel chips have 4 ops/clock and the AMD's only have 2. On a $/GFLOPS basis, the AMD's are about twice the cost of the cheapest Intel system. But you generally get more nodes with AMD than with Intel. Doug thinks I'm nuts, but then again, I have to have a hobby :) It's going to become even more interesting when the AMD quad-core hits the desktop and when Penryn pops up. Jeff > Peter, > > Having some experience with low cost hardware, If you are > doing number crunching multi-core seems to provide the > best bang for buck. The following is the HPL performance that > you can get for $2500. The Kronos and Microwulf clusters > are detailed on http://clustermonkey.net, Norbert is the subject > of a November Linux Magazine article. > > CPU > Cluster Name Clock Release HPL > Processor Speed (MHz) Date Performance > --------------------------------------------------------------------- > Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) > Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) > Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) > > > If you draw a line (3 points I know) you get to 80 GFLOPS > by 2010. Actually with some tweaking I got Norbert > up to 47.7 HPL GFLOPS. And, notice I qaulify the performance > as "HPL GFLOPS" as YMMV. > > With really low cost systems one important aspect is the > interconnect. The PCIe buses on low end motherboards allows > one to use inexpensive PCIe (Intel) Ethernet cards vs > 32 PCI. Some of the on-board GigE implementations are > not very good. > > -- > Doug > > > > > >> Recently, probably you noticed, Walmart began selling a $200 linux PC. >> (Apparently the OS is just Ubuntu 7.10 with a small xindow manager >> instead of Gnome or KDE). Now Slashdot points to >> http://www.linuxdevices.com/news/NS5305482907.html, the MB being sold >> separately for $60 ("development board"). It has 1.5GHz CPU, >> unpopulated memory (slots for 2GB), one 10/100 connection. Does this >> look to y'all like fair FLOPS/$ for a kitchen project? I'm thinking 6 >> of them as compute nodes per 8 port router, with a bigger head node >> for fileserving. (actually I'll use a spare room but you know what I >> mean). An arrangement like this might be faster RAM access per core, >> compared to multicore, since each core has no competition for is't own >> memory, right? >> Thanks, >> Peter >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> !DSPAM:473349b8291457110611695! >> >> > > > From diep at xs4all.nl Thu Nov 8 19:38:33 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: <7ABF4B5F-4310-48C6-A36A-53BD33692AFE@xs4all.nl> Message-ID: On Nov 9, 2007, at 1:27 AM, Chris Dagdigian wrote: > > It is dangerous to project *your* particular use cases and > workflows upon the community at large. > > Most of the clusters I end up building or working on (academic, > government and corporate sites) are intended to support periodic > spikes in computing demands. For university sites this could be the > end of each semester as student projects become due and for > research labs it may be for a 10 day period proceeding the > submission of a major paper or grant applications. For EDA > companies entire clusters may lay idle until some massive > validation process needs to kick off. > Clusters built to meet peak demand rarely hit 90% utilization > (averaged over time) and often have lots of idle capacity sitting > around waiting for a peak period to arrive. That is why we pay > particular attention to things like Project Hedeby from Sun and the > EGO stuff from Platform Computing along with the various homegrown > based systems that people have built to power-on and power-off > nodes (via IPMI) based on the length of the pending job list A few remarks: a) the 70% usage comes from "supercomputer report europe". b) "the community" that posts here has like NEAR TO ZERO $2500 clusters at their work, so if you happen to know 1+ then that already gives statistical significant bragging rights. Ever seen a company say: "heh here you got $2500 build me the fastest cluster you can get for that money"? Actually i'm typing at an ex-company machine, a macbookpro 17'', which i got from my previous employer and it is still 2600 euro in the shop here, which soon is a dollar or 5000. In general when crunching becomes important to a company, definitely a billion euro company, then they're gonna invest quite some more than $2500 into crunching power. c) homegrown clusters usually are not built at the same manner like the $2500 project says. Usually people buy a machine now, buy one a year later and so on, and just cluster 'em, so instead of trying to stick strict in some $2500 budget it's more like: "which cpu at what cheapo mainboard gives most dang for my bucks". that'll be a quadcore. d) for $2500 private clusters (private as in: at home) energy costs play a big role. e) another result of increased energy costs is of course sound concerns. Regarding C: A far more interesting theoretic question is of course whether you should see most homegrown clusters as a cluster or as some jbon (just a bunch of cheapo nodes), as i can't really recall most 'private house clusters' to have even MPI installed. When is it a cluster? heh you sure you still want your 2 cents in dollar currency rather than euro's? > In systems built for peak power and not constant throughput power > control is a big deal and the eventual goal I'd love to see is more > grid schedulers and resource brokers becoming hardware aware to the > point of being able to power on and off nodes based on a given > policy. It's coming and I've seen some neat homegrown solutions > already. > > My $.02 as always > > -Chris > > > > On Nov 8, 2007, at 7:02 PM, Vincent Diepeveen wrote: >> Building a $2500 cluster in order to NOT run software at it then >> nonstop beats the idea of building a cluster. That assumes >> you actually RUN software and that you have a lack of processing >> power nonstop. So the machines are running all the time. >> Additionally it's a private cluster, not some government type thing. >> >> I tend ro remember the government model assumed in the end 70% >> usage effectively of processing power. That's not real true for >> private users of clusters. You really get far above 90% usage. >> >> So you can argue the idle states do matter in the end for energy >> costs, but you definitely can't assume it's idle majority of the >> time. Building a $2500 cluster in order to then not let it run day >> and night definitely is a thrown away $2500. > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From larry.stewart at sicortex.com Thu Nov 8 19:42:10 2007 From: larry.stewart at sicortex.com (Lawrence Stewart) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] I know you guys are out there In-Reply-To: <47338440.90707@tamu.edu> References: <47338440.90707@tamu.edu> Message-ID: <66276A57-7836-443A-9F29-3CCFABDAA8DA@sicortex.com> On Nov 8, 2007, at 4:48 PM, Gerry Creager wrote: > How many of our system integrators/vendors who lurk on the list (or > post) are going to be in Reno next week? I need to start chatting, > and getting some information. > > We're considering building out a high throughput cluster and need to > discuss Intel/AMD options (with documentation/benchmark results), > interfconnects, minimum RAM configs, power, thermal, etc. > > A lot of this has gone past the list in the past, both distant and > recent, and I'm starting to get the cycles to dig into the archives, > but I'd like to start talking 1-on-1 to get an idea of what we're > spending what our options are, who I should (and shouldn't) be > talking to, etc. > > Off-list responses probably are best... > > Thanks gerry > -- > Gerry Creager -- gerry.creager@tamu.edu > Texas Mesonet -- AATLT, Texas A&M University > Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 > Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf SiCortex will be there. We're next to Microsoft, go figure. We'll have 72, 648, and 5832 processor machines on the floor. -Larry From diep at xs4all.nl Thu Nov 8 19:52:01 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: <47350A81.70608@charter.net> References: <46692.192.168.1.1.1194554219.squirrel@mail.eadline.org> <47350A81.70608@charter.net> Message-ID: <318C314D-25A8-4C78-A921-9FA87BB24C4F@xs4all.nl> Would it be interesting to open a discussion how to reduce sound of cheapo clusters as well? Now i guess for most us-guys in this group that is perhaps less interesting as their big mansion/castle in the states has probably plenty of rooms, including a nerd-go-crazy-room from which the net can get spammed, but a lot of people don't have this luxury. The real problem is that articles about getting things silent always assume some sort of 'gamer pc' that's ultra expensive where you can throw big bucks to get it quiet; cheap crunching hardware never gets attention in that sense. Of course it's logical, a loud $1 fan is gonna save more costs than a quiet 'noiseblocker' fan from germany, to just give an example. is it an idea to write something about that as well? just a thought, Vincent On Nov 10, 2007, at 2:33 AM, Jeffrey B. Layton wrote: > I was going to hold this back for after Thanksgiving, but I've been > working > on a small project all year to track the costs of AMD and Intel > systems. > I chose 3 AMD CPU models (4200+, 5000+, and 6000+) and 3 Intel > models (Cores 2: 2.33 GHz, 2.66 GHz and Quad-core 2.4 GHz). I then > chose a couple of memory configurations to see the effects of memory > fluctuations. The system configurations have everything - cases, DVD, > GigE switches (I used 2 GigE switches - one for computation and one > for storage), hard drives, cables, motherboards, etc. Only the head > node > has hard drives and the compute nodes are all diskless. > > The spreadsheet computes the total cost (including shipping) from the > component costs for systems from 1-8 nodes and a 16 node system > (I switched from 8-port GigE switches to 16-port GigE switches for > this last configuration). > > I've been tracking the costs for these systems since about Feb. > I've also > added the theoretical system performance (GFLOPS) so I can compute > $/GFLOPS as well as $/node, and $/core. I wish I had power estimates > though. > > I've also been finding the $2,500 systems from this overall group. > It's > fun to look at how many nodes you can get for $2,500 as the year > went along. I'm hoping to publish an article on ClusterMonkey later > this year with the data and the plots. It's actually quite > interesting. > Think of it as finding the optimal system for $2,500. > > The most interesting thing is that the best $/GFLOPS is the Intel > quad-core Q6600. But that shouldn't be too surprising since the > Intel chips have 4 ops/clock and the AMD's only have 2. On a > $/GFLOPS basis, the AMD's are about twice the cost of the cheapest > Intel system. But you generally get more nodes with AMD than > with Intel. > > Doug thinks I'm nuts, but then again, I have to have a hobby :) > It's going to become even more interesting when the AMD quad-core > hits the desktop and when Penryn pops up. > > > Jeff > >> Peter, >> >> Having some experience with low cost hardware, If you are >> doing number crunching multi-core seems to provide the >> best bang for buck. The following is the HPL performance that >> you can get for $2500. The Kronos and Microwulf clusters >> are detailed on http://clustermonkey.net, Norbert is the subject >> of a November Linux Magazine article. >> >> CPU >> Cluster Name Clock Release HPL >> Processor Speed (MHz) Date Performance >> --------------------------------------------------------------------- >> Kronos/Sempron 2500+ (8) 1750 7/2004 14.90 GFLOPS (Atlas) >> Microwulf/Athlon64 X2 3800+ (4) 2000 8/2005 26.25 GFLOPS (Goto) >> Norbert/Core Duo E6550 (4) 2333 7/2007 45.55 GFLOPS (Goto) >> >> >> If you draw a line (3 points I know) you get to 80 GFLOPS >> by 2010. Actually with some tweaking I got Norbert >> up to 47.7 HPL GFLOPS. And, notice I qaulify the performance >> as "HPL GFLOPS" as YMMV. >> >> With really low cost systems one important aspect is the >> interconnect. The PCIe buses on low end motherboards allows >> one to use inexpensive PCIe (Intel) Ethernet cards vs >> 32 PCI. Some of the on-board GigE implementations are >> not very good. >> >> -- >> Doug >> >> >> >> >> >>> Recently, probably you noticed, Walmart began selling a $200 >>> linux PC. >>> (Apparently the OS is just Ubuntu 7.10 with a small xindow manager >>> instead of Gnome or KDE). Now Slashdot points to >>> http://www.linuxdevices.com/news/NS5305482907.html, the MB being >>> sold >>> separately for $60 ("development board"). It has 1.5GHz CPU, >>> unpopulated memory (slots for 2GB), one 10/100 connection. Does this >>> look to y'all like fair FLOPS/$ for a kitchen project? I'm >>> thinking 6 >>> of them as compute nodes per 8 port router, with a bigger head node >>> for fileserving. (actually I'll use a spare room but you know what I >>> mean). An arrangement like this might be faster RAM access per core, >>> compared to multicore, since each core has no competition for >>> is't own >>> memory, right? >>> Thanks, >>> Peter >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >>> !DSPAM:473349b8291457110611695! >>> >>> >> >> >> > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From landman at scalableinformatics.com Thu Nov 8 20:17:29 2007 From: landman at scalableinformatics.com (Joe Landman) Date: Sat Jul 5 01:06:34 2008 Subject: [Beowulf] The Walmart Compute Node? In-Reply-To: References: <7ABF4B5F-4310-48C6-A36A-53BD33692AFE@xs4all.nl> Message-ID: <4733DF59.6080000@scalableinformatics.com> Vincent Diepeveen wrote: [...] > Ever seen a company say: "heh here you got $2500 build me the fastest > cluster you can get for that money"? ... from this I gather that you are not in the cluster computing business. > Actually i'm typing at an ex-company machine, a macbookpro 17'', which i > got from my previous employer and it is still 2600 euro in the shop > here, which soon is a dollar or 5000. > In general when crunching becomes important to a company, definitely a > billion euro company, then they're gonna inves