From kus at free.net Sat Sep 1 06:16:53 2007 From: kus at free.net (Mikhail Kuzminsky) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] Reading raw binary files in Fortran (Intel compiler)? In-Reply-To: Message-ID: In message from Mark Hahn (Fri, 31 Aug 2007 13:29:56 -0400 (EDT)): >> guessing that my raw binary read trick does not work on Intel >>Fortran? Is there >> another option I need to pass (e.g. perhaps form='binary')? > >I haven't looked closely, but have heard that different compilers >store array dimensions differently in such cases. I worked some years ago w/"binary" but *sequential* files - using Visual Fortran and ifc (some years ago they was just different ;-)). And for successful reading a list of variables (in Linux w/ifc) from created in Windows w/cvf "binary" file, as I remember, it was necessary to perform some tricks like insertion of dummy variables in the list. Now ifort and cvf are the same compiler, but I don't know how they realized 'binary' format :-( Yours Mikhail it was necessary >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Sep 1 06:23:36 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> Message-ID: On Fri, 31 Aug 2007, Jim Lux wrote: > At 02:21 PM 8/31/2007, Peter St. John wrote: >> I'm imaging this system as a computer, and the headaches of it's operator >> (the guy who scripted the worm, maybe) whose million nodes are infested by >> a million hostile users (the refeverse of a users desktop infested by a >> worm, is a worm's virutual supercomputer infested by users). > > I'm sure that the architect sees the value of the Beowulf mailing list for > such things. After all, we all like a challenge, right? Heterogenous hardware > configurations, non-deterministic latency interconnects with an ever changing > topology, configuration management issues galore. By the time we're done, > rgb will have to add another chapter to his book on building Beowulfs. I'm already planning it. The probable chapter title is "Cluster Wars" since of course this is a >>terror weapon<<. Remember, the internet is the basis of much commerce. There are any number of tasks one could assign such a viral bot-cloud cluster for good or evil, but evil (given the second law of thermodynamics:-) is a lot easier... Let's see. RSA and DES and MD5 are considered "probably uncrackable" by anyone with less than NSA-class resources, but of course this bot-cloud is several orders of magnitude more powerful than NSA's probable setup. If we go with the gaudy end of the estimate and give it 10^7 node/members, MDA is done -- one can pretty much crack anything one could have cracked with the old crypt library, even if one can only test 1 password per second per node -- nearly 10^12 passwords a day. Similarly lots of other problems become tractible to a brute force search algorithm when you can displose of order of 20 petaclocks worth of cycles. (Am I multiplying that out right? 10^7 times 2x10^9 = 2x10^16, 9 is giga, 12 is tera, 15 is peta. Yup. Petacycles.). Brute force searches require minimal IPCs, although I'm sure there are interesting problems associated with IPCs and data harvesting when it has to be done in "stealth" mode and not lead investigators back to you and when you need to make it robust against nodes dropping out (being cleaned by their owners) and popping back in (as yet another virus propagates). Then there is denial of service. Everybody knows that this is an attack, but few recognize its potential terror value. Just remember the >>cost<< of some of the countdown viruses of years past. Some of them literally shut down the Internet for close to a day -- clogging all the main arteries and switch points until hosts were run down one at a time and isolated by their hosting ISPs. The cost of those incidents in real dollars, lost productivity, and human misery was easily a billion dollars each (I read estimates that were much higher, but I don't want to be hyperbolic so let's stay conservative here). A bot-cloud attack could be far more costly and last far, far longer, in part because if it were well-designed it could shape-shift every five minutes and vary e.g. IP number, signature, target. It could also turn on and off at random times to make it very difficult to track each bot back to its infected host. If it times itself to take advantage of one of those two-month long window vulnerabilities (yes, a lot of them last for PLENTY of time for this to be feasible) so that it can essentially instantly re-infect a wide class of hosts at will as they are cleansed, it could force the shutdown of nearly every Windows system in the world until it is hand-cleaned and patched -- the Internet itself would be useless in fixing the problem. The cost of such a complete attack would be staggering -- banking, commerce, education, defense -- all at a standstill. It would probably trigger a full depression (led of course by the complete collapse of Microsoft as the full cost of its appalling and perpetual vulnerability is finally laid bare). Truthfully, I've been waiting for foreign terror powers to figure this one out and attempt such an attack, but so far we've been lucky. Bot driven attacks on individual systems of course happen all the time -- check out the logs of pretty much any server and count the number of times per day some system in Korea or South America or God Knows Where tries to probe its way down your ssh ports and standard accounts in search of an idiot who left in a default password (or put a stupid password or root). These folks aren't looking for fun, they're looking for money. Finally, there is viral spam (which might even be that very foreign power attack:-). Viral bot-driven spam is ubiquitous, and IMMENSELY costly. It costs me personally -- AFTER putting all my mail through a fairly aggressive spam filter -- at least fifteen minutes a day just hitting the d button on what gets through. That's AFTER filtering 50 MB a week so I never see it. The cost -- missed messages (filtered incorrectly) and 2-3% of my net productivity. They're going to cost me close to a year of my productive life by the time all is said and done. Multiply that out by hundreds of millions, and SPAM easily costs around a billion dollars in lost productivity and wasted resources a day. Sounds like an attack to me... My own "solution" to this is pretty draconian -- a "final solution" of sorts. I would legislate an "acceptable use agreement" for the Internet at the federal level (to be used for state models as well). It would not be worded to compromise the rights to free speech, it would leave pornography mostly alone (tempting a prize as that would be to idiot lawmakers) and would focus strictly on the issues above that are clearly attacks and which clearly cost a fortune. The law would put individuals knowingly participating in viral-bot theft of network resources for any purpose at all in jail -- for theft! What an idea! Fine them 10x the estimated value of the theft plus its extrapolated cost in lost productivity (cleanup at $150/hour, anywhere up to millions of dollars for something that is widely propagated) and then put them in jail for anywhere from a year for first/minor offenses to twenty years, with an explicit stipulation that >>they<< will never be permitted an information processing device or network connection in their jail cell, no computer, internet, ipod, cell phone, pda. Push it out there in the form of international treaties. Create a special branch of the FBI just to deal with the worst cases, create state BI branches to handle the rank and file, and then go after the bastards. And finally, establish some REASONABLE procedure for handling internet vulnerabilities in a timely way, and provide protection (on the one hand) for companies and operating systems that demonstrably crush exposed vulnerabilities within a reasonable time window and >>make companies that fail to do so liable<< for a crushing class action suit in the event that they do NOT deal with known threats in (say) two weeks and four weeks in an exploit costs a hundred thousand users of said operating system a couple hundred dollars each of wasted time and productivity and services from local systems people to clean up and repair their systems. [Yes, I've been dreaming of this for years, ever since I learned first hand just how COSTLY a successful crack is within an organization. I've never spent less than 1-2 DAYS FTE dealing with successful cracks over the last 20 years, and in some cases just debriefing the crack took a committee of four or five high end sysadmins and a couple of University administrators several three hour meetings PLUS all the time required to actually clean things up -- that one was in our medical center.] Cluster wars indeed. >>> So besides preferring to call it a "Virtual Special-purpose (mail-bombing) > Supercomputer" instead of a "(General purpose) Supercomputer" I'd also be > skeptical of all performance metrics. If you can't measure the number of > nodes within an order of magnitude then other metrics are perforce dubious. > > Well, there IS that, but then, it's more a matter of scale of dubiousness > rather than whether any sort of single cluster metric is "truth". > > > >> And I'm pretty sure that Deep Blue could beat it at chess, if someone >> managed to MPI a chess program on Storm Bot. But I"m sure I can't prove it. > > > Come on.. someone needs to throw down the guantlet. Challenge the botnet to a > smackdown duel. race for pinks* or something. The gauntlet is down and has been for years. The stormbotnet is already constantly probing, seeking to grow, seeking to add your nodes to its supply. Linux nodes are probably "hard" enough that if it doesn't find something really stupid (open guest accounts, toor as the root password) it tests for a few holes and then moves on, but I'm sure that it would love to eat your clusters. And you get to duel whether you want to or not, and get to pay the cost of duelling whether you want to or not! The cost is ensuring that you have a "trustworthy" update stream -- Red Hat's reason for existence, currently. Not using Mosix-like kernels that get too far behind the production kernels (that are generally VERY rapidly patched when a vulnerability is discovered). Using only ssh, or building your rsh-based clusters inside a firewall and hoping to God that the doorway is never cracked. Running nightly (or even hourly) monitoring software primed to look for symptoms of a successful crack and then reading the damn logs! (Hours of wasted sysadmin productivity right there.) Ditto syslog.ng -- doesn't do you any good unless you use it AND read the reams of garbage thus produced to look for anomalies. And then there is the BIG cost when you blink and something slips in and you've got to do a full reinstall and close the vulnerability, the extended big cost as you then devote even more of your time keeping up with security digests and security mailing lists so you can proactively close the next configurational hole before it is exploited. It's all shades of William Gibson. As the network continues to get bigger and faster, as closed source operating systems and networking device companies like Cisco continue to push the utterly failed paradigm of security by obscurity, arguing that they are better off not publishing their sources for open review (which is utter nonsense), as the network starts to include ever more SMALL devices running SMALL operating systems with the same vulnerabilities, we'll get closer and closer to the point where Internet-spanning bot-clouds get a life of their own. There is actually a really interesting problem in evolution being played out here -- at what point will someone drop something out there that is capable of existing at a level that is tolerated by the host so it is not immediately obviously sick, that can live in your cell phone, your PDA, your linksys WAP where tools to discover it simply do not EXIST, and that can exchange codons and evolve as it propagates? We're really close to that already -- some "spyware" is distressingly like this, living quietly enough on the host and collecting information. Not quite botware, though. But it is coming. rgb > > > >> *impromptu drag racing, winner takes possession of loser's vehicle (i.e. >> the certificate of ownership, which used to be pink in California (aka pink >> slip).. now they're a harder to forge rainbow color) > > http://en.wikipedia.org/wiki/Pink_slip > > I like the linked interesting quote from Tony Blair, in re: his pink slip > (the other kind). Apparently in the UK, there's actually an official form > (P45) for such things. The wikipedia didn't say if the P45 is pink, though. > Here in the U.S., we have forms for lots of other things, but not that. > > This list is so educational, in ways that one cannot even begin to describe. > Now I have something to talk about with my wife's relatives in Surrey, > stories about bureacracy having universal appeal, and much more cheery than > death and taxes. > > > James Lux, P.E. > Spacecraft Radio Frequency Subsystems Group > Flight Communications Systems Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From csamuel at vpac.org Sat Sep 1 06:38:46 2007 From: csamuel at vpac.org (Chris Samuel) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] Node not answering... In-Reply-To: References: Message-ID: <200709012338.47390.csamuel@vpac.org> On Fri, 31 Aug 2007, Nestor Waldyd Alvarez Villa wrote: > mpirun: cannot start a.out on n1: No such file or directory NFS (or other network/distributed fs) mountpoint not mounted ? -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From James.P.Lux at jpl.nasa.gov Sat Sep 1 07:38:51 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> Message-ID: <6.2.3.4.2.20070901071940.030af6e8@mail.jpl.nasa.gov> At 06:23 AM 9/1/2007, Robert G. Brown wrote: >On Fri, 31 Aug 2007, Jim Lux wrote: > >Similarly lots of other problems become tractible to a brute force >search algorithm when you can displose of order of 20 petaclocks worth >of cycles. (Am I multiplying that out right? 10^7 times 2x10^9 = >2x10^16, 9 is giga, 12 is tera, 15 is peta. Yup. Petacycles.). Brute >force searches require minimal IPCs, although I'm sure there are >interesting problems associated with IPCs and data harvesting when it >has to be done in "stealth" mode and not lead investigators back to you >and when you need to make it robust against nodes dropping out (being >cleaned by their owners) and popping back in (as yet another virus >propagates). There is a fair amount of literature on such communications problems. For instance, the classic Byzantine Generals problem deals with how to reliably communicate through (potentially deliberately) unreliable channels. And if the seamier side of the internet isn't byzantine, what is? >Then there is denial of service. Everybody knows that this is an >attack, but few recognize its potential terror value. Just remember the >>>cost<< of some of the countdown viruses of years past. Some of them >literally shut down the Internet for close to a day -- clogging all the >main arteries and switch points until hosts were run down one at a time >and isolated by their hosting ISPs. The cost of those incidents in real >dollars, lost productivity, and human misery was easily a billion >dollars each (I read estimates that were much higher, but I don't want >to be hyperbolic so let's stay conservative here). When speaking or writing of world domination, a bit of hyperbole is called for, no? > A bot-cloud attack >could be far more costly and last far, far longer, in part because if it >were well-designed it could shape-shift every five minutes and vary e.g. >IP number, signature, target. It could also turn on and off at random >times to make it very difficult to track each bot back to its infected >host. If it times itself to take advantage of one of those two-month >long window vulnerabilities (yes, a lot of them last for PLENTY of time >for this to be feasible) so that it can essentially instantly re-infect >a wide class of hosts at will as they are cleansed, it could force the >shutdown of nearly every Windows system in the world until it is >hand-cleaned and patched -- the Internet itself would be useless in >fixing the problem. The cost of such a complete attack would be >staggering -- banking, commerce, education, defense -- all at a >standstill. It would probably trigger a full depression (led of course >by the complete collapse of Microsoft as the full cost of its appalling >and perpetual vulnerability is finally laid bare). I'm sure we'll have plenty of time to discuss this through the chainlink walls of our future accommodation at points south. I hope hurricane season is over by then. >Truthfully, I've been waiting for foreign terror powers to figure this >one out and attempt such an attack, but so far we've been lucky. Bot >driven attacks on individual systems of course happen all the time -- >check out the logs of pretty much any server and count the number of >times per day some system in Korea or South America or God Knows Where >tries to probe its way down your ssh ports and standard accounts in >search of an idiot who left in a default password (or put a stupid >password or root). These folks aren't looking for fun, they're looking >for money. And that's the problem. Say you have the ultimate DoS machine. It's not feasible to call up, say, Bank of America and tell them: send us X million or we shut down your consumer website (or your intranet, or whatever). First, you have the classic ransom pickup problem. It's pretty straightforward to move <$100K without leaving too much of a trail, much tougher to do it with $100M, unless the recipient has a substantial investment and preparation, which is hard to do on a "low budget" sort of scale. And it's tough to move from the $10K to the $10M bracket without travelling through the $100K-$1M zone without attracting a lot of attention. Second, if you ask for huge sums from one victim, they're going to have a big incentive to not pay. So you're back to the how to extort smallish sums from lots of victims and get it collected. That's a bigger administrative headache than running the botnet. their own. So, it seems that while the SuperBotNet is amazingly effective as a device for forcing millions of dollars of extra sysadmin time in terms of keeping up with the continuous and pervasive annoyances, it's not particularly profitable for its operator. In the lingo: they haven't figured out how to monetize the botnet. It's more like one of those James Bond novels where Blofeld creates a virus that will decimate the world's population of chickens. Unlike in the novel, though, there's no way to collect the ransom. From andrew at moonet.co.uk Sat Sep 1 09:03:11 2007 From: andrew at moonet.co.uk (andrew holway) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <6.2.3.4.2.20070901071940.030af6e8@mail.jpl.nasa.gov> References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> <6.2.3.4.2.20070901071940.030af6e8@mail.jpl.nasa.gov> Message-ID: Does the mass replication of an exploit really constitute a supercomputer? Has it reached the point where a computing environment capable of supporting programs is created or is it simply a mechanism of attack controlled by a human operator? Andy On 01/09/07, Jim Lux wrote: > At 06:23 AM 9/1/2007, Robert G. Brown wrote: > >On Fri, 31 Aug 2007, Jim Lux wrote: > > > >Similarly lots of other problems become tractible to a brute force > >search algorithm when you can displose of order of 20 petaclocks worth > >of cycles. (Am I multiplying that out right? 10^7 times 2x10^9 = > >2x10^16, 9 is giga, 12 is tera, 15 is peta. Yup. Petacycles.). Brute > >force searches require minimal IPCs, although I'm sure there are > >interesting problems associated with IPCs and data harvesting when it > >has to be done in "stealth" mode and not lead investigators back to you > >and when you need to make it robust against nodes dropping out (being > >cleaned by their owners) and popping back in (as yet another virus > >propagates). > > There is a fair amount of literature on such communications problems. > For instance, the classic Byzantine Generals problem deals with how > to reliably communicate through (potentially deliberately) unreliable > channels. And if the seamier side of the internet isn't byzantine, what is? > > > > >Then there is denial of service. Everybody knows that this is an > >attack, but few recognize its potential terror value. Just remember the > >>>cost<< of some of the countdown viruses of years past. Some of them > >literally shut down the Internet for close to a day -- clogging all the > >main arteries and switch points until hosts were run down one at a time > >and isolated by their hosting ISPs. The cost of those incidents in real > >dollars, lost productivity, and human misery was easily a billion > >dollars each (I read estimates that were much higher, but I don't want > >to be hyperbolic so let's stay conservative here). > > When speaking or writing of world domination, a bit of hyperbole is > called for, no? > > > > > A bot-cloud attack > >could be far more costly and last far, far longer, in part because if it > >were well-designed it could shape-shift every five minutes and vary e.g. > >IP number, signature, target. It could also turn on and off at random > >times to make it very difficult to track each bot back to its infected > >host. If it times itself to take advantage of one of those two-month > >long window vulnerabilities (yes, a lot of them last for PLENTY of time > >for this to be feasible) so that it can essentially instantly re-infect > >a wide class of hosts at will as they are cleansed, it could force the > >shutdown of nearly every Windows system in the world until it is > >hand-cleaned and patched -- the Internet itself would be useless in > >fixing the problem. The cost of such a complete attack would be > >staggering -- banking, commerce, education, defense -- all at a > >standstill. It would probably trigger a full depression (led of course > >by the complete collapse of Microsoft as the full cost of its appalling > >and perpetual vulnerability is finally laid bare). > > > I'm sure we'll have plenty of time to discuss this through the > chainlink walls of our future accommodation at points south. I hope > hurricane season is over by then. > > > >Truthfully, I've been waiting for foreign terror powers to figure this > >one out and attempt such an attack, but so far we've been lucky. Bot > >driven attacks on individual systems of course happen all the time -- > >check out the logs of pretty much any server and count the number of > >times per day some system in Korea or South America or God Knows Where > >tries to probe its way down your ssh ports and standard accounts in > >search of an idiot who left in a default password (or put a stupid > >password or root). These folks aren't looking for fun, they're looking > >for money. > > And that's the problem. Say you have the ultimate DoS machine. It's > not feasible to call up, say, Bank of America and tell them: send us > X million or we shut down your consumer website (or your intranet, or > whatever). First, you have the classic ransom pickup problem. It's > pretty straightforward to move <$100K without leaving too much of a > trail, much tougher to do it with $100M, unless the recipient has a > substantial investment and preparation, which is hard to do on a "low > budget" sort of scale. And it's tough to move from the $10K to the > $10M bracket without travelling through the $100K-$1M zone without > attracting a lot of attention. Second, if you ask for huge sums from > one victim, they're going to have a big incentive to not pay. So > you're back to the how to extort smallish sums from lots of victims > and get it collected. That's a bigger administrative headache than > running the botnet. > their own. > > > this kind of thing> > > So, it seems that while the SuperBotNet is amazingly effective as a > device for forcing millions of dollars of extra sysadmin time in > terms of keeping up with the continuous and pervasive annoyances, > it's not particularly profitable for its operator. In the lingo: > they haven't figured out how to monetize the botnet. > > It's more like one of those James Bond novels where Blofeld creates a > virus that will decimate the world's population of chickens. Unlike > in the novel, though, there's no way to collect the ransom. > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From rgb at phy.duke.edu Sun Sep 2 08:38:44 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <735238.28092.qm@web37904.mail.mud.yahoo.com> References: <735238.28092.qm@web37904.mail.mud.yahoo.com> Message-ID: The following is yet another in a long line of: and can be skipped by the terminally busy...;-) On Sat, 1 Sep 2007, Ellis Wilson wrote: > My own "solution" to this is pretty draconian -- a "final solution" of > sorts. I would legislate an "acceptable use agreement" for the Internet > at the federal level (to be used for state models as well). It would > not be worded to compromise the rights to free speech, it would leave > pornography mostly alone (tempting a prize as that would be to idiot > lawmakers) and would focus strictly on the issues above that are clearly > attacks and which clearly cost a fortune. > ----------- > > I personally feel that attempting to establish which actions online > are within the realm of attacks is a job that will be unfortunately held > (and run poorly) by human beings. Perhaps in the beginning all will be > well, and indeed free speech will continue to reign. However, I would > not at all be suprised if the entire system went awry and some federal > body (I presume you are making this argument American centered, which > also presents an interesting thought: America filters "acceptable" > action on the net; China anyone?) decides to waver from its obviously > unbiased stance and attempt to benefit a company moreso than another. > > Take this example. Let's say such an agreement was established at the > "beginning of the internet" (yes, I know, I'm young :). One of the > markets that seriously suffered (I'm talking billions here too) was term > life insurance. One of those pesky sites came along, simply presenting > the rates of each of the many companies in a quickly rational table, > making competition much more steep. Previously, the haziness of truth > had allowed these companies to avoid competition largely, but alas, the > internet brought a swift and complete end to that. Let's also assume > that as this was coming together, someone in the federal governement of > wherever decided that this action would "clearly cost a fortune" to > certain persons he/she was in league with, and thus, it would "clearly" > constitute an attack. Thats the end of that site. > > To us, clearly, the above would abruptly disrupt free speech. > However, it is of my belief that clarity is often viewed through a > dollar bill (or other paper currency). The clarity of a Duke professor > or say, a college student such as myself, might be well estranged from a > money hungry politician, and thus able to achieve fair objectivity. > Unfortunately, the "feds", run by politicians in large, I believe have > an alterted slide rule to help them understand objectivity. > > Call me an internet darwinist. I certainly don't think taking over > computers (even if they are largely unprotected) is awesome, and > definitely don't make a past time out of it. However, if that is being > done, and those victims cannot defend themselves properly, that is > completely due to the lack of security on the victim's part. I don't > care whether the code is "good" or "bad", if such things exist, I just > cannot see (even the most well intended) restrictions on the internet > ending up unabused. An excellent essay and rebuttal! Let me clarify -- it isn't the money per se that motivates this view on my part. It is (since you seem inclined to pit "human rights" such as free speech against the humble dollar) a matter of things like the "right" to privacy, the "right" of human ownership, to be able to the use resources that we paid for, the "right" to be able to specify the rules for the use of resources purchased in the commonwealth. There is a lovely old Sci Fi classic book you might read called "The Space Merchants", by Pohl and Kornbluth. In it, in some not so great a time into a future, where the world is literally run by giant multinational corporations and where the only career path open to poets is to become a copywriter, it is quite literally illegal to inhibit advertising, including advertising that is projected onto your eyeballs by means of holography while you are driving or riding in a car. Hmmm... don't we live in a world dominated by giant multinational corporations already? Don't we fight a perpetual war against those corporations now to ensure that every single square foot of interstate visible from the road isn't taken up with a billboard? Hasn't the ratio of actual television to advertisement slowly deteriorated in my lifetime to the point where (extrapolating the curve) my grandchildren will likely watch more ads than show? Oh wait, there are now several cable channels that are NOTHING but ads. Also note that the "rights" you are defending are the rights of soulless barbarians to insert software and data onto hardware that I own. Soulless in the literal sense -- this discussion is about BOTS, not people. If somebody wants to take the time to sit down and type out a lovely personal email to me, one character at a time, I am pleased to receive it, even if it is advertising. The problem with the internet and computing is that anybody at all can craft a program that does nothing whatsoever but email at the maximum sustained bandwidth at their systems' point of presence. Some of us with high bandwidth and big clusters could (were we soulless barbarians) absolutely saturate your mailbox, I promise, with junk email. Somebody already is. Now I personally deny that any sane statement of human rights includes the right to send robot cars to drive by and wedge their physical mailbox (on the street) full of manure (literally), to insert bombs therein that blow the mailbox up, to create a line of robots seeking to deliver mail so long that the legitimate postman has a hard time getting through, to send me letters containing deadly anthrax or just the common cold, to send so much stuff that I'm forced to employee a full-time professional to open and presort my mail for me in order to NOT be infected with herpes, in order NOT to inadvertently open a box filled with cockroaches wired with little cameras and microphones ("The Fifth Element"), in order not to have my sanity overwhelmed. Mail of all sorts is a human invention intended to be useful -- surely we have the "right" to set it up and regulate it so that it can perform that intended use? Noting well that Federal Law in at least the United States actually makes doing nearly any of the things listed above a fairly serious crime. The only reason advertising via paper mail hasn't been overtly regulated is that it is not a zero-marginal-cost expense to would be advertisers. And don't think that people haven't thought about it. Six years ago I, like most Americans, was in the process of quietly being driven nuts because direct phone marketing was out of control. We were literally getting anywhere from five to ten advertising phone calls a day, and naturally they always occurred during dinner and one always has to at least stand up and go look at caller ID even to filter them without answering, and of course phone advertisers know this and would always "block" their incoming phone number so we were forced to answer if only to find out whether or not it was bad news about a relative. People revolted. Legislation was passed. The national "do not call" list was created -- a silly enough concept since who in their right mind WANTS to be called during dinner by strangers wanting to sell them yet another credit card? Now I and pretty much everybody I know eats dinner in peace. In other words, once again we ALREADY have regulations prohibiting the abuse of a commonwealth resource (phones being a public utility) to violate the reasonable expectations of privacy and acceptable use on the part of us, the members of the commonwealth. Indeed, as things stand right now there are already regulations prohibiting nearly everything I suggest that we make a stronger effort to control at the federal level. Just about every citizen or corporation who is attached to the Internet in any way inherits one form or another of the toplevel acceptable use agreements of the backbone providers. In many if not most cases, there are additional contractual obligations added on top of these AUAs. So what I'm really calling for is making "Acceptable Use" clear at the level of law, not just contract, and to impose criminal penalties in addition to the more or less nonexistent sanctions available to the toplevel providers. All a provider can do, basically, is disconnect you, and as we have long since seen, the Resident Evil on the Internet just reconnects, new name, new IP number (and in many foreign countries with even less regulation and toplevel administrative control than we have here there isn't even that). As for your last two remarks, that this is somehow something that can be blamed on the victims, that I'm responsible for building a chainlink fence around my house and hiring poor people to sit at my gate as footmen and butlers to selectively admit those who have a "card", and the bit about Darwin: The first is clearly absurd. Anarchy is not freedom, and life in a state of nature (which you seem to advocate) is indeed ugly, nasty, brutish and short. Do you really want an "Internet" where the rich and strong can do anything to your system that they can get away with? Where if I'm clever enough to write a program that figures out how to intercept your personal and professional email messages and append an advertising message (so the actual content is embedded inside an offer for a really great credit card) well then I DESERVE to get rich from it? Where inserting spyware via web connections is "just part of the cost" of using the web (and besides, if I really wanted to control it I'd go ahead and pay a third party $50+ a year, right -- buy myself a butler and footman, like everybody else, buy a firewall (or take the human time to set one up -- even open source firewalls aren't "free") to act like a chain-link fence around my property)? The second -- well, the beauty of Darwin is that he is SO right that he is inescapable. We DO live in a state of nature at all times. As Hobbes and his philosophical successors clearly observed, the things that we call "human rights" are really just the axioms we INVENT upon which to base a society and are ultimately consensual fictions imposed, in fact, by brute force. Perhaps that force is exerted by the tyranny of the "majority", perhaps it is a reasoned and highly functional fiction (so that it >>survives<< in a hostile world) but evolution happens whether or not one believes in it. Memetic evolution (the kind that governs human rights themselves and more practically human law and contract) is an ongoing process whereby we can in fact propose new ways of doing things. I absolutely agree that there are infinite opportunities for governmental control over the Internet to go awry -- to try to make sending pornography over the net a federal crime, to legislate the use of crackable encryptions ciphers so big brother can watch over us, to LEGALIZE SPAM (remembering that those giant multinational corporations have a strong interest in doing so, or would if they ultimately weren't made up of human beings too). But I do not see a stark choice -- no regulation at all, a continuing degradation of Internet functionality, the waste of billions of dollars worth of PRIVATE resources (remember that those bots invade personally owned systems, and that private parties are forced to spend large amounts of money to defend them) OR the end of "freedom" on the Internet. This is simply a matter of reality. The "freedom" of the Internet is already largely an illusion, especially for the private citizen. You want to see Federal control in action, try sending kiddie porn out to somebody and prepare to have your door kicked in and computer seized. You want to be ignored? Craft a letter that begins "Hello dear friend. I am writing you because my third cousin, Hailie Sumboddy, recently passed away leaving behind $37,000,000 that alas My Government wishes to seize..." Or send out a letter offering to show somebody your (implied) nude pictures if only they would click the following link. Or hell, just send 'em an exe that will directly invade their system -- there must still be SOMEBODY stupid enough to click it, if only an eight year old. To me this is a no brainer. I am on the goddamn front lines. Largely because of ten years of participation on this list, my email address is on every spambot's list in the known universe, grazeable eight thousand different ways from archives, reposted lists, web links, whatever. I have spam assassin fronting my mailbox with evolutionary rules. It works -- 50-100 megabytes A WEEK of SPAM is removed as noise (along with some unknown amout of signal, but I'll be damned if I go through that mess looking for it). That cuts it down to where only 50-100 messages A DAY make it THROUGH. Is it "my fault" that I have to waste hours of every week sorting it out and deleting it by hand? Is it a "lack of adequate security" on my part (let's blame the victim, yeah)? I hope not -- I've been a Unix sysadmin for over twenty years, sat in on two committees that helped establish Duke's security and acceptable use policy, ran a gopher drop that became a website that distributed about twenty key documents associated with network security for a decade or so, and have directly defended against internet attack with successes and many, many failures over all that time. In fact, part of my salary and time over those years is just ONE of the costs of security for many other users. If my security is "inadequate", then God help the 99.99% of all internet/computer users who don't actually know what the Morris worm is, who are clueless about buffer overwrite attacks, who think that it is "normal" to have to buy an antivirus program to paste a thin veneer of "security" over what is a badly, badly broken operating system. I'm fed up with it. In my opinion we have long since passed the point where we need real police and real penalties that pass the real costs back to the freeloading soulless bastards that fill up my email box with crap, that periodically shut down the net with timed viruses, that steal identity, invade private medical records, steal privately owned computer systems and networks for their own use, that force every corporation to effectively hire "private police" (in the form of security tools, hardware, and systems administrators) in order to function because there ARE no police out there in the grand old anarchy of the Internet. The wild wild west was a lot more romantic and attractive in retrospect than it was in reality -- that's why it went away when the law moved in. People don't, really, think that being shot down in the street, with bullets zipping by and sometimes striking the innocent, is a "price of freedom". rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From 3lucid at gmail.com Sun Sep 2 13:02:43 2007 From: 3lucid at gmail.com (Kyle Spaans) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> Message-ID: <5a1205b30709021302v5300ad72w70fc2be612e2284c@mail.gmail.com> On 9/1/07, Robert G. Brown wrote: > RSA and DES and MD5 are considered "probably uncrackable" by > anyone with less than NSA-class resources, but of course this bot-cloud > is several orders of magnitude more powerful than NSA's probable setup. Why a "probable setup" ? I would agree that the NSA likely has a hefty HPC sitting around somewhere - but why don't we know about it? If the NSA goes through all the work of putting together a (not trying to sound pessimistic/conspiracy theorist) good-enough-to-crack-your-encryption-for-public-safety cluster, why wouldn't they have it up on the top500 list? Not wanting to gloat? Not wanting the "badguys to know what we've got"? Or do they simply not have systems that are that impressive? :P From James.P.Lux at jpl.nasa.gov Sun Sep 2 14:34:53 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <5a1205b30709021302v5300ad72w70fc2be612e2284c@mail.gmail.co m> References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> <5a1205b30709021302v5300ad72w70fc2be612e2284c@mail.gmail.com> Message-ID: <6.2.3.4.2.20070902143220.030d0ee0@mail.jpl.nasa.gov> At 01:02 PM 9/2/2007, Kyle Spaans wrote: >On 9/1/07, Robert G. Brown wrote: > > > RSA and DES and MD5 are considered "probably uncrackable" by > > anyone with less than NSA-class resources, but of course this bot-cloud > > is several orders of magnitude more powerful than NSA's probable setup. > >Why a "probable setup" ? I would agree that the NSA likely has a hefty >HPC sitting around somewhere - but why don't we know about it? If the >NSA goes through all the work of putting together a (not trying to >sound pessimistic/conspiracy theorist) >good-enough-to-crack-your-encryption-for-public-safety cluster, why >wouldn't they have it up on the top500 list? Not wanting to gloat? Not >wanting the "badguys to know what we've got"? NSA has a generic aversion to publicity of any kind. Read "The Puzzle Palace" by James Bamford. Another good book that provides some insight into the way of thinking is Crypto by Steven Levy Jim From gerry.creager at tamu.edu Sun Sep 2 20:36:15 2007 From: gerry.creager at tamu.edu (Gerry Creager) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <5a1205b30709021302v5300ad72w70fc2be612e2284c@mail.gmail.com> References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> <5a1205b30709021302v5300ad72w70fc2be612e2284c@mail.gmail.com> Message-ID: <46DB812F.6070308@tamu.edu> I have it on good authority that they *do* have the systems and just use them for, well, work! And that, for all the cycles at their in-house disposal, they don't have nearly enough for the data they get in that has to be processed. gerry Kyle Spaans wrote: > On 9/1/07, Robert G. Brown wrote: > >> RSA and DES and MD5 are considered "probably uncrackable" by >> anyone with less than NSA-class resources, but of course this bot-cloud >> is several orders of magnitude more powerful than NSA's probable setup. > > Why a "probable setup" ? I would agree that the NSA likely has a hefty > HPC sitting around somewhere - but why don't we know about it? If the > NSA goes through all the work of putting together a (not trying to > sound pessimistic/conspiracy theorist) > good-enough-to-crack-your-encryption-for-public-safety cluster, why > wouldn't they have it up on the top500 list? Not wanting to gloat? Not > wanting the "badguys to know what we've got"? > > Or do they simply not have systems that are that impressive? :P > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From gerry.creager at tamu.edu Sun Sep 2 20:38:07 2007 From: gerry.creager at tamu.edu (Gerry Creager) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <6.2.3.4.2.20070902143220.030d0ee0@mail.jpl.nasa.gov> References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070831171347.031c0e68@mail.jpl.nasa.gov> <5a1205b30709021302v5300ad72w70fc2be612e2284c@mail.gmail.com> <6.2.3.4.2.20070902143220.030d0ee0@mail.jpl.nasa.gov> Message-ID: <46DB819F.9010105@tamu.edu> Jim Lux wrote: > At 01:02 PM 9/2/2007, Kyle Spaans wrote: >> On 9/1/07, Robert G. Brown wrote: >> >> > RSA and DES and MD5 are considered "probably uncrackable" by >> > anyone with less than NSA-class resources, but of course this bot-cloud >> > is several orders of magnitude more powerful than NSA's probable setup. >> >> Why a "probable setup" ? I would agree that the NSA likely has a hefty >> HPC sitting around somewhere - but why don't we know about it? If the >> NSA goes through all the work of putting together a (not trying to >> sound pessimistic/conspiracy theorist) >> good-enough-to-crack-your-encryption-for-public-safety cluster, why >> wouldn't they have it up on the top500 list? Not wanting to gloat? Not >> wanting the "badguys to know what we've got"? > > NSA has a generic aversion to publicity of any kind. > > Read "The Puzzle Palace" by James Bamford. > > Another good book that provides some insight into the way of thinking is > Crypto by Steven Levy Both should be required reading (and it's time to re-read "The Puzzle Palace") before asking open-ended questions about the organization whose very name was once classified, and whose acronym was expanded to, in most instances, "No Such Agency". gerry -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From csamuel at vpac.org Sun Sep 2 22:10:13 2007 From: csamuel at vpac.org (Chris Samuel) Date: Sat Jul 5 01:06:20 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <46DB819F.9010105@tamu.edu> References: <20070831194837.GB12988@leitl.org> <6.2.3.4.2.20070902143220.030d0ee0@mail.jpl.nasa.gov> <46DB819F.9010105@tamu.edu> Message-ID: <200709031510.16418.csamuel@vpac.org> On Mon, 3 Sep 2007, Gerry Creager wrote: > Both should be required reading (and it's time to re-read "The > Puzzle Palace") before asking open-ended questions about the > organization whose very name was once classified, and whose acronym > was expanded to, in most instances, "No Such Agency". Yesterday upon the stair, I met a man who wasn't there. He wasn't there again today, I think he's from the NSA. With apologies to Hughes Means. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://www.scyld.com/pipermail/beowulf/attachments/20070903/860a0c74/attachment.bin From rgb at phy.duke.edu Mon Sep 3 05:28:40 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <313159.48779.qm@web37902.mail.mud.yahoo.com> References: <313159.48779.qm@web37902.mail.mud.yahoo.com> Message-ID: On Sun, 2 Sep 2007, Ellis Wilson wrote: > This is all very true, and also very contingent upon the fact that the > postal service, roads, and phones are public items that each and every > one of us pays for (even though there are phone companies, they pay > taxes to the government heavily and follow its rules). Whether or not > it is readily quantifiable, we all most certainly pay for one government > with one set of rules to police these commonwealth resources and ensure > their proper use. However, as Thomas Paine puts it, "Government, even > in its best state, is but a necessary evil; in its worst state, an > intolerable one." Unlike roads, mail, and even perhaps telephones, I do > not feel that the Internet is such a necessary means of communication > (yet) that the government need interfere. Actually, IIRC the phone companies "are" the internet, and the internet "is" at least originally a government project (the latter being well known -- DARPA and all). The internet is composed of e.g. fiber optic trunk lines interconnecting switches that almost without exception belong to phone companies. Use traceroute to follow the path of your packets nearly anywhere. Use whois (if necessary) to resolve just who a lot of the core links are. Recognize anyone? And most of the exceptions are things like NCREN -- government funded organizations as well (who still lease the actual lines outside of their own organization. Don't get me wrong -- I too love (being a "romantic" after all) a lot of the things about the way the Internet evolved as a mostly unregulated utility that slipped between the overt regulation cracks in government and was in fact governed by the young idealists who ran (and in many cases still run) the core switches. They created an anonymous resource that the world has somehow managed to avoid monetizing, at least at the price per packet level that would spell doom in many ways. But that model has broken down to some extent, largely because at any given instant a signficant fraction of the attached client systems are "infected", largely because the operating system they are running is a piece of crap that openly invites invasion (nowadays it actually FORCES you to DELIBERATELY have it invaded by things like the Genuine Windows Authentication Tool, and whazzup with the spyware thing? Do they actually try to engineer in holes that can be exploited by the corporate masses)? I can't really do another rant today, though -- gotta teach (sigh) and really must prepare. I'm just a bit more optimistic that "the people" can, when they must, craft things LIKE speed limit and Do Not Call laws that actually work, mostly, sort of. I'm also a true believer in the virtue of "dynamic tension" that permits the laws to be crafted in their imperfect way and then gradually fixed until they do what we want. What I'm NOT optimistic about is that this is a self-healing problem or self-limiting problem, and the extrapolation of its observed growth should be very troublesome from the sheer viewpoint of engineering if nothing else. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From peter.st.john at gmail.com Mon Sep 3 10:44:40 2007 From: peter.st.john at gmail.com (Peter St. John) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] a "microwulf" posted In-Reply-To: <56028.192.168.1.1.1188614523.squirrel@mail.eadline.org> References: <56028.192.168.1.1.1188614523.squirrel@mail.eadline.org> Message-ID: Thanks Doug; great job. Instead of wanting my face on the cover of Rolling Stone, I want it on Slashdot :-) Instead of the FLOPS per dollar per Watt metric I'd consider either amortizing the capital of the equipment, to get dollars per FLOPS per year, and then express watts in dollars per year; or compute the present real value of the watts, to get an equivalent dollar capital investment. You'd have to assume something like continuous operation, and in real life the capital investment and the operating costs will be administred and taxed differently, but it would make for apples-to-apples. Peter On 8/31/07, Douglas Eadline wrote: > > > You may also find the write on ClusterMonkey informative. > > http://www.clustermonkey.net//content/view/211/33/ > > It should be viewable now as we have been experiencing a > 2nd degree slashdoting today -- probably hit 15,000 unique > views before the day is over. (the Microwulf press release link on > slashdot has a link to the CM article.) > > BTW, the article is pretty much a recipe for building > your own system. > > -- > Doug > > > > I saw at wiki http://en.wikipedia.org/wiki/Beowulf_%28computing%29 that > > someone posted a link to > http://www.calvin.edu/~adams/research/microwulf/, > > a > > nice description of a "microwulf" at Calvin College. It has brief but > > useful > > descriptions of it's design, cost broken down by parts in the Manifest, > > and > > price/performance specs. > > If LLNL is an Epic maybe this is only a limmerick, but it's a witty > > limmerick. > > Peter > > > > > > !DSPAM:46d8450d89171409419350! > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > !DSPAM:46d8450d89171409419350! > > > > > -- > Doug > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20070903/c7d0b13c/attachment.html From rgb at phy.duke.edu Mon Sep 3 11:42:19 2007 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: <313159.48779.qm@web37902.mail.mud.yahoo.com> References: <313159.48779.qm@web37902.mail.mud.yahoo.com> Message-ID: On Sun, 2 Sep 2007, Ellis Wilson wrote: A very few things that I will reply to on list (really) and then I'll quit, I promise. > My friends also insist that I purport anarchist suggestions; rather, I > feel that that the government is best kept as small as is possible. The > quote previously hints at the same. As far as the rich and the strong, Libertarian, not anarchist, although yes they blend together on one end of the continuum where a "rabid libertarian" (as in foaming at the mouth) meets the bomb thrower somewhere around the thinkings of Bakunin. You don't sound like a Bakunite, though...:-) > these are often not identical. The rich enlist the strong (or more > appropriately, the clever) and desire to do whatever it takes to > continue in the path to money. People (such as you and me) desire to > use the Internet without great hindrance, and thus will either pay the > government or a private organization (or do it ourselves, if the latter > is possible) to secure our experience. Last, plenty of people who don't > deserve to get or be rich do (on and off the Internet) and thats a > totally different argument. Hmmm, so many things to address. Darwin doesn't care about "deserve" -- that's an overlay of strictly human values on a natural stochastic process that happens independent of human value or desire even as it embraces them (they too arise out of a genetic optimization process, in fact -- see "The Lucifer Principle" by Bloom for a lovely argument to this effect). As for the rich and the strong vs the people, well, how much of Hobbes, Locke, John Stuart Mill should we work through? Again evolution as a mechanism doesn't care, but the optima being pursued are natural self-consistent social constructs, not quite the "nature" Hobbes was talking about, and they're pursued because they provide huge evolutionary advantages compared to Hobbes's nature. You may be much bigger and stronger and meaner than I am as an individual, but mess with me and you mess with my whole community and collectively we are vastly stronger than ANY individual. Ditto richer, ditto bigger, ditto meaner. We the people are bad-assed tyrants whenever we choose to be. So ultimately, philosophically the issue is our CHOICE as to just when and where we wish to invoke the power of that community, that tyranny over the individual. One thing that makes the United States so beautiful a country is its discovery that a balance that preserves the lovely illusion of "human rights" that are held to be inalienable (which is semantically null, of course, but near perfect poetry) and strictly limits the power of the many over the few "works" amazingly well. Well enough to survive in a hostile world, if only by being able to remarket this bit of seductive hogwash to many other competing culturals by means of a bit of memetic sex and being lucky enough and strong enough and rich enough to kick the butts of the handful of serious contenders for a global social paradigm long enough to MAYBE have flipped the world into the post-feudal era and into a state where nonviolent social evolution is strongly favored over the violent kind. I am unashamedly proud of my country for this invention, and deeply revere Thomas Jefferson even as I recognize up front that his declaration is ultimately a vision that nucleates its own artificial reality, not the underlying reality itself. No problem -- happens all the time in complex systems. Nevertheless, from a purely pragmatic point of view the measure of success is success, and Jefferson himself said not to get to attached to any particular set of words representing lofty principles (not even his), as he fully expected things to CHANGE -- he was perhaps unique among all philosophers before and since in actually stating that up front. And so it is with the Internet. It is what we make it. By its nature it cannot be linearly controlled as it is intrinsically a complex landscape upon which individuals can seek personal advantage. It has at least some modes that are well-known to be self-destructing from e.g. feedback (packet storms, anyone?). It has more than enough resources and complexity to enable nonlinear growth and chaotic dynamics. It is, in a word, potentially unstable, and there are a number of historical cases that demonstrate that instability. It can be DELIBERATELY destabilized by anyone that understands it well enough and who controls more than a certain critical mass of attached resources. Morris did so accidentally but catastrophically. Any number of timed viruses since have done so deliberately but on a less catastrophic scale. At the same time, its growth has created an environment where the "noise" (deliberately driven and random) has increased fairly steadily as individuals seek any one of a variety of personal advantages via exploiting it as a resource. As I argued, the deliberatedly introduced noise has become extremely expensive, in part because a lot of it is engineered to be -- just like many creatures in nature, it seeks to propagate itself and seek advantage at the expense of others. However, it is NOT nature, or rather it is an artificial nature. If you analyze the nature of the threat, you can see that one reason these threats and annoyances are enjoying strong growth is because there is little disincentive to bad practices. Basically, if one abuses the hell out of the network, nothing happens to you. Even if you get caught. With a tiny handful of noteworthy exceptions -- kiddie porn, overt identity theft, identifiable fraud, stealing credit card numbers. I personally do not think that this can be solved as a problem in engineering, which is what everybody is trying. Some aspects of it, sure. RSA helps. Closing e.g the hole that permitted the Morris worm to self-propagate certainly helped. Antivirus software "helps", although it is always behind the threat by its very nature and is a very costly bit of engineering that clearly has not been successful (in part because it isn't intrinsic and software engineering itself is inevitably flawed). People have been trying to solve it with engineering almost from the beginning, some of them very bright people, and it isn't working. If anything, we're losing relative ground. If it weren't for Moore's Law, the network would probably have crashed to the ground long ago, in fact -- we get by by growing the resource at a rate competitive with the growth in the destructive noise (yes folks, another major cost). Faster computers and network just mean faster bots, after all. And bot-coders and virus hackers and system crackers, whether they are motivated by personal chops or a real dollar payout (and oddly enough from a strictly evolutionary point of view, there are both kinds) are hardly devoid of talent, and it is easier to destroy than it is to build, easier to introduce noise and entropy than to filter it or reduce it. Second law, no free lunches. So yeah, I think the engineering problem needs a bit of help to shift the Darwinian "fitness" of negative activities from a gently graded plateau where one can simply step from one point to another instead of being penalized even as those activities are locally blocked to at least a possibility of a really strongly negative outcome. A sucker rod upside the head. Loss of freedom and fortune. Social condemnation. I say so not out of malice -- it isn't about getting even. It is pure control theory. As a parent of three boys, I learned long ago that there comes a point in the middle of a chaotic spiral of argument, fighting, and openly dangerous behavior where a single swat on the bottom is worth a million words of persuasion in the absence of sanction, entire vials of Ritalin, any amount of love or trying to protect your head as balls start to knock vases off the shelves and onto it. There is an additional non-engineering dimension that can be used for control that we are not using and that is KNOWN to be the only workable control dimension for many closely related problems. There are costs to using it, as well as benefits, but as with most human affairs the costs are not all black and the benefits aren't all white and human judgement can radically alter the ratio of one to the other as well. We've tried the Internet without meaningful sanctions for a rather long time. Sure, we can probably just muddle on through. At the moment it is killing only 1% of me, 2% of you, half a percent of that guy (given that the life lost hitting "d" keys or clicking delete checkboxes is a form of death, the loss of a bit of that precious freedom we all cherish). I can tolerate it, just like I can tolerate cancer and taxes. But I have long since reached the point where I think that our lives would radically improve if a number of the worst offenders in this life-sucking s**t-parade of SPAM and viral botware were tracked, arrested, fined, and put prominently in jail. Human judgement can easily titrate -- there is no need to crush teen-age lives with felony raps here. However, there is nothing wrong with stern warnings, hefty fines, and jail as a final sanction if nothing else works, and making the punishment in some measure proportional to the cost and damage done (no matter who the doer) is also not unreasonable. > Indeed, and such continues the power struggle, or more appropriately, > the power balance, between invented "human rights" and basic human > instincts. Let one or the other grow too massive, and it will cause > enslavement to ourselves or complete chaos. Exactly. Human judgement and balance is everything, and in the end (as things change!) one has to remember the words of Jefferson and not get married to any particular schema. Maybe we have too few sanctions right now. Maybe trying to add some, we overcompensate and are too draconian. Maybe we then correct again and again until we find something that "works" -- isn't too costly, preserves function. We all have the right to bear arms, just not strapped to our belts when we ride on planes, not fully automatic weapons we have mounted to the roofs of our SUVs when we drive our kids to school. We have the right to freedom of speech, but not to shout fire in a theater or to express that freedom by publishing photographs of children having sex with adults ("consensual" or not). We have the right to liberty, but not if we use that liberty to rob, to rape, to murder, to injure others. I give up MY freedom to rob, rape, or murder you if I can get away with it (always possible in the state of nature, and we ALWAYS live in a state of nature!) in exchange for not being robbed, raped, or murdered in exchange, and we agree to get together and beat up anyone who tries to rob, rape or murder either one of us (whether or not they are "stronger") because together we are stronger and potentially meaner than they are. So what I'm basically saying here is that I'm perfectly happy to give up my rights to spam people and infect their systems with viruses and spybots of my own design if they'll give up theirs, and I'm to the point where I'm willing to get together with you and agree to take a sucker rod to the head of anyone who disagrees enough to do it anyway. A bald statement, but true enough anyway (well, I wouldn't REALLY hit them with a blunt instrument, I'd hit them with a bill). I proposed that they make this a part of Duke's AUA, by the way, although I was unable to sell it here either. Long, long ago I thought that the best way to deal with student crackers who waste a lot of sysadmin and network resources with silly games is to just bill them. If they trash their dorm rooms we just bill them. If they crash their car into J.B. Duke's statue we'd bill them. If they steal or deface or lose library books we bill them. So if they manage to crack some hapless user's account and we catch them -- just assess them a few thousand dollars and add it to their bill... loudly and publically. This IS still an attractive non-governmental solution. If every ISP out there added a clause that certain kinds of unacceptable use would be greeted with a bill of up to $500 (that had to be guaranteed up front with a lock on a credit card, say, or some other form of collateral if only your credit rating should you fail to pay it off) then it wouldn't take a whole lot of occurrences of that clause being triggered for the problem to damp right down. But alas, those clauses only exist BETWEEN ISPs (where they DO exist) -- ordinary users inherit them but don't know it and it would never be enforcable. > Also remember a great number of "private" resources are gained via > advertisement on the Internet, and also protection from the abuse > advertisement on the Internet. Embedded advertising on the internet is fine. That's a consensual agreement between users and providers of resources. I choose to visit websites, and have to accept the content I find, within limits, when I arrive. They do not seek me out and force them into my browser. On the same note though, pop up advertising that creates unsolicited new browser sessions (sometimes five or ten at a time, sometimes looping) that trigger when I visit a website offered up for some other purpose I'd outlaw in a heartbeat. This just sets up the rules for the resource -- things would "work" either way. Passive OK, active bad. I seek it out OK, it seems me out bad. Simple enough rule, actually. > The simple reality is that human nature will continue to attempt to > steal or misuse resources that you purchased. The trick is to pay as > little possible to make your resources safe and your experience sane. > The question is: One government to put our combined faith in (and faith > in common ass) or many private companies to compete to gain our trust > and business. So I assume that you hire pinkertons to guard your house at night? Again, one end point of rabid libertarianism is that we abolish the police, after all, and let people hire private police if they can afford them or want them. And armies are just plain silly -- each of us is perfectly capable of defending ourselves in the event that the US is ever invaded. Again, in common law for some three or four thousand years now, coveting thy neighbor's ox has for all practical purposes been regulated by government with only a few exceptions, and most of those exceptions are horror stories. I also disagree that the issue is one only of cost. Jefferson's lovely words (or the words of the Old Testament, for that matter) create a higher level ethical construct regulating the law. The cheapest way to make my resources safe is to kill anyone who attempts to steal them, or maybe just chop off their hand. If that sort of thing appeals to you, there are countries where this is still the basis of law. Low cost, efficient, effective (I guess). And personally I find it repellant. Private police protect only those that can afford them, often badly and unfairly. It is precisely because I'm viewing theft of my personal and internet resources (or indirectly, Duke's resources that are paid for out of grants or the tuition stream) as THEFT instead of some crazy Russians just having fun, some enterprising Koreans seeking to make a few dollars, some US pornographers trying to get the word out, some mischievous college kids trying to show how good a coder they are that I advocate invoking the same mechanism that we use, in general, to control theft. That is, the law. Not me coding mail bombs or trying to "get even" with my would be attackers by getting into THEIR system, not by finding out where they live and firebombing their houses (ahhh, precious dream:-). Not by ignoring it and hoping it will go away. Not by building up elaborate and expensive walls at great personal expense that end up not working anyway. > Hahaha. I personally enjoy watching windows users go through pain > (and pay a stupidity tax), don't you? Not really. I don't like my state's lottery (an even more overt stupidity tax, if you like) either. Most of them have no choice, after all. That's what "monopoly" is all about. Try buying a PC over the counter without Vista-of-Evil. Can't be done, really -- or rather it is terribly difficult and requires a lot of work and expertise. Remember, roughly half the population has an IQ of less than 100 (by definition). Do I think that they should be punished for this? Of course not. What I resent is the way the laws permit Microsoft to basically lock in those VENDORS so that they cannot afford not to offer alternatives as this is anti-competition, and we've learned the hard way that robber-baron capitalism has a nasty self-perpetuating attractor in social space when that sort of thing is permitted. I resent the fact that my state thinks that it is fairer to tax the poorest and stupidest of people by exploiting their dreams of striking it rich and their addiction disorders. I resent the fact that the massive publication industry has managed to get truly absurd copyright laws passed that give them (the publication companies) a monopoly on books and movies and other content long after the author is dead and the author's heirs grown up and die in turn, and the collusion that is gradually building electronic controls right into the very hardware we use to access it. Even as the Internet created an explosion of information that promises to fuel a golden age, this sort of wickedness could easily end it, meter it, mete it out for money, monitize it, and create an entire generation of self-perpetuating fortunes that exist to do NOTHING but control how and what we think while forcing us to pay for the privilege. > The wild wild west is an interesting example; and so is Britain or > France in the feudal era (or even just before the revolution with regard > to the latter). Plenty of law existed then, it just was abused and > convoluted. My choice is that of moderation. Public when it is most > absolutely necessary; private otherwise. I agree with the moderation part. I just think that history has repeatedly proven that the "libertarian dream" is as silly and dysfunctional as the "communist dream". On paper either one sounds simply lovely. When you factor in human nature and the distribution of abilities, what works better (which is all that really matters, not the idealism expressed either way) is a constitutional democracy with a strong bill of rights and a vigilant and educated population exercising the franchise. Our "Jeffersonian ideals" or concept of federalism and centralization can betray us either way. The main idea is to preserve enough freedom for people to get things done without overt control all of the time (avoiding the "tyranny of the majority" along with other tyrannies) and yet have enough controls and sanctions to prevent the slide into anarchy. Our fundamental problem is that in opposition to this we have a global culture that lacks a common center. There is, really, no widely accepted ethical standard, no single set of axioms that we all agree should be the basis of "society". Hence we muddle along, some people adopting a "Christian" point of view, others revering Jefferson and the Bill of Rights and libertarianism, still others wishing for socialism or communism, then there is Islam, Buddhism, Judaism, and heck, there are still dictatorships and feudal societies and don't even get me STARTED on women. The lack of any such center creates a condition of perpetual cultural/memetic war that breaks out in real wars, of ethical chaos that breaks out as SPAM and viral bots as much as in senators that are closet gays even as they vote against gay issues. Do I trust Our Government? Of course not. However, I do recognize it as the most important, and often the most efficient, vehicle for getting a lot of things that need doing done. Public debate (like this one) on just where one should draw the line and whether or not to create a government sanction for any particular kind of behavior is a sign of a HEALTHY democracy, one that should be able to try things out and then change its mind and try something else. In the present case we've tried out the alternative for a long time. Perhaps we should continue in this manner, but (as a reasoning being) I'd like some TECHNICAL reason to believe that this is a feasible alternative, that there is some hope of my recovering my wasted life without legal sanction. I'd certainly be pissed if somebody broke into my bank and stole 1-2% of my life savings. I'd guess that you would be too (depending on just how big those savings are at this point:-). I'd think that it is perfectly justifiable to spend public monies and energies to find, fine and punish the individual that commited this theft, both to recover my direct damages and to create a social construct that deters others from doing the same (a recipe for social and economic chaos). Why is somebody "stealing" my lifeforce by filling minutes to hours of my week, week after week, with spam-control measures and other crap any different? Why should I "just live with it"? rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From hvidal at tesseract-tech.com Mon Sep 3 13:46:57 2007 From: hvidal at tesseract-tech.com (H.Vidal, Jr.) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Erlang as a language for Beowulf applications Message-ID: <46DC72C1.6090702@tesseract-tech.com> Hello. I have been exploring a range of technologies for parallel applications, some production-level, some experimental. I am curious if anyone on this list has done any work with the language Erlang and/or considers it viable for scientific apps. It seems to be quite mature, has well developed 'process' based semantics with intrinsic message passing, is light-weight for multi-process creation, support application-level fault tolerance (quite applicable for failures in long computations...) and is production level, though not well known in the US. Any comments? hv From diep at xs4all.nl Tue Sep 4 10:56:15 2007 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Erlang as a language for Beowulf applications References: <46DC72C1.6090702@tesseract-tech.com> Message-ID: <004f01c7ef1c$e6334170$0900a8c0@objection> This language i do not know at all. However... ...as i can type at a keyboard i'll make the internet a more safer place :) I'd like to react onto this posting, even though this wasn't your question nor answers it, apologies for that. Please realize why people build a cluster. A cluster is a hard thing to build currently. When it changes in future in effort to build a cluster, then obviously the next statement also might change; namely that considering the effort it takes to build a cluster (huge linux knowledge needed) and the huge price of running a cluster (consider not only the price of the cluster itself but also the running costs, systemadministration and so on), that it's kind of pathetic to run applications at such a cluster that can't get the maximum out of the software. So anything in C#, JAVA that is doing number crunching on such a cluster, is basically a total waste of money, as you could have saved tons of money in that case by making a smaller cluster. Additionally a cluster outdates very quickly. Within a year or 2 you already must start upgrading to a new cluster. So those tons of money that a cluster has as a cost price, you keep spending. In short having software that is as low level as possible, makes sense to have. Having low level software, means you usually also have a capable person of parallellizing that application using the fastest communication technology that you have available. Usually that will be MPI of course. So there is not many advantages in supporting other languages than C, C++ and MPI for clusters. Of course if something would be interesting to make, then it is a shared memory model, as that's the only thing real interesting at a cluster to have. Any other language will always suffer from inferior compiler technology. That said, things might change of course, when it gets cheaper to build a cluster and easier to set it up. I currently don't see microsofts cluster technology as 'real easy'. Vincent ----- Original Message ----- From: "H.Vidal, Jr." To: Sent: Monday, September 03, 2007 10:46 PM Subject: [Beowulf] Erlang as a language for Beowulf applications > Hello. > > I have been exploring a range of technologies for parallel > applications, some production-level, some experimental. > > I am curious if anyone on this list has done any work > with the language Erlang and/or considers it viable > for scientific apps. It seems to be quite mature, has > well developed 'process' based semantics with intrinsic message > passing, is light-weight for multi-process creation, > support application-level fault tolerance (quite applicable > for failures in long computations...) > and is production level, though not well known in the US. > > Any comments? > > hv > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From eric-shook at uiowa.edu Tue Sep 4 12:23:12 2007 From: eric-shook at uiowa.edu (Eric Shook) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Erlang as a language for Beowulf applications In-Reply-To: <46DC72C1.6090702@tesseract-tech.com> References: <46DC72C1.6090702@tesseract-tech.com> Message-ID: <46DDB0A0.7060402@uiowa.edu> I am engaged in similar explorations and I too have found the Erlang language as a possible solution to scaling issues. I have been experimenting with the language for almost a month so my evidence below is not by any means conclusive. With that said in my opinion the language has a fatal flaw and that is data handling. According to the documentation and my brief experience there are certain instances where the data being passed to and from processes / functions is copied and not referenced. You can imagine the problems if you are manipulating a multigig data set. There are certain ways to handle bigger data sets (mnesia, et al.) but I have not fully explored these. If large data handling is not a problem for you I believe you will be pleasantly surprised by Erlang. As you stated error handling and fault tolerance are mature within the language and the lightweight processes (300 bytes) are great for multicore machines, particularly heterogeneous machines. In addition to these great features Erlang has built-in functionality for distributed (e.g. across the network) communication. The distributed communication is also platform independent (e.g. Windows, Linux, etc) By combining fault-tolerance, distributed communication, and lightweight processes one could easily enable an application to adapt to multiple heterogeneous clusters, desktop computers, and grid environments without modifications. I feel this language is very promising. It will simply take an individual or group to fully exploit the languages potential. That is the end of my rant :) Eric H.Vidal, Jr. wrote: > Hello. > > I have been exploring a range of technologies for parallel > applications, some production-level, some experimental. > > I am curious if anyone on this list has done any work > with the language Erlang and/or considers it viable > for scientific apps. It seems to be quite mature, has > well developed 'process' based semantics with intrinsic message > passing, is light-weight for multi-process creation, > support application-level fault tolerance (quite applicable > for failures in long computations...) > and is production level, though not well known in the US. > > Any comments? > > hv > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From James.P.Lux at jpl.nasa.gov Tue Sep 4 13:35:53 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Erlang as a language for Beowulf applications In-Reply-To: <46DC72C1.6090702@tesseract-tech.com> References: <46DC72C1.6090702@tesseract-tech.com> Message-ID: <6.2.3.4.2.20070904133215.030c7ee0@mail.jpl.nasa.gov> At 01:46 PM 9/3/2007, H.Vidal, Jr. wrote: >Hello. > >I have been exploring a range of technologies for parallel >applications, some production-level, some experimental. > >I am curious if anyone on this list has done any work >with the language Erlang and/or considers it viable >for scientific apps. It seems to be quite mature, has >well developed 'process' based semantics with intrinsic message >passing, is light-weight for multi-process creation, >support application-level fault tolerance (quite applicable >for failures in long computations...) >and is production level, though not well known in the US. I know some experimenters are looking at it as the core of a software radio project (dttsp), but I don't know that they've actually got hard core long term experience. http://www.g3ukb.co.uk/ A query to Bob might be useful. >Any comments? > >hv >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf James Lux, P.E. Spacecraft Radio Frequency Subsystems Group Flight Communications Systems Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 From James.P.Lux at jpl.nasa.gov Tue Sep 4 14:33:14 2007 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Erlang as a language for Beowulf applications In-Reply-To: <6.2.3.4.2.20070904133215.030c7ee0@mail.jpl.nasa.gov> References: <46DC72C1.6090702@tesseract-tech.com> <6.2.3.4.2.20070904133215.030c7ee0@mail.jpl.nasa.gov> Message-ID: <6.2.3.4.2.20070904143226.031c95c0@mail.jpl.nasa.gov> At 01:35 PM 9/4/2007, Jim Lux wrote: >At 01:46 PM 9/3/2007, H.Vidal, Jr. wrote: >>Hello. >> >>I have been exploring a range of technologies for parallel >>applications, some production-level, some experimental. >> >>I am curious if anyone on this list has done any work >>with the language Erlang and/or considers it viable >>for scientific apps. It seems to be quite mature, has >>well developed 'process' based semantics with intrinsic message >>passing, is light-weight for multi-process creation, >>support application-level fault tolerance (quite applicable >>for failures in long computations...) >>and is production level, though not well known in the US. > > >I know some experimenters are looking at it as the core of a >software radio project (dttsp), but I don't know that they've >actually got hard core long term experience. > >http://www.g3ukb.co.uk/ > >A query to Bob might be useful. > I should clarify.. dttsp is the dsp engine, and srlink is the Erlang radio implementation, with dttsp being a (C-language) Erlang node. Jim From ballen at gravity.phys.uwm.edu Wed Sep 5 02:25:48 2007 From: ballen at gravity.phys.uwm.edu (Bruce Allen) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Big storage In-Reply-To: <20070904153512.A3565@ccdevli5.in2p3.fr> References: <46CEEE98.1050406@charter.net> <46CF4AD7.10504@charter.net> <46D33D74.4090909@charter.net> <8719.38.101.140.130.1188254867.squirrel@webmail.sonic.net> <20070828105016.GL21979@unthought.net> <20070830161435.A2589@ccvli02.in2p3.fr> <20070904153512.A3565@ccdevli5.in2p3.fr> Message-ID: Hi Loic, > [...] >> >> In a system with 24 x 500 GB disks, I would like to have usable storage of >> 20 x 500 GB and use the remaining disks for redundancy. What do you >> recommend? If I understand correctly I can't boot from ZFS so one or more >> of the remaining 4 disks might be needed for the OS. >> > This is (in my opinion) probably the only real issue with the X4500. > The system disk(s) must be with the data disks (since there are "only" > 48 disks slots) and the two bootable disks are on the same controller > which effectively make this controller a single point of failure (there > are easy ways to move the second system disk to another controller, but > you still need a working "first" controller to boot). Can you boot from a USB device? You can have an inexpensive RAID-1 USB device for the root and OS. > Although in our experience, controller failures are rare on the X4500 > (one failure in over a year with a few tens of X4500). Did you lose data with a controller failure? I assume can you just move the 48 disks to another box. It will take me some time to digest your other comments. But I made a mistake in what I wrote. I want to have a 48 disk box with 500 GB disks. >From this (raw) 24 TB of storage I want to get 20 TB usable (eg, lose no more than 8 disks of the 40 for redundancy and the OS). I mistakenly wrote 20/24 disks and 10 TB in my email. How would you revise your recommendations for 20TB of usable storage? Cheers, Bruce From landman at scalableinformatics.com Wed Sep 5 04:16:13 2007 From: landman at scalableinformatics.com (Joe Landman) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Big storage In-Reply-To: References: <46CEEE98.1050406@charter.net> <46CF4AD7.10504@charter.net> <46D33D74.4090909@charter.net> <8719.38.101.140.130.1188254867.squirrel@webmail.sonic.net> <20070828105016.GL21979@unthought.net> <20070830161435.A2589@ccvli02.in2p3.fr> <20070904153512.A3565@ccdevli5.in2p3.fr> Message-ID: <46DE8FFD.8000100@scalableinformatics.com> Bruce Allen wrote: > Hi Loic, > >> [...] >>> >>> In a system with 24 x 500 GB disks, I would like to have usable >>> storage of >>> 20 x 500 GB and use the remaining disks for redundancy. What do you >>> recommend? If I understand correctly I can't boot from ZFS so one or >>> more >>> of the remaining 4 disks might be needed for the OS. >>> >> This is (in my opinion) probably the only real issue with the X4500. >> The system disk(s) must be with the data disks (since there are "only" >> 48 disks slots) and the two bootable disks are on the same controller >> which effectively make this controller a single point of failure (there >> are easy ways to move the second system disk to another controller, but >> you still need a working "first" controller to boot). > > Can you boot from a USB device? You can have an inexpensive RAID-1 USB > device for the root and OS. FWIW: Other solutions can boot from USB, Flash, ... and provide the 48 drive capability. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From ballen at gravity.phys.uwm.edu Wed Sep 5 06:59:04 2007 From: ballen at gravity.phys.uwm.edu (Bruce Allen) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Big storage In-Reply-To: <20070905154639.A22654@ccdevli5.in2p3.fr> References: <46CF4AD7.10504@charter.net> <46D33D74.4090909@charter.net> <8719.38.101.140.130.1188254867.squirrel@webmail.sonic.net> <20070828105016.GL21979@unthought.net> <20070830161435.A2589@ccvli02.in2p3.fr> <20070904153512.A3565@ccdevli5.in2p3.fr> <20070905154639.A22654@ccdevli5.in2p3.fr> Message-ID: Loic, Thanks. In what I wrote I was using 1 TB = 10^12 bytes and 20 TB = 2 x 10^13 bytes. When you say that the max capacity with two system disks is 19.6 TB, do you mean 19 600 000 000 000 bytes or do you mean 19.6 x 1024^4 = 21 550 427 904 410 bytes? Once again, I need to study what you wrote... Cheers, Bruce On Wed, 5 Sep 2007, Loic Tortay wrote: > According to Bruce Allen: > [...] >>>> >>> This is (in my opinion) probably the only real issue with the X4500. >>> The system disk(s) must be with the data disks (since there are "only" >>> 48 disks slots) and the two bootable disks are on the same controller >>> which effectively make this controller a single point of failure (there >>> are easy ways to move the second system disk to another controller, but >>> you still need a working "first" controller to boot). >> >> Can you boot from a USB device? You can have an inexpensive RAID-1 USB >> device for the root and OS. >> > You can boot from a USB device, there are 4 ports available (2 on the > front side, 2 on the back). > > We booted a machine from an external DVD drive (there are also virtual > floppy and DVD drives available through the service processor). > >>> Although in our experience, controller failures are rare on the X4500 >>> (one failure in over a year with a few tens of X4500). >> >> Did you lose data with a controller failure? I assume can you just move >> the 48 disks to another box. >> > We did not loose any data due to the controller failure. > > The problem occured a few days before a scheduled downtime, the > mainboard was replaced during the downtime and the machine rebooted > just fine. > > Even if we hadn't been close to a scheduled downtime, the applications > running on most of our X4500 are fault tolerant enough that we can > offline a machine for some time without a significant impact. > >> >> It will take me some time to digest your other comments. But I made a >> mistake in what I wrote. I want to have a 48 disk box with 500 GB disks. >>> From this (raw) 24 TB of storage I want to get 20 TB usable (eg, lose no >> more than 8 disks of the 40 for redundancy and the OS). I mistakenly >> wrote 20/24 disks and 10 TB in my email. How would you revise your >> recommendations for 20TB of usable storage? >> > With 48 disks, there are also many different possible configurations. > > The default one (which only remains if you use the bundled Solaris > installation), is quite good and gives globally good results. > It's obvious that Sun has given a lot of thought to this. > > If you really want 20 TB (10*2^41 bytes) of usable space, then you > either need to: > . wait until Sun provides 750 GB or 1 TB disks (750 GB should be > available soon if I'm not mistaken); > . use a less redundant configuration that will not make the machine > resistent to controller failures and probably less resistant to > disks failures. > > We have an actual usable space of 16.9 TB on our machines (we mostly > use a minor variation of the Sun layout). > > > The largest possible usable space you can get from a X4500 with 48x500 > GB disks, two system disks and "some" redundancy is 19.6 TB. But this > is certainly NOT a configuration you want to use: > +-----------------------------------------------+ > | Controllers | > +-----------------------------------------------+ > | c5 c4 c7 c6 c1 c0 | > +-------------+-----------------------------------------------+ > ^ 7 | v1 | v1 | v1 | v1 | v1 | v1 | > | -------+-----------------------------------------------+ > | 6 | v1 | v1 | v1 | v1 | v1 | v1 | > | -------+-----------------------------------------------+ > | 5 | v1 | v1 | v1 | v1 | v1 | v1 | > | -------+-----------------------------------------------+ > D 4 | Sys2 | v1 | v1 | v1 | v1 | v1 | > i -------+-----------------------------------------------+ > s 3 | v2 | v2 | v2 | v2 | v2 | v2 | > k -------+-----------------------------------------------+ > s 2 | v2 | v2 | v2 | v2 | v2 | v2 | > | -------+-----------------------------------------------+ > | 1 | v2 | v2 | v2 | v2 | v2 | v2 | > | -------+-----------------------------------------------+ > | 0 | Sys1 | v2 | v2 | v2 | v2 | v2 | > +-------------------------------------------------------------+ > > That's two "raidz1" (single parity) vdevs of 23 disks (2 x 22+P). > > This a very bad idea if you consider basic best practices, Sun > engineers recommendations and, of course, current hardware reliability > as outlined in the (previously mentionned) article by Bianca Schroeder > and Garth Gibson. > > > If you want roughly 20 TB but can cope with less, then I suggest you > use the Sun configuration or one of its minor variation: moving the > second system disk and/or having 7 identically sized vdevs instead of 6 > (7 x 5+P + 1 x 3+P instead of 6 x 5+P + 2 x 4+P). > > We have tested about 25 different ZFS configurations with various I/O > workloads and unless you're willing to sacrifice available space or > data security, the Sun layout is the best balanced and also gives > good or acceptable performance for most workloads. > > > Lo?c. > From hahn at mcmaster.ca Wed Sep 5 09:47:46 2007 From: hahn at mcmaster.ca (Mark Hahn) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Big storage In-Reply-To: References: <46CEEE98.1050406@charter.net> <46CF4AD7.10504@charter.net> <46D33D74.4090909@charter.net> <8719.38.101.140.130.1188254867.squirrel@webmail.sonic.net> <20070828105016.GL21979@unthought.net> <20070830161435.A2589@ccvli02.in2p3.fr> <20070904153512.A3565@ccdevli5.in2p3.fr> Message-ID: > Can you boot from a USB device? You can have an inexpensive RAID-1 USB > device for the root and OS. it sounds strange, but I really like booting storage servers diskless. that is, have their OS state exported from some other machine, so all the disks in the server are purely data disks. this offers some nice properties when reconfiguring machines, even tolerating some kinds of failures. I note that this is how HP configures their SFS (product based on Lustre.) regards, mark hahn. From xclski at yahoo.com Sat Sep 1 20:57:04 2007 From: xclski at yahoo.com (Ellis Wilson) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: Message-ID: <735238.28092.qm@web37904.mail.mud.yahoo.com> My own "solution" to this is pretty draconian -- a "final solution" of sorts. I would legislate an "acceptable use agreement" for the Internet at the federal level (to be used for state models as well). It would not be worded to compromise the rights to free speech, it would leave pornography mostly alone (tempting a prize as that would be to idiot lawmakers) and would focus strictly on the issues above that are clearly attacks and which clearly cost a fortune. ----------- I personally feel that attempting to establish which actions online are within the realm of attacks is a job that will be unfortunately held (and run poorly) by human beings. Perhaps in the beginning all will be well, and indeed free speech will continue to reign. However, I would not at all be suprised if the entire system went awry and some federal body (I presume you are making this argument American centered, which also presents an interesting thought: America filters "acceptable" action on the net; China anyone?) decides to waver from its obviously unbiased stance and attempt to benefit a company moreso than another. Take this example. Let's say such an agreement was established at the "beginning of the internet" (yes, I know, I'm young :). One of the markets that seriously suffered (I'm talking billions here too) was term life insurance. One of those pesky sites came along, simply presenting the rates of each of the many companies in a quickly rational table, making competition much more steep. Previously, the haziness of truth had allowed these companies to avoid competition largely, but alas, the internet brought a swift and complete end to that. Let's also assume that as this was coming together, someone in the federal governement of wherever decided that this action would "clearly cost a fortune" to certain persons he/she was in league with, and thus, it would "clearly" constitute an attack. Thats the end of that site. To us, clearly, the above would abruptly disrupt free speech. However, it is of my belief that clarity is often viewed through a dollar bill (or other paper currency). The clarity of a Duke professor or say, a college student such as myself, might be well estranged from a money hungry politician, and thus able to achieve fair objectivity. Unfortunately, the "feds", run by politicians in large, I believe have an alterted slide rule to help them understand objectivity. Call me an internet darwinist. I certainly don't think taking over computers (even if they are largely unprotected) is awesome, and definitely don't make a past time out of it. However, if that is being done, and those victims cannot defend themselves properly, that is completely due to the lack of security on the victim's part. I don't care whether the code is "good" or "bad", if such things exist, I just cannot see (even the most well intended) restrictions on the internet ending up unabused. Ellis --------------------------------- Ready for the edge of your seat? Check out tonight's top picks on Yahoo! TV. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20070901/834e2a0c/attachment.html From gbyshenk at byshenk.net Mon Sep 3 02:37:24 2007 From: gbyshenk at byshenk.net (Greg Byshenk) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] [AMD64] Gentoo or Fedora In-Reply-To: <200709011646.37506.csamuel@vpac.org> Message-ID: <20070903093724.GG58515@core.byshenk.net> On Sat, Sep 01, 2007 at 04:46:37PM +1000, Chris Samuel wrote: > http://www.intel.com/pressroom/archive/releases/20040803net.htm > > # SAN FRANCISCO, Aug. 3, 2004 - Intel Corporation and Red Flag > # Software Co., Ltd, today announced that Red Flag is the first > # company to use the Intel???? C++ Compiler 8.0 for Linux* to compile a > # commercial version of its Linux operating system. Red Flag used > # Intel's tools to optimize its Red Flag Server 4.1 series products. > > I have no idea if they're still doing it, but given that the Red Flag > site hasn't had any new news since 2005, I doubt it.. Just wanted to point out that while the Red FLag _English_ site isn't being updated very regularly, the Chinese site appears to be much more current. This may be an indication only that Red Flag is focussing on the Chinese/Asian market, and not on western users. -- greg byshenk - gbyshenk@byshenk.net - Leiden, NL From betrussell23 at gmail.com Mon Sep 3 06:04:19 2007 From: betrussell23 at gmail.com (Bertrand Russell) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Diskless booting help: Message-ID: Dear Beowulf members, I am trying to cluster up 4PCs of SUN hardware (Opteron processor) running on CentOS-5. I am trying to do diskless booting. I have configured one machine as the Head node (with OS and other programs) with DHCP and its running TFTP-server. I don't have EPROM writer and planned to use ISOLINUX CD-ROM boot. I have made the bootable CD-ROM with the following files, boot.cat, initrd.img, vmlinuz (from CD#1 fo CentOS-5), isolinux.bin(syslinux directory-downloaded form its official site), and pxelinux.cfg (copied from centos wiki). But when I boot the client I am getting the following error message, "kernel panic - not syncing: VFS: Unable to mount root fs on unknown block (9,2). I could able to guess, this is due to the kernel could not able to mount the root file system which is not in the CD-ROM. But I want the root file system along with other directories to be mounted form the head node. Could anyone guide me to solve this? -- Miles to go......... Millions to meet........... Bertrand. P. S. Russell +91 - 98943 98441 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20070903/b7c94d09/attachment.html From xclski at yahoo.com Mon Sep 3 19:53:01 2007 From: xclski at yahoo.com (Ellis Wilson) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] [tt] World's most powerful supercomputer goes online In-Reply-To: Message-ID: <763552.13721.qm@web37906.mail.mud.yahoo.com> "Robert G. Brown" wrote: Maybe we then correct again and again until we find something that "works" -- isn't too costly, preserves function. We all have the right to bear arms, just not strapped to our belts when we ride on planes, not fully automatic weapons we have mounted to the roofs of our SUVs when we drive our kids to school. We have the right to freedom of speech, but not to shout fire in a theater or to express that freedom by publishing photographs of children having sex with adults ("consensual" or not). We have the right to liberty, but not if we use that liberty to rob, to rape, to murder, to injure others. I give up MY freedom to rob, rape, or murder you if I can get away with it (always possible in the state of nature, and we ALWAYS live in a state of nature!) in exchange for not being robbed, raped, or murdered in exchange, and we agree to get together and beat up anyone who tries to rob, rape or murder either one of us (whether or not they are "stronger") because together we are stronger and potentially meaner than they are. Upon reading certain concepts or ideas that are exceptionally rational yet heretofore unknown or misunderstood, I have a great tendency to laugh. I suppose it is a strange reaction not shared by most persons: to revere something so much that you laugh out loud, but I must say, the above truly and honestly made me laugh for sometime. And then I took a nap, not just because I had run 19 miles with my team at 6:05 pace, but also because that had sent my previously misconstrued and over idealistic concept of "freedom" for a loop and my head needed some time to mull it all over :). I must jump into the fire of a few more rants in the near future in hopes that I will learn half as much as this. Much thanks, Ellis P.S. - I hope I can contain myself someday if I have an urge to laugh upon the awe of my future wife bearing a child; I cannot imagine that one going well. --------------------------------- Choose the right car based on your needs. Check out Yahoo! Autos new Car Finder tool. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20070903/9b53c989/attachment.html From dkondo at lri.fr Tue Sep 4 00:45:52 2007 From: dkondo at lri.fr (Derrick Kondo) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] EuroPVM/MPI'07 -- Call for Participation (Early Registration Deadline: Sept 7) Message-ID: <60ec14620709040045ta31a2acxe4d4768a624e1e3d@mail.gmail.com> Call for Participation: EuroPVM/MPI'07 http://www.pvmmpi07.org Please join us for the 14th European PVM/MPI Users' Group conference, which will be held in Paris, France from September 30 to October 3. This conference is a forum for the discussion and presentation of recent advances and major challenges in Message Passing programming of clusters and other parallel machines. *** The deadline for early registration is September 7. So there is less than one week left to register at the reduced price. *** The conference will feature six keynote talks from pioneers and global leaders of message passing and parallel machines, namely: Tony Hey, Microsoft Research, USA Al Geist, Oak Ridge National Laboratory, USA Ewing Lusk, Argonne National Laboratory, USA Satoshi Matsuoka, Tokyo Institute of Technology, Japan Bernd Mohr, Central Institute for Applied Mathematics, Germany George Bosilca, University of Tennessee, USA Alongside the technical program, there will be three exciting tutorials led by experts in high-performance computing on Sunday, September 30th on the following topics: Verifying Parallel Programs with MPI-Spin Advanced MPI programming Using MPI-2: A Problem-Based Approach To conclude the conference, there will be an open forum where attendees can discuss recent modifications to the message passing standards and future directions. Also, the conference is a unique opportunity to meet the major developers and designers of communication libraries for HPC (such as PVM and MPI) and the major high-speed network interface builders to shape future research and development. The conference program and registration information can be found at: http://www.pvmmpi07.org Register soon to take advantage of the discount rates offered by the conference hotels. PC Chairs of EuroPVM/MPI'07: Thomas Herault, University of Paris Sud-XI / INRIA Futurs, France Franck Cappello, INRIA Futurs, France From tortay at cc.in2p3.fr Tue Sep 4 06:35:12 2007 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Big storage In-Reply-To: ; from ballen@gravity.phys.uwm.edu on Fri, Aug 31, 2007 at 05:26:48AM -0500 References: <46CEEE98.1050406@charter.net> <46CF4AD7.10504@charter.net> <46D33D74.4090909@charter.net> <8719.38.101.140.130.1188254867.squirrel@webmail.sonic.net> <20070828105016.GL21979@unthought.net> <20070830161435.A2589@ccvli02.in2p3.fr> Message-ID: <20070904153512.A3565@ccdevli5.in2p3.fr> According to Bruce Allen: [...] > > In a system with 24 x 500 GB disks, I would like to have usable storage of > 20 x 500 GB and use the remaining disks for redundancy. What do you > recommend? If I understand correctly I can't boot from ZFS so one or more > of the remaining 4 disks might be needed for the OS. > This is (in my opinion) probably the only real issue with the X4500. The system disk(s) must be with the data disks (since there are "only" 48 disks slots) and the two bootable disks are on the same controller which effectively make this controller a single point of failure (there are easy ways to move the second system disk to another controller, but you still need a working "first" controller to boot). Using ZFS for "/" is not easily done yet (as far as I know it's only available in OpenSolaris at the moment and it's not even available at installation time), so you need to use SVM (Solaris Volume Manager) if you want to mirror the system disk. The ZFS configurations we use minimize the impact of a single failing controller (which becomes more likely since there are 6 of these). Although in our experience, controller failures are rare on the X4500 (one failure in over a year with a few tens of X4500). The controllers are "simple" SATA controllers, there are probably less likely to fail than more advanced RAID controllers. The most frequent failure are (obviously) disks failure (about 3/week). Below is a X4500 disk tray as seen from "above", the columns are the controllers (they're "physically" that way), the rows are the SCSI targets (for instance the "Sys1" cell which is the first bootable device is -- in Solaris lingo -- c5t0 aka c5t0d0). The "vX" marks in the cells are used to specify membership to a "vdev" (a ZFS "virtual device" which can be a single disk or metadevice, a n-way mirror or a "raidz" volume). Here the vdevs are all raidz. This is the default configuration provided by Sun, which is globally pretty good (in terms of redundancy, reliability and performance): +-----------------------------------------------+ | Controllers | +-----------------------------------------------+ | c5 c4 c7 c6 c1 c0 | +-------------+-----------------------------------------------+ ^ 7 | v1 | v1 | v1 | v1 | v1 | v1 | | -------+-----------------------------------------------+ | 6 | v2 | v2 | v2 | v2 | v2 | v2 | | -------+-----------------------------------------------+ | 5 | v3 | v3 | v3 | v3 | v3 | v3 | | -------+-----------------------------------------------+ D 4 | Sys2 | v4 | v4 | v4 | v4 | v4 | i -------+-----------------------------------------------+ s 3 | v5 | v5 | v5 | v5 | v5 | v5 | k -------+-----------------------------------------------+ s 2 | v6 | v6 | v6 | v6 | v6 | v6 | | -------+-----------------------------------------------+ | 1 | v7 | v7 | v7 | v7 | v7 | v7 | | -------+-----------------------------------------------+ | 0 | Sys1 | v8 | v8 | v8 | v8 | v8 | +-------------------------------------------------------------+ All our machines have 48 disks but we tested a beta version of the X4500 with only 24 disks a bit more than one year ago. Only the "lower" 24 disk slots were populated (disks on rows 0 to 3). I'm not sure how the system handles system disk failure in such a case, since the second bootable disk slot is empty, but you could use something like this: +-----------------------------------------------+ | Controllers | +-----------------------------------------------+ | c5 c4 c7 c6 c1 c0 | +-------------+-----------------------------------------------+ ^ 7 | empty | empty | empty | empty | empty | empty | | -------+-----------------------------------------------+ [...] | -------+-----------------------------------------------+ D 4 | empty | empty | empty | empty | empty | empty | i -------+-----------------------------------------------+ s 3 | v1 | v1 | v1 | v1 | v1 | v1 | k -------+-----------------------------------------------+ s 2 | v2 | v2 | v2 | v2 | v2 | v2 | | -------+-----------------------------------------------+ | 1 | v3 | v3 | v3 | v3 | v3 | v3 | | -------+-----------------------------------------------+ | 0 | Sys1 | spare | spare | spare | spare | Sys2 | +-------------------------------------------------------------+ That is only 15x500 GB of usable space, but with lots of security. In order to match your usable space requirement, you can either use two or three (which would be better) of the spare disks in the vdevs but this makes the machine globally less resilient to controller failures. You can also avoid the second system disk altogether and use the last 5 disks on row 0 as a 4th vdev (ZFS allows vdevs to be of differents sizes, even though this requires a '-f' -- "force" -- flag to the "zpool create" call), which would yield 19x500 GB of usable space. It's of course better to have vdevs of similar size but the available space is not limited by the smallest vdev (unlike most RAID-5/RAID-6 implementations). The size difference has a small (but visible) impact on performances but, depending on your I/O workload, you can still get more throughput from the disks than the 4 on-board Gigabit interfaces can handle. According to several Sun engineers, it's also highly recommended to have (raidz) vdevs of only a few disks (less than 10). A more interesting configuration with 24 disks would be: +-----------------------------------------------+ | Controllers | +-----------------------------------------------+ | c5 c4 c7 c6 c1 c0 | +-------------+-----------------------------------------------+ [Empty slots] D -------+-----------------------------------------------+ i 3 | v1 | v1 | v2 | v2 | v3 | v3 | s -------+-----------------------------------------------+ k 2 | v1 | v1 | v2 | v2 | v3 | v3 | s -------+-----------------------------------------------+ | 1 | v1 | v1 | v2 | v2 | v3 | v3 | | -------+-----------------------------------------------+ | 0 | Sys1 | v1 | v2 | spare | v3 | Sys2 | +-------------------------------------------------------------+ ZFS hot-spares are global for a "pool", the disk in "c6t0" can replace any data disk. This gives 4 security (3 parity+1 spare) disks and 3 identically sized vdevs with a usable space of 18x500 GB. A bold configuration would be (starting from the previous one), to use the "spare" and "Sys2" disks in "v2" and "v3" respectively, to get 20x500 GB of usable space but at the expense of no hot-spare or security for the system disk. If I'm not mistaken, Sun now sells (again) X4500 with 250 GB disks. In order to get 10 TB/server, it's probably better (in terms of performance and data security) to have 48x250 GB, although I guess that you plan to buy a "half-full" machine to be able to add 24 larger (and cheaper) disks later. There are several very interesting blogs from Sun engineers about ZFS (linked from ). For instance this entry deals with the balance between data security and available (raw) space using some hard disk drives reliability figures: . Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From tortay at cc.in2p3.fr Wed Sep 5 06:46:39 2007 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Big storage In-Reply-To: ; from ballen@gravity.phys.uwm.edu on Wed, Sep 05, 2007 at 04:25:48AM -0500 References: <46CF4AD7.10504@charter.net> <46D33D74.4090909@charter.net> <8719.38.101.140.130.1188254867.squirrel@webmail.sonic.net> <20070828105016.GL21979@unthought.net> <20070830161435.A2589@ccvli02.in2p3.fr> <20070904153512.A3565@ccdevli5.in2p3.fr> Message-ID: <20070905154639.A22654@ccdevli5.in2p3.fr> According to Bruce Allen: [...] > >> > > This is (in my opinion) probably the only real issue with the X4500. > > The system disk(s) must be with the data disks (since there are "only" > > 48 disks slots) and the two bootable disks are on the same controller > > which effectively make this controller a single point of failure (there > > are easy ways to move the second system disk to another controller, but > > you still need a working "first" controller to boot). > > Can you boot from a USB device? You can have an inexpensive RAID-1 USB > device for the root and OS. > You can boot from a USB device, there are 4 ports available (2 on the front side, 2 on the back). We booted a machine from an external DVD drive (there are also virtual floppy and DVD drives available through the service processor). > > Although in our experience, controller failures are rare on the X4500 > > (one failure in over a year with a few tens of X4500). > > Did you lose data with a controller failure? I assume can you just move > the 48 disks to another box. > We did not loose any data due to the controller failure. The problem occured a few days before a scheduled downtime, the mainboard was replaced during the downtime and the machine rebooted just fine. Even if we hadn't been close to a scheduled downtime, the applications running on most of our X4500 are fault tolerant enough that we can offline a machine for some time without a significant impact. > > It will take me some time to digest your other comments. But I made a > mistake in what I wrote. I want to have a 48 disk box with 500 GB disks. > >From this (raw) 24 TB of storage I want to get 20 TB usable (eg, lose no > more than 8 disks of the 40 for redundancy and the OS). I mistakenly > wrote 20/24 disks and 10 TB in my email. How would you revise your > recommendations for 20TB of usable storage? > With 48 disks, there are also many different possible configurations. The default one (which only remains if you use the bundled Solaris installation), is quite good and gives globally good results. It's obvious that Sun has given a lot of thought to this. If you really want 20 TB (10*2^41 bytes) of usable space, then you either need to: . wait until Sun provides 750 GB or 1 TB disks (750 GB should be available soon if I'm not mistaken); . use a less redundant configuration that will not make the machine resistent to controller failures and probably less resistant to disks failures. We have an actual usable space of 16.9 TB on our machines (we mostly use a minor variation of the Sun layout). The largest possible usable space you can get from a X4500 with 48x500 GB disks, two system disks and "some" redundancy is 19.6 TB. But this is certainly NOT a configuration you want to use: +-----------------------------------------------+ | Controllers | +-----------------------------------------------+ | c5 c4 c7 c6 c1 c0 | +-------------+-----------------------------------------------+ ^ 7 | v1 | v1 | v1 | v1 | v1 | v1 | | -------+-----------------------------------------------+ | 6 | v1 | v1 | v1 | v1 | v1 | v1 | | -------+-----------------------------------------------+ | 5 | v1 | v1 | v1 | v1 | v1 | v1 | | -------+-----------------------------------------------+ D 4 | Sys2 | v1 | v1 | v1 | v1 | v1 | i -------+-----------------------------------------------+ s 3 | v2 | v2 | v2 | v2 | v2 | v2 | k -------+-----------------------------------------------+ s 2 | v2 | v2 | v2 | v2 | v2 | v2 | | -------+-----------------------------------------------+ | 1 | v2 | v2 | v2 | v2 | v2 | v2 | | -------+-----------------------------------------------+ | 0 | Sys1 | v2 | v2 | v2 | v2 | v2 | +-------------------------------------------------------------+ That's two "raidz1" (single parity) vdevs of 23 disks (2 x 22+P). This a very bad idea if you consider basic best practices, Sun engineers recommendations and, of course, current hardware reliability as outlined in the (previously mentionned) article by Bianca Schroeder and Garth Gibson. If you want roughly 20 TB but can cope with less, then I suggest you use the Sun configuration or one of its minor variation: moving the second system disk and/or having 7 identically sized vdevs instead of 6 (7 x 5+P + 1 x 3+P instead of 6 x 5+P + 2 x 4+P). We have tested about 25 different ZFS configurations with various I/O workloads and unless you're willing to sacrifice available space or data security, the Sun layout is the best balanced and also gives good or acceptable performance for most workloads. Lo?c. -- | Lo?c Tortay - IN2P3 Computing Centre | From tortay at cc.in2p3.fr Wed Sep 5 06:50:09 2007 From: tortay at cc.in2p3.fr (Loic Tortay) Date: Sat Jul 5 01:06:21 2008 Subject: [Beowulf] Big storage In-Reply-To: <46DE8FFD.8000100@scalableinformatics.com>; from landman@scalableinformatics.com on Wed, Sep 05, 2007 at 07:16:13AM -0400 References: <46D33D74.4090909@charter.net> <8719.38.101.140.130.1188254867.squirrel@webmail.sonic.net> <20070828105016.GL21979@unthought.net> <20070830161435.A2589@ccvli02.in2p3.fr> <20070904153512.A3565@ccdevli5.in2p3.fr> <46DE8FFD.8000100@scalableinformatics.com> Message-ID: <20070905155009.B22654@ccdevli5.in2p3.fr> According to Joe Landman: > Bruce Allen wrote: > > Hi Loic, > > > >> [...] > >>> > >>> In a system with 24 x 500 GB disks, I would like to have usable > >>> storage of > >>> 20 x 500 GB and use the remaining disks for redundancy. What do you > >>> recommend? If I understand correctly I can't boot from ZFS so one or > >>> more > >>> of the remaining 4 disks might be needed for the OS. > >>> > >> This is (in my opinion) probably the only real issue with the X4500. > >> The system disk(s) must be with the data disks (since there are "only" > >> 48 disks slots) and the two bootable disks are on the same controller > >> which effectively make this controller a single point of failure (there > >> are easy ways to move the second system disk to another controller, but > >> you still need a working "first" controller to boot). > > > > Can you boot from a USB device? You can have an inexpensive RAID-1 USB > > device for the root and OS. > > FWIW: Other solutions can boot from USB, Flash, ... and provide the 48 > drive capability. > Sure, and so can the X4500. :-) But, with only two RAID controllers there is no way your machine will survive a controller failure when using RAID-5 or RAID-6 unless you're willing to half the available space (or add the optional extra controllers). The density of the X4500 is also slightly better (48 disks in 4U instead of 5U). As of today we have 112 X4500, 112U are almost 3 racks which is quite a lot due to our floor space constraints. Besides, Sun machine have been available for one year and they can deliver and maintain theirs reliably wherever we want (specifically in Lyon, France). The main performance figure on your website is quite pointless (sorry) since there is no mention of the number of files, file(s) size(s), number of threads, block size and amount of bytes written. I do 1.7 GB/s with random writes on a X4500,