Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

A Look At CERN's LHC Grid-Computing Architecture

Soulskill posted more than 4 years ago | from the going-for-the-gusto dept.

Supercomputing 53

blair1q writes "Using a four-tiered architecture (from CERN's central computer at Tier 0 to individual scientists' desk/lap/palmtops at Tier 3), CERN is distributing LHC data and computations across resources worldwide to achieve aggregate computational power unprecedented in high-energy physics research. As an example, 'researchers can sit at their laptops, write small programs or macros, submit the programs through the AliEn system, find the necessary ALICE data on AliEn servers, then run their jobs' on upper-tier systems. The full grid comprises small computers, supercomputers, computer clusters, and mass-storage data centers. This system allows 1,000 researchers at 130 organizations in 34 countries to crunch the data, which are disgorged at a rate of 1.25 GB per second from the LHC's detectors."

cancel ×

53 comments

Sorry! There are no comments related to the filter you selected.

I wonder when.. (1)

Superken7 (893292) | more than 4 years ago | (#32219458)

I wonder when we will have the equivalent computing power at home? :)

Re:I wonder when.. (1, Funny)

Anonymous Coward | more than 4 years ago | (#32219478)

According to Moore's Law.. about 4 Hitlers away

Damn thats Godwins.. Moore's says about 12 years.

Re:I wonder when.. (1)

William Robinson (875390) | more than 4 years ago | (#32219548)

I wonder when we will have the equivalent computing power at home? :)

When you will create a black hole at home. Simple !!!

I have a dream... (1, Funny)

Mathinker (909784) | more than 4 years ago | (#32219554)

Your post makes me wonder about a future where I have a home computer powerful enough to run an algorithm which downloads as many tracks off of iTunes as it needs and then can compute by extrapolation the future hits of RIAA, before they are released.

One wonders whether the courts would find that such a program is a circumvention of DRM for the purposes of the DMCA. Unfortunately, the computer, which can answer that question, will be destroyed by the construction of a .....

(Ouch. I should go get some sleep....)

Re:I have a dream... (1)

jvin248 (1147821) | more than 4 years ago | (#32221544)

put enough monkeys on typewriters and you'll get shakespeare (http://en.wikipedia.org/wiki/Infinite_monkey_theorem_in_popular_culture)

And someone started such a simulator : http://everything2.com/title/Monkey+Shakespeare+Simulator [everything2.com] That's what you use this for. And Pr0n.

Re:I wonder when.. (1)

Just_Say_Duhhh (1318603) | more than 4 years ago | (#32220250)

You can have that kind of computing power at home TODAY. Stumble over to http://boinc.berkeley.edu/ [berkeley.edu] to see where LHC and other projects get their global computing power (SETI@HOME is probably the most famous). From there you can start your own project, and in no time you'll have millions of computers around the world crunching away for you! Maybe that'll help you pick next week's winning lottery numbers?

Speaking as an LHC physicist... (1)

Roger W Moore (538166) | more than 4 years ago | (#32223476)

...I already do! One of the great things about the LCG is that you can submit and monitor jobs from anywhere. It is used by far more than the 1,000 physicists the article mentions. There are 2,500+ on ATLAS alone and then there is CMS and LHCb to count as well.

Re:I wonder when.. (1)

splogic (1797526) | more than 4 years ago | (#32229520)

And I wonder when somebody will think of a better waste of time, energy, and computer cycles. I think the LHC is pretty much a Black Hole for money that gets pumped into various "projects" that otherwise would not get funded. Other than that, it doesn't serve much scientific value. A little blip, in a sea of particles is not enough evidence to verify anything.

1.25GB/sec not that much. (0)

Anonymous Coward | more than 4 years ago | (#32219530)

A single 10gb ethernet connection can handle that quite easily. a single symmetrix/shark should be able to keep up.

Re:1.25GB/sec not that much. (1)

$RANDOMLUSER (804576) | more than 4 years ago | (#32219738)

Apparently the LHC detectors spew out 1.25GB of data per second. That's about six times the contents of Encyclopedia Britannica including the index every second.

That's 3.7 Libraries of Congress per hour for those of us on the other side of the pond.

Re:1.25GB/sec not that much. (3, Insightful)

Alef (605149) | more than 4 years ago | (#32219854)

A single 10gb ethernet connection can handle that quite easily.

Eh. A 10 Gb ethernet connection can't handle 1.25 GB/s at all, not to mention doing it reliably. Theoretically, 10 Gb is exactly 1.25 GB, but then you need to account for protocol overhead, packet loss and so on.

Re:1.25GB/sec not that much. (0)

Anonymous Coward | more than 4 years ago | (#32219974)

mmm, channel bond 2 of them together, then you have a 'single' logical 20Gbps link!

Re:1.25GB/sec not that much. (2, Interesting)

evilviper (135110) | more than 4 years ago | (#32221244)

Theoretically, 10 Gb is exactly 1.25 GB, but then you need to account for protocol overhead, packet loss and so on.

You don't have to use TCP/IP over ethernet, you know. AoE & FCoE come to mind.

There are very few ways you might lose packets in a well-built local data-link network. Collisions/congestion are a thing of the past. Modern networking gear is fully capable of forwarding packets at full speed. Packets don't just go flying out of CAT-6A, never to be seen again.

And yes, bonding two 10Gbps ethernet interfaces would give you greater than 1.25GByte/s, though certainly NOT double that...

Re:1.25GB/sec not that much. (1)

Alef (605149) | more than 4 years ago | (#32221598)

You don't have to use TCP/IP over ethernet, you know. AoE & FCoE come to mind.

Mechanisms for ensuring reliable transfer of data aren't exclusive to TCP/IP. There is also some overhead in the packet headers (yes, it could be made very small if you use non-standard Ethernet frames).

There are very few ways you might lose packets in a well-built local data-link network.

Well, yeah, depending on how you define local. I don't know what distances they need to transfer the data at CERN, and I imagine there could be all sorts of nasty EM fields around a 14 TeV particle accelerator.

Anyway, to say you could easily handle this data stream on a single 10 Gb/s Ethernet connection is a bit of a stretch, don't you think?

truly amazing (5, Funny)

Goldsmith (561202) | more than 4 years ago | (#32219532)

I was having lunch with some CERN guys a couple weeks ago, and was asking them about the speed of their analogue to digital converters. I don't remember what the number was, but it seemed low to me, something like 200kHz. So, of course, I had to point out that *my* cheapo converters ran faster than theirs by more than an order of magnitude. They responded with "well, each of our converters does 200kHz on all of our 4000 channels at the same time, so we're really recording at..."

They won.

Re:truly amazing (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#32219728)

I grew up in a Christian home as a Christian of course. But these last three years I kind of took on the title of an Atheist. Being a Bio-Major in college most of what science explains about life all seemed to make more sense then what the bible was telling me. Lately I have been finding myself under some really bad luck. I first got into a fist fight with my mom and then she kicked me out. My car was in her name so she called and canceled my registration. They cut back my hours at work while i was in school and now there is no way for me to get them back. So working 20 hours a week and homeless wasn't a pretty site. My bf let me stay with him and his dad, but while Im there I started noticing pregnancy symptoms. I told him and he want me to get an abortion. I am highly opposed to abortions but he insist that im not gonna mess up his life. Then he starts getting sick. And i mean extremely sick sore throat, body aches, fever, chills, swollen gums and canker sores and pain in his mouth.

About five days into him getting sick and seeing his condition not getting any better, I started to pray. I felt so bad praying and furthermore i felt like no one was listening. But two days later his condition got better. No more body aches, no more sore throat, or fever... but the swollen gums, and ccanker sores were still there. It was so painful he refused to eat. So i prayed some more but nttn happened. i figured the first time it was just a coincidence and there was still no God.

Yesterday while we were at work, my boyfriend texted me about the abortion and I told him yes I would get one if Im pregnant.(we still arent sure dr. appt monday). He also texted me and told me while he was at a grocery store with his cousins a weird crazy guy shouted "LIAR" in his face. Which was weird but where im from alot of crazy ppl do crazy things so i didnt really think anything of it. After he told me that I poured my heart out to him about how I been praying for him, my bf a somewhat Atheist too was shocked. He said he would pray with me tonight about his health. At that moment I realized if there was a God us praying is so unfair. We came back to him when everything started going wrong but when things start to go right will we walk away again.

When i left work it started raining extra heavy. Like as soon as I left out the door and once I got in shelter under the bus station it stopped. Again didnt think anything of it because it was raining on and off for about a hour while i was at work. I caught the bus to my BF apartment and while i was walking home it began to lightening. And anybody who knows me Know that I am afraid of lightening. As a kid you couldn't put me near a window during a thunderstorm. Now that I am older its easier to compose myself but im still scared. So im walking extra fast to get to my BF apartment complex. Then a loud thunder cloud clapped. I asked in my head "God are you mad at me?" and this big lightening bolt came down and struck the electrical pole. Next thing I knew it was pitch black outside. It scared the living daylights out of me and I ran to my BF apartment.

Of all this stuff happening, do you think its just a coincidence or are these signs? And believe me this is a true story. everything that i said happen really did happen. I am not a christian trying to change the beliefs of Atheists or any one I just want to know if you were in my position and all this happened would you take it as being God or merely a coincidence?

Re:truly amazing (0)

Anonymous Coward | more than 4 years ago | (#32220018)

Um.

This is a really weird post and so dramatically offtopic in this forum that it probably makes people either annoyed or uncomfortable. Perhaps it is even an automatic bot that picked up on the words "truly amazing". As such it should probably not get a reply.

In the case I am wrong:
1. It appears you have no support, and you are unlikely to get support here on a public offtopic forum. Find a person you can talk to directly that either loves you but has no vested interest in your situation, or has training in counselling but no bias in the decisions you have to make. Internet forums cannot help you here.

2. Your perception of God are dark and punitive. That God will do nothing for you or your mental health. If you pray, pray to the one that wants the best outcome for you and those around you.

3. The Christian/Atheist debate is not helpful for any strong or healthy person, and even worse for you if you feel not strong or not healthy. Stay away from it. Beware of labelling yourself or others, because those labels create unattainable expectations and do not reflect all that you or other people are. Labelling is a dehumanizing technique.

Re:truly amazing (0)

Anonymous Coward | more than 4 years ago | (#32220088)

YHBT YHL HAND

Re:truly amazing (1)

shentino (1139071) | more than 4 years ago | (#32228180)

This, coming from someone who is too chicken to post their troll material under their real name and take the karma hit like a man.

Re:truly amazing (-1, Troll)

McTickles (1812316) | more than 4 years ago | (#32219932)

IT people at LHC are the biggest liars in the universe, they are also pretty incompetent and get confused by simple questions... One actual example (specifics removed to protect the innocent): "So you configure the IP like this thru this software?" "Oh dear no! I am a hardware guy, I dont deal with software! IPs are a hardware problem..." "No they are a software setting..." "No they are hardware!" and so on...

Re:truly amazing (0)

Anonymous Coward | more than 4 years ago | (#32220576)

Sounds like someone is jealous...

200kHz x 400 channels is nothing (1)

Roger W Moore (538166) | more than 4 years ago | (#32223298)

It depends on which CERN guys you talk to. When I was a grad student we had a 1 GHz ADC (with fewer channels and only 8 bits IIRC) reading out scintillator which timed protons in a beam to O(10ps) timing resolution. I've been more involved with triggers than front end digitization since then but 200 kHz and 400 channels is nothing - the ATLAS calorimeter alone has 110,000 channels and its ADC's operate in the 10's MHz range (IIRC - you'd have to look up the ATLAS detector paper for exact numbers).

Re:200kHz x 400 channels is nothing (1)

Goldsmith (561202) | more than 4 years ago | (#32228032)

You know, I don't really remember the exact numbers we were talking about at lunch. I'm sorry if I got some of the numbers wrong, particularly if they ended up being far to small, that would annoy me.

I just remember being very impressed at the insane amount of total data coming in. I'm definitely more used to the setup of a single GHz ADC, switching between a handful of channels.

There's a big difference in scale between a condensed matter experiment, where I get to do absolutely everything myself, and something like ATLAS. I'm amazed that these various groups (hardware, software, design, construction, cryogenics, analysis... all on each detector) are able to work together so well.

In other words: The LHC Grid (1, Funny)

Anonymous Coward | more than 4 years ago | (#32219596)

is a botnet !

Thanks in advance.

Yours In Akademgorodok,
K. Trout

That's awesome... (0)

Anonymous Coward | more than 4 years ago | (#32219676)

That's awesome, but will it run Crysis at max settings?

Good boffin! Who's a smart boffin? Yes you are! (0)

$RANDOMLUSER (804576) | more than 4 years ago | (#32219706)

Good thing they didn't use PS3s to build [slashdot.org] it.

Oblig. (0)

lloydsmart (962848) | more than 4 years ago | (#32219712)

Wow, imagine a Beowulf cluster of those!

I want more! (1)

lemur3 (997863) | more than 4 years ago | (#32219726)

I was quite saddened to find that this 'Look at CERN's LHC GRID' ....didnt include any pictures. :-(

Re:I want more! (4, Interesting)

Arathrael (742381) | more than 4 years ago | (#32221154)

Have a look here: http://lcg.web.cern.ch/LCG/image.htm [web.cern.ch] for Google Earth based dashboards showing WLCG live grid sites, links, data transfer and job activity.

Just out of curiosity what kind of db? (0)

Anonymous Coward | more than 4 years ago | (#32219732)

What kind of database do you think they would be using? would it be sql based, as they would obviously need to query? but storing that amount of data? at that extremely fast rate??

LCH@Home (1)

Luyseyal (3154) | more than 4 years ago | (#32219742)

I just wish they would send some more work units down the LHC@Home pipe. None of my computers have done any work for that project in ages.

-l

Re:LCH@Home (0)

Anonymous Coward | more than 4 years ago | (#32220514)

You might want to check your statistics page. I've had (let's see...) four work units complete in the last two weeks. Granted this is after months of silence, but there seems to be a trickle of new data coming.

Re:LCH@Home (1)

Luyseyal (3154) | more than 4 years ago | (#32220708)

Hrm, maybe they think my computers are too slow or something. It's been well over a year since I did any LHC work.

-l

Re:LCH@Home (1)

sznupi (719324) | more than 4 years ago | (#32256492)

Not likely, one Athlon XP 1700+ with 768 MiB of RAM, that I keep around here, got maybe a dozen workunits in the last month.

It seems now the project is gearing up specifically for calculations of later stage of LHC, when it will transfer to operating at greater power.

Re:LCH@Home (1)

Luyseyal (3154) | more than 4 years ago | (#32257262)

I have another theory. I don't leave any computers on at night so perhaps the jobs are going out during the European daytime when mine are offline and there is simply not enough work to go around, yet.

-l

LHC IT people are far from top notch (0)

Anonymous Coward | more than 4 years ago | (#32219820)

IT people at LHC are the biggest liars in the universe, they are also pretty incompetent and get confused by simple questions...
One actual example (specifics removed to protect the innocent):
"So you configure the IP like this thru this software?"
"Oh dear no! I am a hardware guy, I dont deal with software! IPs are a hardware problem..."
"No they are a software setting..."
"No they are hardware!"
and so on...

Re:LHC IT people are far from top notch (1)

lepidosteus (1102443) | more than 4 years ago | (#32220800)

Anonymous Cowards people at Slashdot are the biggest liars in the universe, they are also pretty incompetent and get confused by simple questions...
One actual example (specifics removed to protect the innocent):
"So you have any proof of such IT incompetence by them, it's not just jealousy right ?"
"Oh dear no! I am a story teller guy, I dont deal with proof! Proofs are a wikipedia problem..."
and so on...

Correction (2, Informative)

mike260 (224212) | more than 4 years ago | (#32219822)

Using a 4-tiered architecture (from CERN's central computer at Tier 0 to individual scientists' desk/lap/palmtops at Tier 3) [...]

Sorry for being pedantic, but the article says there are three tiers between the central computer and the scientist's machines (which are tier 4, not 3).

Re:Correction (1)

olsmeister (1488789) | more than 4 years ago | (#32220254)

Tier 0
Tier 1
Tier 2
Tier 3

Re:Correction (1)

mike260 (224212) | more than 4 years ago | (#32220824)

And finally Tier 4 which comprises, according to TFA, the scientists desk/lap/palmtops.

Re:Correction (1)

Arathrael (742381) | more than 4 years ago | (#32221138)

Correct. Tier 3 is local batch farm facilities, etc., which aren't really part of the project as such.

Re:Correction (1)

blair1q (305137) | more than 4 years ago | (#32240878)

Interesting. I don't remember writing it that way. Did someone edit the submission before it was posted? Or has it just been a long weekend?

Data to crunch (4, Interesting)

Lord Byron II (671689) | more than 4 years ago | (#32219840)

As someone who worked on the processing of HEP experimental data for awhile, let me say that there is a ton of work to do. You have particles entering the detector every ~40ns and hundreds of different instruments making measurements, which leads to a ton of data very quickly. You then have to reconstruct the path of the particle based off of the detector information, but it's not straight-forward. The detector can have gaps in coverage; neutrinos (which are undetectable) can be created removing momentum; particles from the previous event can still be in the detector et cetera.

And all of the data crunching you do must be done in 40ns, so that you're ready for the next set. (Of course, you can do some processing offline, but if you don't maintain a 40ns average, then your data will start piling up.)

Re:Data to crunch (0, Flamebait)

McTickles (1812316) | more than 4 years ago | (#32219896)

More like "noise crunching", all I can see LHC producing right now is recordings of noise... Now I am not saying there is nothing interesting in that noise but honestly is it worth all that money in trouble to perhaps find some sort of relevant thing in there? One can find any sort of pattern in noise if one really wants to... Doesn't prove anything.

Re:Data to crunch (0)

Anonymous Coward | more than 4 years ago | (#32220280)

Dear LHC scientists,
Well, it was fun while it lasted but it's time to pack up the LHC and head home. Why, you ask? You need look no further than the incredibly subtle insights of esteemed professor/gibbering slashdot troll McTickles. Next time you make one of these funny contraptions, remember to fill your it with expanded polystyrene and talk no louder than 30dB - that should help cut down on all of this noise he is so worried about.

Re:Data to crunch (1)

$RANDOMLUSER (804576) | more than 4 years ago | (#32220042)

...there is a ton of work to do ... which leads to a ton of data...

Holy crap! I had no idea that the relativistic speeds involved would cause the mass to increase that much!

Re:Data to crunch (5, Informative)

klazek (1134141) | more than 4 years ago | (#32220492)

You have particles entering the detector every ~40ns and hundreds of different instruments making measurements, which leads to a ton of data very quickly.

Not exactly true. It's running at 40 MHz, so that's 25 ns bunch spacing. Further, you don't exactly have to 'crunch' the data as it comes in, there are multiple triggers that throw lots of data away based momentum cuts and other criteria before it ever makes out of the detectors.

In ATLAS, for example, there are ~ 10e+9 interactions/sec. The Level1 Trigger, consists of fast, custom electronics programmed in terms of adjustable parameters to control filtering algorithms. Input is from summing electronics in the EM and hadron calorimiters, and signals from the fast muon trigger chambers. The info is rather coarse at this point (transverse momentum cuts, narrow jet criteria, etc), and at level one the info rate is decreased in about ~2us (including communication time), from 40MHz to about 75KHz. Level2 now does a closer look, taking more time and focusing on specific regions of interest (RoIs). This process takes about 10ms, and data rate is reduced to about 1KHz for sending to the event filter. Here, the full granularity of the detector (the 'detector means all the bits - Inner detectors: Pixels, strips, Transition Radiation tracker - The calorimiters - The muon tubes at the outside radius) and runs whatever selections algorithms are in use. This takes a few seconds, and output is reduced to about 100Hz and written to disc for a gazillion grad students (like myself) to analyze endlessly and get our PhDs.

There is much more to it of course, but you can find info about it on line if you really are interested in the details. Have a look at the ATLAS Technical Design Report: http://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/TDR/TDR.html [web.cern.ch]

USENIX LISA presentation (2, Interesting)

Anonymous Coward | more than 4 years ago | (#32220048)

There was a good presentation at LISA '07 on this entitled "The LHC Computing Challenge":

http://www.usenix.org/event/lisa07/tech/tech.html#cass

It was given by Tony Cass, who is/was "responsible for the provision of CPU and data storage services to CERN's physics community". They're planning on collecting 15PB/year.

lhc infrastructure history? (1)

flok (24996) | more than 4 years ago | (#32220478)

I wonder what they at cern were expecting to have for infrastructure at the end of the project (e.g. now).
I mean: who could have guessed the processorspeed and diskspace we have now.

Re:lhc infrastructure history? (3, Funny)

TheRaven64 (641858) | more than 4 years ago | (#32226570)

I mean: who could have guessed the processorspeed and diskspace we have now.

Gordon Moore?

As a Tier-3 manager... (2, Interesting)

imevil (260579) | more than 4 years ago | (#32221916)

I can say that the article doesn't explain it very well. Since CERN has been calling the sites "Tier", this terminology has become a buzzword, and everything is a Tier (the managers call their services "Tiered" just to make them sound important).

Tier0 and Tier1 are well described by the article. Tier2 are mostly computing clusters, with of course big storage, but they're mainly for analysis. Tier3 are like Tier2 but not really. They are "uncertified" Tier2 in the sense that they do not strictly adhere to the Tier2 standards in terms of middleware and configuration and policies.

Tier4... never heard of that, I think the buzzword Tier backfired and they're calling their desktops Tiers. When I started managing the Tier3 we did not even call it like that... it was just a cluster.

sounds familiar (0)

Anonymous Coward | more than 4 years ago | (#32222164)

in fact it sounds just like what i do to get
my latest TV-series (from bit-torrent).
-
oh come ON! anybody that still believes IONIZING,
thus changing and INTERFERING with the "to-be-studid"
object at hand is FUNDAMENTAL is plain ... STFU.

Imagine (1)

MistrX (1566617) | more than 4 years ago | (#32252164)

Imagine a Beowulf cluster of these! \o/

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>