Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Australia The Internet IT

Extreme Heat Knocks Out Internet In Australia 103

Posted by Soulskill from the some-really-don't-like-it-hot dept.
An anonymous reader writes with news that bad weather caused internet connectivity problems for users in Perth, Western Australia on Monday. But it wasn't raging storms or lightning that caused this outage — it was extreme heat. Monday was the 6th hottest day on record for Perth, peaking around 44.4 C (111.9 F). Thousands of iiNet customers across Australia found themselves offline for about six and a half hours after the company shut down some of its systems at its Perth data center at about 4.30pm AEDST because of record breaking-temperatures. ... "[W]e shut down our servers as a precautionary measure," an iiNet spokesman said late Monday night. "Although redundancy plans ensured over 98 per cent of customers remained unaffected, some customers experienced issues reconnecting to the internet." ... Users in Western Australia, NSW, Victoria and South Australia took to Twitter, Facebook and broadband forum Whirlpool to post their frustrations to the country's second largest DSL internet service provider.
This discussion has been archived. No new comments can be posted.

Extreme Heat Knocks Out Internet In Australia More

Extreme Heat Knocks Out Internet In Australia

Comments Filter:

  • One ISP is not 'the internet' (Score:5, Informative)

    by pipedwho ( 1174327 ) on Tuesday January 06, 2015 @01:23AM (#48743293)

    Yes, many people were affected, but iiNet is not 'the internet'. All the other big providers were still running just fine.

    You could post the same headline every time someone's modem cable gets knocked out or their router crashes.

    • The recent heatwave will have knocked plenty of people offline at the pole, the first hot day in a Melbourne summer is always chaos on the trains for similar reasons (hot metal expands). However this incident just appears to be iiNet's server room air-con that fell over, nothing to do with the weather.

    • Re: (Score:2)

      by kv9 ( 697238 )
      What do you mean, it has "inet" right in the name!

    • You could post the same headline every time someone's modem cable gets knocked out or their router crashes.

      A large chunk of Northern VT and NH were knocked out for most of a day a few weeks back when some server monkey at Fairpoint bend the connectors on a blade server (and they apparently have no redundancy). The restore time was the drive time from Boston on their SLA plus a winter storm going on (coincidentally, but not good timing for maintenance).

      But, shit happens. Was there a Slashdot front pager o

  • It's unbelievable that a data centre can't cope with an extra degree or two. What sort of idiot designs these places? Haven't they heard of tolerances?

    • Re:(in)Tolerance (Score:5, Informative)

      by talis9 ( 166451 ) on Tuesday January 06, 2015 @01:36AM (#48743341)

      they had multiple failures. The primary air con failed, and so did the backups

      http://blog.iinet.net.au/statement-chief-technology-officer-mark-dioguardi/

      • Re: (Score:1)

        by Anonymous Coward

        http://blog.iinet.net.au/statement-chief-technology-officer-mark-dioguardi/

        Take the statements with a huge cup of salt. "Network redundancy plans ensured over 98% of our customers’ broadband services were unaffected - See more at: http://blog.iinet.net.au/statement-chief-technology-officer-mark-dioguardi/#sthash.FSMpB8sc.dpuf" is using some big weasel words.

        They turned off the PPP auth servers. So any users that weren't already logged in, couldn't log in. And with the PPP servers offline, they have

    • did the severs get shut down or did a they do a hard power off after tripping the overheat shutdown system in the box?

    • It's unbelievable that a data centre can't cope with an extra degree or two. What sort of idiot designs these places? Haven't they heard of tolerances?

      They had air conditioners fail. They probably needed more redundancy, but they shut down some systems as a precaution when the AC failed.

    • Re:(in)Tolerance (Score:5, Insightful)

      by itzly ( 3699663 ) on Tuesday January 06, 2015 @02:46AM (#48743609)
      Yes, it's unbelievable that something, somewhere goes wrong.

  • Statement from CTO of iiNet (Score:4, Informative)

    by Anonymous Coward on Tuesday January 06, 2015 @01:38AM (#48743351)

    http://blog.iinet.net.au/statement-chief-technology-officer-mark-dioguardi/

    Basically both main and backup aircon went down.

    • "Basically both main and backup aircon went down."

      Which is only, oh, so unsual.

      There're two (basic) kinds of high avaliability: load balancing and redundancy with their typical failure modes:
      1) load balancing: the surviving part can't cope with the aggregated load and it also goes down.
      2) redundancy: once the main fails, either the migration protocol fails or the reserve doesn't work.

      This is HA 101 knowledge but as long as CTOs can go with an "oh! who could expect two failures in a row!" instead of being f

      • Re: (Score:2)

        by paziek ( 1329929 )
        Why would you fire someone who just got more experienced - especially in such important field - than when he was being hired?

      • Because a customer's internet went down due to equipment failure during record breaking weather conditions the CTO should automatically resign?

        May I suggest get a grip and come on back down to reality. From here you can assign and prove fault / wilful negligence.

        As for your HA 101 comment, the world doesn't work on hazards, it works on risk, and it works on minimum acceptable risk. I would rather see the CTO get fired if he spend countless dollars gold plating a system so a few measly customers can get four

        • "Because a customer's internet went down due to equipment failure during record breaking weather conditions"

          No, the problem didn't come from record-breaking conditions but because the N+1 high avaliability system designed to cope with those conditions not only failed, but failed in the most expectable way due to lack of forethought.

          • You're assuming a true N+1 system was what was designed. Back to risk since you clearly don't understand the concept. We have here a case of a very small number of customers experiencing a problem for a very small time period. You can now go up to upper management and justify to them why they should spend money on this.

            Not everything needs to be gold plated, and not everything needs to be N+1. Quite the opposite actually, if you head into many substations or building complexes in the country you'll likely f

            • "You're assuming a true N+1 system was what was designed."

              I do, because that is exactly what the CTO himself said: "Our Perth data centre was subject to a partial failure of both the mains and backup air-conditioning systems yesterday".

              "Not everything needs to be gold plated, and not everything needs to be N+1."

              Quite true. But here there *was* a N+1 system just to allow the "+1" part to fail, which begs the question of why the expenditure on the "+1" part was allowed.

              • Backup does not mean anything. What backup, what capacity? Does the backup assume the same worst case as the main? Is the backup susceptible to the same failure as the primary?

                You see, you don't actually know anything. You think you do because you read two soundbites and now you're asking for the CTO's head.

        • "I would rather see the CTO get fired if he spend countless dollars gold plating a system"

          How do you call, then, paying for an aircon system bigger than needed that demonstrates that it doesn't work when needed? He was doing exactly what you claim he shouldn't do.

          • Why are you looking at the aircon? You should be looking at the customers. Now how many customers did iiNet lose as a result of this short outage affecting only a tiny percent? None?

            So tell me again what the design case was? Companies exist to make money, not to provide four 9s uptime for their clients. This so far was a one-off case. iiNet is one of the more reliable providers in the country, so what is the business decision behind gold plating the system so it never goes down? How much money are you willi

            • "Why are you looking at the aircon? You should be looking at the customers."

              No, I shouldn't. The air conditioner was an already paid for expenditure which was approved upon a -now demonstrated, failed ROI. And the fact the CTO mixes apples to oranges makes the clear case that he is just in CYA mode, both about himself and his company.

              • Tell me again how it failed its ROI. It has successfully kept the datacentre cool for all but a couple of hours of the year.

                You're starting to use all fancy words and are beginning to talk more and more crap in the process. But since you know everything tell me again what are all the design decision that went into the investment. You seem to know it failed it's ROI so I assume you're intimately familiar with the design of that datacentre and the business case for operating it. So please do share with us.

                • "You seem to know it failed it's ROI"

                  Of course I do. The investment of a backup, whatever the investment plan is, is to cover for the time when the mains is in failing or maintenance mode. Here the mains went nuts and the backup was not there to cover it. It's no need for more "intimate familiarity" than that.

                  • Please continue to show your ignorance. A "backup" has many definitions especially those used for maintenance. If you think all "backup" systems should run regardless of the what is happening to the primary system then you're deluded. Many backups installed are half-size, or even quarter-sized to allow work in scheduled periods. Systems that are full sized are not designed to copy with preventing failure due to overload and never have been.

                    But you said of course you know the ROI equation so please tell me,

                    • "Please continue to show your ignorance. A "backup" has many definitions especially those used for maintenance."

                      It might be the general case but certainly it is not the current one, CTO stating that the backup was used to cover for the mains failure -and failing at that, so it is not a "maintenance-only" setup.

                      "Anyway I'm out. This conversation is going nowhere. I suggest you maybe enrol in a project management"

                      That's true. You just stay saying the same stupid things, babbling out some disconected concepts

      • 0.4% design temperature for Perth is 36.2C. A DX system with condensers on the roof would be designed for a temporary stature of 41-43C typically. Once you get much above that, there isn't much you can do with DX; you will overload your compressors quickly. A cooling tower should be more robust, but your envelope load could have exceeded the primary system capacity.

        Typically in extreme temperatures a Tier 3 data center will need to eat into its redundancy for cooling. Tier 4 facilities should be more robu

  • Coincidence? (Score:2, Funny)

    by Anonymous Coward

    Someone call Al Gore--as he's an expert on both the Internet and Global Warming--he'll know what to do.

  • " Users in Western Australia, NSW, Victoria and South Australia took to Twitter, Facebook and broadband forum Whirlpool to post their frustrations to the country's second largest DSL internet service provider."

    Obviously it wasn't the whole of Australias internet that was affected

    How do they get on in the outback? It must get near 50C there

    One thing about global warming though - when it gets hot enough the ocean will dry up and they should be able to spot MH370 easily

    • Re: (Score:3)

      by mjwx ( 966435 )

      How do they get on in the outback? It must get near 50C there

      The hottest place in Australia is Marble Bar.
      http://en.wikipedia.org/wiki/M... [wikipedia.org]

      It average (maximum) summer temperatures in excess of 41 C. Average yearly temps are around 35 C so it doesn't get much cooler in the winter.

      I'm certain that they would see the odd day above 50 C there.

      • Re: (Score:2)

        by _merlin ( 160982 )

        I've experienced 50'C outside in Finley once. It makes you feel horribly lethargic, just don't want to do anything.

        • Re: (Score:2)

          by mjwx ( 966435 )

          I've experienced 50'C outside in Finley once. It makes you feel horribly lethargic, just don't want to do anything.

          I used to live up north in a Pilbara mining town. I saw 50 C a few times. Not a day you really want to spend outside unless your job was in a giant tin shed.

  • last summer (Score:3)

    by slashmydots ( 2189826 ) on Tuesday January 06, 2015 @01:43AM (#48743379)
    Last summer in Wisconsin, believe it or not, it got around 100F for several days and it knocked out our internet. It wasn't some morons with inadequate server cooling though. Apparently Time Warner equipment runs on 90V lines and our energy company's equipment that drops to 90V was overheating. Unbelievable! Our digital phones were down too.

  • It is a big cover up. (Score:1)

    by Anonymous Coward

    They were really clearing a nest of drop-bears from the server room and had to turn reverse the air-con to drive them out so the servers shut themselves down to prevent overheating.

    • Re: (Score:1)

      by talis9 ( 166451 )

      Every Australian knows, if you take the proper precautions and put Vegemite behind your ears then you don't get attacked by Drop Bears

  • It's not the heat! (Score:3)

    by Vinegar Joe ( 998110 ) on Tuesday January 06, 2015 @02:03AM (#48743451)

    It's the humidity.

    • Re: (Score:3)

      by mjwx ( 966435 )

      It's the humidity.

      And if that doesn't get you we have sharks, snakes, spiders, jelly fish, drop bears and backpacker murderers.

    • Re: (Score:1)

      by Anonymous Coward

      It's a dry heat

  • I used to work for an ISP in very hot Texas, but we planned ahead. We kept all of our servers indoors so they were out of the heat. It's a really good idea. I think everyone should do it.
  • Let me just fix that for you:

    Extreme heat, as compared to the climate in a specific part of Australia, causes one ISP to panic and shut down internet for a small percentage of customers in said specific part of Australia.

    • Re: (Score:1)

      by Anonymous Coward
      It was users all over Australia. There only are 6 states so those examples of where people complained from actually cover the whole place! Everyone I know using iiNet at home, on the other side of the country from Perth (6 hour flight) was affected.

  • Click bait...Nothing to see...Move along people (Score:5, Informative)

    by Anonymous Coward on Tuesday January 06, 2015 @03:15AM (#48743685)

    Sigh... first world problems. Few services were shutdown for precaution as result of A/C failure (primary/backup).

    Hi All,

    Due to heat in Perth we have lost a number of services and precautionary shut others down.

    Customer will notice impact to the following services:

    - iiNet Toolbox and Westnet MyAccount [RESTORED 7:00PM WST]
    - Westnet Email [RESTORED 7:45PM WST]
    - Westnet Hosted Websites [RESTORED 9:00PM WST]
    - iiNet hosted email [RESTORED 7:45PM WST]
    - iiNet/Westnet/Adam/Netspace Webmail [RESTORED 7:00PM WST]
    - Customers may be unable to re-authenticate after disconnecting from the internet [RESTORED 8:00PM WST]

    A number of internal tools are also affected, which will impact our ability to respond to certain customer enquiries.

    Update 6pm WST: Due to issues with staff access, some contact centre queues have been closed. Affected queues will be reopened once the incident has been resolved.

    Update 8pm WST: Most services have been restored, Engineers are continuing to review all services impacted by the incident. Customers that were off-line are recommended to perform a modem power-cycle to get back on-line.

    Thanks,

    Basically few gen Y's screaming that they can't post their sweaty selfies for a few hours.
     

  • At 44'C ambient a lot of air conditioners fail (Score:3, Informative)

    by ZombieEngineer ( 738752 ) on Tuesday January 06, 2015 @03:19AM (#48743695)

    Depending on the refrigerant used it is possible that the condenser temperature (the bit exposed to the outside air) exceeded the critical point of the gas at which point it is impossible to tell the difference between liquid or gas. The trouble is phase change cooling works best (most efficient) the closer to the critical point you can go but not past it.

    The second problem is the condenser pressure would increase with increasing ambient air temperature. In the past this was enough to stall the compressor motors on a hot day.

    My guess is they went for a system with a high efficiency that should work for 99.9% of the time, that last 0.1% is the 8 hours of the year when the temperature is above 42'C (normally for Perth it is normally only an hours before the sea breeze kicks in and drops the temperature by at least 5'C). This time the temperature went up and stayed up for a period of time.

  • Fuck you slashdot you media hungry whore (Score:1)

    by Anonymous Coward

    I remember when slashdot
    - didn't treat their readers like mindless media starved zombies
    - had content targeting intelligent geeks, not the retarded Inbred 7 year old crowd.
    - had stories with more than a catchy but blatantly false headline.

    Fuck you.

  • They should have been using fiber. Specs on a popular GPON chassis says maximum operating ambient air temp is 131f @ 95% relative humidity.

    The problem seems to be routers. A regular router seems to be limited to about 121f and "core" routers are more around 104f.

  • We don't start complaining about the heat until it hits 50C.

  • Sounds like the temps we get here each and every year... Las Vegas Nevada *lives* between mid June and late October with temps over 100F, and frequent transients to over 110F... Guess the Aussies aren't used to such temps...

  • posting to suppress mis-moderation

Slashdot Top Deals

Old programmers never die, they just hit account block limit.

Close