How Can NSA Protect Our Power Grid from Cyberattack When It Can’t Keep Its Own Power On?

/34 Comments/in , /by

In the United States, it is usually a safe bet to attribute massive government fuck-ups to the bloated contractors we’ve outsourced our projects to.

And the electrical problems plaguing NSA’s new UT data center — described as lightening in a box that has caused $100,000 of damage each of the 10 times it has happened — do seem to stem from poorly supervised contractors.

The Army Corps of Engineers is overseeing the data center’s construction. Chief of Construction Operations, Norbert Suter said, “the cause of the electrical issues was identified by the team, and is currently being corrected by the contractor.” He said the Corps would ensure the center is “completely reliable” before handing it over to the NSA.

But another government assessment concluded the contractor’s proposed solutions fall short and the causes of eight of the failures haven’t been conclusively determined. “We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents,” said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.

[snip]

It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

[snip]

Contractors have started installing devices that insulate the power system from a failure and would reduce damage to the electrical machinery. But the fix wouldn’t prevent the failures, according to project documents and current and former officials.

Now, don’t pee your pants laughing.

But I did have two thoughts as I read this.

First, this extended confusion sounds similar to that which Iranian nuclear scientists experienced as they tried to figure out why their centrifuges kept blowing up, thanks to StuxNet. While I think the chances some kind of hack caused this are small (but not zero), I do find it ironic that we cause ourselves the same kind of havoc we cause our worst enemies.

And consider the mission!

Back in February, Keith Alexander warned of the possibility of cyberattacks on our grid (which, anonymous sources made clear, could probably only be launched by China or Russia, but that didn’t stop Alexander from suggesting Anonymous might launch such attacks). The NSA needs more authority to protect against attacks that might bring down our power sources, the head of the NSA suggested.

But the entity that proposes to wield that authority, it seems, can’t even build a brand spanking new electrical system immune from some kind of failure.

34 replies
  1. greengiant says:

    It takes only one wiring mistake to create a shrapnel explosion. Miss wiring the phases from the emergency power supply for one and then “testing” it.
    The Army corp of Engineers and their contractors have probably discovered a few other interesting methods as well.
    As for blaming Iranians, who would have put such stuff under software control in the first place?

  2. orionATL says:

    in a building that special who knows.

    in ignorance, i’d bet on a serious, systemic grounding problem, perhaps associated with rebar in concrete. large-scale, high voltage wiring is both dangerous and tricky. perhaps particularly so in a building that may be laced with copper “spook” wiring to prevent the martians from learning there is unintelligent life on earth.

    but of course the contractor knew that, didn’t it?

  4. William Ockham says:

    The idea that cyberattacks on the grid could only come from China or Russia is ludicrous. Any competent developer with the right incentives could do it. Bringing down a major portion of the grid is easier than what I do on a day-to-day basis.

  6. orionATL says:

    @William Ockham:

    particularly if it involves those little fairly simple industrial computers – i forget how they’re denominated – that rayne has written about here which control so much industrial activity here (as with, also, the iranian centrifuge computers).

  11. YY says:

    For a country so paranoid about the compromising of the grid, there is absolutely no excuse to have a grid vulnerable by having IP addresses for operational hardware (I’m assuming that’s the case if they are paranoid).
    The grid by definition is a well wired network that can operate independent of existing data/communications infrastructure. How difficult can it be to have separate network for communications/control of the grid by just modulating the data on the grid itself? Forget about IP addresses, just use proprietary formats.

  14. thatvisionthing says:

    The Army Corps of Engineers is…

    Ha! I got this far and burst out laughing.

    Because Katrina, the result of their engineering. And because

    http://www.correntewire.com/harry_shearer_interviews_maria_garzino_on_hurricane_isaac_e_mail_foias_uh_oh

    August 2012. New Orleans. Hurricane Isaac. First real test of the $14 billion rebuilt flood protection system since Katrina. You have reporters from the New York Times and the Times-Picayune embedded in your control center, and you’re getting reports of failures in your pumps, your gates, your water level gauges, and your electrical system. Water’s rising and the public is watching. What do you do?

    Answer: Turn the data off and tell the reporters everything’s working fine.

    You are the Army Corps of Engineers.

    Okay, now I’ll continue reading

  15. thatvisionthing says:

    reading…

    But another government assessment concluded the contractor’s proposed solutions fall short and the causes of eight of the failures haven’t been conclusively determined. “We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents,” said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.

    cf the hydraulic pumps the Army Corps of Engineers installed in New Orleans:

    Maria Garzino, ACE whistleblower: …to simplify it, the moment you turn the hydraulic pumps on, they essentially start destroying themselves.

    ok, reading on

  17. thatvisionthing says:

    reading…

    It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.

    hmmm…is there Congressional oversight?

    http://www.wwltv.com/news/eyewitness/Corps-lacks-money-to-finish-work-on-3-NO-pump-stations-SWB-plant-172180871.html

    NEW ORLEANS – The Corps of Engineers acknowledged to 4 Investigates Monday that it will not have enough money to finish strengthening three New Orleans pump stations and the Sewerage and Water Board’s power plant so they can withstand Category 3 winds.

    Congress and former President George W. Bush gave the Corps more than $200 million after Hurricane Katrina specifically to harden New Orleans pump stations and the old Carrollton Power Plant that runs the sewer and water system.

    But in a closed-door meeting last week, the Corps told the three contractors handling the work that there isn’t enough money to retrofit the walls and roofs to withstand a Category 3′s 156-mile-per-hour winds.

    Sen. David Vitter, R-La., said he found out about it from Channel 4, and he is not happy.

    “Congress authorized this structure work, including this hardening work, and for the Corps to mount these cost overruns and then simply cancel the work, not even notify Congress what it will take to finish the work properly, I think, is totally irresponsible,” Vitter said.

    Vitter, Sen. Mary Landrieu, D-La., and Rep. Cedric Richmond, D-New Orleans, just held a meeting last week to call for more funding for the Corps. But they are frustrated by the Corps’ constant problems with cost overruns.

    Read more at http://www.nakedcapitalism.com/2012/11/harry-shearer-preventing-another-sandy-the-lessons-new-orleans-can-teach-new-jersey.html

  18. thatvisionthing says:

    @orionATL:

    After the attacks of September 11, 2001, the government’s authority to collect, keep, and share information about Americans with little or no basis to suspect wrongdoing dramatically expanded.

    Except the govt approached the telecoms in February 2001 to do warrantless wiretapping, and the only one who said no was Joseph Nacchio at Qwest. So the beginning of the article (I just started reading) already implies “because terror,” because 9/11, which cannot be true.

    http://www.emptywheel.net/2013/10/04/upstream-us-person-collection-eo-12333-andor-fisa/#comment-637168

  19. thatvisionthing says:

    reading on…

    Introduction

    The attacks of September 11, 2001, and the intelligence failures preceding them, sparked a call for greater government access to information.

    Oh, fuck. This is stupid. Because “intelligence” backed by NSA compartmented walnut-shell paranoia is stupid. Bradley Manning was the one guy in his intelligence post in Iraq who surfed the web and tried to see a big picture, and everybody else in his unit with paranoid tunnel vision came to him to find out what was going on in the world, iirc from the trial.

    I guess I should say mho. But the trouble with the paranoid IC is that nobody says mho, they have to say “I speak for all, and this is it.” At least by the time they report it anonymously and seriously to us. And if you do something that fucks their spin…

    http://www.correntewire.com/doj_to_press_if_you_preempt_my_ability_to_spin_out_a_story_the_way_i_want_to_i_m_going_to_ruin_your

    DOJ to press: “If you preempt my ability to spin out a story the way I want to, I’m going to ruin your source base” – Marcy Wheeler on Scott Horton Show

    Oh, fuck.

  20. omphaloscepsis says:

    A WSJ blog that isn’t behind a paywall:

    http://blogs.wsj.com/washwire/2013/10/08/nsa-data-center-failures-follow-history-of-electrical-troubles/

    “the NSA has a history of poorly managed electrical projects.

    In 2006, the agency found itself running out of electricity at its Fort Meade, Md., headquarters, was unable to plug in new supercomputers and resorted to shutting down “non-essential” uses of power. That problem had been understood for at least a decade but hadn’t been prioritized high enough to fix.

    The problems flared again in the summer of 2007, when NSA had to resort to rolling brownouts and delayed the deployment of data-processing equipment to keep its systems running.

    At the Utah Data Center, some engineers have suggested that the best way to fix the electrical failures would be to replace the problematic parts of the system, but the idea gained little traction when project officials realized it would cost hundreds of millions of dollars, a project official said.”

    http://www.forbes.com/sites/kashmirhill/2013/10/07/the-nsas-hugely-expensive-utah-data-center-has-major-electrical-problems-and-basically-isnt-working/

    “The problem, and we all know it, is that they put the appliances too close together,” a person familar with the database construction told FORBES, describing the arcs as creating “kill zones.” “They used wiring that’s not adequate to the task. We all talked about the fact that it wasn’t going to work.”

    More stories from local media:

    http://www.sltrib.com/sltrib/news/56973004-78/center-utah-data-corps.html.csp

    “The Utah Data Center was scheduled to be operational last month or this month — the NSA would not specify — and employ about 200 people.

    The Salt Lake Tribune has previously reported that the 1 million-square-foot data center will use 65 megawatts of electricity (a single megawatt is enough to run appliances in 500 to 750 homes). Rocky Mountain Power’s largest Utah customer is Kennecott Utah Copper, which uses about 200 megawatts.”

    http://www.deseretnews.com/article/865587919/Utah-Data-Center-experienced-electrical-failures-NSA-confirms.html

    http://www.ksl.com/?sid=27160192&nid=960&title=electrical-failures-confirmed-at-nsa-data-center

  21. thatvisionthing says:

    @thatvisionthing:

    https://twitter.com/kgosztola/status/354322507857268736

    Kevin Gosztola ‏@kgosztola

    Sgt. David Sadtler testifies staff in Brigade would come to #Manning “if they needed to know what was going on in the world”

    […]

    https://twitter.com/emptywheel/statuses/354326833216495617

    emptywheel ‏@emptywheel

    Really interesting testimony in Manning trial per @carwinb. Manning was the only one in INTELLIGENCE UNIT that kept up w/current events.

    – – –

    soonergrunt ‏@soonergrunt 8 Jul

    @emptywheel @carwinb Not remotely surprising to anyone who knows three or more intel types. Tend to be VERY focused on what they are doing.

    – – –

    emptywheel ‏@emptywheel 8 Jul

    @soonergrunt Sure. But it also explains why our IC misses large forests while they’re obsessed w/classified trees.

    – – –

    Grunge Junkie ‏@GrungeJunkie385 8 Jul

    @emptywheel @carwinb That is truly appalling. The drone program now includes intelligence agents too apparently – Never questioning anything

    – – –

    Magnum Innominadum ‏@LastBoneStands 8 Jul

    @emptywheel @carwinb @usdayofrage Hmmm, something seems weird with the title of that department.

    – – –

    etc … etc … etc …

  25. orionATL says:

    @omphaloscepsis:

    thanks for the informative cites.

    from the shape of the data center in the forbes article i conclude that the arc fault problem results from nsa/architects having accidentally built a cyclotron.

  27. thatvisionthing says:

    Back in February, Keith Alexander warned of the possibility of cyberattacks on our grid (which, anonymous sources made clear, could probably only be launched by China or Russia, but that didn’t stop Alexander from suggesting Anonymous might launch such attacks).

    He’s leaving out plain old American constitutional democracy and the people of the state of Utah.

    Most intriguing idea I’ve heard yet — using state’s rights to stop the NSA. Utah could turn off the NSA’s water spigot.

    http://offnow.org/2013/10/07/nullifynsa-now-open-business/

    Because of the vast nature of the Orwellian snooping machine, it takes a tremendous amount of resources to maintain it. Aside from destroying our environment by depleting our precious natural resources, this also leaves Big Brother vulnerable to decentralized political resistance.

    The feds cannot force state and local governments to do their bidding. This notion is so uncontroversial that the Supreme Court has even agreed multiple times. This gives us an opportunity to hit Big Brother through legislation at the state and local levels. By stopping local and state governments from aiding and abetting the NSA’s unconstitutional behavior, we can make it impossible – or at the very least far more difficult – for them to spy on us.

    In the case of the Utah data center, it is bigger than several Pentagons. Thus, it takes an astounding 1.7 million gallons of water per day to maintain the facility. Most of that water comes from a political subdivision of the state of Utah. That means we can cut the NSA off. Many other NSA centers have similar agreements with local and state governments. They can be cut off as well.

    I got that from listening to Michael Boldin interview on The Scott Horton Show (the radio one, not The Other Scott Horton), who cited California Gov. Jerry Brown signing AB351 into law to refuse California’s participation in NDAA.

    http://scotthorton.org/2013/10/04/10413-michael-boldin/

    http://tenthamendmentcenter.com/2013/10/04/coalition-building-for-the-win/#.UlRdlWYnaRJ

    Earlier this week, California Governor Jerry Brown signed AB351 into law – the California Liberty Preservation Act. It makes a new state policy to refuse to help the federal government implement indefinite detention under any federal act which “purports to authorize” such.

    My governor! My left wing and right wing coalition! We can fly!

  28. lysias says:

    Let’s hope it never works. Maybe some idealistic techies working on the project have thrown a spanner (or several spanners) in the works?

  29. thatvisionthing says:

    @thatvisionthing:

    Ad at the beginning of the program:

    sound of water running
    Narrator: Fact. The new NSA data center in Utah requires 1.7 million gallons of water every single day to operate. Billions of Fourth Amendment violations need massive computers and the water to cool them. That water is being supplied by the state of Utah. Fact: There is absolutely nothing in the Constitution which requires your state to help the feds violate your rights. Our message to Utah: Turn – it – off. No water equals no NSA data center. Visit offnow.org.

  30. thatvisionthing says:

    @lysias: Or Mother Nature. All that drilling and fracking in Utah — remember what Tim DeChristopher was trying to disrupt, in Utah? What could possibly go wrong with the computer center’s critical water supply? No techies or citizen votes required. Think Fukushima. One bad day.

  31. thatvisionthing says:

    @thatvisionthing: Another day, another window – craazyman on Yves Smith:

    Think about it. One thin blonde alone in a New York apartment with two cats, a telephone and internet connection did more to ferret out financial fraud and mayhem for five straight years than the U.S. Department of Justice, Securities and Exchange Commission, Office of the Comptroller of the Currency and Federal Reserve combined. And youze guys in Washington are the ones in charge of Ahmurrica?

    Read more at http://www.nakedcapitalism.com/2013/10/craazyman-im-thinking-at-least-5-million.html

  32. thatvisionthing says:

    @thatvisionthing:

    Michael Boldin: It passed 71-1 in the assembly, 37-0 in the state senate. I actually thought Jerry Brown was going to veto it, because it was taking a stance against the federal government. What it does is it creates a state policy that the state of California will not put any resources or participate in any way in the implementation not only of indefinite detention under NDAA, but as the wording of the bill says, “any federal law which purports to authorize indefinite detention.” So it’s future proof. And it’s a very strong first step by saying, “Hey, we’re going to say no to this, rendering that law null and void.” And we’ve seen in the past, people like Rosa Parks have proven that when you say no you can create a movement and change the world.

  33. thatvisionthing says:

    @thatvisionthing: What could possibly go wrong… Just occurred to me, has anyone checked to see if this thing that takes 1.7 million gallons of water a day to operate might be… nuclear powered?

    Slap me.

Comments are closed.