Posts

Cloud Computing and the Single Server

[NB: Check the byline, thanks. /~Rayne]

I’ve been meaning to write about this for a while. Push came to shove with Marcy’s post this past week on Roger Stone and the Russian hack of the DNC’s emails as well as her post on Rick Gates’ status update which intersects with Roger Stone’s case.

First, an abbreviated primer about cloud computing. You’ve likely heard the term before even if you’re not an information technology professional because many of the services you use on the internet rely on cloud computing.

Blogging, for example, wouldn’t have taken off and become popular if it wasn’t for the concept of software and content storage hosted somewhere in a data center. The first blogging application I used required users to download the application and then transfer their blogpost using FTP (file transfer protocol) to a server. What a nuisance. Once platforms like Blogger provided a user application accessible by a browser as well as the blog application and hosting on a remote server, blogging exploded. This is just one example of cloud computing made commonplace.

Email is another example of cloud computing you probably don’t even think about, though some users still do use a local email client application like Microsoft’s proprietary application Outlook or Mozilla’s open source application Thunderbird. Even these client applications at a user’s fingertips rely on files received, sent, managed, and stored by software in a data center.

I won’t get into more technical terms like network attached storage or storage area network or other more challenging topics like virtualization. What the average American needs to know is that a lot of computing they come in contact every day isn’t done on desktop or laptop computers, or even servers located in a small business’s office.

A massive amount of computing and the related storage operates and resides in the cloud — a cutesy name for a remotely located data center.

This is a data center:

Located in Council Bluffs, Iowa, this is one of Google’s many data centers. In this photo you can see racks of servers and all the infrastructure supporting the servers, though some of it isn’t readily visible to the untrained eye.

This is another data center:

This is an Amazon data center, possibly one supporting Amazon Web Services (AWS), one of the biggest cloud service providers. Many of the sites you visit on the internet every day purchase their hosting and other services from AWS. Some companies ‘rent’ hosting space for their email service from AWS.

Here’s a snapshot of a technician working in a Google data center:

Beneath those white tiles making up the ‘floor’ are miles and miles of network cables and wiring for power as well as ventilation systems. More cables, wires, and ventilation run overhead.

Note the red bubble I’ve added to the photo — that’s a single blade-type server inserted into a rack. It’s hard to say how much computing power and storage that one blade might have had on it because that information would have been (and remains) proprietary — made to AWS specifications, which change with technology’s improvements.

These blades are swapped out on a regular maintenance cycle, too, their load shifted to other blades as they are taken down and replaced with a new blade.

Now ask yourself which of these servers in this or some other data center might have hosted John Podesta’s emails, or those of 300 other people linked to the Clinton campaign and the Democratic Party targeted by Russia in the same March 2016 bulk phishing attack?

Not a single one of them — probably many of them.

And the data and applications may not stay in one server, one rack, one site alone. It could be spread all over depending on what’s most efficient and available at any time, and the architecture of failover redundancy.

~ ~ ~
Some enterprises may not rely on software-as-a-service (SaaS), like email, hosted in a massive data center cloud. They might instead operate their own email server farm. Depending on the size of the organization, this can be a server that looks not unlike a desktop computer, or it can be a server farm in a small data center.

(The Fortune 100 company for which I once worked had multiple data centers located globally, as well as smaller server clusters located on site for specialized needs, ex. a cluster collecting real-time telemetry from customers. Their very specific needs as well as the realistic possibility that smaller businesses could be spun off required more flexibility than purchasing hosted services could provide at the time.)

And some enterprises may rely on a mix of cloud-based SaaS and self-maintained and -hosted applications.

In 2016 the DNC used Microsoft Exchange Server software for its email across different servers. Like the much larger Google-hosted Gmail service, users accessed their mail through browsers or client applications on their devices. The diagrams reflecting these two different email systems aren’t very different.

This is a representation of Google’s Gmail:

[source: MakeInJava(.)com]

This is a representation of Microsoft Exchange Server:

Users, through client/browser applications, access their email on a remote server via the internet. Same-same in general terms, except for scale and location.

If you’ve been following along with the Trump-Russia investigation, you know that there’s been considerable whining on the part of the pro-Trump faction about the DNC’s email server. They question why a victim of a hack would not have turned over their server to the FBI for forensic investigation and instead went to a well-known cybersecurity firm, Crowdstrike, to both stop the hack, remove whatever invasive tools had been used, and determine the entity/ies behind the hack.

A number of articles have been written explaining the hacking scenario and laying out a timeline. A couple pieces in particular noted that turning over the server to the FBI would have been disruptive — see Kevin Poulsen in The Daily Beast last July, quoting former FBI cybercrime agent James Harris:

“In most cases you don’t even ask, you just assume you’re going to make forensic copies…For example when the Google breach happened back in 2009, agents were sent out with express instructions that you image what they allow you to image, because they’re the victim, you don’t have a search warrant, and you don’t want to disrupt their business.”

Poulsen also quantified the affected computing equipment as “140 servers, most of them cloud-based” meaning some email and other communications services may have been hosted outside the DNC’s site. It would make sense to use contracted cloud computing based on the ability to serve widespread locations and scale up as the election season crunched on.

But what’s disturbing about the demands for the server — implying the DNC’s email was located on a single computer within DNC’s physical control — is not just ignorance about cloud computing and how it works.

It’s that demands for the DNC to turn over their single server went all the way to the top of the Republican Party when Trump himself complained — from Helsinki, under Putin’s watchful eye — about the DNC’s server:

“You have groups that are wondering why the FBI never took the server. Why didn’t they take the server? Where is the server, I want to know, and what is the server saying?”

And the rest of the right-wing Trumpist ecosphere picked up the refrain and maintains it to this day.

Except none of them are demanding Google turn over the original Gmail servers through which John Podesta was hacked and hundreds of contacts phished.

And none of the demands are expressly about AWS servers used to host some of DNC’s email, communications, and data.

The demands are focused on some indeterminate yet singular server belonging to or used by the DNC.

~ ~ ~
The DNC had to shut down their affected equipment and remove it from their network in order to clean out the intrusion; some of their equipment had to be stripped down to “bare metal,” meaning all software and data on affected systems were removed before they were rebuilt or replaced. 180 desktops and laptops had to be replaced — a measure which in enterprise settings is highly disruptive.

Imagine, too, how sensitive DNC staff were going forward about sharing materials freely within their organization, not knowing whether someone might slip and fall prey to spearphishing. There must have been communications and impromptu retraining about information security after the hack was discovered and the network remediated.

All of this done smack in the middle of the 2016 election season — the most important days of the entire four-year-long election cycle — leading into the Democratic Party’s convention.

(This remediation still wasn’t enough because the Russians remained in the machines into October 2016.)

If the right-wing monkey horde cares only about the DNC’s “the server” and not the Google Gmail servers accessed in March 2016 or the AWS servers accessed April through October 2016, this should tell you their true aim: It’s to disrupt and shut down the DNC again.

The interference with the 2016 election wasn’t just Russian-aided disinformation attacking Hillary Clinton and allies, or Russian hacks stealing emails and other files in order to leak them through Wikileaks.

The interference included forcing the DNC to shut down and/or reroute parts of its operation:

(excerpt, p. 22, DNC lawsuit against Russian Federation, GRU, et al)

And the attack continues unabated, going into the 2020 general election season as long as the right-wing Trumpists continue to demand the DNC turn over the server.

There is no one server. The DNC shouldn’t slow or halt its operations to accommodate opponents’ and suspects’ bad faith.

~ ~ ~
As for Trump’s complaint from Helsinki: he knows diddly-squat about technology. It’s not surprising his comments reflected this.

But he made these comments in Helsinki, after meeting with Putin. Was he repeating part of what he had been told, that Russia didn’t hack the server? Was he not only parroting Putin’s denial but attempting to obstruct justice by interfering in the investigation by insisting the server needed to be physically seized for forensic inspection?

~ ~ ~
With regard to Roger Stone’s claims about Crowdstrike, his complaints aren’t just a means to distract and redirect from his personal exposure. They provide another means to disrupt the DNC’s normal business going forward.

The demands are also a means to verify what exactly the Special Counsel’s Office and Crowdstrike found in order to determine what will be more effective next time.

The interference continues under our noses.

This is an open thread.

Thursday: Alien Occupation

Since I missed a Monday post with a movie clip I think I’ll whip out a golden oldie for today’s post.

This movie — especially this particular scene — still gets to me 37 years after it was first released. The ‘chestburster’ as scene is commonly known is the culmination of a body horror trope in Ridley Scott’s science fiction epic, Alien. The horror arises from knowing something happened to the spacecraft Nostromo’s executive officer Kane when a ‘facehugger’ leapt from a pod in an alien ship, eating through his space helmet, leaving him unresponsive as long as the facehugger remained attached to his face. There is a brief sense of relief once the facehugger detaches and Kane returns to consciousness and normal daily functions. But something isn’t right as the subtle extra scrutiny of the science officer Ash foreshadows at the beginning of this scene.

Director Ridley Scott employed a different variant of body horror in his second contribution to the Alien franchise, this time by way of a xenomorph implanted in her mimicking pregnancy in scientist Shaw. She is sterile, and she knows whatever this is growing inside her must be removed and destroyed or it will kill both her and the remaining crew. The clip shared here and others available in YouTube actually don’t convey the complete body horror — immediately before Shaw enters this AI-operated surgical pod she is thwarted by the pod’s programming for a default male patient. In spite of her mounting panic and growing pain she must flail at the program to enter alternative commands which will remove the thing growing inside her.

I suspect the clips available in YouTube were uploaded by men, or they would understand how integral to Shaw’s body horror is the inability to simply and quickly tell this surgical pod GET THIS FUCKING THING OUT OF ME RIGHT THE FUCK NOW.

I don’t know if any man (by which I mean cis-man) can really understand this horror. Oh sure, men can realistically find themselves host to things like tapeworms and ticks and other creatures which they can have removed. But the horror of frustration, being occupied by something that isn’t right, not normal, shouldn’t continue, putting its host at mortal risk — and not being able to simply demand it should be removed, or expect resources to avoid its implantation and occupation in one’s self? No. Cis-men do not know this terror.

Now imagine the dull background terror of young women in this country who must listen to white straight male legislators demand ridiculous and offensive hurdles before they will consider funding birth control to prevent sexual transmission of Zika, or fund abortions of Zika-infected fetuses which put their mothers at risk of maternal mortality while the fetuses may not be viable or result in deformed infants who’ll live short painful lives. Imagine the horror experienced by 84 pregnant women in Florida alone who’ve tested positive for Zika and are now being monitored, who don’t know the long-term outcomes for themselves or their infants should their fetuses be affected by the virus.

Body horror, daily, due to occupation not only by infectious agents alien to a woman’s body, but occupation by patriarchy.

I expect to get pooh-poohed by men in comments to which I preemptively say fuck off. I’ve had a conversation this week about Zika risks with my 20-something daughter; she turned down an invitation this past week to vacation with friends in Miami. It’s a realistic problem for her should she accidentally get pregnant before/during/immediately following her trip there.

We also talked about one of her college-age friend’s experiences with Guillain–Barré syndrome. It’s taken that young woman nearly three years to recover and resume normal function. She didn’t acquire the syndrome from Zika, but Guillain–Barré’s a risk with Zika infections. There’s too little research yet about the magnitude of the risk — this vacation is not worth the gamble.

But imagine those who live there and can’t take adequate precautions against exposure for economic reasons — imagine the low-level dread. Imagine, too, the employment decisions people are beginning to make should job offers pop up in areas with local Zika transmission.

What’s it going to take to get through to legislators — their own experience of body horror? Movies depicting body horror don’t seem to be enough.

Wheels
Put these two stories together — the next question is, “Who at VW ordered the emissions cheat device from Bosch before 2008?”

Pretty strong incentives for Volkswagen to destroy email evidence. I wonder what Bosch did with their emails?

Self-driving electric cars are incredibly close to full commercialization based on these two stories:

  • Michigan’s state senate bill seeks approval of driverless cars (ReadWrite) — Bill would change state’s code to permit “the motor vehicle to be operated without any control or monitoring by a human operator.” Hope a final version ensures human intervention as necessary by brakes and/or steering wheel. I wonder which manufacturer or association helped write this code revision?
  • California now committed to dramatic changes in greenhouse gas emissions (Los Angeles Times) — State had already been on target to achieve serious reductions in emissions by 2020; the new law enacts an even steeper reduction by 2030 in order to slow climate change effects and improve air quality.

I don’t know if I’m ready to see these on the road in Michigan. Hope the closed test track manufacturers are using here will offer realistic snow/sleet/ice experience; if self-driving cars can’t navigate that, I don’t want to be near them. And if Michigan legislators are ready to sign off on self-driving cars, I hope like hell the NHTSAA is way ahead of them — especially since emissions reductions laws like California’s are banking heavily on self-driving electric cars.

Google-y-do

  • Google’s parent Alphabet-ting on burritos from the sky (Bloomberg) — No. No. NO. Not chocolate, not doughnuts, not wine or beer, but Alphabet subsidiary Project Wing is testing drone delivery of Chipotle burritos to Virginia Tech students? Ugh. This has fail all over it. Watch out anyhow, pizza delivery persons, your jobs could be on the bubble if hot burritos by drone succeed.
  • API company Apigee to join Google’s fold (Fortune) — This is part of a big business model shift at Google. My guess is this acquisition was driven by antitrust suits, slowing Google account growth, and fallout from Oracle’s suit against Google over Java APIs. Application programming interfaces (APIs) are discrete programming subroutines which, in a manner of speaking, act like glue between different programs, allowing programmers to obtain resources from one system for use in a different function without requiring the programmer to have more than passing understanding of the resource. An API producer would allow Google’s other systems to access or be used by non-Google systems.
  • Google to facilitate storage of Drive content at cloud service Box (PC World) — Here’s where an API is necessary: a Google Drive user selects Box instead of Drive for storage, and the API routes the Drive documents to Box instead of Drive. Next: imagine other Google services, like YouTube-created/edited videos or Google Photo-edited images, allowing storage or use by other businesses outside of Google.

Longread: Digitalization and its panopticonic effect on society
Columbia’s Edward Mendelson, Lionel Trilling Professor in Humanities and a contributor at PC Magazine, takes a non-technical look at the effect our ever-on, ever-observing, ever-connected technology has on us.

Catch you later!

DOJ’s Untracked Email Spying

As Wired reports, DOJ blew off the requirement that it tell Congress how many pen registers and trap and trace devices they used for the entire Bush Administration.

[…]the Justice Department was not following the law and had not provided Congress with the material at least for years 2004 to 2008. On the flip side, Congress was not exercising its watchdog role, thus enabling the Justice Department to skirt any oversight whatsoever on an increasingly used surveillance method that does not require court warrants, according to Justice Department documents obtained via the Freedom of Information Act.

But just as interesting as DOJ’s failure to follow the law on disclosing these surveillance tools are two details from the emails Chris Soghoian liberated to make all this clear.

First, note the December 23, 2009 email from Janet Webb (on PDF 4) revealing that DOJ’s agencies weren’t tracking email pen registers (that is, lists of who was emailing each other), and one of them–they speculate DEA–still wasn’t in 2009.

FBI only began keeping computer intercept stats a couple of years ago. The other agency may be DEA.

From which we might assume DEA is engaging in a ton of email tracking they don’t want to tell anyone about?

Wired suggests why they may not be tracking such information.

Another feature of [the Electronic Communication Privacy Act] had once protected Americans’ electronic communications from the government’s prying eyes, but it has become so woefully outdated that it now grants the authorities nearly carte blanche powers to obtain Americans’ e-mail stored in the cloud, such as in Gmail or Hotmail — without a court warrant.

That is, we probably should assume these email numbers are so small–and DEA isn’t tracking them at all–because they’re just taking them, with no court oversight at all.

The other detail to remember about these reports is they include only criminal surveillance, not intelligence surveillance. Russ Feingold staffer Lara Flint makes that clear in her request, and DOJ staffer Mark Agrast makes it clear in his response. They’re getting that information via other means, presumably NSLs or Section 215.

So while they’re hiding a lot of the cloud computer spying they’re doing in the name of criminal investigations, that doesn’t even scratch the surface of the degree to which they’re tracking who emails whom.