Still Rattled: Fallout and Pushback
[NB: Note the byline. Portions of this post may be speculative. / ~Rayne]
The tech industry and technology journalism outlets remain rattled by Bloomberg Businessweek’s The Big Hack article.
Bloomberg Businessweek’s Jordan Robertson and Michael Riley published a second article last Tuesday in which a security expert went on the record about compromised servers with Supermicro motherboards in an unnamed telecommunications provider. Do read the article; the timing of the discovery of the unexpected network communications and the off-spec covert chip fit within the timeline of Apple and Amazon problems with Supermicro motherboards.
The FBI’s and DHS’ responses are also interesting — the first refused to comment and the second offered a tepid endorsement of Apple’s and Amazon’s denials.
The second article hasn’t assuaged industry members or journalists, though, in spite of a source on the record about a third affected entity.
The main criticisms of Bloomberg piece are:
— No affected equipment or firmware has been produced for review;
— Too much of Bloomberg’s sourcing remains anonymous;
— The claims cannot be validated by other journalists, technology companies, persons at Apple and Amazon who have been contacted and interviewed by non-Bloomberg journalists;
— Contacts inside the companies in question continue to deny knowledge if they don’t express confusion about the alleged hack;
— Apple and Amazon have published firm denials, including Apple’s preemptive letter to Congress.
However,
— Something drove both Apple and Amazon to change their relationship with Supermicro within a fairly tight time frame;
— The uniformity of their early denials in which they avoid mentioning hardware and lean toward web application as a point of conflict is odd;
— Neither of these enormous firms nor Supermicro have filed a lawsuit against Bloomberg for libel that the public can see, preventing questioning of Bloomberg’s journalists and sources under subpoena;
— Securities and Exchange Commission doesn’t appear to have been engaged to investigate the claims (although it’s possible the SEC is on this and may simply not have disclosed this publicly);
— None of the other unnamed companies alleged to have received compromised motherboards have uttered a peep to defend (or rebut) Apple or Amazon.
I have not seen in any reporting I’ve read to date — from either Bloomberg Businessweek in The Big Hack or subsequent articles examining the claims or rebutting them — that any journalist, tech industry member or infosecurity community member has asked whether Apple, Amazon, or the other affected companies ordered customized motherboards or servers with customized motherboards made to their company’s specifications. Supermicro has also said nothing about any possible differentiation between motherboards for different companies which would affect the scenario. The silence on this point is confounding.
This piece in Ars Technica captures many of the concerns other tech news outlets have with the Bloomberg reports. Complaints that software — meaning firmware — is easier to hack than adding off-spec hardware miss two key points.
Made-to-order components or assemblies in Just-In-Time lean manufacturing enterprises make it easier to ensure that adulterated products reach their intended mark because each order represents an identified, traceable batch. Adherence to ISO standards in manufacturing processes may even make traceability easier.
We know Supermicro uses lean manufacturing techniques because it’s in job postings online (lousy pay, by the way, which may also say something).
Does Supermicro use the same lean manufacturing approach overseas? Do any of its suppliers also use lean manufacturing?
In contrast, release of firmware (without corresponding adulterated hardware) to a single target is more difficult to control than hardware — the example given is Stuxnet (excerpt here from Ars Technica).
Why wouldn’t a determined nation-state ensure there was a failover, a Plan B method for accessing specific intelligence from a narrow range of sources instead of betting the farm on one method alone? Given the means to deploy both malicious firmware and adulterated hardware, why wouldn’t they try both?
~ | ~ | ~
In spite of tech industry and journalists’ criticisms of Bloomberg’s reporting, these facts remain:
1 — Technology supply chain has been compromised;
2 — U.S. government has known about it (pdf);
3 — U.S. government has not been forthcoming about it or the blacklists it has implemented;
4 — U.S. government has tried to investigate the compromise but with insufficient success;
5 — Some companies are also aware of the compromised supply chain.
We’re no closer to resolving this question: has the compromise of the supply chain remained limited to counterfeiting, or does the compromise now include altered products?
At what point will the tech industry and infosecurity community begin to take supply chain hacks more seriously?
_________
[AN: I still have to analyze both Apple’s letter to Congress and its second response posted on their website along with Amazon’s published response. More to come./~Rayne]
I’d bet that both tech and infosec, both private and governmental, are investigating this very closely, just not publicly. The implications that data was (or had the potential to be) compromised would have significant ramifications for the affected company(s), not the least of which would be stock price.
More critical could be the loss of faith in the company’s reliability by it’s customers. Non-denial denials are risk management practices designed to avoid a crisis of faith. It will be interesting to see if SuperMicro survives.
There should be no question as to why the government isn’t more forthcoming with information. Telling your adversary that you’ve broken their hack causes massive clean-up and limits the ability to investigate to determine what other tools and techniques of which you aren’t aware. Think Enigma in WW II. That the Allies had one and were listening in was one of the most closely guarded secrets of the war.
The more important question is whether congress has figured out that allowing ‘free market’ access to US communication networks to quasi-state owned foreign companies should be outlawed.
I’d bet that both tech and infosec, both private and governmental, are investigating this very closely, just not publicly.
There’s an awful lot of denials flying around. I’m sure somebody is but then somebody is probably lying, too.
Asking a question like:
is like, ah, uh, hmmm, I can’t think of an analogy stupid enough to fit in here.
Congress is a multi-headed critter incapable of complex thought due to the number of non-thinking members of the body. Even the more intelligent and logical members of Congress are typically uneducated regarding high tech. I’m a software guy, which leads to some awareness of hardware and how software and hardware are linked to create computers/phones/etc, etc. Even so, this was complicated enough a story I had to study on the methods described pretty hard to see what had been done and how it worked.
A wicked smart interference with properly operating hardware, though. Smart. I don’t know what to do about needing new hardware now!
Agreed.
A bunch of ‘lead by following’ power mongers whose ability to think rationally is impacted by the glare of the klieg lights and the headlines. I find their belief that somehow they are able to actually legislate on complex issues with an hour or two of background briefings to be ludicrous.
I too am in software, but on the marketing rather than the tech side. Is there a market for entirely US sourced and built hardware? Would the majority of US consumers be willing to give up secure communications in order to save a couple of bucks on the price of their new iWhatever?
Unfortunately, methinks the answer is low price wins all.
But Apple’s prices are already very high. It’s obviously not due to paying workers. They are raking it in. Make something you can continually update, rather than a product based on planned obsolescence, and I’d buy it. As it is, the cost is too high replacing old with new (over a couple of years) and then we have the cost of toxic shit leftover to contaminate the Earth.
I find their belief that somehow they are able to actually legislate on complex issues with an hour or two of background briefings to be ludicrous.
This is where gifted staffers and Congressional Research Service of the Library of Congress are invaluable for members who aren’t technically adept.
Not when you include the full life-cycle costs, and how little is paid to workers who do the work – and what that costs their communities and governments.
It’s easier to shovel “low-cost shit” when those who produce it live half a world a way. That way, buyers only see the upside, not the full cost of their buying habits.
Congress is a multi-headed critter incapable of complex thought due to the number of non-thinking members of the body.
John McCain and Carl Levin would have been all over this; they were investigating the counterfeiting if you’ve been following the links I’ve posted. Ron Wyden gets this, I’m sure, along with Adam Schiff and Eric Swalwell. Congress is not entirely stupid — just big portions like Devin Nunes and Tom Cotton.
Nadler would be interested in this, too.
Yup, but I think his focus given committee assignments will be elsewhere. He’d follow the lead of the relevant in-the-know committee members.
It’s clear that a certain silence about all this from the US government or various computer tech sources is because the US itself does the very things it accuses China of doing. This was spelled out in an article at IEEE Spectrum four years ago. https://spectrum.ieee.org/tech-talk/computing/hardware/us-suspicions-of-chinas-huawei-based-partly-on-nsas-own-spy-tricks
“Other NSA hardware exploits include hardware implants for Dell and Hewlett-Packard servers. Such implants must be installed in person by NSA operatives during an ‘interdiction’ process described by Der Spiegel. That means the NSA intercepts certain shipping deliveries of new computers or related accessories so that it can load spyware or install hardware components that provide back door access for U.S. intelligence agents later on.”
That may have been the implication PRC made in their response, by saying China was also a victim.
The public knows the NSA has made implants but what has been reported so far shows the NSA intercepting equipment — that’s ham-handed as supply chain hacks go. Think about it: shipments from Amazon can be tracked minutely door-to-door and nobody will notice an extra 4 to 12 hours handling on network equipment?
And any nation-state — including key businesses which are tightly associated with government — receiving network equipment made and shipped from the U.S. would have to be stupid not to assume adulteration.
Which is why I am amused at the naivete of tech industry pros and journalists who struggle to believe it could happen to the U.S.
I still don’t see tech industry media moving to the next level of inquiry: if this may have been going on for some time, can any odd circumstances over the last 5-10 years be attributed to “interventions” using adulterated products?
China is a state actor. I know, my grasp of the obvious is excellent. I say that however to contrast it to the US which has no defined state interest anymore, outside of what the most powerful corporations and their owners want. As far as I have heard none of these Supermicro boards seemed to be aimed at US government. Why bother really, with what calls itself government?
As far as I have heard none of these Supermicro boards seemed to be aimed at US government.
Bloomberg’s “Big Hack” said,
We don’t know who the other affected entities are in addition to Apple and Amazon. Given the possibility more than one government contractor was affected, these boards may have been aimed at government intelligence.
ADD: From my first post on “The Big Hack” —
I can’t be certain but the way the Bloomberg piece is worded I think “government contractors” means other additional entities besides Amazon’s AWS and Apple.
Part of my point is that the lines between corporations and government in the US are so blurred as to be essentially non existent over vast swaths the political economy. Especially in the National Security and military realms.
Perhaps the disappearance of distinction between government and corporations in National Security matters owes as much to the nature of digital information and communication.
With China a distinction is better understood in the relationship of the government to the Party. There they are embraced as one and the same. Here the continued insistence that the corporate and government realms are separate is an ever more threadbare cover.
Information technology, both design and production, should have been designated part of critical infrastructure protection program under the 2001 definition within the Patriot Act. So much of that Act was and remains wrong, like this gaping hole. Much of our information technology had its seeds in U.S. government-funded projects — the entire internet spawned from government-funded programs — and our government should have done more to ensure our investment was protected.
As part of critical infrastructure we needed a far more public, open discussion about transferring intellectual property to other countries as a trade-off for improved relations while addressing concerns about the same infrastructure’s security. But our increasingly fascist government — government of, by, and for business — didn’t want to go there because profits über alles.
Barn door open. Horse long gone. Now what to do?
The Fifth Risk by Michael Lewis, just published by Allen Lane details some of the array of US government activities many of us are barely aware of, all of which be vulnerable to these infiltrations and attacks to our great disadvantage. The US government is huge and in many places very competent indeed.
Bloomberg’s second “hardware hack” story certainly does it’s best to damage any credibility that you would concede to the allegations of the first Businessweek report.
Quote:
That’s ridiculous. Every single connector of any serious server-grade hardware is housed in metal. “The usual plastic ones” (connectors) are only to be found on the cheapest consumer devices. Metal housing is needed for shielding and mechanical robustness. If a “spy chip” would consume that much power to need a separate heat sink (meaning: significantly more than even the proper network chip), than it wouldn’t stay hidden for long.
So when Bloomberg (and their named source) obviously perceive metal jacks as an important technical detail of hardware implants, you can conclude that they’re far out of their field of expertise here. And this gives security researcher Joe Fitzpatrick, named source of 1st report, concerns about Bloomberg’s reporting even more weight.
Fitzpatrick states that many hypothetical technical details he described to the journalists about how hardware implants could be made in theory, he then found in the published report as facts about the purported real hardware implant, anonymously sourced.
Quote (from the single named expert source of the first report):
Link: https://risky.biz/RB517_feature/
– kisch
Fundamental problem with your argument and Fitzpatrick’s: it’s not clear whether the suspect motherboards are standard stock with adulteration by addition of a chip alone, or if the boards were made to a client’s specification which may affect what rogue component was used and how/where it was implanted.
Neither Apple nor Amazon have produced their own standard motherboards currently in production environment to demonstrate the lack of any adulteration.
What Bloomberg ran of security expert Appleboum’s feedback also doesn’t indicate if the telecom provider’s board was a stock item or made to order for them with additional off-spec modification. This is the extent of detail Bloomberg reported on the altered boards based on Appleboum’s remarks:
I focused on the lack of info regarding stock versus custom/made-to-order boards in my post; I might as well be talking to myself.
I’ll point back to my original post in which I noted academic Nicholas Weaver thought the rogue chip was possible. He elaborated more fully in a Lawfare blog post:
Emphasis mine.
While Weaver has become a little skeptical since the original report, Appleboum has since said Bloomberg’s report didn’t fully convey his concerns:
Appleboum didn’t have a problem with claim about the physical possibility of a rogue chip; he had a problem with accusing Supermicro alone.
If we didn’t already know about the massive problem of counterfeit electronics going back nearly a decade, I’d be more skeptical. But no, I’m still open to the idea adulterated motherboards may have made it to as many as 30 companies. I suspect I am very much not alone.
If so many tech pros are skeptical about Bloomberg’s claims about Supermicro, why is its stock still languishing around $13/share?
And if Amazon hadn’t bought Elemental in order to control its server supply chain more effectively, would Amazon be this close to winning a $10 billion dollar contract for cloud service from the U.S. government?
Hey Rayne, Tiny equates an autocratic tyrant in a historically misogynist country to Kavanaugh. Nice:
Julie PaceVerified account @jpaceDC
WASHINGTON (AP) — Trump tells AP that Saudi Arabia being blamed for missing journalist is another case of ‘guilty until proven innocent’
False assumptions about what hostile actors or nation states ‘c/wouldn’t do’ are what got us 9-11. Or more accurately, they caused warnings to go unheeded.
But these are the kinds of arguments I see a lot making- e.g. in the Ars Technica linked article, but have seen similar in other places about this including Bloomberg. The ideas that it is too costly, difficult, blah blah supply chain are ones I don’t buy as being decisively falsifying for that reason, and because plenty of knowledgeable tech folks – including Rayne- have shown in various ways how it is not as difficult or impossible as it would seem per some critics. (I am sticking to general terms like ‘it’ because it’s not clear anyone agrees upon whether firmware and or hardware is really ruled in or out, and other authors alternate between the two or three options.)
Also at Ars Technica (above) is the argument that the scheme must be clean, hitting nearly purely the intended targets, with Stuxnet given as an example of the ‘likelihood/cost of detection’ consequences otherwise:
“The articles don’t explain how attackers ensured the altered equipment shipped broadly enough to reach intended targets in a distant country without also going to other unintended companies. ”
But elsewhere it’s uncritically noted that dozens of other unnamed companies would be affected beyond Amz & Apple. Are the former “unintended companies?” Is it possible that that is merely evidence of some target waste?
We also don’t know that this is some rare irreplaceable spyware, i.e. that plenty of other options aren’t waiting in the wings once something like this is detected.
Then of course there is the issue of plausibility, as many have addressed.
I don’t have the technical knowledge, so must rely on other writers/commenters in that regard. But some of the arguments about ‘costs,’ ability, motivation etc. that one can parse are crap.
It’s the assumptions, man… (cue italics)
Excellent point about 9/11 — our intelligence community, law enforcement, military, elected and appointed officials had all been conditioned to a worldview which was changed dramatically because they didn’t leave their worldview’s confines. It was too easy for the application of Shock Doctrine to disrupt and remold their perspectives.
It happened again with 2016 and the hacking of our election — blindsided by assumptions no other country would try to rig our races with information warfare and hacking. American Exceptionalism bit us in the ass and may be biting us again with tech industry’s blinkered thinking about supply chain hacks.
It’s not as if we didn’t have a little warning the way we had a few warnings which were blown off ahead of 9/11. Remember the piece, How to Hack an Election circa March 2016 in which Andrés Sepúlveda said he’d done it in Mexico and he was certain others were doing it in U.S.?
Oddly enough, same Bloomberg journalists Jordan Robertson and Michael Riley worked on that piece as on The Big Hack. What a coincidence.
I’m glad you linked this piece (and hadn’t realized the shared authorship), as I wanted to reread it and cross-check some timelines after reading Molly McKew’s wired article on the ratfuckers’ efforts. A couple of points on the McKew piece: while she does a commendable job of weaving complex strands into the 2016-forward info warfare coalition, she does write as if the nefarious strands passively coalesced around Stone at the RNC. Perhaps that’s due to journalistic restraint, but of course we know from _How to Hack an Election_ and e.g. Marcy’s piece on Gates soliciting Israeli psy-ops that the campaign was, shall we say, effortful all over the place long before the RNC. (McKew also starts off the ne’er-do-well pack (?PAC) at gamergate, though there was that active ‘pedo-hunter’ strand (branched off from anonymous) beforehand, which she doesn’t mention, and which plays an essential role in ongoing info warfare rhetoric.)
And so back to the supply-chain issues: while the overall topic is like watching a tennis match for the non-experts like me, at the very least we have a lot of creative-analytic thinking going on, as throughout your series. Through those exercises, we can hopefully divert worse losses here.
Tariffs be damned: information is always our largest ‘export’ to China.
From How to Hack an Election, so “last year” means 2015:
I also meant to mention that PBS’ Great Conversations has an hour-long interview with Malcolm Nance, a relatively early warner of ~’Russia is trying to info warfare pwn us.’ (I listened to it falling asleep, and something he said about March (or April) 2016 made me go, ~’Oh that makes sense now.’ Lost it to the REM but maybe it’ll get spit back out.)
Thanks Jeffrey Kaye, glad to see an IEEE member chiming in. I’m a life member, meaning I’m old enough to remember that we were talking about supply chain hardware security in the ’80s and ’90s. If you send the masks (for a board or a chip) and all they do is “print” them, no risk. As soon as they have the hardware design tools (purchased) and the knowledge (send students to the USA), the game is over. No security, period. And folks, we’re not talking about consumer equipment here, they’re not doctoring your iToy.
So what I’m saying is that as of about 1990, NOBODY interested in security should have been manufacturing hardware offshore. That includes everybody buying servers. Or Cisco routers, except the home variety.
As a tech type, I never understood the economics of manufacturing electronics (NOT consumer stuff) offshore. Cost of plant, zillions; percentage of labor, zip.
Thanks Rayne, you’re doing a great job.
Thanks, Anne. WRT economics, it wasn’t just labor costs. It’s the other externalities like environmental and safety regulations China did not have in place when electronics and other manufacturing were offshored.
I’ve had a visitor from Asia a few times during the last decade who remarked each visit how much they loved it here because our skies were so blue. Felt terrible knowing the reason why. Also shocking to hear from an engineer how women carried concrete in basket-like hods on their heads to pour an ag company’s factory floor in India, risking injury for pennies a day — happened within the last 15 years. It wasn’t just the labor cost but the expenses related to negotiating a flexible short-term workforce along with no safety regulations.
One thing I need to spend more time on researching is thin-film technology for manufacturing electronic substrates. Supermicro uses this (it appears in the excerpt of the job posting in my post) as do other manufacturers. I wonder what new opportunities for adulteration this technology poses over previous substrate manufacturing processes. So much food for thought.
To Anne’s point: now that we have made the mistake of offshoring the manufacturing and giving away our IP (something that always worried me, too) we need to spend whatever it takes to rebuild that capability in the States. I am amazed at all the craven protests that we “can’t afford to do this” or, that it’s “too expensive.” What kind of national security do we really have, if, for example, our shiny warplanes might be controlled by hacked circuit boards? Maybe that’s a reason why China has been so confidently, and brazenly, moving into the South China Sea, for example? Because they have real reason to believe that our military truly IS the “paper tiger” Mao said it was?
We have become a less developed country than we used to be, before we offshored our most advanced technologies. And when less developed countries want to catch up to the advanced countries, they make expensive investments, and even (gasp!) sacrifice a little consumption here and there. Maybe it’s time to repeal Trump’s tax cuts for the rich and use some of that money to rebuild a tech infrastructure that will make products able to, at a minimum, service the military, and critical industries such as communications, banking & securities trading, transportation, energy, and utilities. Mandating that all critical economic sectors use electronics made with secure components would guarantee a market for such electronics and provide manufacturers with economies of scale.