Rattled: China’s Hardware Hack – SMCI’s Response
[NB: Note the byline. Portions of my content are speculative. / ~Rayne]
The following analysis includes a copy of an initial response Bloomberg Businessweek received from Super Micro Computer in response to its story, The Big Hack. In tandem with the Bloomberg story this was published on October 4 at this link. Super Micro Computer’s response is offset in blockquote format. No signer was indicated in the published response. Additional responses to Bloomberg’s story will be posted separately.
__________
Supermicro
While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard.[1] We are not aware of any customer dropping Supermicro as a supplier for this type of issue.[2]
[1] (a) “we are not aware” “nor have we been contacted” — who is we?
(b) “nor have we been contacted by any government agency” — has Supermicro been contacted by customers or their auditors or their security teams, contract or not, about security problems?
[2] Were one or more of Supermicro’s customers dropped by their customers because of security concerns including problems with firmware? Are any of the customers or customers of customers U.S. government entities?
Every major corporation in today’s security climate is constantly responding to threats and evolving their security posture. As part of that effort we are in regular contact with a variety of vendors, industry partners and government agencies sharing information on threats, best practices and new tools. This is standard practice in the industry today. However, we have not been in contact with any government agency regarding the issues you raised.[3]
[3] Has Supermicro been in contact with any government agency regarding any security issues including firmware updates?
Furthermore, Supermicro doesn’t design or manufacture networking chips or the associated firmware and we, as well as other leading server/storage companies, procure them from the same leading networking companies.[4]
[4] Interesting pointer about networking chips. What other motherboard content does Supermicro not design or manufacture, procuring from other companies? What procured motherboard components have firmware associated with them?
I have to say of the three corporations offering rebuttals at the time Bloomberg went to print, this one was the tightest but mostly because it was three paragraphs saying the least amount possible. It also leans forward rather than backward in perspective.
I think it was just a PR stunt on the day Pence gave his China strategy speech. I could be proven wrong of course. I’m sure there is a there, there, somewhere.
The Register has put up a piece describing the NatSec community responses from the US and the UK. As they note the denials fall short of “It did not happen” and actually parse out to “We believe the companies when they say it.”
https://www.theregister.co.uk/2018/10/08/super_micro_us_uk_intelligence/
Yep. Kind of like GOP members of Congress saying they believe assault survivors…
Rapier, I suspect you are correct.
If there were a large number of tampered servers in service, I think it would have been very likely that at least some of the more security conscious users would have detected questionable traffic and identified the source at an early stage.
The claimed implementation seems a bit odd – the additional components in question would have likely introduce delays and glitches that would noticeably degrade the networking performance and hiding them in the circuit board would implicate the manufacturer (not a good thing for a commercial business). If one really wants to go to the trouble of doing this sort of thing, one could clone the network interface chips and add the additional functionality desired or possibly hide malware in the system firmware. Such approaches would have the advantage that, it discovered, it would be much less clear as to who was responsible – there would be lots of potential culprits in the supply chain.
No white paper or security researcher has demonstrated such an implant in public.
Every party has publicly denied it, by law publicly traded companies cannot lie to public share holders and shareholders are allowed to sue. If they had a NSA/CIA gag order they would simply refuse to make public statements a gag order cannot force you to lie nor does it allow you to violate laws by lying
30 major companies, with hundreds to thousands of servers each which means a lot of grunts know about this. We live in the age of the internet these grunts could spill the truth anonymously over the internet. Yes executives can keep tight lips, but its thousands of grunts who are inspecting motherboards for implants, throwing out thousands of motherboards etc…
Adding a chip (as described in the original Bloomberg story) requires modifying the circuit board to accept it, so that’s another factory and a big design challenge, as every trace on the multiple layers of the circuit board is designed to a specific length. It must also be routed so it doesn’t pick up interference from adjacent traces, leading to failures.
Next the modified circuit board and the spy chip have to be integrated into the assembly line. First comes a custom solder template, which is a sheet with tiny holes to let the solder be ‘printed’ in tiny dots onto the circuit board in a process like screen printing a t-shirt. These solder dots also act as a temporary glue to hold the components in place until heated enough to melt and make a permanent bond.
Then the spy chip would be placed by a robot that places all the other tiny components on the board. The components come on reels of hundreds to thousands of parts so the robot can quickly place them in the specific location needed. That requires getting the reel of spy chips to the factory, changing the robot’s programming to accommodate the new chip and changes to the location of other chips, and matching up the new circuit board, new solder mask, new chip reel, new placement instructions, and new testing procedures to verify each stage of assembly.
Now the modified and populated circuit boards will be shipped to yet another factory to assemble into servers destined for a specific customer. It can’t be a random delivery to Apple or Amazon, so it must match and be substituted for an existing order from one of these companies. If Apple orders 5000 servers, then 5000 counterfeit motherboards will have to be ready to deliver, just the same as if a standard part is ordered from Super Micro Computer, Inc. If Apple has to wait an extra 30 days while the boards are made and assembled, Apple will ask Super Micro about the delay and the conspiracy will be revealed
Thanks for the feedback. I suggest reading the researchers I linked.
It’s also quite easy to make all the motherboards for a particular company to the same spec if they order in large enough batches — it’d be business as usual.
Doesn’t it strike you as odd that only the two largest companies in the world, ones big enough to create their own economies and with cash to burn, are the ones who’ve piped up and not the other ~28 affected companies?
Doesn’t it strike you as odd the second publicly-known response by Apple is to write a letter to Congress instead of suing Bloomberg for defamation?
拜拜
Look up “UEFI Rootkit”. Systems that have been set up (or designed) poorly by OEMs are vulnerable right off the assembly line.
While you are right that adding a new chip might be difficult, modifying an existing chip would not. Yes, there would be grunts involved. And those grunts may be PRC Army grunts who modify chips. … or specifications. While the NSA hasn’t dropped to physical coercion (that we know of), the PRC might not have such qualms, further reducing the number of people who know enough beans to spill them.
And they need not inflict this vulnerability upon just specific customers. In fact, it is much easier all around if the entire product line is affected.
Regular reader. Rare commenter. But, I have some pretty deep expertise in this area. I have run multiple tech-hardware companies including circuit board manufacture and assembly firms. The Bloomberg article was in my feed the day it hit. I read it. My initial thought was… damn. Well played China. Here is why I had that thought. The spy-chip shown was not “added” it was supposed to be there, but replaced with an “evil twin”. To insert, or add a chip would be…beyond maximal order of difficulty. China to do that would have pulled off compromising all entities in the entire design-source-assemble-test supply chain. That chain likely involved 10+ inter-related vendors and possible dozens.
What they did instead was redesign a pretty standard and wholly expected signal conditioning component, which every single vendor, designer, tester and end-user would have signed off on, designed and expected to be there.
The PRC RA would have had to do an extremely sophisticated, very compact, but wholly doable design of a small-mem code injector integrated into the signal conditioner. The rolls of components would have shown up. They would have visually appeared correct. They could be set up to test correctly at test bed stage, check. They get inserted in the place designed per the design all parties agreed to, Check. They would have to properly pass signal conditioning test on sub-assembly and final assembly, Check. IT IS HIGHLY UNLIKELY they would have been tested to find out if they were doing MORE than just signal conditioning. Testing doesn’t look for that. It looks for a positive response it is doing the right thing, it does not look to see if it is doing additional and unexpected things.
This also makes sense in light that some of the discovery happened over oddities discovered after a firmware update. The inserted mini-mem conditioner would have been designed with a pretty narrow range of signal capacity based on the original signal architecture of the unit. Any variation in firmware would NOT have known to update the PRC mini-Mem, and the mini-Mem would be unlikely to operate transparently in the signal stream after the update. For those who may be super-non-technical: If the whole board was a human choir, all with firmware that had them all singing a D-Flat. And new firmware was installed on all the KNOWN singers on the board, telling them to sing a D-Minor, the PRC’s mini-Mem would not have gotten the update. It would have continued singing a D-Flat and been easily detected.
Spoonamore
I’m curious what you mean by a “signal conditioning component”? Some of the reports referred to “couplers”, though, in my experience, such devices would tend to be radio frequency components or maybe simple connectors (though this doesn’t seem to be the case). I can think of things such as termination resistors and common mode chokes, but can’t see how one could implement usable intelligence in such components. Any intelligent component would need power supply and ground connections and input and output lines – it would take significant board modifications to embed such a device and some really good engineering to avoid an obvious performance degradation.
I don’t doubt that there are people in a number of countries looking at ways of embedding malware and that some of these have been used. It is just that the Bloomberg report raises more questions than it answers.
I have not examined the actual piece of hardware. So, I am only reacting the article. I see a lot of tech-reporting that I just don’t believe. This one seemed near-impossible until I read it. I assume you have seen the pictures online provided by Bloomberg. The hardware insert (physically) looks like a common signal conditioning coupler. Look at the TDK – HHM22152A2. But cut open, it has packed both a signal conditioner and something more. I don’t know how they did the networking to-from the device. I wish I did know. But from what I see, it would be unlikely to require significant modification to the board. Or any. As to performance degradation, clearly the board performed within tolerance when first built. If it were me, the “Added Functions” on the compromised component would only come on after 100 hours+ of operation, and only for the shortest possible periods. It would just be inert, doing the proper signal conditioning during testing and ring-out phases. Then come on briefly, do it’s alternative function, and return to dormant. I would assume the the PRC designers would have worked hard to make sure it would not be detectable pre-operational install. In the article, it notes the discovery is reported to have stemmed from unusual network communications and problems arising after a customer driven firmware update, but I also wonder if those two things in the article are from two sources looking at two pieces of the same event. The customer did a firmware update, testing the update found the unexpected signal. I am sure follow on articles will be out as more customers uncover compromised units.