FBI Cyber Division’s Enduring Blue Pill Mystery
I’m writing a post on the technical analysis John Durham included in his report purporting to debunk the white papers submitted via Michael Sussmann to, first, the FBI and, then, the CIA. But first I’m going to do something even more tedious: Try to track down FBI’s persistent blue pill problem — or rather, the FBI’s apparent failure to ever analyze one of two thumb drives Sussmann shared with Jim Baker in September 2016, the Blue one.
Last year, before Sussmann’s trial, Durham had FBI’s top technical people review what he claimed were the data Sussmann had shared. He cited those reports in his own report, claiming they debunk the white papers.
Here’s how they are described in footnotes.
- 1635 FBI Cyber Division Cyber Technical Analysis Unit, Technical Analysis Report (April 20, 2022) (hereinafter “FBI Technical Analysis Report”) (SCO _ 094755)
- 1671 FBI Cyber Technical Operations Unit, Trump/Alfa/Spectrum/Yota Observations and Assessment (undated; unpaginated).
Not only doesn’t the YotaPhone report have a date, but it doesn’t have a Bates stamp reflecting that it was shared with Sussmann. I’ll get into why that is interesting in my follow-up post.
Below is a summary of the materials Sussmann provided to both agencies. By description, the Technical Analysis Report only reviews the white paper and the smaller of two sets of text DNS logs included on the Red Thumb Drive. By description the Trump/Alfa/Spectrum/Yota Observations only review the Yota White Paper.
The data FBI’s technical people reviewed appear to be restricted to what is marked in blue.
They did review the actual thumb drives turned over to the CIA, because they found hidden data on one; there’s no indication they reviewed the thumb drives provided to the FBI.
In fact, it’s impossible that they reviewed the data included on the second thumb drive Sussmann shared, the Blue one.
That’s because the FBI analysis claims Sussmann only provided 851 resolutions, which is the 19-page collection of text files included on the Red Thumb Drive, not even the larger set.
Similarly, the FBI experts told us that the collection of passive DNS data used to support the claims made in the white paper was also significantly incomplete. 1657 They explained that, given the documented email transmissions from IP address 66.216.133.29 during the covered period, the representative sampling of passive DNS would have necessarily included a much larger volume and distribution of queries from source IP addresses across the internet. In light of this fact, they stated that the passive DNS data that Joffe and his cyber researchers compiled and that Sussmann passed onto the FBI was significantly incomplete, as it included no A-record (hostname to IP address) resolutions corresponding to the outgoing messages from the IP address. 1658 Without further information from those who compiled the white paper data, 1659 the FBI experts stated that it is impossible to determine whether the absence of additional A record resolutions is due to the visibility afforded by the passive DNS operator, the result of the specific queries that the compiling analyst used to query the dataset, or intentional filtering applied by the analyst after retrieval. 1660
1659 The data used for the white paper came from Joffe’s companies Packet Forensics and Tech Company-I. As noted above, Joffe declined to be interviewed by the Office, as did Tech Company-2 Executive-I. The 851 records of resolutions on the USB drive were an exact match for a file of resolutions sent from University-I Researcher-2 to University-I Researcher- I on July 29, 2016, which was referred to as “[first name of Tech Company-2 Executive-l]’s data.” Id. at 7.
1660 Id. [bold]
There’s no way they would have come to this conclusion if they had seen the Blue Thumb Drive, which had millions of logs on it.
In fact, it appears that the FBI never did review that Blue Thumb Drive when they were investigating the Alfa Bank anomaly.
They didn’t do so, it appears, because the Cyber Division Agents who first reviewed the allegations, Nate Batty and Scott Hellman, misplaced the Blue Thumb Drive for weeks.
That may not have been an accident.
Batty and Hellman’s initial review, which they completed in just over a day, was riddled with errors (as I laid out during the trial). Importantly, they could not have reviewed most of the DNS logs before writing their report, because they claimed, “the presumed suspicious activity began approximately three weeks prior to the stated start [July 28] of the investigation conducted by the researcher.”
Even the smaller set of log files included on the Red Thumb Drive showed the anomaly went back to May. A histograph included in the white paper shows the anomaly accelerating in June.
Had anyone ever reviewed the full dataset, the shoddiness of their initial analysis would have been even more clear.
Here’s how the FBI managed to conduct an investigation on two thumb drives without, it appears, ever looking at the second one.
As the chain of custody submitted at trial shows, Jim Baker accepted the thumb drives, then handed them off to Peter Strzok, who then handed them off to Acting Assistant Director of Cyber Eric Sporre, who at first put the thumb drives in his safe, then handed them over to Nate Batty.
Within hours (these logs are UTC), Batty and Hellman started mocking the white paper but also complaining about the “absurd quantity of data.”
Hellman, at least, admitted at trial that he only knows the basics about DNS.
The next day, Batty told Hellman that their supervisor wanted them to write a “brief summary” of what he calls “the DNC report.” Batty appears to have known of Sussmann from other cases and he was informed that Sussmann was in the chain of custody.
In spite of the clear record showing Batty was informed who provided the thumb drives, in 2019, he told Durham that he and Hellman — whose analysis was so shitty — had considered filing a whistleblower complaint because they weren’t told what the documentary record shows he was clearly informed. And Durham thought that was sufficiently credible to stick in his report.
Before writing an analysis of this report, Batty admitted, they should first “plug the thumb drives” in and look at the files before they wrote a summary.
The documentary evidence shows that these guys formed their initial conclusion about the white paper without ever reviewing the data first.
A day later, Curtis Heide texted from Chicago and asked them to upload the thumb drives, plural, so they could start looking at them.
They only uploaded one, the Red Thumb Drive.
That’s clear because when Kyle Steere documented what they had received on October 4, he described that his report is, “a brief summary of the contents of the USB drive,” singular. The contents match what were on the Red Thumb Drive.
Two hours and 16 minutes later, after uploading the Red Drive, Batty asked if he should send the actual thumb drives to Chicago.
48 minutes later, Batty asked Hellman if he had the Blue Thumb Drive.
The chain of custody shows that Batty didn’t send anything on September 22, when he and Hellman were panicking about the missing Blue Thumb Drive. Instead, he put something in storage on October 6, two weeks later. That he put them in storage makes no sense, because when he wrote an Electronic Communication explaining why he was sending the thumb drives to Chicago on October 11 (by that point, 19 days after saying they would send the thumb drives to Chicago that day), he claimed,
Due to case operational tempo, and the need to assess the data at ECOU-1 prior to referring the matter to the [Chicago] division the evidence was not charged into evidence (at the NVRA) until October 6, 2016.
Not a shred of evidence in the available record supports that claim and a great deal shows it to be false.
But he didn’t send the physical thumb drives until October 12, FedEx instead of internal BuMail.
By October 12, the FBI had decided there was nothing to these allegations.
Somewhere along the way, there was some confusion as to whether there was one or two thumb drives. At the time the case ID was added — the case was opened on September 23 — it seems to have been understood there was just one thumb drive.
Batty does seem to have sent two thumb drives, one Red and one Blue, to Chicago after that 20-day delay, though.
At trial on May 23, Alison Sands dramatically pulled two thumb drives — a Red Thumb Drive and a Blue Thumb Drive — out of the evidence envelope where she put them years earlier.
Q. Ms. Sands, I’m showing you what’s been marked for identification as Government’s Exhibit 1. Do you recognize that?
A. Yes.
Q. What is that?
A. This is the la envelope.
Q. Do you know what this envelope contains?
A. Yes, it contains the thumb drives. So I basically took them out of evidence and put it into this envelope.
[snip]
Q. Now, Ms. Sands, do you recall how many thumb drives there were?
A. Yes, there’s two.
Q. Do you recall if they had any particular colors?
A. One is blue and one is red.
On the stand, Sands also introduced Steere’s memo, the one that documented the contents of the Red Thumb Drive. In doing so, though, she falsely claimed (at least per the transcript) that the memo described both thumb drives.
Q. Do you recognize what Government’s 206 is?
A. Yes.
Q. What is that?
A. It is the EC documenting what information was on the thumb drives that were provided.
She also introduced the items included on the Red Thumb Drive, one after another, into evidence.
Except for the 19-page set of text files used for technical analysis.
When prosecutor Brittain Shaw got to that file in Steere’s memo, she tried to move it into evidence, but both Judge Cooper and Sussmann attorney Michael Bosworth noted it was already in evidence.
MS. SHAW: Could we go back to Government’s Exhibit 206, please? Moving down the list —
BY MS. SHAW:
Q. The second item, what is that?
A. It is data that was provided as alleged evidence of these DNS lookup tables.
Q. After number 2, is that the title that was given to the file or is that something you assigned?
A. I believe that’s something we assigned.
Q. Okay.
MS. SHAW: And if I could have Government’s Exhibit 208, please. If you’d just blow that up a little bit. Thank you.
BY MS. SHAW:
Q. And, Ms. Sands, do you recognize what that is?
A. Yes, these are the DNS lookups that I just described.
MS. SHAW: All right. I would move Government’s Exhibit 208 into evidence.
MR. BOSWORTH: It may be —-
THE COURT: I think it’s probably in.
MS. SHAW: All right.
It was already in.
Almost a week earlier, Scott Hellman introduced what he called “a portion” of the data included with the exhibit. It was the 19-page text file of DNS logs that reviewed in the Technical Analysis included on the Red Thumb Drive. He didn’t describe it as one stand-alone document included on the thumb drive. He seemed to imply this was a selection the FBI had made.
Q. And if I could show just to you on your screen what’s been marked Government Exhibit 208. And Agent Hellman, this is about an 18- or 19-page document. But you just see the first page here. Do you recognize this?
A. It appears to be a portion of the technical data that came along with the narrative.
MR. DeFILIPPIS: All right. Your Honor, the government offers Government Exhibit 208.
MR. BERKOWITZ: No objection.
THE COURT: So moved.
Q. And if we look at that first page there, Agent Hellman, what kind of data is this?
A. It appears to be — as far as I can tell, it looks to be — it’s log data. So it’s a log that shows a date and a time, a domain, and an IP address. And, I mean, that’s — just looking at this log, there’s not too much more from that.
Q. And do you understand this to be at least a part of the DNS data that was contained on the thumb drives that I think you testified about earlier?
All the while, he and DeFilippis referred to this as “a part” of the DNS data and referred to the thumb drives, plural.
And that, it appears, may be all the data anyone at the FBI ever analyzed.
Update: I erroneously said there were texts between Batty and Hellman that may have gotten deleted. I’ve corrected that error.
Update: I added details from the Lync files showing Batty provided a claim that conflicts with all public evidence about why he didn’t check the thumb drives into evidence until after the investigation was substantively done.
Update: I’ve updated the table to show what Sussmann shared. Particularly given FBI’s shoddy record-keeping and Durham’s obfuscation, it’s not clear on which drive GX209 was, nor is it clear whether there was a separate set of CSV DNS logs on the Blue Drive and if so how many logs they included.
Maybe Alice knows… ;D
Since a simple statistical analysis of the DNS records should have only taken a few minutes (for millions of records), it appears that someone was purposefully stonewalling. The other explanation is pure incompetence, but I hate to think that our civil servants making very big bucks are incompentent – I’ll accept malevolence.
IMO that’s firing levels of incompetence or malevolence.
I’d like to point out that this claim (that Durham attributes to the FBI) is absolute nonsense:
That’s literally the opposite of how DNS works. According to the FBI (and undisputed by anyone), none of the “documented email transmissions from IP address 66.216.133.29 during the covered period” came from mail1.trump-email.com. If zero email transmissions came from mail1.trump-emai.com, then zero DNS lookups for mail1.trump-email.com would have resulted from those transmissions.
I’m having a hard time figuring out an “incompetence” explanation for that passage. It looks like a very deliberate smear.
Right.
Isn’t the current argument that CenDyn no longer owned the IP which was still registered under the Trump name, Listrak did, but was sending emails — from whom, it cannot now and claims never to have been able after 30 days — from that IP for someone else?
That’s a dodge in my opinion. They were using a group of servers which shared a set of IP addresses to send emails for various clients. The “emails from that particular IP address” is a nonsensical question in this context. The question to ask whether or not there were emails or other network traffic from those servers in response to the DNS queries. Given that the servers were used for email marketing campaigns, at least the email side of that question would be an easy question to answer. Unfortunately, the “other network traffic” is probably not answerable now.
On a somewhat related note, what is going on with Durham’s list of DNS queries (Govt. Exhibit 0208)? They aren’t ordered by timestamp, name queried, or originating IP address (the three pipe delimited fields displayed) and yet there’s obviously some sort of grouping. Otherwise, there wouldn’t be runs of entries on the same date.
That’s from April Lorenzen, not Durham, tho.
And I should say the fact that that data was effectively just cut-and-paste txt file is one of the reasons why people have scoffed at these claims.
Which is why the potential existence of a Blue Thumb Drive that has the source files on it, one that via magical means has never to this day been reviewed by the FBI, changes the story dramatically.
“……Before writing an analysis of this report, Batty admitted, they should first “plug the thumb drives” in and look at the files before they wrote a summary…..”
Ya think so?
The image of text messages that follows “48 minutes later, Batty asked Hellman if he had the Blue Thumb Drive.” is stunning. That “uh oh” message in particular is damning.
It also seems to sum up the post quite well.
And given how the DOJ folks involved seem to have trouble counting up to two, it is not surprising that they can’t figure out DNS log files, or recognize that the stuff they want to introduce into evidence is already in evidence.
This, to me, is one of the most tantalizing mysteries out there. Simply because it just doesn’t add up from any angle. To this day, all these years later, I have yet to see a credible suggestion as to what, innocent or otherwise, would create this trail of evidence.
And surely by “plug the thumb drives in” they must have meant “mount the forensic images from our server”, right?
Anyone with the faintest knowledge of forensic data analysis would wholly and reasonably expect they meant “mount the forensic images from server”, versus actually “plug in the usb drives”. There’s dozens of reasons why it MUST be done that way.
(Mods: TL;DR, I know, but seemed worth mentioning, edit as desired…)
The ingestion of forensic data on something like a USB drive should _never_ allow for “just mounting the drive” on an arbitrary machine, full-stop. Doing so not only opens that machine to very real risks of destruction, malware/infection, even masks possibly destruction and/or corruption of the USB drive’s contents.
The proper forensically-safe/-secure method of intaking a USB drive involves (at least some variant of) the following steps, and sequence IS important:
1. Physical drive is tested to confirm it is not a “USB Killer” device intended to destroy the receiving machine’s circuitry (yes such exist — search on “usb killer device toms hardware” for further details on USB Killer Devices).
The IC was aware of these devices well prior to 2016, and they are why you NEVER plug in an “unknown” USB drive into any device you value. You can buy these nasty things on Amazon, they’re also not that difficult to make for someone who understands USB electronics.
2. Non-“USB Killer” devices then should be checked (various “security-known” methods — don’t ask, I’m not telling) to see if there are any signs of “access-triggered decrypt-or-self-destruct” mechnisms.
This depends highly on tech level of lab involved, and isn’t done anywhere near as often as it should be done. Failure to do this, combined with access attempts of next step, can lead to loss of device contents, in cases where user could otherwise be compelled to unlock/disable mechanisms present.
3. From “hygenically sandboxed” machine with direct-assigned USB device, forensic tools are used to scan and attempt to gather all data off the device at a “raw-data” access level. This involves various well-known (and a few not-well-known) techniques to identify and map “hidden data” as well as “expected data”, identifies and can confirm formatting, etc.
This is an vastly-oversimplified description of the process, but there are well-vetted tools which do most of the heavy lifting for isolation of execution, and secure extraction of data from a USB device (potentially bypassing “native” USB stack entirely to ensure isolation).
4. Resultant images are uploaded to a COC-secure server, given IDs, and from this point on, ideally only the resultant images are ever accessed unless/until a compelling reason arises to involve the physical devices (which are stored in a COC-secure repository until needed).
The forensic image both ensures a safeguard against subsequent manipulation of data on the drives themselves, as well as being a digital copy that can safely be accessed/replicated given proper tools, and where copies can be easily validated to be exact duplicates of the original.
Sorry for the lengthy post, but too few are aware of all the risks associated with plugging in a random USB device (both to their machine and/or any data inside the USB drive), so seemed worth giving an overview.
Doing so also makes clear just how ridiculously poor the data hygiene applied in this particular situation really was compared to normal “forensic practices” — all of which forms another reasonable basis to seriously question their motives, their analyses, and their basic competency for the work they were doing.
Yes, the sandboxing — there’s no obvious effort to protect the government’s resources because there’s no clear documentation of processing steps for each device which should include a threat analysis on a air-gapped machine.
For the most part I can agree with WilliamOckham above, but it’s the utter absence of infosec which makes it difficult to agree it’s a deliberate smear. The kack-handed operations combined with the lack of information security look more like a convenient versus deliberate smear.
(In re length — your comment was broken into concise chunks with adequate paragraph breaks. It’s a lot easier to read than a run-on block of text. Thanks.)
Make sure you click through. There’s a bit more on what they did. At the very least, though, someone should go back and check on who was using non-standard device readers.
All that said, this would have been in a lab environment. So maybe they have dirty machines?
Maybe that’s why they were going to use “Frank’s computer” to maybe look at the files before writing a report. Poor Frank.
great info and responses.
thanks, all !
Ew,
“Within hours (these logs are UTC), Batty and Hellman started mocking the white paper but also complaining about the “absurd quantity of data.” By Ew. FBI agents Batty and Hellman mocking the white paper while failing to review the derivative data or the Blue pill thumb drive data at all is the epitome of Johnny Durham’s modus operandi. Johnny, ever the sharp pencil concludes Hillary is evil well because the 45th President said so and Johnny is nothing if not a man who can find a smoking gun on a woman. Well in his own mind that is. Two juries and the world at large haven’t bought Johnny’s poor little conclusions because Johnny didn’t begin with the evidence. Because he’s a man with a mission. Well in his own mind he was. 4 years and one FBI agent’s conviction and two acquittals from his conclusive prosecutions will be his legacy. One out of three is 33% success rate. Evan a pass fail no grade law school would call that a failure. Students will learn a lot from Johnny.
Don’t underestimate Durham. From his previous experience investigating torture, it’s clear that he’s a man specializing in finding exactly what he’s supposed to. However, that was much easier when what he was supposed to find was nothing.
It does rather look like a sincere and determined effort … to not find anything.
I can’t help but mention that if men over medicate with the “Blue Pill”, one of the side effects is a “Blue Haze” aka Viagara Vision.!!!
http://www.healthline.com:
It also exerts a minor inhibitory action against PDE6, which is present exclusively in rod and cone photoreceptors. At higher doses, sildenafil causes mild and transient visual symptoms in a minority of patients (mainly blue tinge to vision,
1.DO NOT ASK HOW I KNOW!
2. bmaz. Was Jimmy Hendrix ahead of his short time?
3. Was the FBI suffering from blue on blue bias, thus the blue thumb drive hidden in plane sight! (were they too on the pill)
4. Apologies to Dr.W…but you started this.
5. Have at it Punaise.
LB
Can’t6 speak to most of that, but yes as to Hendrix.
heh. I’ve heard of “seeing red” before, but this could spawn a new idiom.
Blue Drive
That look on my face
Report from my lips
Pretending is all I do
I’m painting the town red
To hide a drive that’s blue
I prey on the crowd
I play with the crowd
But oh, if they only knew
I’m painting the town red
To hide a drive that’s blue
Can I go on carryin’ on?
My damn report is just pretend
Loving it so, wanting it so
Where will it end, how will it end?
That look on my face
Report from my lips
Will history break through
I’m painting the town red
To hide a drive that’s blue
https://www.youtube.com/watch?v=UMEHxtAT8M0
“1935 Richard Himber – I’m Painting The Town Red (To Hide A Heart That’s Blue) (Stuart Allen, vocal)”