Posts

The Other Servers and Laptops FBI Never Investigated: VR Systems and North Carolina Polling Books

/40 Comments/in /by

Ron Wyden had a lot to say in his minority views to the SSCI Report on election security released yesterday, mostly arguing that there need to be national standards and assistance and that no one can make any conclusions about the effects of Russia’s efforts in 2016 because no one collected the data to make such conclusions.

But there’s one line in his section raising questions about the 2016 conclusions I find particularly interesting, pertaining to VR Systems (which he doesn’t name).

Assessments about Russian attacks on the administration of elections are also complicated by newly public information about the infiltration of an election technology company.

Since the Mueller Report came out, Wyden has been trying to chase down this reference in the report to the VR Systems hack.

Unit 74455 also sent spear-phishing emails to public officials involved in election administration and personnel a~ involved in voting technology. In August 2016, GRU officers targeted employees of [redacted; VR Systems], a voting technology company that developed software used by numerous U.S. counties to manage voter rolls, and installed malware on the company network.

In May, he sent a letter to VR Systems President Mindy Perkins, asking how the company could claim, in March 2018, that it had not experienced a security breach when the report said it had been infected with malware in August 2016. In response, the company told Wyden (according to a letter he and Amy Klobuchar sent FBI Director Chris Wray) that they had alerted the FBI that they found suspicious IPs in their logs in real time, but that FBI had never explained the significance of that.

In a May 16, 2019, letter to Senator Wyden, VR Systems described how it participated in an August 2016 conference call with law enforcement. Participants in that call were apparently asked by the FBI to “be on the lookout for certain suspicious IP addresses.” According to VR Systems, the company examined its website logs, “found that several of the IP addresses had, in fact, visited our website” and as a result, the company “notified the FBI as we had been directed to do.” VR Systems indicates they did not know that these IP addresses were part of a larger pattern until 2017, which suggests the FBI may not have followed up with VR Systems in 2016 about the nature of the threat they faced.

The implication from Wyden’s letters is that VR Systems only hired FireEye to conduct an assessment of what happened after Reality Winner leaked an NSA document making it clear they had been targeted by GRU in 2017. [Update: Kim Zetter actually reported this here.]

In their June 12 letter, Wyden and Klobuchar asked Wray whether the FBI followed up on VR Systems’ report.

  1. What steps, if any, did the FBI take to examine VR Systems’ servers for evidence of a successful cyber breach after the company alerted the FBI, in August of 2016, to the presence of suspicious IP addresses in its website logs? If the FBI did not examine VR Systems’ servers or request access to those servers, please explain why.
  2. Several months after VR Systems first contacted the FBI, electronic pollbooks made by the company malfunctioned during the November 8 general election in Durham County, North Carolina. In the two and a half years since that incident in Durham County, has the FBI requested access to the pollbooks that malfunctioned, and the computers used to configure them, in order to examine them for evidence of hacking? If not, please explain why.
  3. VR Systems contracted FireEye to perform a forensic examination of its systems in the summer of 2017. Has the FBI reviewed FireEye’s conclusions? If so, what were its key findings?

It’s unclear how Wray answered (or didn’t). But just before Wyden sent this letter, the WaPo reported that no one had yet conducted a forensic examination of the laptops used in the VR Systems polling books in North Carolina. After Democrats took over control, they finally persisted in getting DHS to agree to check the laptops.

On Tuesday, the Department of Homeland Security told The Washington Post it will conduct a forensic analysis of the laptops used in Durham County elections in 2016. Lawson said North Carolina first asked the department to conduct such a review more than 18 months ago, though he added that DHS has generally been a “good partner” on election security.

“We appreciate the Department of Homeland Security’s willingness to make this a priority so the lingering questions from 2016 can be addressed in advance of 2020,” said Karen Brinson Bell, the newly appointed executive director of the State Board of Elections.

After the election, Durham County hired a firm called Protus3 to dig into what happened. The security consultant said it appeared the problems were caused by user error but ended its 12-page report with a list of recommendations that included examining computers in a lab setting and interviewing more election workers.

Durham County elections director Derek Bowens said he is comfortable with the report’s conclusions. Even so, in 2017, the county switched to electronic poll books created by the state. Bowens said in an interview that the state’s software would save money and is, in his view, better.

But for North Carolina officials, concerns resurfaced in June 2017 when the website Intercept posted a leaked National Security Agency report referencing “cyber espionage operations against a . . . U.S. company in August 2016.” The NSA report said that “it was likely that at least one account was compromised.”

VR Systems soon acknowledged that hackers had targeted the company but insisted that its network had not been breached.

North Carolina officials weren’t so sure.

“This was the first leak that indicated anything like a nation-state actor targeting a voting systems vendor,” Lawson said.

The state elections board soon launched its own investigation, seizing 40 laptops from Durham in July. And it suspended the certification that allowed more than 20 North Carolina counties to use VR Systems’ poll books during elections, an action that would later land in court. “Over the past few months there has been a considerable change in the election security landscape and the level of scrutiny we receive,” the board wrote in a letter explaining its decision to VR Systems.

No one working for the board had the technical expertise to do a forensic examination of the machines for signs of intrusion. Staffers asked DHS for technical help but did not get a substantive answer for a year and a half, Lawson said.

As noted, FireEye appears to have done an assessment at VR Systems itself in the wake of the Winner disclosure. The WaPo reports that FireEye declared VR Systems hadn’t been hacked, but wouldn’t share any information with Wyden or–apparently–DHS.

VR Systems said a cybersecurity firm it hired to review its computer network in 2017 found no evidence of a hack. A subsequent review by DHS also found no issues, the company said. VR Systems declined to give Wyden documentation of those reviews, citing the need to protect proprietary information.

Wyden in a statement to The Post accused VR Systems of “stonewalling congressional oversight.”

A senior U.S. official confirmed DHS’s review of VR Systems’s network to The Post and noted that by the time agency investigators arrived, a commercial vendor had already “swept” the networks. “I can’t tell you what happened before the commercial vendor came in there,” the official said, speaking on the condition of anonymity to discuss a sensitive matter.

The same day as the WaPo report, Kim Zetter reported that VR Systems used remote updates for their software, opening up a possible point of compromise for hackers.

For two years, GRU hack denialists have thought it was the most important thing that the DNC provided FBI Crowdstrike’s forensic images of the hacked laptops, rather than providing the servers themselves.

But that step has, apparently, not been done yet with VR Systems. And the laptops that failed on election day are only now being forensically examined.  Which is why, I presume, that Wyden believes it’s premature to claim no vote totals were affected on election day 2016.

In Discussion of Unmasking Admiral Rogers Gets Closer to Admitting Types of Section 702 Cybersecurity Use

/2 Comments/in , , /by

Last Friday, Director of National Intelligence Dan Coats, Director of NSA Mike Rogers, and FBI Director Christopher Wray did an event at Heritage Foundation explaining why we need Section 702 and pretending that we need it without reasonable reforms. I attended Wray’s talk — and even got my question on cybersecurity asked, which he largely dodged (I’ll have more about two troubling things Wray said later). But I missed Rogers’ talk and am just now catching up on it.

In it, he describes a use of Section 702 that goes further than NSA usually does to describe how the authority is used in cybersecurity.

So what are some examples where we’ll unmask? Companies. Cybersecurity. So we’ll report that US company 1 was hacked by the following country, here’s how they got in, here’s where they are, here’s what they’re doing. Part of our responsibility on the US government side is the duty to warn. So how do you warn US company 1 if you don’t even know who US company 1 is? So one of the reasons we do unmasking is, so for example we can take protective to ensure this information is provided to the appropriate individuals.

What Rogers describes is an active hack, by a nation-state (which suggests that rule may not have changed since the 2015 report based off 2012 Snowden documents that said NSA could only use 702 against nation-state hackers). The description is not necessarily limited to emails, the type of data NSA likes to pretend it collects in upstream (though it could involve phishing). And the description even includes what is going on at the victim company.

Rogers explains that the NSA would unmask that information so as to be able to warn the victim — something that (via the FBI) happened with the DNC, but something which didn’t happen with a number of other election related hacks.

Of course, Reality Winner is facing prison for having made this clear. The FISA-derived report she is accused of leaking shows how the masking works in practice.

In the case of VR Systems, the targeted company described, it’s not entirely clear whether NSA (though FBI) warned them directly or simply warned the states that used it. But warnings, complete with their name, were issued. And then leaked to the press, presumably by people who aren’t facing prison time.

In any case, this is a thin description of NSA’s use of 702 on cybersecurity investigations. But more detail in unclassified public than has previously been released.

 

How to Read the DHS Targeted States Information

/9 Comments/in , /by

Yesterday, DHS informed the states that had their registration databases targeted by Russian hackers last year. There has been an outright panic about the news since states started revealing they got notice, so I thought it worthwhile to describe what we should take away from the notice and subsequent reporting:

  • “Most” of the 21 targeted states were not successfully hacked
  • Some targeted states were successfully hacked
  • Not all swing states were targeted, not all targeted states are swing states
  • These hacks generally do not involve vote tallying
  • These hacks do not involve hacking voting machines
  • These hacks do not involve other voter suppression methods — whether by GOP or Russians
  • Notice needs to improve

The AP has done good work tracking down which states got notice they were targeted, identifying the 21 targeted states. Those targeted states were:

  1. Alabama
  2. Alaska
  3. Arizona
  4. California
  5. Colorado
  6. Connecticut
  7. Delaware
  8. Florida
  9. Illinois
  10. Iowa
  11. Maryland
  12. Minnesota
  13. North Dakota
  14. Ohio
  15. Oklahoma
  16. Oregon
  17. Pennsylvania
  18. Texas
  19. Virginia
  20. Washington
  21. Wisconsin

 

“Most” of the 21 targeted states were not successfully hacked

This list of 21 states does not mean that Russians successfully hacked 21 states. All it means is Russians probed 21 states. And the AP says “most” were not successful. WI, WA, and MN have said the attacks on them were not successful.

Thus, for “most” of these states, the impact is the same as the reports that Russians were attempting, unsuccessfully, to phish engineers in the energy industry: it is cause for concern, but unless new intelligence becomes available, it means that for those “most” states these probes could not affect the election.

Some targeted states were successfully probed

Of course, by saying that “most” attacks were not successful, you’re admitting that “some” were. We only know IL and AZ to have successfully been breached.

This means this story may not be done yet: reporters, especially state based ones, are going to have to get their voting officials to provide details about the attacks and it may take some FOIA work.

Mind you, a successful hack still doesn’t mean that the election was affected (as I believe to be the understanding with respect to AZ, though there is more dispute about IL). It might be that the hackers just succeeded in getting into the database. It may be that they succeeded only in downloading the voter registration database — which in many states, is readily available, and which is nowhere near the most interesting available data for targeting in any case.

In my opinion, the most effective way to affect the outcome of the election via voter registration databases is not to download and use it for targeting, but instead, to alter the database, selectively eliminating or voiding the registration of voters in targeted precincts (which of course means the hackers would need to come in with some notion of targets). Even changing addresses would have the effect of creating lines at the polls.

Altering the database would have the same effect as an existing GOP tactic does. In many states, GOP secretaries of state very aggressively purge infrequent voters. Particularly for transient voters (especially students, but poorer voters are also more likely to move from year to year), a voter may not get notice they’ve been purged. This has the effect of ensuring that the purged voter cannot vote, and also has the effect of slowing the voting process for voters who are registered.  In other words, that’s the big risk here — that hackers will do things to make it impossible for some voters to vote, and harder for others to do so.

Not all swing states were targeted, not all targeted states are swing states

The list of targeted states is very curious. Some targeted states are obvious swing states — WI, PA, FL, and VA were four of the five states where the election was decided. But MI is not on there, and NC, another close state, is not either.

In addition, a lot of these states are solidly red, like AL and OK. A lot of them are equally solidly blue, like CA and CT. So if the Russians had a grand scheme here, it was not (just) to flip swing states.

These hacks generally do not involve vote tallying

DHS has said that these hacks do not involve vote tallying. That means these disclosed probes, even assuming they were successful, are not going to explain what may seem to be abnormalities in particular states’ tallies.

These hacks do not involve hacking voting machines

Nor do these hacks involve hacking voting machines (which is covered, in any case, by the denial that it involves vote tallying).

Yes, voting machines are incredibly vulnerable. Yes, it would be child’s play for a hacker — Russian or American — to hack individual voting machines. With limited exceptions, there been no real assessment of whether individual machines got hacked (though it’d generally be easier to affect a local race that way than the presidential).

These hacks do not involve other voter suppression methods — whether by GOP or Russians

This list of 21 targeted states does not represent the known universe of Russian voting-related hacking.

It does not, for example, include the targeting of voting infrastructure contractors, such as VR Systems (which Reality Winner faces prison for disclosing). There’s good reason to at least suspect that the VR Systems hack may have affected NC’s outcome by causing the most Democratic counties to shift to paper voting books, resulting in confusion and delays in those counties that didn’t exist in more Republican ones.

And they don’t include any Russian social media-related support or suppression, which we’re getting closer to having proof of right now.

Importantly, don’t forget that we know Republicans were engaging in all these techniques as well, with far better funding. Russians didn’t need to hack WI and NC given how much organized suppression of voters of color took place. Republican secretaries of state had the power to purge voters on trumped up excuses without engaging in any hacking.

Do not let the focus on Russian tampering distract from the far more effective Republican suppression.

Notice needs to be improved

Finally, the other big story about this is that some states only got notice they were targeted yesterday, some even after having partnered with DHS to assess their voting infrastructure.

DHS has used classification, in part, to justify this silence, which is an issue the Intelligence Committees are trying to address in next year’s authorization. But that’s particularly hard to justify that many of these same states have run elections since.

Mind you, we’re likely to see this debate move to the next level — to demanding that state officials disclose full details about their state’s infrastructure to citizens.

In any case, if we’re to be able to use democratic pressure to ensure the infrastructure of democracy gets better protected, we’re going to need more notice.

Report from North Carolina Makes Reality Winner Leak Far More Important

/43 Comments/in , , /by

According to NPR, the poll books in six precincts in Durham County, NC, went haywire on election day, which led the entire county to shift to paper poll books.

When people showed up in several North Carolina precincts to vote last November, weird things started to happen with the electronic systems used to check them in.

“Voters were going in and being told that they had already voted — and they hadn’t,” recalls Allison Riggs, an attorney with the Southern Coalition for Social Justice.

The electronic systems — known as pollbooks — also indicated that some voters had to show identification, even though they did not.

[snip]

At first, the county decided to switch to paper pollbooks in just those precincts to be safe. But Bowens says the State Board of Elections & Ethics Enforcement got involved “and determined that it would be better to have uniformity across all of our 57 precincts and we went paper pollbooks across the county.”

That move caused a whole new set of problems: Voting was delayed — up to an hour and a half — in a number of precincts as pollworkers waited for new supplies. With paper pollbooks, they had to cut voters’ names out and attach them to a form before people could get their ballots.

The company that provided the software for the poll books is VR Systems — the company that the document Reality Winner leaked showed had been probed by Russian hackers.

But Susan Greenhalgh, who’s part of an election security group called Verified Voting, worried that authorities underreacted. She was monitoring developments in Durham County when she saw a news report that the problem pollbooks were supplied by a Florida company named VR Systems.

“My stomach just dropped,” says Greenhalgh.

She knew that in September, the FBI had warned Florida election officials that Russians had tried to hack one of their vendor’s computers. VR Systems was rumored to be that company.

Because of the publicity surrounding the VR targeting — thanks to the document leaked by Winner — NC has now launched an investigation.

Lawson says the state first learned of the hack attempt when The Intercept, an online news site, published its story detailing Russian attempts to hack VR Systems. The leaked report said hackers then sent emails to local election offices that appeared to come from VR — but which actually contained malicious software.

[snip]

So now, months after the election, the state has launched an investigation into what happened in Durham County. It has secured the pollbooks that displayed the inaccurate information so forensic teams can examine them.

So this may be the first concrete proof that Russian hackers affected the election. But we’ll only find out of that’s true thanks to Winner’s leak.

Except she can’t raise that at trial.

Last week, Magistrate Judge Brian Epps imposed a protection order in her case that prohibits her or her team from raising any information from a document the government deems to be classified, even if that document has been in the public record. That includes the document she leaked.

The protective order is typical for leak cases. Except in this case, it covers information akin to information that appeared in other outlets without eliciting a criminal prosecution. And more importantly, Winner could now point to an important benefit of her leak, if only she could point to the tie between her leak and this investigation in North Carolina.

With the protection order, she can’t.

Note one more implication of this story.

In addition to the Presidential election last year, North Carolina had a surprisingly close Senate election, in which Senate Intelligence Committee Chair Richard Burr beat Deborah Ross by 6%. Admittedly, the margin was large — over 200,000 votes. But Durham County is the most Democratic county in the state.

Burr, of course, is presiding over one of the four investigations into the Russian hacks. And while I don’t think this story, yet, says that Burr won because of the hack, if the investigations shows VR was hacked in the state and it affected throughput in the most Democratic county, then it means Burr benefitted as clearly from the Russian hacks as Trump did.

The SSCI investigation has been going better than I had imagined. But this seems like a conflict of interest.

Update: I originally said the entire state switched to paper pollbooks. That’s incorrect: just Durham County did, which makes the issue even more important.