In this post, I’m going to examine a claim made in the May 3, 2012 audit report of NSA violations. Through the magic of simple arithmetic, I’m going to show that the report misleads readers about why the number of incidents rose in the first quarter of 2012, wrongly suggesting it was an unpreventable seasonal problem, rather than pointing to the human error and fault that really explained the increase.

On page two, the report shows how many Signals Intelligence Directorate-reported incidents there are across both kinds of authorities: EO 12333 (strictly foreign) and FISA (involving US persons).

As the report acknowledges, there was an 11% increase in incidents for both kinds of authority.

But don’t worry, the report says, the increase is due to Chinese New Year, sort of.

The increase in incidents reported for 1QCY12 was due to an increase in the number of reported Global System for Mobile Communications (GSM) roamer1 incidents, which may be attributed to an increase in Chinese travel to visit friends and family for the Chinese Lunar New Year holiday.

1Roaming incidents occur when a selector associated with a valid foreign target becomes active in the U.S.

On the following page, a section provides further explanation on the roamer problem.

The largest number of incidents in the System Limitations category account for roamers where there was no previous indications of the planned travel. These incidents are largely unpreventable. Consistent discovery through the Visitor Location Register (VLR) occurs every quarter and provides analysts with timely information to place selectors into candidate status or detask. Analysis identified that these incidents could be reduced if analysts removed/detasked selectors more quickly upon learning that the status of the selector had changed and more regularly monitored target activity. This analysis indicates that continued research on ways to exploit new technologies and researching the various aspects of personal communications systems to include GSM, are an important step for NSA analysts to track the travel of valid foreign targets.

On page 6, we get a more comprehensible explanation.

Roamers: Roaming incidents occur when valid foreign target selector(s) are active in the U.S. Roamer incidents continue to constitute the larges category of collection incidents across E.O. 12333 and FAA authorities. Roamer incidents are largely unpreventable, even with good target awareness and traffic review, since target travel activities are often unannounced and not easily predicted.

In other words, the roamer problem stems from the fact that when valid foreign targets travel to the US with their GSM phones, analysts don’t know that and therefore don’t act accordingly. I think (though am not positive) the presence of the target in the US would shift a 12333 intercept into a FISA one (we’d be tracking calls to foreigners with one end in the US), and a FISA Amendments Act target into an illegal one (we’d be tracking calls with both ends in the US, one potentially involving a US person). Since this involves primarily valid foreign targets, it is not the most urgent problem identified in the report.

And, the NSA claims, it is largely unavoidable, so readers of this report should expect the relatively large numbers of roamer problems to continue.

Up to this point — far beyond where most readers will be paying attention, I’d imagine — we might believe (because the report said so explicitly) that the 11% increase in incidents stems from a problem involving valid foreign targets and reflecting an unavoidable technical problem.

It’s only when you get to page 5 and 6 that this narrative falls apart. Here’s how many roamer incidents occurred under EO 12333 for the four quarters reported.

And here’s how many roamer incidents occurred under FISA for the four quarters presented.

Adding the roamer incidents for each kind of authority together, we discover the total roaming incidents, across both authorities, look like this in the last quarter of 2011 and first quarter of 2012:

4QCY11: 582 + 87 = 669

1QCY12: 491 + 95 = 586

In fact, the roaming problem doesn’t explain the 11% overall increase in incidents at all, because the number of roaming incidents under EO12333 actually went down 19%, meaning roaming incidents across the two authorities went down 14%.

The roamer explanation doesn’t even explain the entire increase in FISA incidents, as FISA roamer incidents only went up 9%, as the report admits on page 6. On that page, the report admits that another big source of increased incidents, for both kinds of authorities, comes from database queries.

During 1QCY12, NSAW SID reported an increased of 9% of roamer incidents under all FISA Authorities. There was also a 260% increase in database query FISA Authority incidents during 1QCY12. Human Error accounted for the majority of all FISA Authorities database query incidents (74%).

[snip]

Database Queries: During 1QCY12, NSAW SID reported a total of 115 database query incidents across all Authorities, representing a 53% increase from 4QCY11. E.O. 12333 Authority database query incidents accounted for 84% (97) of the total, and all FISA Authorities database query incidents accounted for 16% (18).

The report goes on to describe the root causes for these database query incidents.

Broad syntax (i.e., no or insufficient limiters/defeats/parameters)

Typographical error

Boolean operator error

Query technique understood but not applied

Not familiar enough with the tool used for query

Incorrect option selected in tool

Lack of due diligence (failure to follow standard operating procedure)

Training and guidance

Resources (Inaccurate or insufficient research information)

Thus, the front page story on this report should not be “oh, it’s all the fault of the Chinese New Year,” but instead, “oh, a bunch of human errors and due diligence (which implies fault) and research problems (which might be somebody else’s fault) resulted in the better part of the reported incident increase.”

There’s one more thing that accounts for the increase: International Transit Switch Collection, which is the problem behind the FISA Court’s October 3, 2011 finding that the program violated the Fourth Amendment (see this report, which I’ll return to). Transit Program violations went from 7 in 4QCY11 to 27 in 1QCY12. But don’t worry about that, the report says; it just reflects a different counting method.

International Transit switches, FAIRVIEW (US-990), STORMBREW (US-983), ORANGEBLOSSOM (US-3251), and SILVERZEPHYR (US-3273) are Special Source Operations (SSO) programs authorized to collect cable transit traffic passing through U.S. gateways with both ends of the communication being foreign. When collection occurs with one or both communicants insider the U.S., this constitutes inadvertent collection. From 4QCY11 to 1QCY12, there was an increase of transit program incidents submitted from 7 to 27, due to the change in our methodology for reporting and counting these types of incidents.

I’m guessing this changed methodology arose in response to the FISC opinion and also guessing that there should have been far more than 7 or even 27 violations reported before the FISC declared all this illegal.

But that still means that another important contributor to the rise in incidents between the two quarters had to do with actually counting the number of times US person communications were collected off of purportedly international transit switches.

Now, I might assume the simple math problem the NSA exhibited here was innocent.

Except it’s the NSA, the biggest math organization in the world. If I — someone who hasn’t played around with math in 27 years — can find the NSA’s simple arithmetic problem, I’m going to assume it is not an accident at all, but stems instead from an effort to hide the more serious problems deeper in the report.

The NSA: failing simple math.

Update: I should add that one other source of the increase — larger in absolute terms than the roamer problem — comes from analysts failing to turn off some or all of a wiretap of a US person when they’re supposed to.