The Biggest Math Organization in the World Has a Simple Arithmetic Problem
In this post, I’m going to examine a claim made in the May 3, 2012 audit report of NSA violations. Through the magic of simple arithmetic, I’m going to show that the report misleads readers about why the number of incidents rose in the first quarter of 2012, wrongly suggesting it was an unpreventable seasonal problem, rather than pointing to the human error and fault that really explained the increase.
On page two, the report shows how many Signals Intelligence Directorate-reported incidents there are across both kinds of authorities: EO 12333 (strictly foreign) and FISA (involving US persons).
As the report acknowledges, there was an 11% increase in incidents for both kinds of authority.
But don’t worry, the report says, the increase is due to Chinese New Year, sort of.
The increase in incidents reported for 1QCY12 was due to an increase in the number of reported Global System for Mobile Communications (GSM) roamer1 incidents, which may be attributed to an increase in Chinese travel to visit friends and family for the Chinese Lunar New Year holiday.
1Roaming incidents occur when a selector associated with a valid foreign target becomes active in the U.S.
On the following page, a section provides further explanation on the roamer problem.
The largest number of incidents in the System Limitations category account for roamers where there was no previous indications of the planned travel. These incidents are largely unpreventable. Consistent discovery through the Visitor Location Register (VLR) occurs every quarter and provides analysts with timely information to place selectors into candidate status or detask. Analysis identified that these incidents could be reduced if analysts removed/detasked selectors more quickly upon learning that the status of the selector had changed and more regularly monitored target activity. This analysis indicates that continued research on ways to exploit new technologies and researching the various aspects of personal communications systems to include GSM, are an important step for NSA analysts to track the travel of valid foreign targets.
On page 6, we get a more comprehensible explanation.
Roamers: Roaming incidents occur when valid foreign target selector(s) are active in the U.S. Roamer incidents continue to constitute the larges category of collection incidents across E.O. 12333 and FAA authorities. Roamer incidents are largely unpreventable, even with good target awareness and traffic review, since target travel activities are often unannounced and not easily predicted.
In other words, the roamer problem stems from the fact that when valid foreign targets travel to the US with their GSM phones, analysts don’t know that and therefore don’t act accordingly. I think (though am not positive) the presence of the target in the US would shift a 12333 intercept into a FISA one (we’d be tracking calls to foreigners with one end in the US), and a FISA Amendments Act target into an illegal one (we’d be tracking calls with both ends in the US, one potentially involving a US person). Since this involves primarily valid foreign targets, it is not the most urgent problem identified in the report.
And, the NSA claims, it is largely unavoidable, so readers of this report should expect the relatively large numbers of roamer problems to continue.
Up to this point — far beyond where most readers will be paying attention, I’d imagine — we might believe (because the report said so explicitly) that the 11% increase in incidents stems from a problem involving valid foreign targets and reflecting an unavoidable technical problem.
It’s only when you get to page 5 and 6 that this narrative falls apart. Here’s how many roamer incidents occurred under EO 12333 for the four quarters reported.
And here’s how many roamer incidents occurred under FISA for the four quarters presented.
Adding the roamer incidents for each kind of authority together, we discover the total roaming incidents, across both authorities, look like this in the last quarter of 2011 and first quarter of 2012:
4QCY11: 582 + 87 = 669
1QCY12: 491 + 95 = 586
In fact, the roaming problem doesn’t explain the 11% overall increase in incidents at all, because the number of roaming incidents under EO12333 actually went down 19%, meaning roaming incidents across the two authorities went down 14%.