How Does NSA (and Its Partners) Catch More Terrorists in Europe with Less Metadata?
In follow-up to yesterday’s I Con, Le Monde reports that France’s spy agency, DGSE and the US, established a data sharing arrangement in 2011-2012 via which France provides call data to the US. It notes that part of the data the US gets comes from the French (apparently, Le Monde has better mastery of the conjunction than American National Security journalists) and that French citizens, as well as other targets, are included.
I suspect this is where the global dragnet may proceed: where we learn, country by country, that the US has side deals with partners, in addition to massive collections done largely (in Europe, anyway) by GCHQ, that allows it access to a lot of metadata.
But there’s something missing.
The US can, so long as it gets away with it, collect as much metadata as it can from France and other foreign countries. In the US, it has to work through the courts (well, that’s the law, one the Bush Administration flouted for 5 years).
And yet, the US collects far more metadata in the US than it does in France. In the last month of 2012, the US (and its partners, including GCHQ and DGSE) collected 70.3 million pieces of metadata in France, or roughly 1.07 piece of metadata on every French person. According to the Guardian, Boundless Informant shows the NSA (and its partners) collected 2.89 billion pieces of data in the month ending March 2013, or roughly 9.32 pieces of metadata on every American. And all that’s apparently before you consider the billions or trillions of pieces of metadata collected in the phone dragnet (which of course collects on “substantially all” the 310 million Americans (though in France, investigators can access phone metadata more readily).
That is, legally, the NSA (and its partners, including GCHQ) are not bound by legal limits on what they collect. But it collects more on Americans than it does on the French.
And yet … NSA finds more terrorists in Europe than in America.
More terrorists, less metadata.
I am sure this is a matter of comparing oranges to orange bouncey balls. Different times of the year, different numbers of terrorists in the country, different complementary tools and investigative skills. That is, there are nuances in all this data that neither the Snowden document recipients nor the NSA are going to be able to explain anytime soon. But they both seem to agree Boundless Informant does provide some picture of how much data the NSA (and its partners) collect where. And that does seem to show that NSA collects relatively more in the US than it does in Europe.
If that’s the case, then why is having a complete haystack of metadata here in the US pursuant to the Section 215 dragnet necessary? Doesn’t the European case show you can find even more terrorists without it?
a little light reading in history:
http://www.washingtonpost.com/national/earl-s-browning-an-army-counterintelligence-officer-in-world-war-ii-dies-at-96/2013/10/29/ca4c6546-3d9d-11e3-b6a9-da62c264f40e_story.html?tid=hpModule_4eb86ade-919e-11e2-bdea-e32ad90da239&hpid=z15
Plus ca change, plus c’est la meme chose.
Let’s get real. The only place one need look for the real terrorists is the US Federal government. Here is living proof…….
http://www.youtube.com/watch?v=LhSe6-mK8wk
Can you even imagine this? As the daily NSA Surveillance State Circus of Liars perform their tricks to keep America spellbound in the center ring of the Greatest Show on Earth, meanwhile, a small group of innocent Pakistani victims of our depraved War on “Terror”, travel to the US in order to BEG our wunnerful USG to stop vaporizing their fellow citizens and family members, during a US Congressional hearing whereby..only SIX of these detestable Congressional bastards had the nerve to show up and stand accountable to these poor human beings. ONLY SIX. And what does our fabulous US media do? NOTHING. ABSOLUTELY..fucking… nothing. Zilch. Zero. Nada.
The mere fact that these victims had to come to the US to BEG our depraved, murdering government to stop killing innocent people is enough to gag a maggot. Living proof King Obomber, the kill list overseer is a vindictive, psychopathic murderer. And then to see only 6 members of our pathetic, sick Congress appear to publicly atone for the depraved actions of our government is living testimony to their cowardice and apathy towards victims of those agency’s whom they fund. Shame on our worthless Congress. Shame on our sadistic, barbaric USG. Shame on the completely useless US media, and SHAME on the pathetic, soulless citizens of this degenerate nation for not rising up and removing these psychopaths from power. I submit, it’s only a matter of time until SOME people in this country decide ..enough is enough.
Shiny objects. Now it’s “listening to foreign leaders”, which is being manipulated into a proposed “lead story” with the perceived side benefit of diverting attention from illegal spying on Americans: witness DiFi making her first “we’ve got to protect our phony-baloney jobs” bloviations about oversight on Merkel-sneaking, while simultaneously starting to mark-up her whitewash NSA/US bill on the quiet.
There are, it would seem, two distinct things going on and a conscious effort to conflate them. The stuff about monitoring foreign countries and their leaders is not, however, the same thing as conducting US data collection in derogation of the Fourth Amendment. The “we’ve always done that and so have they”-ness of the Merkelism stuff is, to a very large degree, true: it’s called “espionage”, it’s essentially been state policy since time immemorial and it’s ultimately conducted subject to the Mission Impossible caveat of “if you or any of your IM squad are captured or killed, the Secretary will disavow any knowledge of your actions” — which implicitly acknowledges that, well, the Germans could arrest our spyguys for espionage or (at the very least) violating German law. That’s a question involving sovereignty, political expediency and straight up technological superiorities — can we do it because we can, subject to the risk of getting busted? That makes the “international” stuff something that can be likened primarily to a technological game of “who’s best” — reminiscent of the 19th century “better cannons/better armor plate” personal challenges glorified by Jules Verne as the backstory for From the Earth to the Moon. Whose government’s communications can withstand the other guys’ hacks or taps? Interesting, yes — but of a different nature from the issues presented when a government turns its technological espionage upon its own citizens, isn’t it?
Certainly the various revelations about international spying are significant, certainly the knowledge of the scope of governments’ efforts is newsworthy and the absence of congressional or administrative oversight troubling. But those are not the same issues as the issues most requiring attention and rectification – which are those involving the effort to turn global capabilities into domestic license, notwithstanding the Constitution we have here. “Because we can” does not cut it and must not be suffered to excuse it by a shift of focus to “newsworthy” titillation over offended (“really” offended or “officially” offended — is that what sells?) nations.
If the governments want to arrest each others’ operatives, have at it. If they want to crystallize “rules” for this knife fight, so be it. But our government has a separate set of things to explain to us here. Whether Merkel is happy isn’t what drives those requirements.
@emptywheel: Marcy, I agree this is odd but there is one possible caveat that I didn’t see mentioned. Just because the NSA says that “they” caught the terrorists doesn’t mean that they did. If we are to judge from the public examples they’ve released (e.g. Zazi) merely having related info counts as having “caught” someone even if nothing useful was done with it.
Welcome to the Corporate world. How will these revelations go over when the French,German and other citizens realize their intel agencies not only allow a foreign power to spy on them, but facilitate and participate? The old saying that spying is the second oldest profession denigrates whores in comparison.
If there were any democracy in the so called democracies of the world, a serious scaling back of intel budgets would be on the table, well maybe except for counter-intel :)
In a June 5, 1986 email message to Colonel Oliver North of the Reagan National Security Council (NSC) staff, which Iran/Contra investigators recovered from the NSC’s IBM mainframe computer after Oliver North thought he had deleted it, an NSC colleague recounts his meeting that day with the top officials of the Meese Office of Legal Counsel (OLC) on obtaining a Top Secret/Code Word legal opinion authorizing NSA to add another 400 major commercial banks to its Follow the Money real-time electronic surveillance of bank transfers and to augment NSA’s SIGINT resources for the expanded coverage with help from European SIGINT agencies. The National Security Archives published the partially redacted email in its publication entitled “White House Emails.”
“Less is more” is not just a silly saying. It’s true in lots of situations, often for rather simple (albeit technical) reasons. For example, if you conduct an experiment with a certain (fixed) number of subjects in an attempt to detect a significant effect of some factor, which I’ll call F-I (for “factor of interest”), then you will have a better chance of achieving significance if F-I is the only factor. Adding a second factor, F-nI (for “factor not of interest”), can only decrease the chance of significance; it can never help.
Too much information – especially when the extra info is irrelevant – always hurts. It hurts in stats; it hurts in rational decision making; and it hurts in pattern recognition. It is not at all surprising that terrorists are easier to find in limited/smaller data-sets. Everything else is easier.
@JTMinIA:
thanks. very useful reference from methodology/analysis of data. would ypu think this would apply to the huge data sets referred to these days as “big data”?
@orionATL: orionATL, in my experience the proponents of “Big Data” ignore exactly the advice that JTMinlA is giving. Instead they place, often quite credulous, faith in a misreading of the law of large numbers and assume that more is always better or that they can/should collect more and then let the machines figure out what to do with it.
@orionATL: Well, to be fair to “big data,” if you are trying to discover new predictors of an outcome, then you need as many measures (variables) as possible. But that’s during what a stats-person would call the “exploratory phase.” Once you have the predictors, then you want to follow the advice that I was giving (which really comes from Jacob Cohen, who is sort of a g*d to us stats geeks [even though I recently proved that one of his famous equations is wrong]): include only those predictors that are useful; include no predictor that does that add to your ability to predict the outcome.
In a nutshell, my guess is that the folks trying to catch bad-guys are using all of their data, instead of only the variables that are known to be useful. They’re thinking that more is better, even if some of the extras are worthless. They’re wrong. Less is more.
Big Data is actually a misnomer.
It should more legitimately be called Voluminous, Fast, Highly Complex, Questionable Quality, Unrelated, and Rapidly Changing, DATA.
Addressing it was not something any sane person would pursue by choice. It was an outcome of the activities that occur on the internet and a requirement that had to be addressed.
Example, the BIG DATA coming at Twitter cannot be reduced into a small sampling and deliver the same insights.
So, less is not always more. More does provide a better trend analysis.
But as I’ve said before, historical trends are almost useless as predictors of the future because they ignore the low-probability but high impact events, Black Swans.
You can never get enough data to predict the next WTC event, or the next Ft Hood, or the next Boston Marathon event.
The whole idea of prediction is farcical. Think about it, if anyone could accurately and repeatably predict anything their powers would appear god-like.
So, handling more data is not a choice, it’s a requirement. But, it will only improve the accuracy of the historical trend line, and then only if it is very well managed, eliminating bad data, excluding unrelated data, recognizing whole new trends, etc.
It will never accurately predict what is going to happen even 1 second into the future. NO way.
I get the distinct impression the NSA thinks it can be predictive and that more data will help achieve this. It won’t.
@C:
@JTMinIA:
@Greg Bean (@GregLBean):
thanks to each of you for taking the trouble to explain from your perspectives. decades ago i did this sort of analytical work on data sets very rarely numbering more than 1500, and often smaller.
i have trouble imaging how one would even approach analyzing a data set of xx millions, let alone the useful output from that analysis.
yet i read of google and facebook analyses, of president obama’s re-election campaign analyses, and of course of nsa’s analyses of “big data”. i can at least envision a gigantic version of cluster analysis, but i can’t see what one would achieve beyond that relatively weak (and easily changeable/manipulable) type of result.