What’s the Relationship Database About?
Atrios asks what the whole dragnet is about.
It’s actually a serious question. Maybe it’s just a full employment program for spooks. Maybe they just do it because they can. But the only “real” point to such an extensive surveillance system is to abuse that surveillance (the surveillance itself is already an abuse of course).
At best it’s a colossal fucking waste of money. At worst?
I actually think there are understandable answers for much of this.
Since Michael Hayden took over the NSA, contractors have assumed an increasingly dominant role in the agency, meaning you’ve got a former DIRNSA at Booz Allen Hamilton pitching future Booz VPs on solutions to keep the country safe that just happen to make them fabulously profitable and don’t happen to foreground privacy. As Thomas Drake showed, we’re pursuing the biggest and most privacy invasive solutions because contractors are embedded with the agency.
I think there’s the One Percent approach we got from Dick Cheney, that endorses maximal solutions to hunt terrorists even while avoiding any real accountability (both for past failures and to review efficacy) because of secrecy. We’re slowly beginning to wean ourselves from this Cheney hangover, but it is taking time (and boosters for his approach are well-funded and publicized).
And, at the same time, criminals and other countries have attacked our weak network security underbelly, targeting the companies that have the most political sway, DOD contractors and, increasingly, financial companies, which is setting off panic that is somewhat divorced from the average American’s security. The accountability for cybersecurity is measured in entirely different ways than it is for terrorism (otherwise Keith Alexander, who claims the country is being plundered like a colony, would have been fired years ago). In particular, there is no punishment or even assessment of past rash decisions like StuxNet. But here, as with terrorism, the notion of cost-benefit assessment doesn’t exist. And this panicked effort to prevent attacks even while clinging to offensive cyberweapons increasingly drives the overaggressive collection, even though no one wants to admit that.
Meanwhile, I think we grab everything we can overseas out of hubris we got while we were the uncontested world power, and only accelerated now that we’re losing that uncontested position. If we’re going to sustain power through coercion — and we developed a nasty habit of doing so, especially under Bush — then we need to know enough to coerce successfully. So we collect. Everything. Even if doing so makes us stupider and more reliant on coercion.
So I can explain a lot of it without resorting to bad faith, even while much of that explanation underscores just how counterproductive it all is.
But then there’s the phone dragnet, the database recording all US phone-based relationships in the US for the last 5 years. In spite of extensive discussion of ways to do this without creating a database of every phone based relationship (and the House Intelligence Committee’s willingness to shorten the retention period to 3 years), Keith Alexander and James Clapper insist we cannot change the way we do this. This, in spite of the almost complete lack of any evidence the database (and its predecessor) has been useful over the last 12 years.
Indeed, in an op-ed, Adam Schiff suggests (given his reference to having urged changes privately before he did publicly, which he did in the first HPSCI hearing after the Snowden leaks) he has been making this point for some time.
As for the effectiveness of the program, the evidence that it has made us safer is limited. The Obama administration cites about a dozen cases in which the database was consulted in an investigation. Although many of the details of these cases remain classified, evidence that the metadata program was an integral part of the success of each of these investigations — or even most of them — is far from clear. Instead, it appears that the utility of the metadata program has been conflated with the success of other collection efforts.
Finally, on the third test of whether the program has been structured to minimize unnecessary intrusion on our privacy, the NSA program plainly fails. Rather than a narrowly tailored effort that reduces the potential for abuse or violations of privacy, the bulk collection regime is vast, touching billions of phone calls made in the United States over the last five years.
This is all the more troubling because there are other less intrusive ways to structure the program. I have urged the administration — privately at first, then publicly — to reconfigure the NSA effort so that the call records remain with the telephone companies that already hold them for business purposes. Under this model, the government could meet its national security needs by asking the companies to run a number once it had been connected to a suspected terrorist plot. The government would neither collect nor retain the phone records.
At the Tuesday hearing, NSA Director Keith Alexander acknowledged that such a restructuring is technically feasible provided the phone companies maintain the data long enough and in an accessible format. Such a system might be less efficient for the NSA, but it could nonetheless provide quick and timely results. And Americans have the right to expect that intelligence-gathering programs are judged on more than efficiency alone. After all, if efficiency were the only priority, there would be no need for a 4th Amendment.
Keith Alexander’s shout out to terrorist supporter Peter King for his vocal support of the NSA in the hearing the other day made me realize that the sole known person caught primarily because of the Section 215 data, Basaaly Moalin, did far less than King did in the 1980s supporting Irish terrorists, and did it (according to a 2009 FBI assessment) for the same reasons — to raise his leadership profile in his tribe. The database simply hasn’t netted any serious threat.
And while I seem to be the only one gravely concerned that the NSA suggested it might use the relationship database to target informants, rather than actual terrorist associates, that does seem to be part of the explanation: “investigative leads” (as Clapper justified the program) are far more useful when they come complete with means to coerce even more useful investigative leads, no matter how unethical that might be.
But ultimately, even that application can’t explain the need for a relationship database encapsulating the entire country.
There seems to be little that justifies that relationship database than the desire to have it, in case, for such time as the government plans to unleash the nuclear bomb of reading every relationship in the country.
reading commentary here and at the guardian, i’ve beginning to wonder if the chink in the mo ster’s armpur plate might not be that there is little useful predictive analysis that can be done with any of these data bases.
all focus jas been on what sa does where and under what authority, together with the absolute torrent of evasions from nsa, ic officials.
we assume some sorts of analysis can be done with this data, bit is that the case?
you can call writing a report summarizing what chancellor merkel told hans-peter schrier analysis, but that is nothing more than a verbal summary.
what effective capacity to analyze the large amounts of data it collects does the nsa have? we hear “algorithm” bruited about. just exactly what useful analysis might these magical math sentences produce – if any?
might the nsa mountain of data be no more than a very unkempt, overstuffed library with its own dewey decimal system (per j. clapper)?
this, of course, is where the secrecy constraint our gov/nna imposes provides the ability to hid incapacity or incompetence under the guise of protecting a possibly-nonexistent large-scale analytical capacity.
i’m repeating the above, bit edited, since the edit function refused to let me send this from the edit page:
reading commentary here and at the guardian, i’ve beginning to wonder if the chink in the monster’s armour plate might not be that there is little useful predictive numerical analysis that can be done with any of these data bases.
all focus has been on what nsa does where, and under what authority, together with the absolute torrent of evasions from nsa and odni officials.
we may assume some sorts of numetical analysis can be done with this data, but is that the case?
you can call writing a report summarizing what chancellor merkel told hans-peter schrier analysis, but that is little more than a verbal summary of a discrete event or two at a discrete time or two.
what effective capacity to analyze the large amounts of data it collects does the nsa have? we hear “algorithm” bruited about. just exactly what useful analysis might these magical math sentences produce – if any?
might the nsa mountain of data be no more than a very unkempt, overstuffed library with its own dewey decimal system (per j. clapper)?
this, of course, is where the secrecy constraint our gov/nsa imposes provides the ability to hid incapacity or incompetence under the guise of protecting a possibly-nonexistent large-scale analytical capacity.
Why? Empire, plain and simple. Unilateral exercise of power over vassals and tributaries.
It’s secret because the true Americans, the ones that believe in democracy, justice, human rights and cooperation would never stand for it.
Why the massive domestic collection? Just like why the FBI operated COINTELPRO, dissent cannot be tolerated by the deep state.
EW, I think you and I define bad faith differently.
I think former NSA officials who take a spin through the revolving door to land highly paid gigs with contractors in order to sell expensive rubbish to taxpayers is bad faith.
I think Cheney’s One Percent solution as you describe it, is not only evidence of bad faith but staggering incompetence, all of which happens to bilk the taxpayer and make Cheney’s chums rich.
I think developing offensive cyberweapons behind your back, while in public shouting to high heaven about others doing so is bad faith.
I think bullies who snitch stuff from others simply because they can, and then go on to use it to embarrass their victims or manipulate and/or blackmail them is perhaps not bad faith, but certainly bad behavior and every bit as bad when done by a government as a kid in the schoolyard.
I don’t think one has to look any further than the trail of money to understand what motivates the NSA. Like all other federal agencies, success as an administrator is measured by money. How much have you grown? How successful have you been at selling your mission to Congress? How much has your budget increased? So, the Successful Administrator spends money as fast as s/he can rake it in, in order to justify an ever expanding budget.
Thomas Drake understood this when the more expensive, less effective option was chosen. More expensive? Good. Less effective? Better. Less effective means further solutions will be required in the future and that of course means… More Money. Perfect.
We have not progressed one iota since Watergate. Follow. The. Damn. Money. When you do, it explains a lot of the poor decision making that occurs by administrators (whether politically appointed or promoted from within by those appointees) that leaves the rank and file public servants in the agencies either scratching their heads or sputtering with irritation. Or… turns those public servants into whistleblowers.
In terms of capabilities, here’s an NSA presentation on its storage/search software (Accumulo) and its ability to efficiently massive (“web scale”) social (aka relationship) graphs (PDF): http://www.pdl.cmu.edu/SDI/2013/slides/big_graph_nsa_rd_2013_56002v1.pdf
TL;DR version: It does very well, thank you very much.
I think Atrios has a point when he says that they do it because they can. Put aside the moral/legal/constitutional concerns, this is a cool technical problem to work on.
And, I think that they do it because they hope to make predictions from the data. Forget whether or not that’s actually possible, if they think it’s possible, they’re going to try. If it doesn’t work, well, there just wasn’t enough data, so if you include more…
Suddenly you have a never ending justification to collect more and more.
@phred: I was thinking along the same lines.
It PISSES me off when Alexander, Clapper, and that bunch of hooligans called Congress keep lying to us about this. When they aren’t trying to be too cute by half, they are running it around the bushes, hedge groves, and forests to keep us from actually seeing the one tree.
I am LIVID! Well, that is the term they like to use when describing the anger of Foreign Leaders, so the American public may as well use the same term. Maybe they will hear it that way as they have been playing word games for so long.
I am to the point to where if our very own Government is not going to come clean and give true answers to legit questions, then we should ask those contractors. Carlyle Group, Chertoff Group, and all the other entities making a killing off the fear and unknowing of Americans will come to a point of being plastered across the globe if they don’t walk straight in DC about this!
How many American lives could you save in a year if you had a $50 billion budget?
And at what cost, in terms of lives lost, is the annual charge of $50 billion abstracted from American pockets?
Cheney said more than he realised when he called his security policies the 1% solution.
Excuse me, o/t update: http://www.emptywheel.net/2013/10/28/has-john-mccain-been-chatting-up-bibi-on-a-tapped-phone/#comment-649452
:-)
Michael Hayden, the Voice of Terror. Spymaster and serial liar Michael Hayden should review the history of the Stasi and the Secret Police in Rumania when those dictatorships fell.
Hayden is also a torturer and assassin. Did Hayden murder Pat Tillman? Did Hayden murder Michael Hastings? Hayden aided and abetted mass murder in Irak and Afghanistan and Yemen and everywhere there is a war on terror. Hayden did plan criminal wars using False Flag terror. Hayden and Barack Obama certainly murder people on a daily basis.
@Saul Tannenbaum:
For another article / point of view on Apache Accumulo and PRISM see
http://www.informationweek.com/big-data/commentary/big-data-analytics/defending-nsa-prisms-big-data-tools/240156388
RHIL
@Saul Tannenbaum:
thanks.
i understand now your point about a fascinating challenge that will be pursued.
but 8+ MEGAwatts for a single cray or ibm blue array? these nsa guys have got to be skating right at the edge of possible.
@Frank33: From Hayden’s Wall Street Journal Op-Ed (http://stream.wsj.com/story/latest-headlines/SS-2-63399/SS-2-369456/):
“Going forward we need to remember how U.S. intelligence suffered a similar crisis of conscience in the 1990s when the CIA’s human-intelligence (humint) collectors were told to stand down and not talk to “bad” people, a deficit from which the agency had to recover after 9/11. We can create a similar effect now if we tell signals-intelligence (sigint) collectors in the NSA that they cannot listen to any “good” people.”
Hayden’s “good” people reference is surely about wiretapping our allies like German Chancellor Angela Merkel, but Hayden’s evidently unwitting ironic admission also fits the very definition of the collection of all of the “good” American public’s phone records.
@RHIL:
this is a helpful discussion of nsa software.
however, henschen’s sly implication that the fact that the accumulo software is proven and available software implies there are few serious reasons to question nsa’s behavior raises questions in my mind about his credulity and good judgement.
and his phrase “privacy hawks” is pure propaganda.
@Snoopdido:
“..going forward we need to remember how U.S. intelligence suffered a similar crisis of conscience in the 1990s when the CIA’s human-intelligence (humint) collectors were told to stand down and not talk to “bad” people, a deficit from which the agency had to recover after 9/11…”
i have never heard this tale told before. told by whom? “talk to bad people”? the nsa at least doesn’t talk; it listens.
the nsa/cia had the info in hand to stop the wtc bombing and refused to release it to the fbi. it wasn’t restrictions on them; it was their own bureaucratic fuccup.
the nsa’s (which is the organization of interest at the moment) “crisis of conscience” wasn’t in the 1990’s; it was in the 1970’s when it was discovered the nsa had broken their charter and broken faith with the american people and gone after american anti-war protesters at the behest of the president.
@orionATL: The NSA doesn’t run on Crays. It runs on massive, globally dispersed Linux clusters. If one hadn’t already surmised that, Snowden’s documents confirm it.
@RHIL: Thanks for the link. There’s also this: http://gigaom.com/2013/06/07/under-the-covers-of-the-nsas-big-data-effort/
Back to what the database is for, there’s this: (pdf) http://arxiv.org/pdf/1310.6753v1.pdf. (NY Times version here: http://bits.blogs.nytimes.com/2013/10/28/spotting-romantic-relationships-on-facebook/?_r=0 ) In it, a Facebook engineer and a Cornell computer scientist demonstrate the ability to figure out who your romantic partner is from the structure of your Facebook social graph. The NSA is certainly doing things like this, too. There is tremendous predictive value in this information, assuming you have the data and the capabilities to process it. If they happen to have multiple social graphs – Facebook, phone, email, twitter, whatever – they have far richer information than this paper worked with. That’s another reason you hoover all this stuff up.
Would an organization acting in good faith spend as much effort as the national security apparatus has spent (over decades) in misleading Congress and the courts?
There also are myopia and obtuseness at work here. Just as for General Curtis LeMay every geopolitical problem could be solved with air strikes, so Clapper, Alexander, et al are convinced that the unlimited collection of data will protect the US, for the simple reason that that’s what they do for a living.
And let’s not overlook the power-tripping elation that comes with running an agency that’s beyond reach. Which gets to the core of the issue: if its power remains unrestrained, the apparatus will continue at will. The question is whether anyone has the mettle to do the substantive things necessary to wrest power back from an apparatus that has zero intention of ceding it voluntarily.
@Saul Tannenbaum:
i’m glad to learn about the nsa’s computering source, but i was not referring to the nsa, but rather to the pdf you posted. the pdf does note, towards the back, precisely those two machines and their power requirements, which astonished (and informed) me. from that i guessed that the nsa might be working near the limits of available electrical and computing power, but maybe not.
The thing about models is that they are in general 99% wrong. So when you create a profile using a model you are pretty much going to get primarily wrong answers. And, in order for the answer to be statistically valid you need at least 100 right answers to even prove that your model works. 100 terrorists all with similar characteristics. Good luck NSA!!
@orionATL: That pdf seems to be structured to convince someone that, for graph data analysis, cloud computing works better than massive supercomputing.
The thing about cloud architectures like the NSA’s (and Google’s, etc) is that you can scale out massively and put data centers near good power sources, making former limits fall away.
@Saul Tannenbaum: & Orion – NSA’s first Cray along with their closet sized gb tape storage are on display at their museum. It is outside the gate at Ft. Meade, you can go see it.
“The NSA… runs on massive, globally dispersed Linux clusters” should make those folks believing that Linux will keep them safe from snooping have second thoughts, not quite as naive as their first.
“these nsa guys have got to be skating right at the edge of possible.” – and they’ve made their bones over the years again and again by moving the edge.
The real issues are law, policy and oversight as DiFi’s moronic legislation in the headlines today illustrates.
The hubris part requires a minor change: “for such time as the government plans to unleash the nuclear bomb of reading every relationship in the world [country].”
Thank you for this. (Atrios’s answers to the posited questions were pathetic.)
This is absolutely about private company-generated priorities. These include maximizing their power and influence over government and its intel operations. Just as this army can’t go to war without beds, food, showers and fuel supplied by outside contractors – at hugely profitable rates – vendors collectively want the government dependent on their services for data collection, storage, analysis and use. That’s an enormously powerful and, hence, profitable arrangement.
Expansion, growth are essential to this in several ways. It does far more than populate the database and excuse billion dollar, privacy invading tinkering with it. Other effects would include these: Normalizing (a role patented by Mr. Obama) the role of business in so-called public/private partnerships. Normalizing the information flow to them and the dependency on them to make and carry out normally governmental decisions. Normalizing the deletion of reasonable and customary public expectations of privacy and a modest role in influencing government conduct.
Well, you see, we’ve got all these contracts, you understand, with all of these contractors, and we owe it to them to not renege on the terms of the contracts and to fulfill our full obligations therein, of these contracts that is, which by the way the contractors themselves wrote, and these obligations include the complete digital de-opacification of all information from all sources for whatever purposes this unprecedented procurement may or may not entail. It’s all good! It’s! All! Good!
There are two ways for this to end.
(1) Elect a congress which defunds the NSA entirely and sacks everyone — same for the CIA and most of the other unintelligence agencies — while giving a couple billion in funding to a special grand jury to prosecute all the criminals and traitors like Alexander and Clapper.
(2) The unwieldly insecurity agencies will collapse during any revolution, coup, civil war, or invasion, and if the US is mismanaged long enough, we’ll get one.