I’m working on some updates to this, but in the meantime, here’s a great post suggesting what the original logic of hte data collection program was. Apparently, AT&T didn’t adequately hide the following claims made in the Hepting case;

â€AT&T notes that the facts recited by plaintiffs are entirely consistent with any number of legitimate Internet monitoring systems, such as those used to detect viruses and stop hackers,†the redacted pages say.

In other words, I think that’s what the original justification for letting the NSA into the circuits (it does have a defensive role like this, and it is probably necessarily from a technical standpoint).

kim

They’re collecting the data from the circuits, from the roughly ten places in the country that all data have to physically pass through. So it’s not like they have to go to different places to collect the data. Also, they’ve got a contractor helping with the monitoring process, using some fancy software.

Golly, Moses! This conjures up the image of millions of grimlins hovering over computers to catch names. Surely, even if the computers sift out all the chaff, there is too much â€wheat†for analysts to â€eat.†And, how many of these grimlims speak Arabic?

Considering how inefficient government bureaucracy is, I haven’t much fear about it, in general. However, the creepiness of it all
is offensive. Data pornography.

They’re also collecting overseas (the major responsibility of the NSA it still overseas I hope!), as well as in the US… they have to store the raw data (not just connection info) before analysing it, they also have to be able to integrate all locations, then be able to data mine with changing criteria, then probably search for specific words or voices… I guess all this happens after bringing it all back to MD. Sometimes it seems like they’d need an entirely separate second â€shadow†net to accomplish this.

ew,

I suspect that your taxonomy is incorrect.

First, the data vacuuming (as described by Slate and USA Today) is pointless if there is not an associated data mining effort. Actually, calling this data mining is somewhat misleading (in the same way that there are no wires involved in the â€wiretappingâ€). What we’re really talking about is a sophisticated pattern recognition system that differs in some important ways from what â€data mining†has traditionally meant in software circles, but probably on real geeks care about that.

Second, there is a scenario that would make the pre 9-11 data collection efforts legal and legitimate. Unfortunately, we’ll probably never know for sure. On the other hand, we can be pretty sure that the way the program was used after 9-11 was illegal, unconstitutional, and absolutely ineffective. I don’t have time to back up that assertion now, but if anybody seriously disagrees, I’ll get back to it later.

Third, if the warrantless wiretapping is illegal and unconstitutional, the data collection effort is assuredly illegal (and probably unconstitutional, but that’s not clear to me). Even after the â€Patriot Act†changes, collecting call detail records still requires identifying the specific individual that the government is interested in. This is the basis for my response to Kagro X’s post. I don’t think it matters that the judge accepted the â€state secrets†claim with respect to the data mining. If her decision stands, the NSA will shut down the data collection effort.

Fourth, based on what Alberto Gonzales refused to say at the Senate Judiciary Committee hearing, what Hayden said in his defense of the program and what Jay Rockefeller implied in his â€memo to fileâ€, I believe that the Administration is attempting to foist off a false distinction. After 9-11, the NSA merged these two programs into a single effort. The data collection effort doesn’t just involve internet traffic, the wiretapping isn’t just about voice communications, and the patterns aren’t based on analysis of terrorists’ calling patterns. as difficult as I find to say this, I think Harman is right about her description of the program. I wish she was more helpful in seeing this for what it really is: An authoritarian abuse of our civil liberties.

To sum up, although the data collection started earlier than the warrantless wiretapping, the wiretapping should be seen as an outgrowth (and a particularly malignant one at that) of the earlier effort.

WO, I agree with everything you say, particularly your distinction between data mining and pattern recognition. I suspect they’ve used the word â€data mining†as a bit of a red herring that is more familiar to people. But I’ve tried to make this simple, at the risk of losing the nuance you’re pointing to, and for that reason I chose the language both the judges and the press are using to refer to this part of the program.

But I don’t think that challenges my taxonomy itself. The laws are still different, and the basis for court challenges (both potential and actual, what we’ve seen) are different. You and I have standing in a class action case against AT&T. It’d be a lot harder to argue we have standing for a case against the NSA on the illegal wiretapping. This distinction is important to make, not least because there are different laws at stake. We know, for sure, that the illegal wiretaps violate FISA. But the AT&T case is more ambiguous.

And I suspect there are actually two different factors that would be more accurate to include. First, there’s the presence of NSA on the network. Its cybersecurity function alone would justify this, and it would also allow the telecoms to claim a proper business function (that is, to access pen register data for the normal functioning of their business–if they say they’re doing it to prevent hackers, they can at least argue the case; as the first comment I made points out, that appears to be what they were prepared to do). Furthermore, NSA’s mandate to spy internationally requires that it have access to the circuits–you can’t tap using traditional analog methods anymore and presence in the circuits is the only thing that’ll give you that access and the only place to get it is here in the US.

Then there’s the collection of all data, domestic and international. This is where you start violating Electronic Communications Privacy Act because you’re collecting domestic as well as international traffic (FWIW, one of Lichtblau’s and Risen’s articles included a quote that suggested this was the beginning of the end, the technical impossibility to separate international and domestic data).

Then there’s the data mining (pattern recognition). The reason this is important, in this program, because no one has ever made a legal case for the validity of pattern recognition as a justification for probable cause. BushCo had the option to try, and thereby to continue this surveillance under FISA. But they didn’t do so, apparently because they were warned (I think by the FISA judges) it wouldn’t fly.

And then there’s the wiretapping. Again, it’s legally distinct, because this is the first time they’re going beyond call data to call content (ambiguities about whether Internet â€call dataâ€â€“things like email headers and web searches) can be considered just register data aside).

And I do’t see how what you say makes Jane Harman correct. There are two cases challenging the illegal wiretapping program. One has declared it illegal, the other hasn’t yet ruled. All the other decisions are irrelevant to this program. The illegal wiretapping is a subset of all the activities (some legally defensble and arguably necessary, some illegal) made possible by the collection of the data. A ruling on the collection of the data (which we haven’t had yet) would affect all the programs subsidiary to it, but a ruling on the subset of illegal wiretapping does not affect the larger class of activities associated with the data collection.

But they were able to look back into global conversations after 911 and find references to an attack, specific words referring to an attach, right, are any other anecdotal examples of this sort of thing? I’m also sure that they use frequency-spectrum type voice recognition to search conversations for particular targets.

WO

Let me make my point a different way.

You and both agree that the warrantless wiretap program is a subset of a bunch of activity made possible by the collection of data.

The ruling yesterday pertained to the wiretapping process. I need to go read the ruling, but from what I’ve heard, she’s talking about the legal process of tapping. I don’t think she addressed the means by which they got to that tap.

Thus far, no one has issued a ruling on the AT&T cases, beyond whether or not they can go forward (that is, whether or not the Administration’s state secrets claim holds up). If those cases rule that that data collection was illegal (and not just a violation of written privacy policies, for example), then the warrantless wiretap program will also be illegal, because it requires the data collection. But that hasn’t happened. The only ruling thus far affects a subset of the activity.

Although it’s all of course mostly unknown, I’ll stand by my belief that the NSA can store and scan conversations for words and particular voices from the very beginning of the process (globally and within the USA). One argument would be that actual capabilities are always well beyond public info and this is particularly true for digital technology – and ignoring mundane things like the Baltimore power grid going down.

Update: the text is now posted to Harman’s website.

WO,

The problem with the data mining part of the equation is that so long as the state secrets privilege bars justiciability, the question of its legality is moot.

What shuts down data mining? Assuming the generally understood function of law, an order from a federal court shuts down data mining.

What prevents the issuance of a federal court order shutting down data mining? Its non-justiciability under the state secrets doctrine.

Nobody will infer its illegality from the parallel illegality of the TSP. Least of all this â€administration,†which continues to insist that Hamdan doesn’t touch torture or the â€inherent powers†doctrine.

Kagro X,

Patience. I’m getting there. This is complicated. I don’t disagree with your judgment of the this â€administration†and its lack of respect for the law.

Next piece of information: The call detail information. From a NYT article in May 2006:

One senior government official, who was granted anonymity to speak publicly about the classified program, confirmed that the N.S.A. had access to records of most telephone calls in the United States. But the official said the call records were used for the limited purpose of tracing regular contacts of â€known bad guys.â€

â€To perform such traces,†the official said, â€you’d have to have all the calls or most of them. But you wouldn’t be interested in the vast majority of them.â€

That’s not data mining. That’s collecting a bunch of information in advance so that when you identify a suspect, you don’t have to go get at warrant to get his call detail records. The â€data mining†is being applied to the internet traffic (where they have all the details and the data is in a digital format more amenable to software analysis).

Remember the stories about the FBI being given a bunch of leads to pizza places, etc.? The call detail database is where that crap came from. So, the question gets back to this: How do those shift supervisors that Hayden talked about decide which calls to listen in on? I think the easiest way to answer that question is to look it how the NSA has operated in the past with its surveillance of foreign communications. We know that what they like to do grab all the communications traffic they can and then pick out the stuff they find interesting.

Here’s what I think the program looks like. The NSA has tapped into to all voice communications going into and out of the US. They filter through it based on a database of phone numbers they’ve built (through the internet program, through captured cell phones, human intelligence, whatever). When they get a hit (i.e. one of the numbers in their database) makes or receives an overseas call, they record the call. If it is an call to or from a number that’s not already in their database, it gets added to their database. If the number is a domestic number, they’ll run it through the call detail database, looking for other numbers to add. They know they can’t ever use any of this stuff to get a warrant, even from the FISC. If the court pulls the plug on the warrantless eavesdropping, the whole program is essentially worthless. It just won’t be cost-effective.

How long does it take to run a domestic number through the call detail database, looking for other numbers to add?

Less than 72 hours?

If so, then who cares if you lose on your warrant request before the FISC?

But they did tell the court about the detail database. And then they told them it was covered by the state secrets privilege, and Judge Taylor agreed. So there it is, illegal or not, but either way untouchable.

And wiretaps are for people who intend to take you to court. What are the chances your buddy’s cousin sees the inside of a courtroom? Or indeed the outside of anything? Ever again?

Even without that little diversion, the claim is that the legality of the database is nonjusticiable. So long as that cover remains intact, they can do what they like. And I’m not even entirely sure that evidence that an illegal but nonjusticiable program uncovers couldn’t provide probable cause for a FISC warrant. What if a defendant attacked the warrant as without foundation, since the program that produced the probable cause is illegal, but no judge was willing to pass on the program’s legality because they all felt bound to recognize the state secrets privilege?

EW – re ’falafel searches, here’s Risen:

That is essentially what Echelon is – Echelon is the public name for what the NSA has been doing overseas for many years. They have been doing keyword searches all over the world, outside of the US in the past – using their vast computing power and their ability to monitor foreign communications. Now it’s possible they are doing the same thing inside the US. (link)

ew,

Here’s where I think we disagree. I don’t think there is any way that the collection of the call detail records by the government is legal. Even as amended by the odious â€Patriot†act, the government has to specify a particular individual when requesting call detail records from carriers. Qwest is entirely correct in refusing to play along. As an aside, it is possible that the class action suit can breach the â€state secrets†claim by using the public statements of the (former) Qwest officials. In any event, I wish there were more judges who understood technology, because there is absolutely nothing in this program that needs to be protected for technological reasons.

Furthermore, I think by buying into the Administration’s frame that there are multiple programs, we prolong the ability of the government to claim â€state secretsâ€. If you take a look back at the evolution of their public statements, you can see that the â€multiple programs†lie was ginned up when they realized that the AUMF rationale wouldn’t work and, that by letting Bush brag about his authoritarian toy, they had screwed themselves on the â€state secrets†defense.

Here’s why I think the recent decision is worth uncorking your champagne (or in my case, a nice bottle of Fuller’s ESB). Three critical things were decided in our (the people’s) favor. First, she ruled that somebody had standing without having to prove the more or less impossible (that they were definitely spied on). Second, she ruled that by talking about the program publicly the government had given up its â€state secretsâ€. Third, she ruled that the program was unconstitutional, meaning that there is no way for this hapless Congress to â€fix†the problem.

I certainly realize that any one of these issues could get reversed on appeal, but we should celebrate our victories when we get the chance. This battle with the authoritarian impulse will never end. If you wait for the final victory, you’ll never get to celebrate.