Monday’s document release provided mounting evidence that when the hospital confrontation “heroes” moved the Internet dragnet they had deemed to be illegal under the auspices of the FISA Court, neither they, nor Judge Colleen Kollar-Kotelly believed it was legally sound. But they traded those truly crummy legal claims to bring the program under court oversight. Since then, boosters of the scheme have claimed the oversight serves to eliminate violations quickly.

We already knew that’s not true.

Still, Monday’s release — particularly this John Bates opinion written around July 2010 — makes that even more clear. After Kollar-Kotelly sacrificed judicial wisdom for court oversight on July 14, 2004, the government continued breaking the court’s rules for five years, until Reggie Walton shut the program down, sometime in fall 2009.

First, let’s lay out the dates. I’ve done a rough timeline below, based on the known start-date (July 14, 2004) and the rough end point with John Bates’ opinion (around July 2010). The bulk of the other dates impose the timeline laid out in the Bates opinion on a few known dates taken from the phone dragnet production (plus, the geniuses at ODNI not only left the date of the June 22 Internet dragnet order in its URL (CLEANED101.%20Order%20and%20Supplemental%20Order%20%286-22-09%29-sealed.pdf), but it’s the same document as the June 22 phone dragnet order, which has different redactions but most dates intact — see the three bolded entries below).

As you’ll see, there were two known violations in the Internet dragnet before the before the discoveries of the problems started in earnest in 2009. That’s not that big a deal — there was at least one phone violation before 2009 too, except in the case of the Internet dragnet, NSA overcollected from the very start.

The examination of the Internet dragnet started in response to the first phone dragnet disclosures in January 2009 (with the change in Administration, it should be remembered). Reggie Walton told NSA to see if the Internet dragnet had the same compliance problems as the phone dragnet did.

From that point until June 2009, the discoveries seemed to work in parallel (the NSA was working on End-to-End reports for both programs at the same time, and they share some common databases). But with the discovery that both dragnet programs were sharing information freely with other agencies, it became clear the violations were much worse on the Internet dragnet side, with reports going out with US person information that did not even remotely comply with minimization requirements.

Then sometime after that — and after Walton issued what would be the last Internet dragnet order for a year (that was sometime after June 22, 2009) — NSA discovered they had been receiving “metadata” far outside the permitted scope, which surely included content. Note this may have happened around the same time as NSA reported that one phone provider had overproduced (including international data in addition to domestic, I think) on July 9, 2009, so I wonder if they were only then reviewing returned data on receipt.

In any case, it was around that time that NSA “discovered” the Internet metadata program had never ever been in compliance. From Bates:

Notwithstanding this and many similar prior representations [made on the summer 2009 reauthorization] there in fact had been systemic overcollection since [redacted]. On [redacted] the government provided written notice of yet another form of substantial non-compliance discovered by NSA OGC on [redacted] this time involving the acquisition of information beyond the [redacted] authorized categories.

[snip]

This overcollection, which had occurred continuously since the initial authorization in [redacted] included the acquisition of [long redaction]. [my emphasis]

Never.

If my math is correct, the application the NSA withdrew was submitted not long after September 20. There are briefings for the Intelligence Committees that likely alerted them to the scale of the Internet dragnet problems around that time. But as of October 5, some of the most assertive House Judiciary members seem to have had no idea about the problems with the Internet dragnet. If they found out about it with the notice to Congress on December 17, 2009, it explains why the PATRIOT Act reauthorization process stalled.

There’s one more very important thing in this timeline. You’ll see below that almost at exactly the same time as NSA “realized” it had never complied with program requirements, it started a pilot project that would be rolled out on January 3, 2011, analyzing metadata with no special protections for US persons or limit for use only on counterterrorism.

Specifically, these new procedures permit contact chaining, and other analysis, from and through any selector, irrespective of nationality or location, in order to follow or discover valid foreign intelligence targets. (Formerly analysts were required to determine whether or not selectors were associated with US communicants.)

[snip]

In the second place it enables large-scale graph analysis on very large sets of communications metadata vwithout having to check foreignness of every node or address in the graph. Analysts in S2 have used this to great benefit over the past year and a half under a pilot program. [emphasis original]

In other words, at the moment they were coming clean with the FISC that they had never ever complied with the PR/TT orders, they were beginning the pilot project that would move metadata collection overseas, under EO 12333. (This document goes back to this NYT story on social network analysis.)

So much for the notion that putting all this under court oversight would accomplish a damn thing. All it did was degrade the law and provide NSA cover until they developed the technology to do all this overseas.

Update, 11/22: More dates added to timeline.

Update, 11/26: More dates added to timeline.

---

July 14, 2004: Colleen Kollar-Kotelly approves Internet dragnet, specifies categories of metadata

Before October 12, 2004: the government provides notice it exceeded scope included in first order, in follow-up declarations attributes overcollection to poor management (response probably includes Paul Wolfowitz, Michael Hayden, and Joel Brenner)

Around October 12, 2004: Government reapplies without some collection, promises monthly spot checks

November 17, 2007: Executive begins (internal) approval process for contact chaining on already-collected data.

December, year uncertain: Another compliance problem due to typographical error; government asks FISC to adjust order, but Court refuses

January 15, 2009: DOJ reports phone dragnet compliance problem to FISC

January 28, 2009: Walton order in response to phone dragnet violations requires a report by February 15; Walton also orders NSA to check Internet dragnet for similar problems:

The Honorable Reggie B. Walton of this Court ordered the government to verify that access to the bulk PR/TT metadata complied with comparable restrictions, noting “the similarity between the querying practices and requirements employed”

February 4, 2009: DNI Blair receives more info from DNI General Counsel Benjamin Powell

February 10, 2009: USD/I James Clapper briefed on problems

February 12, 2009: DNI Blair receives more information;

February 15 (17), 2009: submission to FISC in phone dragnet notes NSA will conduct a similar review of other sensitive programs

SIGINT Director has directed similar reviews for some of the other sensitive activities NSA undertakes pursuant to its SIGINT authorities, to include certain activities that are regulated by the FISA, such as NSA’s analysis of data received pursuant to the [redacted] If the Agency identifies any compliance issues related to activities undertaken pursuant to FISC authorization, NSA will bring such issues to the attention of DoJ and the Court.

Before February 25, 2009: NSA alerts FISC of manual queries involving US persons who had been in contact with RAS approved selectors; Walton authorizes continued Internet dragnet surveillance; government announces End-to-End report on Internet dragnet

February 25, 2009: Congressional notification regarding both phone and Internet dragnet programs; only one violation of Internet dragnet identified

March 2, 2009: Walton phone dragnet order

March 5, 2009: Submission to Congressional Committees on significant FISC filings, including both Section 215 and pen register; includes February 15 phone dragnet submission and March 2 order, probably includes Walton renewal of Internet dragnet

March 31, 2009: DOJ lawyers do first spot check of PR/TT program; NSA’s own systemic check had found nothing

April 10, 2009: According to a notification to Congress, NSA had not yet found major violations in PR/TT

May 7, 2009: Congressional notification regarding implementation of Section 215 authority (does not mention Internet dragnet)

May 8, 2009: Notice to FISC of phone dragnet defeat list

May 2-12, 2009: Quarantining of phone dragnet FISA derived defeat list terms

Before May 29, 2009: NSA alerts FISC of using identifiers that met RAS but didn’t have tie to foreign power, defeat list tech work (perhaps on May 8 in conjunction with same issue on phone dragnet side?)

May 29, 2009: Walton opinion reflecting two additional violations, addressing both phone and Internet dragnet

June 12, 2009: NSA alerts Congress to Internet dragnet master defeat list

June 16, 2009: NSA notifies of access by CIA, FBI, and NCTC to both the phone and Internet dragnet databases

June 22, 2009: FISC Order on dissemination outside of NSA (phone dragnet version; Internet dragnet version)

June 25, 2009: Phone dragnet End-to-End report

June 29, 2009: While providing phone dragnet End-to-End report (which still needed one new section), NSA tells Congress it has recently started Internet dragnet End-to-End report

After June 22, 2009 (and after June 29, 2009): NSA finished Internet End-to-End report, Walton approves new Primary Order

July 9, 2009: Walton halts production of some phone dragnet production (probably one provider who provided international data)

Unknown date: NSA alerts FISC to substantial overproduction violation that has continued since program’s inception; ceases querying and receipt of metadata (though possibly only some of it)

~July 2009: Pilot program on new contact chaining begins

August 4, 2009: Discovery of fourth hop in beta test in phone dragnet

August 19, 2009: Phone dragnet submission to FISC

September 1, 2009: Briefing materials for FISC

September 3, 2009: Phone dragnet Primary order

September 3, 2009: Submission to Congressional Committees regarding various matters, including implementation of Section 215 authority. [Cover released]

September 10, 2009: Notification to a Congressional Committee regarding implementation of Section 215 authority

September 14, 2009: DOJ provides recommendations Feinstein and Bond asked for in March; also provides to Pat Leahy (claiming only that SSCI members aware of secret pograms)

After September 20, 2009: Government submits new application in Internet metadata; after Walton expresses concern, government chooses not to submit final application

October 5, 2009: House Judiciary Committee members Conyers, Nadler, Scott only know of problems with phone dragnet, not Internet dragnet

October 19, 2009: FBI General Counsel Valerie Caproni briefs Senate Judiciary Committee members in closed session, and SJC and Senate Intelligence Committee staffers on PATRIOT’s expiring provisions

October 21, 2009: Statement for the record before a Congressional Committee closed hearing on PATRIOT Reauthorization

November 2009: Per training program (page 15), date before which Internet dragnet data must received special treatment (suggesting collection cut off in late October or November)

December 17, 2009: Letter to Conyers, Nadler, Scott refusing to make public more on Section 215; Letter to Intelligence Committee Chairs admitting phone and Internet dragnet

December 17, 2009: Latest possible date before Internet dragnet expired (120 days after June 22)

Before July 2010: Government submits application substantially similar to July 2010 one

est. July 2010: Application that would lead to Bates opinion (see post for explanation on date)

October 2010: Date after which PRTT data stored/treated differently (see page 15)

November 29, 2010: NSA signs management directive rolling out new metadata program

December 1, 2010: In notice to SSCI, NSA references FISC opinion describing approved categories of Internet metadata; also reveals test geolocation program

January 3, 2011: NSA rolls out new contact-chaining approach, using EO 12333 collected data without restrictions on either foreign intelligence purpose or nationality