Working Thread on WH EMails "Compliance"
Much of this is Greek to me–and it’ll take a while to upload it all–but I’m sure MadDog and WO can make some sense out of what the Bush Administration says is proof they’re complying with requirements to save their emails. Here’s the document explaining what this inventory is.
My very first glance at this makes me think this is an effort to drown the court and archivist with data. I’ll look closely now.
Do you use a snorkle when diving through this stuff?
I realize that this is nearly a hundred comments later, but I couldn’t resist:
WO’s prolly wearin’ a HAZ Matt suit…
I’m going to have to add the rest of them in threads. Geneva 2.
Innovative 1
btw, your first two links point to the same document.
Innovative 2
Here’s that second one:
Copy Tapes
Legato Tapes
What I find interesting is some of these are dated, and some aren;t. And note hits one has some tapes as “out.”
Sequester 1
Sequester 2
Note this one says it is web output–has some dates (but not years).
And is it normal to be using so many different kinds of media to save stuff?
It does seem a little odd to have that many different tapes for an organization the size of the EOP.
So some of these are company names, right? EDM?
ANd then others are “type” names: damaged, possibly sequester.
any idea what Geneva is?
Legato Networker and EDM are both brands of backup software (both owned by EMC now, I think). Checking on the others.
I sure hope it isn’t this:
Here’s another possibility. I located these in some job ads for computer analysts.
It’s enough to make one think systems were replaced or duplicated at a whim in an effort to make an accurate catalog of the president’s business impossible to assemble, isn’t it?
The types of media would be based upon the drives, the actual hardware, on each server. Yes, it is unfortunately common that different servers have different media. Also makes it harder to reconstruct the data. Other than Geneva’s list, nothing here identifies which tape goes with which server on which date. Time for a renewed discovery motion.
Sequester 3
Standalone
Ah, this might help figure this out:
Tape Category Number of Boxes *Quantity
Networker/Legato 144 14,058
EDM 565 25,543
VMS/ARMS 265 20,189
Standalone 68 4,825
Geneva 1 62
Copy Tapes 1 14
Sequester Set 1 33 1,888
Sequester Set 2 23 2,982
Sequester Set 3 49 3,968
Innovative 1 49
Damaged Tapes 1 16
Total 1151 73,594
Note EDM box count include originals and copies
Can you find this document:
Just did–should have opened that one first.
Here’s the PDF of that last one: Tape Master Inventory
VMS 1
Last one!
VMS 2
VMS 1
VMS 2
Looks like a manual log kept in excel that tracks what tapes “Volser” are checked in or out and what box number they live in when checked in.
VMS 1 has five columns, column 1 (sequential line count ) A “Volser #“, B “IN“, C “OUT“, D “BOX #“. The clerk enters an “X” in the IN or OUT column to indicate the “Volser” us checked-in or checked-out.
VMS 2 has those same columns, A,B,C,D plus columns S,T,V. The only the info in S,T,V columns is the word “Damaged” associated with a particular “Volser”, otherwise these columns are blank.
It looks like VMS 2 was printed so that columns E-R were set to HIDE so that any data in those coluns ould not be printed. The ssame could be true for VMS 1.
Some tapes are still checked out.
One might wonder whether hidden column indicate who the tapes were checked out to.
VMS is the operating system for the Digital VAX (minicomputer). VMS has it’s own mail utility that can send/receive over the Internet but VMS can also act as a file server for a PC network. It would be surprising if Executive Office PCs loaded drivers for Microsoft networks/Novell Networks AND VMS networks. The VMS machine could also be the platform for the archive.
I’ve forgotten a crapload of VMS — especially backup — honestly think I am unconsciously blocking it because it was so tedious. You brought back some of it with your comment, thanks (I think).
Volser = VOLSER, volume serial number, wasn’t typically used as a reference in VAX VMS environments, though. Must be a holdover from a point earlier when everything would have been mainframe. Would be the ID of the tape, but it could explain too why we don’t see “volsers” across all of the tapes uniformly.
I don’t know that the E-R columns are hidden; for some reason the letters S, T, V jog a memory, like they are headings not row identifications. They’d be relevant to a VAX/VMS system, but I can’t bloody remember what they mean.
S = system? stored?
T = tape?
V = vault, volume?
Only time I can ever remember regretting leaving behind my enormous VAX/VMS manual at the big IT firm…
Rayne, I used VMS BACKUP with reel to reel media and Digitals “next generation” DECtape casette (4gb? 8gb? 64GB? cartridges.) Both used the same software VMS BACKUP just different driver for the device and media.
Reel to Reel tapes got a software label when they were “initialized” up to eight characters long. To format, you would enter something like this.
$ format/drive=$MTA0:/density=2400/volume=”12345678″
I think VMS BACKUP could also label it for you.
$ backup/drive=$MTA0:/Volume_label=”123456″ $DISK2:[*]*.*;* MTA0: /FULL
After initializing the tape you’d write the label on the cover too so you could pull the tape you needed.
I don’t remember DEC documentation using the term “Volser” but that’s not surprising that a shop would have its own terminology, especially a multi-vendor shop.
I worked on VMS at college, at DEC, at an Investment Company, and a law firm.
I was the guy at the firm who was chosen to produce electronic document discovery in response to a subpoena for materials relating to a case one of the partners was named in. Can I tall you how much it sucked to mount, tape by tape, the firms complete inventory of backup and archive tapes in search of documents relating to a client/matter? That was the beginning of my awareness about the nature of electronic documents, the importance of making deliberate decisions about retention even in a business that has the protection of attorney/client privilege.
NARA has been provided that database, although OA cannot provide a copy to the Court because of the size of the database and the sensitive information that could be disclosed in many of the file names.
Sensitive file name e.g. Non-Bushie US Attorneys we plan to fire.doc
VMS is a term for a IBM mainframe operating system. I know the old IBM mail system PROFS used to run on VMS. I have heard of companies using Lotus Notes for e-mail on a mainframe, but its not a typical practice. There could be some advantages to scale, and there certainly would be advantages from a security perspective.
Not sure if that helps.
Not IBM, DEC. The old (pre-Bush43) recordms management system ran on VMS.
IBM’s VM Ran Profs. If its VM it will be in VM DDR Format.
VMS is a Dec operating system.
Legato is a storage back up system from EMC.
Thats right Vax/VMS IBM – VM. Using VMS anytime in the last 8 years would be really weird.
It’s not out of the realm of possibility. When the Bushies took office I was still using VMS backup systems as well as a VAX production system for a small manufacturing plant.
But we weren’t running email on it, we were using HP tape drives on HP servers in 2000/2001 time frame.
Have to remember these guys deliberately avoided moving to new technologies because they wanted stability — not to mention the contracts with providers would encourage “stickiness” of technology.
Quick take. The descriptions in the filing to the court emphasize the physical tapes themselves, largely “back-up tapes”, not whether the documents recorded on them comply with the requirements or include any or all of the documents the court asked for.
Deletions include “sensitive” server information, which might include, for example, details on the identity of the RNC or other non-White House, non-official government servers. That information alone seems relevant to whether the White House complied with the PRA.
Just the list of “every file” on 26,000 backup tapes supposedly creates a database of 6 terabytes.
I thought we’d been through the malarky that the executive branch cannot give info. to the judicial branch because it might be “sensitive”, however the hell Richard Bruce Cheney defines it.
Short take, the White House is telling the court, “Here it is”, while doing a Clinton by not defining “it”. They seem to be just keeping the ball rolling, in order either to run out the clock, interest, or the money needed to pursue them.
If this were a corporate defendant, the DOJ would take the position it wasn’t acting in good faith or complying with the requirements of the old sentencing guidelines, subjecting them to more severe penalties.
I think you’re right on the money with this comment. I can’t tell if they just threw this stuff together haphazardly or intentionally mucked it up to make it impossible to understand. I’m betting a little bit of both
+1 on the tapes-are-not-emails point. Tapes can be intentionally bulk-erased, or inadvertently made unreadable through exposure to humidity, weather, magnetic fields, etc. There are many many potential points of failure to pass through between a tape inventory document and a recovered email of evidentiary interest. So what producing the inventory proves almost attains the high exalted threshold of squat.
All of these pdfs are printed from Excel spreadsheets. If you look in Adobe at the Document Properties, the titles all end in .xls. Yet, the formatting is wildly varying, sometimes within the same document. If you look at p. 66 of sequester-3, you’ll even see that they hid a bunch of columns before they printed out one section [column headings skip from B to AE].
Genson is quiting Blago.
http://www.nytimes.com/aponlin…..enson.html
Sam E. Adams done the job, wittingly or not is not clear.
but then what is clear where blago is concerned.
Blago to boycott own impeachment because impeachment is unfair. Yeh, that’ll show ‘em all right.
This is for criminal now.
I know, I was referring to the impeachment boycott more as another facet in the fascinating, sparkling gem that is Blago’s ever-destructing self.
Shit. Those DLT-IV tapes are also circa VMS/VAX era, old DEC machines.
I remember them well, too.
A quick OT on the earlier Tice/FISA/TIA thread… Jello Jay told Tweety today…
Saw that from yesterday. Need to come back to it.
Must be Friday today when a enormous docudump like this shows up at EW’s place. *g*
Keeping you busy, are they? LOL!
Questions we should be asking:
Multiple flavors of tape backup machines — why?
Are there as many server types that these machines support?
Which departments or offices were served by which kinds of servers and backups?
Were there more than one email system in operation at a time, across different platforms?
Not that the answers to the questions would indicate any criminal behavior, only the possibility that we are being snowed under with a different kind of greymail, and//or that the problems of missing mail are compounded by a patchwork of multiple platforms and generations of software/hardware.
We’ve got tapes from:
DEC (Digital Equipment Corp. now HP) — multiple generations
IBM — 3480 and 3590
HP (or Dell) — LTO1, LTO3
Have no idea what the 4mm and 8mm could be, many different options.
The different types of tape backups reflect three things: (1) there are normally different tape drives for different servers – just look at a current Dell or HP catalog; (2) tape drives evolve over time, such as the 4mm and 8mm DAT drives, with different capacities too; (3) some software (O/S or third party) only supports some drives and not others for a given environment.
The hardware can support many different operating systems, Unix/Linux, VMS, NT, W2K, etc. One problem, due to federal policy, is that without ADMIN permissions, most of the data will be unuseable, even if you read it and restore it to a new disk drive. The reason is that it will be encrypted, and on NT/W2K machines, it may be encrypted with a different key for each user. The good news is that deleted e-mails can most often be reconstructed.
The which departments and which offices question can only be answered by the IT guys for the EOP.
Odds are there were several email systems operational at the same time on different servers, for different clients. I think I read about that once in Information Week or something.
None of the above information or questions remotely nears criminal behavior. The fact that vast majorities of official business was conducted using RNC email accounts, and that those servers are completely outside the scope of this inquiry, is criminal (Hatch Act), and also as obstruction of justice.
I think you’re assuming these guys are state of the art. They are not.
I think you’re also assuming that most places of business will upgrade regularly. They don’t.
And they clearly were not top shelf experts in security, or they wouldn’t have such a hodge-podge of old crap; they’d be more worried about disaster recovery if they were.
This stuff is likely quite readable by people who are IT forensics, IMO. The only concern I’d have is that there is some obscure custom software package and not a commercial software package involved in this mix.
Frankly, this looks a lot like something an IT department might run into if they acquired another firm with multiple sites. There might be different servers in different buildings and/or departments, depending on budgeting and on volume of use as well as management. There might be multiple locations where backup devices are located and operate, although the numbering system on the tapes doesn’t suggest this.
I wonder if the numbers on the tapes are for tracking this batch and do not reflect the numbers on the actual original tapes. If the entire email system was centralized and run off a single bank of servers with a single tape backup solution, I’d believe more readily this was the original numbering system. In my experience, the numbering system was parallel to the site location and the equipment; if one were to grab a tape from a different site or server, the numbering would reflect this.
agree, and there is no mention of actual raid protected servers that have this data on them where it was dedicated by law.
Yes, in re RAID (redundant array of independent disks) — no idea what their architecture was, although I’d guess there was a minimum of redundancy based on the weird, helter-skelter of tape types. We always used RAID 5 on email systems for the Fortune 100 company at which I once worked, but I know we had small sites that were spun onto the architecture with RAID 1 and RAID 3 (eventually migrated to RAID 5 as it was our standard).
Wish I knew of a more simplistic explanation for the layman who might want to know what this was all about besides this Wikipedia definition.
Since RAID 3 and 5/6 refer to extra drives in the array to maintain data if another fails. Raid 3/5/6 are hardware based and won’t fail if the OS fails, another way to prevent a windows excuse for failure. If the operator wants to keep the data, it might be kept. It is an automatic backup system on the main computer drive storage. And it just might be someone has hidden or copied those existing files, again, just because it might have been setup that way and they don’t know it. Wouldn’t take long to download by fast pipe to secure and perhaps unknown location.
Surprises are likely forthcoming.
I’m just going by my experience with the National Labs out here (10 yrs) and the Forest Service. Shame that the most remote Forest Service locations have better computers than the White House does.
Federal computer management practices have been pretty good for many years, since the great awakening in the 80s. The standardization of practices has allowed lower quality people to perform in positions to which they otherwise could not aspire.
I don’t assume they will upgrade regularly. I assume they will upgrade only when no longer financially viable, or more likely, simply impossible.
I have a MS in CSE and 35 years of experience. Computer forensics in most police departments amounts to little more than using pre-packaged toolkits to copy and restore data. I have most of those tools, including password crackers, bit-level copy, etc. Using some of those tools on a computer you don’t have authorization to access is a felony. Also note that we aren’t being given access to the hard drives, where any forensic tech will tell you the greatest likelyhood of recovering data is, until they’ve been wiped by a mil-spec approved reformatter.
I’m more concerned about the tapes marked “damaged” and “out”. In most cases, you cannot determine the contents without reading the tape, and thus we will never know what is on them. Hopefully there are full backups, not just incrementals, which still have the files.
If they were modern, they’d be using SAN with netbackup, a single backup in a secured location, with offsite archival storage. I don’t expect any of that to happen during this administration either. Yes, it is nothing less than the growth of EOP over time made manifest in hardware.
I would agree that “the numbers on the tapes are for tracking this batch and do not reflect the numbers on the actual original tapes.” The tapes would actually have generated labels embedded in the data. Some tape software keeps libraries of these labels, and use it to perform storage management. Many of the tapes listed seem to be day of week or grandfather-father-son three generation backups, and thus may not be able to reconstruct a point in time – say March 11, 2002 – ever. But the last three weeks would have whatever was left on the server at that time. Likewise, if an incremental for a day is missing, and files were deleted on that day, they could never be reconstructed later.
Isn’t it the FBI system that is notorious in this regard? They’ve been trying to upgrade it for years, millions have been spent, but they continue having to use an old system because the new system can’t be made to work properly? Or am I thinking of some other Federal Dept?
Bob in HI
Yes, it’s the FBI.
http://fpc.state.gov/documents…../32038.pdf
I think you are thinking of the FAA Air Traffic Control fiasco.
So what’s your thinking on why they would have an IBM system and an Alpha system backing up mail?
Different locations? Different servers? Different apps?
Glad to hear Natl Labs and Forest Service are better equipped; after all the whining about the lack of current IT in EOP and the problems with a secure Blackberry for POTUS this week, we shouldn’t be too surprised. I guess the White House didn’t figure they needed much in the way of IT to write history. Or perhaps they bypassed this antiquated stuff far more often than we realize — like Rove’s Blackberry — hence no demand for rotation of equipment on a more frequent basis in this enterprise. (Actually, the stories about Obama’s Blackberry hassles point out a critical security problem at the White House during the Bush years; they had no ready solution, meaning the White House staff who used Blackberries never bothered with security. Morons.)
Agreed on the incrementals — they’ve offered so little info here that we can’t tell when they did fulls and incrementals easily, or at least I can’t at a glance.
I’m assuming that apart from the damaged tapes that we should have most of the data, and if this were a legitimate operation, a damaged tape would have prompted a fix if the tape were damaged at the time the backup was running. If the tape was damaged later when copied for this production, that’s a different and highly problematic kettle of fish.
Next trick is going to be matching up tapes with days of email (where is that spreadsheet anyhow…). Ugh.
Just a SWAG here, but my reading of the PDFs is that the OA’s previous backup system, ARMS, is the critter that was running under VMS.
If I remember correctly, the ARMS was an almost totally “custom” system, probably designed/hacked together by some techie to run on DEC systems just because that particular techie had a VAX/VMS background.
If he/she had an Atari background, doG knows what else would be missing. *g*
That link in my # 70 for the Motion to Dismiss was not correct. Here is the correct one and it has has many of the gory details of which you seek.
Hmmm…that linky stuff is not working. Probably because CREW screwed up; not me. *g*
Here the link http://citizensforethics.org/f…..ismiss.pdf
Jimminy Crickets! Linkys from CREW are just not working.
Let me see if I can somehow get it here:
http://citizensforethics.org/f…..ismiss.pdf
And that isn’t working either.
Ok folks, just go here at the CREW page wrt to “Operating In Total Secrecy, Bush Administration Files Last Minute Motion To Dismiss CREW’s Lawsuit for Missing Emails” and then on the right side of the page under “Related Documents”, click on the link to “1/21/2009 Defendant’s Motion to Dismiss”.
Just a note from the pessimist side of me. Without the hardware, same servers, tape drives, etc., these tapes are worthless. Without the same software (different backup software uses different formats) the tapes are unreadable.
You may be able to find some Vaxes in a computer museum. Last time I used one in a production environment was at the labs back in the ’80s (roughly four computer generations ago). Surprisingly, many Blockbuster video stores still run on Dec/Compaq microVax VMS boxes. They have a company which keeps spares for them and should be able to put a box together called Hallifax.
NT boxes like INNOVATIVE should be easier. Any MS-Server (2000/2003/2008) box should be able to read the tapes, with the right software.
Boy, ain’t dat da trooth. Anyone else got some 5.25 inch floppies with some old work on ‘em?
Bob in HI
Back when I started out in computers, one of the main storage media was six-foot high tape drives with 10 inch reels. But I don’t recall what company made them. Wouldn’t 4mm and 8mm refer to tape widths?
Bob in HI
Those are very common DAT (Digital Audio Tap) data backup tapes types that were/are used by a variety of different vendors and their Operating Systems.
I think Rayne’s point was that one couldn’t infer vendor/Operating Systems from just the designation of 4mm or 8mm.
StorageTek, Louisville, CO, now a part of Sun Microsystems, would have made those tape drives.
Those mm references would indeed refer to tape widths.
If the White House still used this equipment, it’s only because they hauled out of museum. Unless the tape is digital, it can’t be searched electronically. It would have to be searched like an old VHS tape, one foot, one real at a time. Like searching microfiche instead of a hard drive.
Given the money this administration spent, and the depths to which it attempts to hide its actions, choosing this sort of equipment and media could only be to intentional hide what they did. No one would wanted to be able to reference their own past work would have made those choices in the 21st century.
A bit OT, or at least distracted:
Woke up this morning thinking about a remark that the new WH staff found themselves without their Macs and having to work with the equivalent of Ataris. Put that together with the introduction of Windows7, which seems to be “Vista,Fixed” and wondered whether it was feasible to put an operation like the WH on Linux and open-source. I have half a dozen half-formed hunches about why it would be beneficial, but I thought I’d just ask whether it would be utterly impractical for some reason, although I’ve never seen a process that a UNIX couldn’t do better than MS.
Given the Cheney penchant for security, I’m surprised the didn’t move to Linux years ago. Mail servers, even Exchange compatible servers (See Novell SuSE) are much more productive and secure than MS boxes.
So concurring in the opinion.
As one whose career depends on slavishly kowtowing to everything Bill Gates says, take my comments with a grain a salt. *g*
That said, there are significant downsides in deploying, managing and maintaining non-Microsoft (i.e. Windows in all its incarnations) Operating Systems in a medium to large organization environment.
Much of the focus of Microsoft R&D over the last 10 years or so (as well as all the Microsoft-compatible 3rd party vendors) has been in the creation of products to deploy, manage and maintain a Windows-based infrastructure.
There is very little comparable product to provide the same level of corporate/organizational support for end-user computing in the Unix/Linux environment. And don’t even ask about Apple computers for end-users. There’s nada at all there!
At an individual level, one may argue whether a Linux or a Mac end-user system works better, feels better, smells better, etc.
At a medium or large organization systems management level, there is no comparison at all. Neither Linux or Mac end-user systems make it out of the starting gate.
Toys for tots, corporate players NOT!
The impracticality would be that it might be a death sentence for Microsoft.
And as we all know, Microsoft is Too Big To Fail.
(I’m not sure whether I should put a snark tag on that, or not.)
Bob in HI
Sort of like the auto industry, though, you might not want to push that first domino when there are dominoes ready to fall all over the place. We’d survive the failure of Microsoft, but it would be an expensive, disruptive transition. An odd time to encourage such risky behaviour.
Considering that the best that MS can do seems to be Vista and correctives thereto, somebody ought to be thinking that maybe Gates’ Goliath could topple itself. GM. SUV. et al.
Why? Is Microsoft to gig to fail?
O/T: Urf. I went to Raw Story and found a video of the Daily Show with clips of Rush L. and Fox News. It’s chilling (my reaction, anyway).
My question: Is Fox News too big to fail? (And if not, why not?)
Seeing something non-proprietary and open being used prominently might trigger the new generation to start programming. *OK, OK; I see the shoes coming* Personally, I have never compiled a “hello,world” C program on an MS box — I have never been able to figure out what the h**l it wants. Have done image processing libraries on multiple flavors of UNIX, by ontrast. These days, if I want my MS boxes to do something, I use something platform-independent, like Python. I cannot believe that MS is doing our onshore CS talent pool any favors. And if MS is the only game in town for corporations, it sure can’t be because of any particular advantage inherent in the OS.
Quantum is one that deals with the older formats, but they are ancient and magnetic rather than protected.
Quantum is the world’s largest supplier of tape drives, and its DLT®, LTO, DAT/DDS and Travan-based
Geneva 1 and Geneva 2 are titled “Geaneva”. The spelling might be significant.
OT, Bmaz, Clemens lawyer takes top job at DOJ
http://www.forbes.com/feeds/ap…..54644.html
3480 tapes used by mainframes have to be electronically labeled (i.e. stick a tape in and run a program to apply the label you want), and then the label must be provided in order to unload the tape. People put the label on a sticker on the tape so they will know the label to enter to unload the tape.
Two people at different sites would most likely use different numbering systems for their labels, and so like Rayne says, different numbering (or labels) would reflect different locations.
Yes, exactly, although that would apply to the IBM system and not necessarily to any of the backup devices used to create the other tapes. Highly dependent on the tape backup software, would have been some sort of volume label generated that may have been synced or keyed to a barcode; the barcode may/maynot be generated by the software system.
What a mess; I would hope this would end up in the hands of forensics at some point to get this sorted out (preferrably DOJ forensics in tandem with prosecution).
I just don’t think that is ever going to happen. The archivist is going to log this in, check off the “emails sent to the archives” box, and that is going to be the end of it. No one at DOJ is going to want to pick a fight where they have no evidence that says any crime was committed. Only if the AUSA scandal goes forward to prosecution would there be any interest. That may be a few years.
I share your hopes but I drown in my pessimism.
Not true, labels are optional on 3840 tapes. 3480 tape can be (,nl) no lable in MVS, and are in a DDR format under VM. VSE can use lables or not.
If labled under MVS the tape lable is an 80 byite record at the beginning of the tape, and with multi file tapes a possible 80 byte lable for every file on the tape at the beginning of every file.
The tapes could also be an IEBCOPY (backup) dump of an MVS PDS.
The document explaining what the inventory is says that “that inventory shall identify with reasonable specificity and by number or other specification the contents of the tape or media.”
I’m failing to see reasonable specificity of the contents of the media. I see numbers but no contents.
I wonder how much outsourcing went on in EOP. The Federal Government were basically feitshists about outsourcing. I’m not sure whatever happened to the Seat Management Initiative from GSA.
The NMCI (Navy Marine Corps Intranet) program was the largest outsourcing deal in the history of the planet. It was awarded to EDS, and promptly became a disaster. Just because you outsource something doesn’t mean you don’t have responsibility for it. I could tell more, but you’d have to buy me lots of drinks first.
If I remember correctly, the White House Office of Administration (OA) had only about 60 or so employees AND a couple hundred “contractors” taking care of both day-to-day IT stuff and planning longer term strategic stuff.
Just wondering–
The WH was so in love with subcontracting that it subcontracted just about everything but George’s conjugal obligations to his wife (not sure about that). What happened to all those subcontracts at noon on Tuesday?
What happened to any WH communications that may have been in the possession of contractors?
Is there any special deviltry possible connected with the transition because of all this subcontracting?
Is Obama’s transition team gonna have to call in all subcontractors to find out what they’ve got, and if they are still working on contracts for their old bosses?
Just wondering,
Bob in HI
To my understanding (and I’m often wrong, but never shy about being so *g*), it is in fact the contractors who perform most, if not all, of the day-to-day IT operations stuff for the OA.
Stuff like doing the backups!
So, in essence, they are in possession of the crown jewels!
As to the contractors continuing to perform these duties, I’m guessing they have multi-year contracts that moot any change of Administration.
Can’t we just tell you how much we love the University of Buffalo Bulls and Turner Gill? We really gotta buy drinks too?
You could try telling me how much you love Reggie Whiterspoon and the UB Basketball Bulls. Maybe this year will be our first trip to the NCAA tournament. Do we get basketball trash talking threads here?
Maybe when it is bracket building time. But I did one last year and had shockingly little participation. We’ll see.
OMG. You’re an EDS’r too? Or merely burnt by them?
I feel your pain — and I cannot say anything more under the terms of my “package” post-RIF.
No I’m not an EDSer. Although I have been to their offices in Plano and I have worked with lots of them. Even had the pleasure of getting in between EDS and GM at one point.
To keep it simple I was a consultant hired by another consultant who was hired by the government to help justify the deal. Your tax dollars at work.
And btw, I don’t know if folks here caught this from CREW’s site, but it ties into the PDFs EW has provided in this post:
Of a specific tie-in to the subject of EW’s post here, is the Bush Administration’s filing:
This is a large (117 page) PDF, which after you get through the legal mumbo-jumbo (after the 1st 33 pages), you will find the declaration of Steven M. Everett, Chief Idiot Officer of the Office of Administration in which he laboriously details both the processes and the end results of their effort to “find” the “missing White House emails”.
On page 47, there follows a 1 page Powerpoint presentation on the OA’s current email backup process.
On page 48, there follows a 23 page Excel spreadsheet detailing the OA’s “Phase III” effort to “find” those “missing White House emails”.
On page 71, there follows a 33 page analysis entitled “Time Series Analysis of Daily Email Counts” by a Nancy J. Kirkendall, Ph.D., Adaptics, Inc which purports to statistically prove that those “missing White House emails” were not missing at all. She has more mathematical formulae than you could shake a stick at. Just saying. *g*
On page 103, there follows a 4 page MOU (Memorandum Of Understanding) between the National Archives (NARA) and the Office of Administration (OA) “concerning the continuation and completion, after January 20, 2009, of activities to complete transfer of George W. Bush electronic records…”
So, after I read that turkey last night, I thought I’d burden ya’ll with it here on EW’s most appropriate post.
Thanks for that. I just got the email lists and thought I’d post them–but that’s the background for idiots like me who relies on you tech whizzes.
There has been lively discussion of creating a CTO spot, one perhaps lesser known commentary rumination regarding one candidate. Legato I recall from the jukebox aisle at a tradeshow a decade ago. There was a university site at which I worked more than decade ago where gear was hodgepodge as evidently the WH, make that a PDP 744. Academics used to favor the configurability and compactness of DECs, fit in one large room. The xp $1k box thin client nowadays has orders of magnitude more everything. Still need to look at other links, in case there may be something recognizable.
3480 Tapes are probably IBM tapes. God alone knows what the OS was. Could be VM, or MVS. Most likely VM.
Need a mainframe & old 3480 tape drive & controller.
What about 3590’s? newer version, probably no label either — but I wonder if they would have skipped using something that was inherent to the system, especially if they outsourced service to the same vendor…?
Depends on the Operating System. Practice under MVS was to use tape labels, becuse of AVR (Automatic volume recognition, where the OS would not just let an application read any tape, but only the correctly labeled tape).
Practice under VM was not to label tapes.
I’ve done tape conversion between multipe vendors. It’s hard but not impossible, as long as one can read the tapes. Some of machinery will be old.
Once they are read, then the files can be recreated. While the file contents may be encrypted, the file themselves can be recreated.
Have to watch out for code points, IBM stuff was traditionally not ASCII but EDCDIC.
8 mm tapes were used with SCSI tape drives, exabyte was one manufacturer.
I began looking at the data in the Motion to Dismiss. Beginning on page 23 are counts of emails in 2005 and on the completion of Phase III of restoration, as described in the first appendix to the motion. And illustrated on page 47).
The columns are arranged by EOP office, i.e. “components” (WHO, OVP, etc.). Here’s an interesting sample of the counts from the Office of the Vice President:
Office of Vice President (pages 48 onward in Defendant’s Motion to Dismiss)
Zero counts (not included in 2005, zero count from Phase III restoration):
01-Jan-03 through 13-Feb-03
15-Feb-03 through 19-Feb-03
22-Feb-03 through 24-Feb-03
27-Feb-03
01-Mar-03 through 04-Mar-03
07-Mar-03 through 09-Mar-03
13-Mar-03 through 18-Mar-03 and so on …
That seems to me to be a lot of zero counts.
Restored days (lavender color code) – no components coded, must be all components (offices)
13-Sep-03
17-Oct-03
20-Oct-03
08-Jan-04
12-Jan-04
14-Jan-04
16-Jan-04 through 18-Jan-04
23-Jan-04
28-Jan-04 through 29-Jan-04
02-Feb-04 and so on …
2005 Zero Days, Office of Vice President (red color code)
12-Sep-03
01-Oct-03 through 03-Oct-03
05-Oct-03
29-Jan-04 through 31-Jan-04
07-Jan-04 through 04-Feb-04
15-Feb-05 through 17-Feb-05
21-May-05 through 23-May-05 and so on
Seems to me that this chart beginning on page 48 could be compared to some of the timelines that emptywheel and others have compiled to see if there might be reasonable cause to think that something is being held back.
It absolutely lines up. THat second group includes the batch where were know they were talking about Libby being targeted.
Here’s what I think they are saying. There were no named PST files for the OVP during that period, but they reconstructed a whole bunch of emails from other PST files. This is totally unsurprising, but think about what it means. All the OVP originated emails sent to users in other components would show up in the other components’ PST files. There’s nothing in the filings that suggest they found any emails sent from one OVP user to another OVP user (and no one else) on the days without OVP PST files. Maybe they did, but they didn’t say that.
They do a lot of bragging about all the additional emails they found, but they didn’t really address the main issues found in the 2005 review. Sure, they explain some of the unimportant ‘zero email’ days, but sure don’t say something like we recovered X number of component PST files from backup or DR tapes. They don’t even explain where all the emails came from. Nor is there any evidence they did the other stuff they were supposed to do (like ask users for backup media, etc.).
I think that the whole PST file issue is a false trail. Where did the Lotus Notes archives go for this period?
My understanding is that all of the boxes here were mail servers. IBM/VM/MVS, Dec Vax/VMS (perhaps alpha), NT perhaps Alpha. Geneva 1 and 2 were server names, like INNOVATIVE. Willing to bet dollars to donuts “Geaneva 2008″ is a typo.
My bet is that IBM mainframe was used for milcom mail, also possible for VMS box. EOP staff was probably on NT server. Just a Kentucky windage spitball SWAG at how the systems were divided.
Out of curiosity…what ever happened to the emails that were on the RNC and other Republican run servers that were used for official business covered by the Presidential Records Act? Does this order cover those emails per se? How is the HJSC and SJC suits proceeding there–I seem to recall there were some rulings making it harder to recover this info and that the various owners of the Republican run servers were dragging their feet…now what happens there legally…without a Bush DOJ to help the prolonged agony of getting to those emails?
Amazing quantity of three letter combinations in this thread. Two combinations I didn’t see: NSA and TIA.
Since the NSA was apparently sweeping up ALL electronic info (as it passed through the ether or the ethernet cables from point A to all other points and back again — and why not White House coms too — pretty much right from the beginning) in order to achieve Total Information Awareness, why isn’t anybody asking them for their copies of everything. The ‘non-database’ (heap of heaps) they have should be easily searchable with the tools they use every day. I know this is silly because they ‘delete everything from sources that are not a target’, but what the hell, give it a shot.
Otherwise, this looks like an exercise in reconstructing a bound version of the Encyclopedia Britannica after it has been mil-spec incinerated and spread on landfills in 38 countries.
–TA
I believe all that is required at this point is for CREW to be able to prove that in fact the EOP did not deliver the material requested by the court. Therefore the suit cannot be dismissed.
Now, if the suit is not dismissed, what does that practically mean? Who is on the hook for delivering the missing materials?
.. and to Rayne @ 109
Sorry. I guess I need to push my tongue harder into my cheek so that it will be clearly obvious.
We all know we got screwed. We can tell from the pain in our collective ass in the morning. I suppose my frustration about this particular mattress pounding is that we now supposedly have the good guys running things and they, being the smartest or able to hire the smartest (and with access to all the material not actually under the direct physical control of the dick that screwed us), should be making reassuring and supportive noises. That they aren’t is not really a surprise. Having to prove we got raped just adds insult to injury.
Oh well, I guess my best course of action is to go buy a lot of lube.
–TA
They weren’t spying on themselves, they were spying on us. If their communications stayed inside their network, it wouldn’t be anyplace we could get to it. And the NSA would never admit to copying info coming in/out of White House, w/regards to any communications that exited the network, assuming they did anything like that.
The alphabet soup here won’t have anything to do with NSA or TIA — it’s all about the technology inside the EOP.
One person who had known backup media was Bloch. A dilettantish glance at the catalog files appears like a multivendor environment. Maybe I will find a moment to review the Payton topology show and tell.
Yes, multi-vendor, but there may have been as few as two backup solutions. Looks like OpenVMS might have been one, across VAX and Alpha boxes, and whatever IBM runs.
Please someone tell me, can’t we do this?
Question: Can’t the Obama administration hire a forensic geek (computer
genius) and prepare a report that outlines what happened? Jeebus, I don’t
think it’ll be that expensive if you narrow what you want, and it will
be a source document. This fucking circle jerk needs to stop with a remedy.
Get a professional outside opinion (I’ll contribute) like….
We keep going around and around…
Boston Software Forensics
10 Milk Street Suite 416 / Boston, Massachusetts 02108
Providing technical research and consulting services for software litigation and review, including
Experienced expert investigation and exposition Source code analysis: understanding software’s technical purpose from its original design Behavioral analysis: understanding software’s behavior through live experiments Security and privacy audits of Web sites and Internet-enabled software Our clients have engaged us in cases including trade secrets, patents, privacy, internal product reviews, and antitrust.
Historically, according to the documents, they moved from Lotus Notes to MS-Exchange. So the Alpha/Vax boxes were running NT, not VMS. Looks like the IBM mainframe was used as a file store for Journals and PST files. Many of the inconsistencies in traffic volume they claim are due to a failed migration to Exchange, return to production Lotus Notes, then remigration to Exchange.
More problematic is their mootness argument. Suit asks for them to act, they spent 10M$ and thus have acted. Bush will never be president again, thus no “capable of repetition yet evading review” issue.
Many of the emails were never lost, simply not counted. They made their own little tool (we call him George) called PIVIT, which zeroed out anything over 32,000 for a given PST file. No need to restore that which was never lost. They do identify 4 to seven days of emails which are suspect (without naming the dates).
Prediction: Suit will be dismissed for failing to have subject jurisdiction due to mootness. Court will say ball is in NARA’s court. Good luck getting to those emails in anything less than a few years.
Just for those who want to know:
JohnLopresti @ 113 mentioned Payton: Testimony of Theresa Payton, Chief Information Officer, Office of Administration, Before the House Committee on Oversight and Government Reform, February 26, 2008 also the breadcrumbs Suppliers Suggest White House Email Fixes and White House E-Mail Probably Not ‘Lost Forever’
One of the things I would begin to do with respect to tapes inventoried as Out, v. vms-1 and vms-2, would be to migrate the logs into a single Excel document with tabs for searchability among other views.
OT re BPasdar: I wonder what has happened to the q loop.
Still testing…cache cleared, rebooted…
Wonder if Wade’s outfit configured backups.
Interesting link, esp. pp 16 – 19.
Did you mean that Fourth Count (Conspiracy), for the contract to provide support services (database processing, IT, digitizing docs) ‘and sensitive survey work for the NGIC [Dept of Army’s Ground Intelligence Center]? That contract was also supposed to provide support for the DoD Counterintelligence Field Activity (CIFA) which developed and managed DoD Counterintelligence (CI) … ‘programs and functions, including… economic security… (and presumably Iraq-related intel).’
Well, maybe Wade wasn’t behind the entire WH email problem, but that’s more than a cursory can of worms you linked to there, JohnLopresti.
p. 2 notes that Mitchell Wade received $150 million 2002 – 2005.
I didn’t see a specific mention about the OVP, but I do recollect that he had the contract for ‘office supplies’ for FourthBranch.
OT