Pew released a new poll yesterday that has led to some remarkably bad reporting. The most problematic I’ve seen is the WaPo declaring the “Post-Snowden Era” that suggests the concern for civil liberties purportedly sparked by Edward Snowden’s disclosures has shifted in light of the “real fear” Americans have of ISIS.

We’re now just 15 months removed from Edward Snowden’s first bombshell revelation about the United States’ massive surveillance apparatus. But with Islamic extremists putting down roots in Syria and Iraq, Americans are very much reverting to a pre-Snowden attitude toward civil liberties.

Or perhaps we should call it “post-Snowden.”

While the Snowden revelations led to a lot of American soul-searching when it came to just how much of our civil liberties we want to yield in the name of protecting ourselves from terrorism, the soul-searching has largely come to an end, according to a new poll.

[snip]

Given that very real fear, it’s perhaps not surprising to see people willing to cash in some of their civil liberties in exchange for peace of mind when it comes to their safety. But it also suggests the shift toward civil libertarianism and the criticism of the National Security Agency in the aftermath of all the Snowden revelations — of which more could certainly come and change things again— were very temporary.

Before I get into why this is so bad, first, look at what the report said. Amidst reporting that people are increasingly worried about “Islamic extremism,” Pew claims,

The survey also finds a shifting balance between concerns about civil liberties and protection from terrorism. In a reversal from last year after Edward Snowden’s NSA leaks, 50% today say they are more concerned that government anti-terrorism policies have not gone far enough to protect the country, while 35% are more concerned that the policies have gone too far in restricting civil liberties.

It claims to be reporting on a “balance” between “government anti-terrorism policies” and “restricting civil liberties.” But here’s what they actually asked: “What concerns you more about the government’s anti-terrorism policies?” In addition to picking either “They have gone too far in restricting the average person’s civil liberties” or “They have not gone far enough to adequately protect the country,” people apparently answered “Both,” “neither,” “approve of policies” (9% of respondents in this poll answered one of those things; the number has varied from 8% to 13% since Pew started doing this question in July 2004), or “don’t know” or “refused” (6% in this poll, which is the all-time low, with the number ranging up to 13%). So around 10% of respondents have consistently rejected the structure of the question.

I’d say there’s a good reason for that: because there is not necessarily any reason to believe there is a balance between counterterrorism and civil liberties. Not to mention, there are plenty of other legitimate concerns about our counterterrorism policy that Pew didn’t poll. What would the polling look like, for example, if it included “Our anti-terrorism policies have involved far too many illegal wars launched against Muslim countries”?

In other words, Pew is asking people to choose, but it doesn’t actually ask respondents to “balance” these two things. Thus by reporting this as a balance, Pew is imposing its own judgment that it is a balance, a belief which its question isn’t designed to measure. Pew just assumes it is so and reports it as such.

Let me interject and say that I am not doubting the polls reflect a very real change in attitudes in recent weeks. Nor am I doubting that a lot of people do believe this is a balance. Nor do I doubt that some of the poll movement is satiation with a civil liberties focus or even a belief that we do have to double down on the dragnet.

It would be very interesting to measure those things, if someone actually asked questions designed to measure them. I am not doubting Pew’s numbers, just what we can conclude from them.

Now let’s go back to the WaPo. It claimed, in part, that polls reflected people choosing to “cash in some of their civil liberties in exchange for peace of mind.” That adopts the same unjustified “balance” interpretation that Pew did (perhaps because Pew used that language in its report). Some people likely are thinking in terms of cashing in their civil liberties, but this poll didn’t actually measure that.

The WaPo reporting is even worse with respect to its claims that Edward Snowden is the sole explanation for higher support for civil liberties last year. Not only does it have a correlation/causation problem, it doesn’t even have correlation.

Pew and WaPo compare — correctly for measurement purposes — last week’s results with the results from a poll taken in the same series July 2013 (though WaPo gets the timing of that poll wrong), just a month after Snowden’s leaks started. It is true that July was — in Pew’s poll — the high point for civil liberties support in its poll, and that an October 2013 poll showed the beginning of a decline in concern for civil liberties and a rise in concern about protecting the country. Therefore it is true that support for civil liberties since a month after the Snowden leaks first started appearing has declined.

Also Pew did a different series of polls tracking opinion about what Snowden disclosed, which is a fair measurement about changes in perception of spying since Snowden’s leaks. That measured a real decline in support for what Pew inaccurately described in questions as NSA’s counterterrorism spying that persisted at least as late as January. In that series, Pew also presumed factually false details about the dragnet. So a flawed series of polls had actually shown increasing disapproval of the dragnet the last time it was released, but we don’t know how that data has changed in the 8 months since it was polled.

But the real problem with WaPo’s proclamation of a post-Snowden era is it doesn’t cite any polling from before the Snowden stories started (Pew’s previous poll in the civil liberties or counterterrorism series was way back in 2010). To make a claim about how much Snowden influenced civil liberties support, you’d have to cite the same poll from before and from after those stories started. WaPo doesn’t do that at all; it just assumes the record high support for civil liberties was caused by Snowden.

Now I wish Pew had polling from just before the Snowden leaks, because they might show something really remarkable.

Consider this CNN poll, taken (from a much smaller sample) on April 30, 2013, just two weeks after the first successful terrorist attack targeted at civilians since the anthrax attacks. It showed a somewhat elevated level of concern that the respondent or a family member might be the victim of a terrorist attack. (It also showed an all time high in that series — 63% — believing that terrorists would always find a way to attack.)

But the most remarkable part of that poll — one which got a lot of coverage at the time — was this question:

Again, this can’t be compared with the Pew poll; the questions and polling methodology are different. Though to the extent they might be comparable, it would support an interpretation of a decline in relative support for civil liberties. It would also, however, raise real questions about whether Snowden was responsible for all or even most of Americans’ heightened support for civil liberties.

But what a poll taken two weeks after an actual terrorist attack and a month before Snowden’s stories started being reported showed that Americans were far more worried that the response to the attack would be a crackdown on civil liberties than they were about needing new anti-terror policies. Americans already showed a remarkably high degree of support for civil liberties.

Now I agree with the WaPo: a slew of polls do show Americans peeing their pants about perceived threats. As the WaPo notes, this NBC/WSJ poll shows more Americans feel less safe now than they have since 9/11 — almost a 20 point spike from this time last year, a year when terrorists actually succeeded in attacking the US.

And I’d love to know what’s behind the numbers on whether changes have been more good than bad. Are so many people peeing their pants because a general malaise has the susceptible to fear-mongering? Does that mean they like or hate the dragnet? Or just the President?

But here’s the thing.

If there is a tie between the way America is peeing its pants and support or not for civil liberties, this is not about actual threats. Here’s what President Obama said last night.

So ISIL poses a threat to the people of Iraq and Syria, and the broader Middle East — including American citizens, personnel and facilities.  If left unchecked, these terrorists could pose a growing threat beyond that region, including to the United States.  While we have not yet detected specific plotting against our homeland, ISIL leaders have threatened America and our allies.

This is not to say ISIS is not a threat or — more accurately, a very dangerous entity that is currently focused far away from the US. But the President, at least, doesn’t think they’re about to attack Boston.

13 years after 9/11 the American people are far more afraid after a month of fearmongering about an inflated threat than they were last year, weeks after terrorists succeeded in attacking.

But all this seems to be saying that Americans are far more afraid of the fearmongering images than of the actual threat of terrorism. If Americans have changed their relative concern about civil liberties because they are afraid, it’s not the actual threats that are causing that change.

Perhaps Pew should start a new series: Are you more afraid of terrorism, or of what your country will do by inflating the threat of terrorism?