Site Meter

On the NSA and Media Bias: An Extended Analysis

You may also like...

10 Responses

  1. Orin Kerr says:

    If I understand your methodology correctly, the premise of your report is that the following words are “pro surveillance” and are signs of pro-government bias:

    ******
    broad, counter(-)terrorism, counter(-)terrorist, foreign, protect, protecting, protection, secure, security, specific, target, targeted, targeting, terrorism, and terrorist
    *******

    Under that methodolgy, the following would be examples of highly-biased pro-government media coverage:

    1. “The NSA shamefully targets innocent Americans.”
    2. “NSA programs designed to catch foreign terrorists are a sham and would make the Framers of the Constitution turn over in their graves.”
    3. “The government conducts broad monitoring instead of targeted monitoring, understandably enraging foreign governments.”

    Do I misunderstand your methodology, or are these sentences that your study would count as very biased in favor of the government?

  2. Albert Wong says:

    Thanks for the comment.

    Our premise is that there are certain key words that are generally used to justify increased surveillance, including security or terrorism. There are other key words that are generally used to invoke opposition to mass surveillance, such as privacy or liberty. When General Alexander defends the NSA’s actions, he focuses on how these actions are necessary to maintain national security, to protect us from terrorism. When Senator Wyden criticizes the NSA’s actions, he emphasizes how these actions impinge on Americans’ civil rights, Americans’ expectation of privacy. These are overall trends, which is why our technique examines content in totality. Looking at any single cherry-picked sentence can be misleading (in either direction) – as anyone who’s been quoted out of context can attest.

    As a side note, “broad” is one of our anti-surveillance terms (not pro-surveillance).

  3. Orin Kerr says:

    Albert, thanks for the explanation (and sorry for the error on “broad”). I think the difficulty with your methodology is that you can control the result by picking the words you deem “pro surveillance” and the words you deem “anti surveillance.” You can get any result you want just by changing the words you classify and test.

    In the case of your study, it seems to me that you selected fairly neutral words for your pro surveillance terms but selected fairly loaded words as anti-surveillance terms. Words like “foreign” and “target” (allegedly pro surveillance) don’t have any obvious valence one way or the other, it seems to me, while words like “dragnet” and “rubber stamp” (the anti-surveillance terms) are the words of civil liberties advocates.

    That selection of words stacks the deck, I think. Given that, it’s not surprising that Glenn Greenwald — who prides himself on being an advocate and thinks that neutrality is a charade — comes off as being neutral and unbiased. Did you try running the test using different words — say, “treason,” “safe,” and “anti-American” as the pro surveillance words — and if so, did you reach the same result?

  4. Albert Wong says:

    Orin, thanks for following up.

    I believe our terms on both sides are fairly vanilla. There was certainly no attempt to stack the deck – we selected the words first, then ran the analysis. We did not change or switch out any words after we ran the analysis in order to ‘improve’ the data. We did not have any pre-determined outcome in mind – if anything, we were surprised by our results.

    To address your two specific examples, “foreign” does indeed have a valence – the NSA’s defenders are known to emphasize the NSA’s focus on foreign intelligence (which polls have shown Americans tend to be supportive of), while the NSA’s critics highlight the NSA’s domestic activities (which Americans tend to oppose). And “target” is similar: the NSA’s defenders frequently emphasize the targeted nature of the NSA’s surveillance. They note that even though lots of data are collected, the NSA only looks at the data in a targeted fashion, to catch terrorists. Likewise, the NSA’s critics blast the NSA precisely because its activities are broad and indiscriminate – the NSA is collecting everyone’s data, they note, not just suspected terrorists’ data.

    More generally, the pro-surveillance terms were selected precisely because they are the words of the NSA and its supporters. Likewise, the anti-surveillance terms were selected because they are the words of the NSA’s critics. I don’t think “liberty” or “privacy” are any more loaded than “terrorism” or “security”.

    The fact is that if we were to reduce the “anti-surveillance” terms to 4 (domestic, liberty, privacy, and rights), and the “pro-surveillance” terms to 4 (terrorism, foreign, security, and targeted), we’d end up with The New York Times having a pro-surveillance bias of 12.7% – essentially identical to the 14.1% we found with our original analysis.

    If we were to reduce it further, to just 2 completely uncontroversial terms each (liberty and privacy on the “anti-surveillance” side, and terrorism and security on the “pro-surveillance” side), we’d find a pro-surveillance bias in the Times of 13.2%. Again, essentially identical to the 14.1% in our original analysis.

    I hope this shows that the deck isn’t being stacked.

  5. Orin Kerr says:

    Albert, I’m not suggesting that you intentionally stacked the deck, to be clear. I just think that this sort of word-testing is highly likely to lead to results that reflect one’s own priors. The example I’m most familiar with is Timothy Groseclove, a conservative who always manages to find (through a somewhat similar methodology) that the media has a massive liberal bias. The problem is that s/he who chooses the words chooses the frame, and choosing the frame determines the bias. The power to pick the words that you think reflect both sides is the power to manufacture a false impression of “neutrality” based around one’s own priors.

  6. Albert Wong says:

    Orin, thanks for the response. I agree with you that it’s possible to introduce bias into a keyword-frequency analysis. However, in this case, our results remain constant even if we eliminate all potentially ‘arguable’ terms from the equation. As my previous (Nov. 2nd) comment noted:

    “More generally, the pro-surveillance terms were selected precisely because they are the words of the NSA and its supporters. Likewise, the anti-surveillance terms were selected because they are the words of the NSA’s critics. I don’t think “liberty” or “privacy” are any more loaded than “terrorism” or “security”.

    The fact is that if we were to reduce the “anti-surveillance” terms to 4 (domestic, liberty, privacy, and rights), and the “pro-surveillance” terms to 4 (terrorism, foreign, security, and targeted), we’d end up with The New York Times having a pro-surveillance bias of 12.7% – essentially identical to the 14.1% we found with our original analysis.

    If we were to reduce it further, to just 2 completely uncontroversial terms each (liberty and privacy on the “anti-surveillance” side, and terrorism and security on the “pro-surveillance” side), we’d find a pro-surveillance bias in the Times of 13.2%. Again, essentially identical to the 14.1% in our original analysis.”

    So I hope we can agree, based on the above, that our results here aren’t false or misleading.

    Many thanks for the discussion.

  7. Bruce Boyden says:

    To follow up on Orin’s questions though, I think what your study shows is that recently newspapers reporting on NSA surveillance have used the terms “terrorism, foreign, security, and targeted” more than they’ve used the terms “domestic, liberty, privacy, and rights.” But I don’t understand how that leads to the conclusion that the coverage is biased in some particular way unless we know in advance that use of the terms “terrorism, foreign, security, and targeted” more often than “domestic, liberty, privacy, and rights” corresponds to a pro-surveillance bias. Is there evidence to support that premise?

    For example, it might be the case that analyses have been done of the writings of known advocates for each position, engaging in advocacy, and that those studies have found that advocates use the four words you’ve selected for their position more often the other four words. (If so, I’m curious how large the tilt is, and whether there are other sets of four words that provide even more of a variance.) Even assuming that is the case, it would be suggestive, but I’m not sure it leads to a conclusion of likely bias when you see the same four words being used more often in news coverage. I think in order to establish that, you would need to establish that using the four words on one side or the other is *inconsistent* with objective news coverage. Showing that it’s consistent with the words advocates for one side more often use does not strike me as quite the same thing, although as I said, it would be interesting.

  8. Albert Wong says:

    The evidence, I believe, is that the debate itself is about the proper balance between liberty and privacy on the one hand, and terrorism and security on the other. That’s what the entire debate is about.

    As I previously noted, “our premise is that there are certain key words that are generally used to justify increased surveillance, including security or terrorism. There are other key words that are generally used to invoke opposition to mass surveillance, such as privacy or liberty. When General Alexander defends the NSA’s actions, he focuses on how these actions are necessary to maintain national security, to protect us from terrorism. When Senator Wyden criticizes the NSA’s actions, he emphasizes how these actions impinge on Americans’ civil rights, Americans’ expectation of privacy.”

    I don’t think it can be seriously disputed that “security” or “terrorism” are used to justify surveillance. I certainly have never heard anyone attempt to argue that monitoring innocent people is necessary to protect their privacy.

    As for disparate use and objectivity, what we have here is overtly neutral coverage where words like “terrorism” and “security” are used 36% more frequently than words like “liberty” and “privacy”. To me, that strongly suggests that the deck’s not even.

  9. Bruce Boyden says:

    Well, maybe it’s not possible, but I am seriously disputing whether a greater frequency of occurrence of the words “security” or “terrorism” than “liberty” or “privacy” in news coverage indicates bias. Note that this is entirely different from disputing a claim that “security” and “terrorism” are often used to justify surveillance. The question is about what the *disparity* means, not whether any given article features one or more of the words on the list.

    “To me, that strongly suggests that the deck’s not even.” But isn’t that the very thing the frequency of terms is supposed to demonstrate? It seems somewhat precarious to rely on intuition about the meaning of a small disparity (for e.g. the Times) in frequency of use of various terms to arrive at a conclusion of likely bias.

  10. Albert Wong says:

    Well, let’s make this simple. Our results show consistent, double-digit disparities in four major newspapers’ total coverage of the issue over a calendar month. The coverage is overtly neutral. But the security/terrorism half of the story is being mentioned more – getting more ‘airtime’ – than the liberty/privacy half of the story. This is bias.

    This is objective data, not intuition. Why do you think this disparity is insignificant, and what data do you have to substantiate this belief?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

*
To prove you're a person (not a spam script), type the security word shown in the picture. Click on the picture to hear an audio file of the word.
Anti-spam image