Saturday, March 1, 2014

A Journalist Asks "How Do We Know"

Dean Miller, the Director of the Center for News Literacy at Stony Brook University posted this warning on his New York Time blog.  News literacy, whether you are a producer or consumer of news, is based on skepticism and, perhaps more than anything else, a healthy awareness of your own bias.

But it seemed odd to me that he could ask the question "how do we know" and feel obligated to restrict his assessment of veracity to the text itself and not even mention objective sources or empirical evidence outside of the story.  Is that the best we can do?

Taking a deeper look at these journalistic conventions with regard to journalistic "truth" at the level of a story can help us understand how matters of scientific evidence and reason are covered in the media.

In Miller's post he recalls three stories that were widely circulated from the Sochi Olympics.  Which of the following claims turned out to be real?  See if you can tell from these statements:
  • "The man responsible for the opening-ceremony display of Olympic rings was stabbed to death the day after the ring device malfunctioned in the middle of the world-televised event."
  • "Russian officials claim that surveillance video from inside a guest’s bathroom proves that sabotage lead to the complaints about shoddy workmanship in Olympic hotels."
  • "Russian authorities are quoted announcing they will solve prison overcrowding problems by housing convicts in Sochi hotels."
You won't be surprised to learn that your perception will be largely determined by what you think about Russia in the first place.  I wasn't.

When I read the stories the second time, however, I noticed that two of the stories included sources while one did not.  This was deliberate, I suppose.  Miller points out that "News Literacy students spend a lot of time learning to evaluate the evidence provided by journalists and to analyze the authority and independence of people quoted as experts and witnesses in news stories."

This is a great quote and it speaks volumes about how journalists often cover controversial matters of fact and opinion like climate change as a compelling and interesting story.  And that is CERTAINLY how we consumers of journalism read it:  as a literary story.  But is it true?  We humans are exquisitely tuned to consider the social standing and social relationships of the speaker to infer interests and motives as context we use to assess the validity of their account of some event.  But is that REALLY the best we can do?

In controversial matters of fact and opinion in science, for example, it's a problem to rely only on the stature of the scientist or the authority of their scientific institutions to assess their claims.  What are we to make of a reputable and brilliant physicist like Freeman Dyson who continues to criticize the scientific establishment for presenting the theory of anthropogenic global warming (AGW)?  What happens when that authority is undermined by "scandalous" personal emails between scientists, for example?  Or when new findings overturn established science?

It's just as problematic to assess the legitimacy of an argument by how coherent and compelling it seems.  H. L. Menckin wrote:
"The essence of science is that it is always willing to abandon a given idea, however fundamental it may seem to be, for a better one; the essence of theology is that it holds its truths to be eternal and immutable."
But the authority of science is derived not from the standing of the scientist.  And neither is it based on the elegance of the theory or the quality of a "better" idea as Menkin wrote, any more than the veracity of a journalistic account can be determined by compelling quality of the characters or the logical flow of the narrative.

Dyson's stature as a brilliant and influential scientist framed as an iconoclastic, independent thinker, outcast from mainstream climate science for his beliefs makes a compelling story... and it obscures the reasons we should be skeptical of his skepticism.  Perhaps that's why, in dozens of blog posts, Dyson is cited as an authority with very specific critical positions opposed to established models, without reporting how he knows what he know or why his views are rejected by the mainstream.  In this opinion piece by Charles Krauthammer, counter claims and rebuttals are not included.  Instead, these authors adopt a political frame showing Dyson as having been excluded from the scientific mainstream simply because his ideas are deemed politically unacceptable, as if they were involved in a theological dispute of "eternal truths" rather than a scientific dispute based on changing truths and evidence.  But how can a reader figure this out from the text of the story itself?

What IS the basis for scientific "truth" and how can journalists get beyond hearsay?  What about empirical evidence, observation and experimentation?  Can we even distinguish self-delusion from deliberate omission, in the case of Dyson, Krauthammer, or his editors?  How can a reader tell fact from fiction from the story itself?

It turns out that Dyson is not published in prestigious journals not because of his conclusions but because of his arguments.  In fact, his claims HAVE been and are regularly rebutted in scientific literature.  Climate scientists agree with Dyson, for example, that cloud formation is a source of tremendous uncertainty in predicting what will happen to the Earth's climate as global CO2 concentrations continue to rise.  Climate science knows that it's possible to cherry pick ten year sequences of average temperature to show that global warming has ceased, even though the 10- or 100-year temperature record -- not to mention evidence from the distribution of plants, animal and ice -- clearly shows continuation or even the acceleration of warming trends.  However Krauthammer and Dyson know what they know, they appear to be unaware of these rebuttals.  The reader is left to figure this out for themselves, however, no easy task and impossible if they remain within the frame of the story.

Although it's surely not a panacea, as readers we ought to insist that both journalists and their sources tell us how they know what they are reporting, not just why we should trust their authority.  This is easier said than done, however.  In history, there is often no way to ask historical sources how they knew what they recorded was true.  As readers of news, the sources are not there for us to examine.  Good journalists, editors and fact checkers always do ask this question on our behalf;  they just don't always tell us what they learn.  And none of these best practices protect us readers from bias, selective omissions, and deliberate fabrication.

I do not think the failure to include how sources know what they know is a conspiracy;  on the contrary, the REASON to leave out exactly how the authorities in a story know what they know is that often the details detract from the narrative of the story, especially in science journalism.  Kerry Emanuel at a recent even in Lexington told us that, in fact, the margin of error in predictions of climate change are actually growing in some cases. On one hand, we have much more certainty about the most likely climate change outcomes assuming different courses of human behavior.  We are more confident than ever that the most probable outcomes will be devastating for billions of people.  On the other hand, however, Emanuel is very transparent admitting that, as we learn more about the climate, we recognize that there is is actually MORE UNCERTAINTY in our models, not less.  In other words, the range of possible outcomes is growing wider even as we gain confidence in the most probable outcomes.

This is precisely how we know.  This is precisely the texture or quality of scientific knowledge that you hear when you engage directly with scientists but is difficult to find in the media.  In the frame of a policy controversy, often used by the media, each actor must assume positions either for or against a specific policy, it's impossible to report on uncertainty and probability, hallmarks of science.  Although uncertainty is always present in science, it is often cited as a reason not to act and it surely weakens the story.  It's no wonder we leave it out.

Looking back at the three "stories" from Sochi, what are your questions as a reader?

It turns out that the account of the "Russian officials" defending the quality of construction projects in Sochi was based on actual video evidence.  Ironically, revealing how you know something can make your claim more compelling while, at the same time, revealing something about yourself that you may not want to be known.  The first story was without attribution but it turns out to have been a YouTube video posted by American luger, Kate Hansen, video evidence that was completely staged by comedian, Jimmy Kimmel.  But that was information that was deliberately omitted from the story and the original video.  Similarly, the final story was simply satire, completely fake, including fictitious "officials." If it's possible to construct evidence, authorities and even "how do we know" stories from nothing at all, how is the reader supposed to sort out the truth of deliberate lies from the text itself?  And when a scientists falsifies data to support his claims, we loose our way.

Any way you look at it, this is "bad news", pun intended.  In the end it's necessary to go outside any story itself to form an educated opinion from multiple sources regarding it's veracity.


  1. Great post! We use a number of heuristics to determine whether a statement is true. Some are more fallible since they are bigger shortcuts. Dean Miller points out the biggest shortcut, "that sounds right," but points us to another shortcut, "this is a credible source." You take it one step closer, "they've exposed their methodology which checks out," but the real verification is when YOU reproduce the results for yourself, which is absolutely the model used for establishing truth in science.

    You and Dean are right that sometimes people use these heuristics because they don't realize that they're using them, so it is good to make people aware of these heuristics, but sometimes people use these heuristics because they don't have the ability to take it one step closer. For instance if I don't understand the science behind climate change or the computer models used, it might be impossible for me to establish that some methodology checks out. And say I were able understand the methodology, I might not have access to the resources for reproducing the results myself.

    At some point we're forced to defer to some heuristic. Getting a satisfactory answer to the question "How do we know?" is often the closest we can get to verifying a statement.

    1. Thank you, Alex. This is the first level. The next layer, though, takes into account that not all authors are INTERESTED in communicating the truth. We have examples of what I would consider willful omission and outright lying as well, which are IMPOSSIBLE to detect from within the texts themselves. I think there is probably feedback we perceive in the form of reputation: when we test what we read from sources outside the text and regularly find a new author or editorial "frame" consistent with what we know, we trust it.

      However, even this does not protect us from groups of people, even large groups, who are consciously or subconsciously omitting evidence that contradicts their findings or lying outright.

    2. There is another point here, as well. You wrote that the next level of confidence is one where you independently "reproduce the results for yourself." This is the ultimate way of knowing. But you can also trust two independent sources where both have reproduced the result more than a single source. This is more to my liking for readers of science in the media.

      When journalists actually report that more than one independent scientist has reproduced the result it should be more credible. For example, journalists can and should say that specific arguments against the theory of Anthropogenic Global Warming (AGW) have been refuted and reproduced independently by multiple groups over decades of debate in the literature, and they should identify the source. They should also occasionally point out that anyone with specific arguments against AGW that have not been refuted in the literature ARE WELCOME IN THE LITERATURE, that they would be celebrated by the scientific community, and they can cite evidence of that. And finally they should say that non-specific arguments attacking the veracity of AGW by attacking individual scientists or the integrity of scientific institutions such as peer-review may be useful emotional, social and political tactics but are not germane to the veracity of AGW claims because, well, because they are not.

  2. I want to know (sic) who are the "we" that are to know?

    Somewhere up the line, someone edited these "news" stories. Somewhere someone had to read these things before they went public and asked, Does this, um, story make any sense whatsoever?

    "In the end it's necessary to go outside any story itself to form an educated opinion from multiple sources regarding it's veracity."

    Ok, so then, why didn't THEY? Why does it have to be WE?

    And who is this mythical "we", anyway? Scientifically educated, fairminded, LEFT BRAINED upper middle class professionals, comme moi? The general public?

    If it's the former, then one must strain one's neck enough to sprain it to buy a NYT article with a headline like:

    "Pentagon Plans to Shrink Army to Pre-World War II Level."

    at best a misleading headline; at worst, a blatant lie. I mean, it's the NYT! Shouldn't they (we?) know better?

    Yet how many millions of people take their cue as to what's true from headlines such as these?

    (And what happens if those "multiple sources" are all in on the scam, er, I mean, misunderstanding? Look at the end of this article, at the litany of repetition of this fallacy. Am I to believe that this is merely the product of these great institutions of information not understanding..? "Opponents of Pentagon-Budget Cuts Just Played the Entire Media"

    Which brings us to the general public. In a Frontline show, "The Real CSI", the following exchange took place among Lowell Bergmann, the correspondent; Harry Edwards, the primary author of a National Academy of Sciences report questioning the validity of all the "science" underpinning crime scene investigation (except DNA evidence); and Peter Neufeld, founder of the Innocence Project:

    Judge HARRY T. EDWARDS: I think it’s a little bit naive to think that the adversarial system will have smart lawyers on both sides and they’ll duke it out, and we’ll figure out the right answer. We need the science first. And then let them fight about what the good science means, as opposed to struggling with disciplines that don’t have good science undergirding them.

    PETER NEUFELD: Lawyers are scientifically illiterate. Judges are scientifically illiterate. And certainly juries are. So there has to be a fix upstream to make sure that before any evidence gets to a court of law, that it has been validated, that it is reliable, that it does meet national standards, and that we can all have confidence in the— in the result.


    Here we have people, us regular "we", who are subject to a justice system that is riddled thru with experts who aren't, with prosecutors who harbor agenda, and juries imbued with superstition and prejudice. The people who are supposed to be looking out for us are not.

    If it's in the courts, why should we believe it's not in the media?

    You may not call it conspiracy. Indeed, I doubt a lot of it is. But it stretches credulity to accept a model wherein it's just a matter of not understanding. Could it be that there is in fact a layer of those who call the shots, who don't actually conspire - because their interests align? Could it be that they don't conspire because, well, they don't need to...?

    People are consciously trying to manipulate what we believe. And they are in positions of power to do so - be they in the seats of power, in the courts or in the media, as you mention. Could it be that they are getting their marching orders from someplace other than a commitment to the truth, however flawed in well-intentioned practice? This may be an uncomfortable truth - "science" has traditionally not been willing to discuss the role of intent in describing reality.

    That does not make it less real...

    1. Right on, Gavin. All true.

      "We" refers to all of us in our society, writers, publishers, editors and readers.

      I allude to the effect of conscious or subconscious omission in the post, and also the problem of deliberate lying. And it's pretty clear this operates on the level of the individual post or article or book, a blog or publisher, and even at the level of a community of subscribers or readers which can be collectively guilty of omission or self-deception. It has never been an easy problem for an individual to figure out what to believe given so many sources of information which are also deliberately trying to manipulate us.

      However, despite fits and starts, we have made progress, in my opinion. Historically, over the course of our "experience" as post-Enlightenment, post-industrial moderns, we have relied on a mix of teachers, public intellectuals, and INSTITUTIONS to challenge our beliefs with "inconvenient truths" and correct biased and manipulative media. Despite what appears to be progress overall, there have been and will continue to be periods where figuring out what is really happening is harder for the reader.

      We'll have to wait and see what the effect of the Internet is on our society. It's not at all clear. As you point out, historically this function of editing what is published has rested in large publishing houses with a selfish interest in their reputation. Readers in turn relied on that reputation to frame what they read, and trusted it. To some extent, the Internet (and blogs like this one) disrupt those institutions. Hopefully, we'll figure out new ways of constructing trust networks the way we did when radio and television disrupted print.

      To be continued...