Trust in a Post-Truth World

The Oxford Dictionaries have announced the 2016 word of the year:

post-truth (adjective) - relating to or denoting circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief

The concept of post-truth has been in existence for the past decade, but Oxford Dictionaries has seen a spike in frequency this year in the context of the EU referendum in the United Kingdom and the presidential election in the United States. It has also become associated with a particular noun, in the phrase post-truth politics.

A recent article in The Guardian tied this concept to the notion of trust, and discusses results of the Ipsos Mori 2016 Veracity Index, which assesses level of trust in different societal roles. Unsurprisingly, politicians are among the most distrusted, and saw their trust rating fall this year to 15%. Who are the most trusted? Nurses, at 93%, followed closely by physicians at 91%:

The stellar 93% rating for nurses was warmly welcomed by Janet Davies, chief executive and general secretary of the Royal College of Nurses. “Nurses are some of the most caring, hardworking staff in the UK and it is very encouraging to see their efforts reflected in the eyes of the public,” she said.

“A trusting relationship is absolutely essential in healthcare. As pressures on the health service rise, it’s particularly positive that the public have maintained their faith in the frontline staff working tirelessly for them throughout these difficult times. These results highlight the critical role nurses play in the lives of people in the UK.”

Here are the full results (and access more details here):

Academic Dishonesty, Post-Peer Review and Debunking Research

As a researcher, publishing is a very important part of my job and ongoing career options. Though most researchers engage in research honestly, and if their results are incorrect, it's more likely due to error than malice, there are still cases in which researchers have fabricated data and even entire studies (for more, see here and here). Recently, a friend brought to my attention yet another instance of research dishonesty - a case that came to light last year, but I only learned about today. What is surprising to me, in this case, is that both the dishonest researcher and the one who debunked the research are (or were at the time) graduate students:

The exposure of one of the biggest scientific frauds in recent memory didn’t start with concerns about normally distributed data, or the test-retest reliability of feelings thermometers, or anonymous Stata output on shady message boards, or any of the other statistically complex details that would make it such a bizarre and explosive scandal. Rather, it started in the most unremarkable way possible: with a graduate student trying to figure out a money issue.

Michael LaCour, a graduate student at UCLA, talked to David Broockman, grad student at UC Berkley, about a multiphase study he performed in which canvassers were able to change respondents attitudes about gay marriage by revealing their sexual orientation. Broockman, who was fascinated by the results, set out to replicate the study and encountered the first issue: Labour's survey had included 10,000 respondents paid $100 a piece, a rather large grant for a graduate student. So Broockman approached polling firms about the study idea - most said the study they couldn't carry out such a study, and if they could, it wouldn't be feasible on the usual grants grad students could obtain.

So Broockman started talking to people - carefully, because he was informed by many, and suspected himself, that exposing another researcher could get him labeled as a troublemaker or incapable of coming up with his own research ideas. And in fact, LaCour had written the paper on the results with a well-respected political scientist at Columbia, Donald Green. Broockman even said when he described the results to others, they were surprised that the results seemed to fly in the face of previous theory and research, but dropped those arguments when they heard Green was involved. In fact, when Jon Krosnick of Stanford was contacted about the study, he said, "I see Don Green is an author. I trust him completely, so I’m no longer doubtful."

Broockman hit many snags along the way, not just because he was a busy grad student working on his own research and finishing his degree - he was cautioned about exploring these issues by nearly everyone he spoke to. An anonymous post on the poliscirumors.com laying out his suspicions was deleted. And his analyses on the distribution of the data, which looked too clean to be real, failed to uncover major issues.

But still, there were hints that something was wrong. When he messaged LaCour with questions about methodology, the answers were vague and unhelpful. A similar study Broockman conducted with fellow grad student Josh Kalla showed response rates for the first wave around 1%, even though they were offering as much money as LaCour, who reported response rates of 12%. An email to the survey research firm LaCour said he had worked with on the study showed that, not only had LaCour never worked with the firm, the person he claimed to be in contact with (and had emails from) didn't exist. Then, they hit gold: a 2012 Cooperative Campaign Analysis Project that was a perfect match for LaCour's "first wave data."

By the end of the next day, Kalla, Broockman, and Aronow had compiled their report and sent it to Green, and Green had quickly replied that unless LaCour could explain everything in it, he’d reach out to Science and request a retraction. (Broockman had decided the best plan was to take their concerns to Green instead of LaCour in order to reduce the chance that LaCour could scramble to contrive an explanation.)

After Green spoke to Vavreck, LaCour’s adviser, LaCour confessed to Green and Vavreck that he hadn’t conducted the surveys the way he had described them, though the precise nature of that conversation is unknown. Green posted his retraction request publicly on May 19, the same day Broockman, Kalla, and Aronow posted their report. That was also the day Broockman graduated. “Between the morning brunch and commencement, Josh and I kept leaving the ceremonies to work on the report,” Broockman wrote in an email.

So what happened to the grad student who was repeatedly cautioned that debunking research could be a career killer? The response he received was "uniformly positive" and, oh, by the way, he's now tenure track at Stanford University. About this issue, he says: "I think my discipline needs to answer this question: How can concerns about dishonesty in published research be brought to light in a way that protects innocent researchers and the truth — especially when it’s less egregious?” he wrote. “I don’t think there’s an easy answer. But until we have one, all of us who have had such concerns remain liars by omission."

I think many of us in the research field have witnessed activities that were questionable, perhaps even clearly unethical. But rarely are we encouraged to bring our suspicions to light, and there are certainly no safe venues to bring up concerns that may or may not be accurate. While I've never been actively discouraged from reporting ethical issues, I'm sure there are many researchers who have, like Broockman. And for many grad students and post-docs, it's more likely they are working with more seasoned faculty than other grad students, so when ethical dilemmas come up, the power dynamic may discourage them from doing the right thing. While we certainly don't want witch hunts for data that looks "too good to be true," we need to find ways to protect fellow researchers and the public from bad science and false data. Because that hurts all of us.

Don't Tell Momma What You Know

Not long ago, a post went viral that shouted to the masses "New Research Establishes That Intelligence Is Inherited From The Mother." Hopefully everyone called their mom and thanked her for this incredible gift, not because it's true (it's not), but because it's always good to call your mom and thank her for something.

Not the point of the post. The point is that, no, you don't inherit your intelligence from your mom. At least not completely. In a recent post for Forbes, science writer Emily Willingham ripped this post to shreds:

A garbled post from a website called Second Nexus has gone viral in my feeds (and possibly yours), likely because of its eye-catching headline claim that “New Research Establishes That Intelligence Is Inherited From The Mother.” The piece is bylined “Editorial Staff,” presumably because everyone was too embarrassed to put a real name on it.

Ouch. She goes on to offer some education on genetics, including that 1) women "tend to" have two X chromosomes (but not always), 2) one of those X's had to come from the father, 3) and having two X's doesn't double your odds for receiving some trait because cells might shut down most of one X, reducing its influence.

But more than that, she offers a far more complex view of intelligence, reminding her audience that only about half of what we call intelligence comes from genes:

While maybe half of our intelligence as we currently define and measure it is inherited, that proportion is in turn fractured into many many genetic variants scattered across our genomes. These variants operate together in various ways to form what we view as intelligence. And each of those fragments of heredity that contributes is itself subject to a host of environmental factors, both in its immediate molecular world and inputs to the whole organism, that will influence function. And that influence continues after birth as an ongoing mutual interplay of gene variants and environment. It’s layer upon layer upon layer of interacting pieces. So no. Not just your mother. Not just the X chromosome. Not even just genes.

Willingham then traces back the sources the original post used to make its claims, including Psychology Spot (a blog she calls "a dumpster fire of poor information about genetics and embryonic development"), and Cosmopolitan.

I saw many of my friends, including fellow scientists, share this hot mess of a post. Why? Willingham has one potential explanation:

Those headlines. It’s an irresistible invitation to humblebragging, whether you have a mother and think you’re a genius or you are a mother and think your children are geniuses or you’re feeling feminist and want to stake a claim that women bring the smarts to this world. That’s a pretty solid built-in audience ready to click…and share.

This is not to say mothers aren't important for their children's development and intelligence. About half of our intelligence comes from genes, and short of genetic engineering, there's not a lot we can do about that. But the other half seem to be environmentally influenced. Mothers (and fathers, and really anyone interacting with children) have a lot more control here. For instance, some research shows that offering growing infants lots of stimulation can result in more developed brains - the environment can actually influence the physical. Taking children out to see the world, and experience new things, does enrich them and boost their intelligence. (Note: this is real stimulation, not on a screen - though some would argue television has deleterious effects on children, the most we can say from research is that it has no effect. And don't get me started on the Baby Einstein videos...)

In my case, I know my mom is responsible for a lot of my intelligence, not just from genetics, but from offering an enriching world. She would teach me new words and ask me to try to use them in a sentence that day. I was probably the only first-grader who knew "tumultuous." She taught me the difference between independent and dependent variables when I entered a science fair in 7th grade. And she instilled me the curiosity that continues to influence my efforts today. She's also the reason I write.

Regardless of what genes were passed on (or not), there's a lot we can do for children.

CDC, You've Created a Monster (Pun Fully Intended)

At work the other day, I heard a radio ad for Illinois's Click-It-Or-Ticket campaign, in which they discuss the importance of wearing seatbelts in case of a zombie attack (hiding and waiting for unsuspecting drivers before jumping out at their cars - yeah, seriously). People are fascinated with zombies recently and public health organizations are no exception. For instance, CDC released a disaster preparedness campaign last year, in which they discuss the preparations individuals and families should make to prepare for a zombie uprising. Though the purpose of this campaign was to get people thinking about disaster preparedness for less supernatural reasons (floods, tornadoes, etc.), it also was meant to get people's attention - and it worked, though the response on the internet was perhaps more "WTF, CDC?" than "Wow, I never thought about disaster preparedness like that."

Besides, wouldn't wearing a seatbelt be a bad idea if zombies suddenly jump in front of your car and cause you to wreck? I mean, you want to get out of there quickly after wrecking, right? If you're surrounded by a zombie hoard, you're probably not moving very quickly anyway. And now that I think about it, if they wait to jump in front of your car so they can make you wreck and get some food (sounds suspiciously like fishing), wouldn't that involves things like patience, thinking power…? Not exactly what zombies are known for.

But I digress.

Public health organizations are doing whatever they can to get people's attention these days. And the CDC campaign, although weird and random, got attention and perhaps that's all it needed to do. After all, mass media campaigns for public health are generally really expensive to implement but have small effects. Really, any campaign intended to simply educate does little to actually change behavior - in fact, it does little to even change attitudes about a behavior, and even if it does change attitudes, a lot of psychological research demonstrates that attitudes are poor predictors of behavior. That's right, people can hold very strong beliefs and behave in ways completely counter to them. It drives me nuts when people say that psychology research just confirms common sense, but seriously - does this actually surprise you?

How about now?

By using something currently popular, like zombies, they increase the chance that mainstream media will pick up the story and help pass on the message, which could exponentially increase the campaign's reach. But does capitalizing on the current trend actually weaken the message? Does it mean that, once zombies are no longer "cool", the campaign will lose its effect and possibly even have the opposite effect in the long run? I tried to find some research on this - it's what I do.

There's definitely been research on what makes public health campaigns effective, but the focus is more on things like tone (e.g., fear appeals) than content. One exception is a review by Randolph and Viswanath, which examined studies of public health campaigns to determine which factors most clearly related to effectiveness. They identified as the most important factor: "successful manipulation of the information environment by campaign sponsors to ensure sufficient exposure of the audience to the campaigns messages and themes (influencing the information environment and maximizing exposure)." Okay, so perhaps CDC's (and even Illinois's Click-it-Or-Ticket) idea was a good one - do something that gets attention and is picked up by other media outlets. Fine fine, moving on in the list… 2) use social marketing to develop messages appropriate to a specific audience and get the message to that audience (meaning there may need to be more than one message for different demographic groups), 3) create an environment in which the target audience can carry out the recommended change, 4) use sound health behavior theory, and 5) conduct process as well as outcome evaluations to understand the campaign's effect.

But there is a bigger question here. Could CDC have legitimized some people's beliefs that a zombie apocalypse is upon us?

Four or five years ago, I discovered a web group devoted to surviving zombie attacks. As a long-time horror movie fan (see previous blog posts - part 1 and part 2), I figured this was a group run by fellow horror movie fans who would discuss different zombie movies… sadly no. This was a group of people who genuinely believe a zombie apocalypse is imminent, and spend their time 1) posting news stories supposedly providing evidence that zombies are among us, and 2) devising plans to survive attacks. I received quite a few, "This is not what our group is about" messages in response to some of my posts about great movies, or even when I said that my survival approach would depend on if these were slow-moving, Night of the Living Dead zombies, or fast-moving, 28 Days Later zombies. I can imagine that this group must be pretty active given the media's recent focus on particularly heinous attacks involving cannibalism (and I choose each of those words very carefully - no, these recent events are not, in my opinion, evidence of zombies, but rather evidence that, "If it bleeds, it leads" is still the approach taken by mainstream media, and that, given our recent focus on zombies in popular culture, the media IS biased in what stories it chooses to cover).

I come from a behaviorist background - my undergraduate psychology department head was a behaviorist and for the better part of my undergraduate career (and a not-insignificant chunk of my graduate career), I was like a mini-Skinner, arguing that everything could be explained by contingencies of reinforcement. Though I lean more toward cognitive psychology these days, I still remember many of the things I was taught in my behavior classes and believe these topics are still very relevant for understanding human behavior.

One important thing to keep in mind is unintended consequences. When trying to reinforce a certain behavior, you have to watch for other behaviors you may be inadvertently rewarding (or punishing). For example, I had a professor in college who hated it when people showed up late for class. When people would come in late, he would stop class and spend a minute or two interrogating the person about why they are late, lecture them about why their behavior is unacceptable, etc., etc. He would also take points off for tardiness - the same number of points he would take off for an absence. Basically, people who showed up late once never did it again - because after that, if they were going to be late for class, they just skipped. I'm sure if this professor were aware that his punishment, though decreasing tardiness, was actually increasing absences, he would have changed his approach. But I doubt he ever examined his tardiness and absence data to look for this pattern.

The same goes for interventions intended to change human behavior - you want to examine behaviors that might have been affected by your campaign, whether those were behaviors you were hoping to change or not. It is unclear what effect the CDC campaign had on things like belief in zombies, but if I were performing an evaluation of this campaign, that would be something important to assess.

In fact, CDC spoke to a reporter at Huffington Post about the recent events, and said that CDC is aware of no virus or toxin that could cause the reanimation of dead tissue. So perhaps they recognize that selecting zombies for their campaign was a bad idea.

Still, the blame cannot be placed entirely on CDC. As I said earlier, zombies were already getting lots of attention, and this is probably why CDC selected that topic to frame their disaster preparedness campaign. What do you think? Did CDC add an air of legitimacy to the supposed "zombie apocalypse"? Or just capitalize on the zeitgeist?

Thoughtfully yours,
~Sara

Citizen Journalists and Credibility

Five years ago this month, CNN started an initiative called iReport, where regular people could generate news content. Though visitors to CNN.com can access these iReports via the iReport page, CNN will also occasionally place iReport stories on their main homepage as well. In a recent story covering what the iReport news team consider to be the five stories that defined their "year one", many commenters attacked the initiative, saying iReport stories were "non news-worthy", and "lack the credibility and the training to hold such an important place on the website". Others posted support for iReports, saying that these "iReporters" put more efforts into fact-checking and proofreading than regular CNN reporters. This notion of citizen-journalists is definitely very interesting, and could be the starting point for many interesting debates. For another day, I suppose. What I'd really like to delve into today is this idea about credibility: what determines whether someone is viewed as credible and what biases may influence that judgment?

Human beings are regularly called upon to process large amounts of information from the world around us. This information is perceived by our five senses, and the information has to be interpreted by our brain in order for us to make decisions and navigate our environment. We're very good at processing this information. We're so good in fact that people are always looking for supernatural explanations for why we're so good at it - I know a lot of people who believe in ESP, but honestly, I think that people are just very good at picking up on cues in the environment (and some people are better and faster) and that because of their skill and speed, it appears they reacted before whatever they were reacting to even happened. Once again, another post, another day.

That's not to say that biases can't be introduced into our processing. We often see what we want to see and hear what we want to hear. It's true that our current experiences are often colored by our past experiences, which I think is just another example of how awesome our brain is; not only do we process what's directly in front of us, we are processing past experiences in tandem and drawing connections between the two. This means, however, that we can make mistakes.

Want to learn more about some of these "mistakes"? Check out this list of cognitive biases.

Of course, just because we're capable of systematic processing doesn't mean we will always be raring to think things through at that level all the time. On the contrary, thinking through everything at this level would lead us to become quickly overwhelmed. Therefore, we've created mental shortcuts that help us to navigate our environment while saving our cognitive resources for the things that really matter; Susan Fiske and Shelly Taylor referred to this tendency, calling us "cognitive misers". It is important to note, however, that some people really enjoy thinking and engage in it much more than the average person; we would say they have a "high need for cognition". Even these people will occasionally engage in mental shortcuts, they just do so less frequently than the average person.

One mental shortcut is called a heuristic. Heuristics are quick rules or devices, which allow someone to draw a conclusion or solve a problem quickly and without a great deal of thought. Heuristics are not always wrong. For example, if you see someone standing on a busy street carrying a map, you can probably safely assume they are a tourist. True, you could cycle through all the possible reasons a person would be carrying a map, but the cognitive miser in you will probably just think, "Tourist", and move on (or if you're feeling altruistic, offer to help said tourist find his or her next destination). The problem is when we use these shortcuts in situations where more systematic thought is necessary.

For example, imagine you've been called for jury duty. You're presented with lots of competing evidence, and you have to use that information to determine whether the defendant is guilty or not guilty. Now would probably be a bad time to use a heuristic, but some research evidences suggest that people do use heuristics in these situations, especially when the evidence is confusing or points equally to guilt or innocence. One heuristic is referred to as "what is beautiful is good"; it is a belief that, if a defendant is physically attractive, he or she must be innocent.

Another well-known heuristic is called the "availability heuristic". When we believe something is true - such as, "Women are bad at telling jokes" - we can easily think of many examples that support our conclusion, but often have difficulty remembering examples that refute the conclusions - such as times when women showed excellent comedic skills or times when men showed poor skills. Based on having many available examples that support our conclusion, we develop even stronger beliefs that our initial conclusion is correct.

The heuristic that is probably operating here deals with source credibility. When someone is presenting information, and especially, trying to convince you of something, you could 1) listen to her arguments and systematically think through them to determine if her conclusions are valid, or 2) decide whether you agree with her based on her education and/or title. Research shows that arguments presented by someone with, for example, a PhD are judged to be more sound than the same arguments presented by, say, a high school student. True, when studies manipulate how good the arguments really are, people are generally able to differentiate good arguments from bad arguments, but even then, the PhD often still has an advantage over Joe H.S. Student.

I don't often read the iReport stories, so I don't really have any conclusions about whether there is any truth to claims of inaccuracy. It stands to reason, however, that some of these commenters are reacting to job title rather than content. An iReporter could be anybody. CNN does provide information on their vetting process - how they determine whether an iReport is accurate and worthy of being called news - but someone who is applying a heuristic to determine whether to believe this particular story may not be very motivated to read about the vetting process and determine whether that raises their estimation of an iReporter's credibility.

That’s not to say that job title is the only marker of credibility people use. For instance, Elizabeth Smart, whose kidnapping case received widespread media attention, is now a contributor to ABC News on stories involving kidnappings. ABC News clearly feels this experience qualified her to talk about kidnapping in general (and many viewers likely do, as well). Of course, her hiring did prompt the Daily Beast to ask, “Other than fame – as the victim of a horrifying crime – what exactly are her qualifications?” True, the qualifications they would like see, such as a degree in psychology or years of experience in studying kidnapping victims, could be considered a metric of knowledge – we assume someone which such training to have a lot of knowledge on the topic, but this is not always the case; once again, this is just a heuristic at work.

Heuristics are not likely to go away. Despite their flaws, these shortcuts are necessary. Imagine if every decision you made (paper or plastic, boxers or briefs, soup or salad) involved the same careful thought you currently reserve for the important decisions (city or suburbs, car or SUV, Cubs or White Sox). So what is the best way to deal with this dilemma of credibility? Other than what CNN currently does – provide a document detailing the vetting process – what could they do to set your mind at ease?

Thoughtfully yours,
Sara