Bad Science Journalism by Example

The other day, a friend shared the following story: Chocolate makes you cleverer: A 40-year psychological study proves that the sweet treat can boost our grey matter. Of course, as with many scientists, I cringe when I hear the word "prove" because we don't actually prove anything in our research. We generate hypotheses and gather evidence that would allow us to support or refute our hypothesis. We prove nothing, even if we find evidence to support our hypothesis, because the hypothesis could be incorrect. We may think the explanation is one thing but find out it is something different that we didn't even consider or gather evidence about. In fact, I blogged about this pretty extensively here.

So of course, I clicked on the article, thinking it might be an example of bad science journalism.

I had no idea how right I was.

The story is about a longitudinal study of cognitive ability on over 1,000 people in New York State. Yes, it is a 40-year study, but no, the chocolate part was not examined over 40 years. The researchers decided to look at eating patterns in the 6th wave of the study, which began in 2001. Additionally, they didn't even begin collecting the data to find out if chocolate has any relationship to cognitive ability:

The researchers incorporated a new questionnaire – gathering all sorts of information about dietary habits – into the sixth wave of their data collection, which spanned the five years between 2001 and 2006 (there have been seven waves in all, each conducted in five-year intervals). And they revealed an interesting pattern. “We found that people who eat chocolate at least once a week tend to perform better cognitively,” says Elias. “It's significant – it touches a number of cognitive domains.”

Later on, they talk about some analyses they performed to determine direction of causality - whether chocolate consumption causes improved cognitive ability or whether people with higher cognitive ability happen to like chocolate more. They found support for the former, which lends some credence to their claim. But still, the researchers are quick to caveat their results:

“It's nearly impossible to talk about causality with our design,” says Elias. “But our study definitely indicates the direction is that chocolate consumption affects cognitive ability.”

While this is not a "40-year study that proves chocolate makes you smarter," there does appear to be something here worthy of investigation. For instance, they authors used only two levels of chocolate consumption: never/rarely versus once a week or more. So these results don't tell us how much chocolate is beneficial - or whether there are diminishing returns. They also didn't look at type of chocolate consumed - white, milk, or dark. Still, this is a correlation they were not exactly looking for, so there's a danger of p-hacking.

In the meantime, how can you apply these results to your life? Probably the best conclusion is the one the researchers shared:

“I think what we can say for now is that you can eat small amounts without guilt if you don't substitute chocolate for a normal balanced healthy diet,” Elias says.