Statistical Sins: When the Data are Too Perfect

Yesterday, Ars Technica published an article about an investigation into the research of Nicolas Guéguen, a psychologist who has received a great deal of media attention for his shocking findings in gender effects. His research includes findings that men prefer women in heels or wearing red, and that men are more likely to help a woman wearing her hair down instead of up. But, according to James Heathers and Nick Brown, Guéguen's data and high publication rate are suspect:

What they've found raises a litany of questions about statistical and ethical problems. In some cases, the data is too perfectly regular or full of oddities, making it difficult to understand how it could have been generated by the experiment described by Guéguen.

Social media is where it all kicked off, when Nick Brown saw a tweet about a paper claiming that men were less likely to help a woman who had her hair tied up in a ponytail or a bun. When they looked more closely at the paper, something odd jumped out at them: the numbers in the paper looked strangely regular.

When you’re dividing by three, the decimal points will always follow this pattern: either .000, .333, or .666. If you divide by 30, the pattern just moves up a decimal place: the second decimal will always be 3 or 6.

In this study, every average score was divided by 30, because each group (male-ponytail, male-loose, female-bun, and so on) had 30 people in it. But every average number was perfectly round: 1.80, 2.80, 1.60. That’s … unlikely. “The chance of all six means ending in zero this way is 0.0014,” write Heathers and Brown in their critique.

Many of Guéguen's studies involve elaborate situations using confederates - research assistants who pretend to be a participant or random person on the street. But Guéguen publishes many single author papers without acknowledgements. When Heathers and Brown reached out to Guéguen for more information on how he could publish so many elaborate studies on his own, he explained that he supervises many student projects. But why aren't the students at least thanked in the papers? Or listed as a coauthor, as they should be if they're doing a great deal of the work?

Heathers and Brown have repeatedly reached out to Guéguen for some documentation to substantiate that these studies occurred as described, but email correspondence, ethics review committee reports, and original datasets have not been shared in many cases.

Brown will be publishing the results of his and Heathers's examination of Guéguen's work on his blog. The first critique can be found here.

As has happened before, this particular instance of alleged academic dishonesty is liable to lead to a discussion about the problems of the "publish or perish" mentality in academia and research. But, as in previous cases, it's unlikely that such a discussion will result in any real improvements to dissuade such honesty. When the benefit of publishing a great deal is high and the probability of being caught is low, these things will continue to happen. Completely fabricated research is rare and likely to remain so, but tiny slips into academic dishonesty - massaging numbers or dropping cases - will happen, even by the most honest of researchers.