Teamwork and the Reproducibility Problem

It has been known for some time that psychology has a reproducibility problem, though we may not always agree on how to handle or discuss these issues. I remember chatting with another researcher at a conference shortly after I finished my masters thesis on stereotype threat and its impact on math performance in women. I had failed to replicate stereotype threat effects in my study. She, on the other hand, said her effects were incredibly strong; she described a participant experiencing a panic attack when she was told she had to do math problems, and had even noticed her female participants' math performance was negatively affected when her research assistant had been knitting during the session. (I also remember a reviewer telling me I must have performed the study poorly, not because the reviewer found any flaws in my methods, but because I had failed to reproduce the stereotype threat effects in my research.)

Efforts to handle this crisis thus far have included making psychological research more transparent and large-scale meta-analyses. And a new effort is already underway to harness the power of multiple research labs across the world: the Psychological Science Accelerator. Christie Aschwanden of FiveThirtyEight has more:

[Psychologist Christopher] Chartier, a researcher at Ashland University, doesn’t think massively scaled group projects should only be the domain of physicists. So he’s starting the “Psychological Science Accelerator,” which has a simple idea behind it: Psychological studies will take place simultaneously at multiple labs around the globe. Through these collaborations, the research will produce much bigger data sets with a far more diverse pool of study subjects than if it were done in just one place.

The accelerator approach eliminates two problems that can contribute to psychology’s much-discussed reproducibility problem, the finding that some studies aren’t replicated in subsequent studies. It removes both small sample sizes and the so-called weird samples problem, which is what happens when studies rely on a very particular population — like relatively wealthy college students from Western countries — that may not represent the world at large.

So far, the project has enlisted 183 labs on six continents. The idea is to create a standing network of researchers who are available to consider and potentially take part in study proposals, Chartier said. Not every lab has to participate in any given study, but having so many teams in the network ensures that approved studies will have multiple labs conducting their research.

According to the blog, the Psychological Science Accelerator is taking on its second study, this one on gendered social category representation. And if you're attending the Association for Psychological Science meeting in May, you can check out a symposium on "Large Scale Research Collaborations: Applications in Crowd-Sourcing and Undergraduate Research Experience, Replications, and Cross-Cultural Research." (Day and time TBD - APS is still finalizing the program, and is still accepting poster submissions through the end of this month.)