The analysis of the “replication crisis” in the social sciences – at least, as reported by RadioLab – seems to be forgetting the very thing that RadioLab does: reporting on science.
Stick with me here.
The short form of the replication crisis is that when important studies and experiments originally done five, ten, fifteen, or more years ago are repeated now, the results aren’t the same.
Since reproducing results is kind of the basic tenet of science, this is a big deal.
But here’s the part that appears to be overlooked: in the social sciences, doing the same experiment at different times is not automatically doing the same experiment.
The original 1995 study about stereotypes and language in testing – that RadioLab uses as their example – was widely reported in popular media.
Reporting the results changed the sample group being tested in a way that would not be caught by a control group.
Advertising research has long known that people resist attitudinal change when they percieve an attempt to change their attitudes.
Therefore, there should be a hypothesis that if the subjects of the replication study years later are aware of the original study, it will have an effect on the attempt to replicate the original study.
An example where this effect does not occur quickly comes to mind. The results of the “doll preference study” was reproduced sixty years later. But this experiment was using children who were probably unaware of the original study. Many experiments in the social sciences use individuals of college age and above. (The common and often unremarked upon sample bias of using the available pool of college students on campus is a separate issue.) These individuals would be more likely to be aware of prior research and skew results.
This hypothesis would be difficult to test directly. A pretest for knowledge would obviously inform participants of the prior study. A post test would be impacted by the study itself.
However, there is a first step toward testing this hypothesis (and I hope it’s already been done and I’m unaware of it).
What I would like to see is a metastudy determining whether the degree of the replication crisis varies based on age and education of the participants.
If we see that reproducibility decreases with the age and education of the participants, it would indicate that our analysis of the reproducing crisis needs reworking.