In-Class Data Analysis Replications: Teaching Students While Testing Science

Abstract

Science is facing a reproducibility crisis. Overcoming it would require concerted efforts to replicate prior studies, but the incentives for researchers are currently weak, as replicating prior studies requires considerable time and effort without providing the same level of recognition as de novo research. Previous work has proposed incorporating data analysis replications into classrooms as a potential solution. However, despite the potential benefits, it is unclear what the involved stakeholders—students, educators, and scientists—should expect from it. What are the costs and benefits? And how can this solution help benchmark and improve the state of science?In the present study, we incorporated data analysis replications in the project component of the CS-401 Applied Data Analysis course (ADA) taught at EPFL (École Polytechnique Fédérale de Lausanne) enrolling N = 354 students. First, we report preregistered findings based on surveys administered throughout the course. We find discrepancies between what students expect of data analysis replications and what they experience by doing them, along with changes in expectations about reproducibility. Second, we provide information for educators about how much overhead is needed to incorporate replications into the classroom and identify concerns that replications bring, as compared to more traditional assignments. Finally, we discuss potential implications of in-class data analysis replications for scientific communities, such as insights about replication barriers in scientific work that should be avoided going forward.Overall, we demonstrate that incorporating replication tasks into a large data science class can increase the reproducibility of scientific work as a by-product of data science instruction.