These questions are based on the Science News (SN) article “Replication crisis spurs reforms.”

Making initial connections:

1. Based on the article’s title, what background knowledge, if any, do you have about the topic?

Possible student response: Science is founded on finding results that are experimentally testable and reproducible. If one scientist’s experiments show a particular result, other scientists should be able to do similar experiments and get similar results.

2. What would you like to learn from the article?

Possible student response: The difficulties and mistakes that plague real researchers. When we hear about the results of scientific studies in the news, how readily should we believe those results?

Summarizing and citing evidence and structure:

3. What specialized terms are used in the article, and what do they mean?

Possible student response: Social science includes fields such as economics and psychology. Replication or reproducibility means that other scientists can repeat an experiment and get similar results.

4. What is the main idea of the article?

Possible student response: The reported results of many published social science studies cannot be replicated. When other scientists try to repeat a study, it doesn’t work. That raises questions about whether the results of the first study are correct. Social scientists need to be more careful when validating conclusions of studies based solely on statistical results. Journals need to be more careful when reviewing and deciding what studies to publish.

5. What evidence does the article provide to support its central idea?

Possible student response: The scientists picked 21 social science articles from two major journals, Nature and Science, and tried to reproduce their results, even using four to five times as many participants than the original studies. The results of eight of the original studies could not be reproduced, and the results of the other studies were found to be reproducible but less significant than initially reported. This evidence appears to support the central idea that there are widespread problems with published studies in the social sciences.

Integrating and evaluating sources:

6. Scientists present their detailed results in primary research articles in scientific journals and/or presentations or posters at scientific conferences. SN and other news outlets then report some of those results for a wider audience. Citations of the primary research studies are listed and linked at the end of the SN article online. What specific scientists, primary research studies, scientific conference presentations and other sources does the SN article quote, mention or cite?

Possible student response:
Colin Camerer, an economist at Caltech.
Brian Nosek, a psychologist at the University of Virginia in Charlottesville.
Will Gervais, a psychologist at the University of Kentucky in Lexington.
Gerd Gigerenzer, a psychologist at the Max Planck Institute for Human Development in Berlin.
C.F. Camerer et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behavior. Published online August 27, 2018.

7. How well does the SN article explain the scientific work? Does it clearly explain the purpose, methods, results and implications of the scientific work? Does it leave out some details?

Possible student response: The SN article gives a general overview and analysis of the investigative report as well as reactions from the scientific community. The article doesn’t go into detail about the specific studies that scientists attempted to replicate. If the article were longer, it might include more of the data. The statistical results would be helpful in knowing how far off replicated study results were from the original study results.

8. Compare and contrast the SN article to a second SN article that also covers replication research. What are important similarities and differences between the sources? What new things have you learned from fitting all of the information together? (Ask your teacher for the log in credentials: https://www.sciencenews.org/learning/search.)

Possible student response: “Cancer studies get mixed grades on redo tests,” published February 18, 2017, in Science News, takes a similar look at the reproducibility of results from cancer biology studies. In trying to reproduce the results of five prominent cancer studies, the researchers found that the results of one study could not be replicated at all, while some results from the other four studies could not be replicated. Based on these two articles showing replication problems in the social sciences and in cancer research, it seems likely that similar problems could be widespread across many fields of science.

Analyzing content:

9. How could you visually present some of the ideas or results discussed in “Replication crisis spurs reforms”?

Possible student response: A pie chart could show the fraction of the studies that were not reproducible, or a graph could show the percentages of statistics faculty, professors and lecturers, and students who believe that statistical significance means a result requires no replication.

10. Why should people care about the article? What is the importance, impact or significance of the article for you, your community or the scientific community?

Possible student response: The results of social science studies are sometimes used to determine the policies and programs of public and private institutions, including schools. It is important for the results of such social science studies to be truly meaningful, which means the results must be reproducible. A policy or program should be created based on a body of evidence built up over multiple studies, only if those results are representative of a population that is larger than the ones that were studied.

11. What related work might you or other students be able to do, perhaps as a science fair project?

Possible student response: There are many behavioral science projects in science fairs. I could replicate a previous behavioral science project, or possibly collect data from many previous science fair projects and do a meta-analysis of their statistical significance and reproducibility.

For information on making your science fair project reproducible, check out Cookieology.

12. Circle or highlight scientific processes that are covered in the article. Such processes may include asking questions and defining problems; developing and using models; planning and carrying out investigations; analyzing and interpreting data; using mathematics and computational thinking; constructing explanations and designing solutions; engaging in argument from evidence; and obtaining, evaluating, and communicating information. Cite an example from the text for one of these processes.

Possible student response: The article focuses on analyzing and interpreting data. It emphasizes that data analysis and interpretation must be done correctly in order to ensure that the results are truly meaningful and can be replicated in similar experiments.

13. Explain how one of the following concepts is represented in the article: patterns; cause and effect; scale, proportion or quantity; systems and system models; matter and energy; structure and function; rates of change and stability of a system.

Possible student response: The social science studies in question were intended to measure whether certain causes had certain effects, but in many cases researchers apparently mistook random results for actual correlations between causes and effects.