University of Surrey Open Research logo

Reproducibility

An overview of reproducibility and replicability.

Contact the Open Research team

What are reproducibility and replicability?

Definitions of “reproducibility” and “replicability” can vary considerably. These terms are sometimes used interchangeably, both within and across disciplines. Here we provide commonly understood definitions of the terms:

Reproducibility

Refers to obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis.

Replicability

Means obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.

Reproducibility

The reproducibility of both research methods and research results is critical to research, particularly in the experimental sciences with a quantitative focus.

Reproducibility forms part of Surrey’s wider commitment to transparency and rigour in all of our research. We recognise that behaviours in support of transparency and rigour vary considerably across disciplines and methodologies, and encourage our researchers to adopt actions most appropriate to their disciplines.

In the arts, humanities and social sciences, it may be more useful to focus on transparency or academic rigour in the use of research methods and in the whole research process – from the collection of evidence or thoughts through analysis to final conclusions and the publication of findings.

The reproducibility of research methods is required for research to be replicated, i.e., enabling consensus to emerge in scholarly communities. This, in turn, is essential in research contexts where findings must be robust and reproducible in order to form a solid foundation on which to build further knowledge.

In research contexts where reproducibility is possible and appropriate, we strongly encourage researchers to use measures that support it. These include (but are not limited to):

  • Preregistration of study procedures and analysis plans, and use of Registered Reports where appropriate
  • Transparent reporting of research in line with recognised community guidelines
  • Disclosure of all tested conditions, analysed measures and results
  • Transparency around statistical methods (including sample size planning and statistical assumptions and pitfalls)
  • Use of preprints
  • Carrying out replication studies
  • Publication of “null” findings
  • Automate whenever possible.

Replicability

Replications are typically termed “close” or “conceptual”. The decision to pursue one over the other hinges on the intended function of the study. It is important to remember that there is no such thing as an exact replication.

Close replication

  • The study attempts to match the critical elements of the original study (i.e., those believed to be necessary to produce the original effect) as closely as possible
  • This might include matching the sample, procedures, and materials
  • If an effect is robust, it should be observable under the conditions of a close replication (Simons, 2014). Close replications are therefore used to help determine whether the original finding was credible.

Conceptual replication

  • A conceptual replication study tests an extension of the original study or theory to a new context. For example, whether an effect extends to a different population or life stage
  • This is done by varying some aspect(s) of the design (e.g., recruiting older participants when the effect was previously observed in younger participants)
  • A result similar to the original study is informative about the generalisability of the effect. That is, the effect still occurs under the new conditions
  • However, when the results differ from the original it is not possible to establish whether the difference is due to the features the replicator intentionally varied, or that the original study was invalid (e.g., due to sampling or measurement error).

 

Note: The definitions of reproducibility and replicability, and the content on reproducibility were originally published in the UKRN Statement on Transparency in Research, that was developed from the UCL Statement on Transparency in Research, November 2019.

Resources