What I talk about when I talk about reproducibility: A tutorial

IJCAI 2019 Tutorial, August 10-16 2019, Macao, China

The tutorial is given by Odd Erik Gundersen, Norwegian University of Science and Technology.

This web page will be updated irregularly before IJCAI 2019 to include more information such as slides.

Slides

Please find the slides from the tutorial here. I had a great time, and I hope you did as well. Thanks for all the great questions!

Description

There is an ongoing reproducibility crisis. Most research findings are false. Although many experiments can be completely executed on computers when conducting AI research, their results are still not necessarily reproducible. Why is this so?

In this tutorial, the concept of reproducibility will be investigated. The problems related to reproducibility and recommendations for how to solve them are discussed. Some outstanding issues related to reproducibility are presented. Finally, the future of reproducibility research is outlined.

Target audience

This topic is relevant for all researchers conducting empirical AI research.

Outline of the tutorial

Introduction:

• The reproducibility crisis (Nature study, ICLR reproducibility challenge)
• Why is reproducibility needed (science and reproducibility)?

Section 1: Understanding reproducibility

• What is reproducibility (definitions and problems with them)
• A structured definition that can be evaluated empirically.
• Quantifying reproducibility
• The advantages of quantifying reproducibility

Section 2: Causes of irreproducibility in computer science and AI

• The complete experiment is done using a computer – how can this not be reproducible.
• Experiments that reveals some problems with reproducibility.
• Problems of reproducibility that are special for AI and ML (randomness, performance metrics, robotics, reinforcement learning, big data, computing requirements).

Section 3: Recommendations

• Best practices related to study design, evaluations and documentation.
• How to document AI methods, experiments, code and data.

Section 4: Challenges

• Some issues that have not yet been solved (what exactly should be documented, what is meant by “same results”?)
• Which barriers are there to reproducibility?
• How can barriers related to reproducibility be removed?
• Who is responsible for making research reproducible (researchers, publishers, grant makers, academic institutions)?

Section 5: Future

• How does the future of reproducibility look?
• The scientific paper of the future
• Some software platforms for reducing the work related to reproducibility.

Summary

• Reproducibility is a crucial part of science.
• Small steps make huge a difference