What I talk about when I talk about reproducibility: A tutorial

IJCAI 2019 Tutorial, August 10-16 2019, Macao, China

The tutorial is given by Odd Erik Gundersen, Norwegian University of Science and Technology.

This web page will be updated irregularly before IJCAI 2019 to include more information such as slides.


Please find the slides from the tutorial here. I had a great time, and I hope you did as well. Thanks for all the great questions!


There is an ongoing reproducibility crisis. Most research findings are false. Although many experiments can be completely executed on computers when conducting AI research, their results are still not necessarily reproducible. Why is this so?

In this tutorial, the concept of reproducibility will be investigated. The problems related to reproducibility and recommendations for how to solve them are discussed. Some outstanding issues related to reproducibility are presented. Finally, the future of reproducibility research is outlined.

Target audience

This topic is relevant for all researchers conducting empirical AI research.

Outline of the tutorial


Section 1: Understanding reproducibility

Section 2: Causes of irreproducibility in computer science and AI

Section 3: Recommendations

Section 4: Challenges

Section 5: Future