IJCAI2020 Tutorial on Making Your AI Research Reproducible

January 7 2021, Yokohama, Japan

Making Research Reproducible

There is an ongoing reproducibility crisis. Claims supported by research that is irreproducible cannot be trusted. One could expect that experiments in artificial intelligence conducted fully on a computer were reproducible, and that the reproducibility crisis is a problem restricted to research in social sciences and psychology. Counter-intuitively, this is not the case, and also research conducted fully on a computer could be irreproducible. As if that was not enough, artificial intelligence research is among the least reproducible sciences according to Ioannidis in Why most published research findings are false from 2005.

Reproducible research is good research. Making your research reproducible can be achieved by following good research practices. We have proposed a general guideline for making empirical AI research reproducible and to promote good research practices. In this tutorial, we will present the guidelines and provide practical advice on how to implement them in your research.

Outline of the tutorial

The tutorial is based on the General Guidelines for Making Empirical AI Research Reproducible that I co-wrote with Yolanda Gil, past president of AAAI and Mausam, AAAI Program Co-Chair. It can be found here:


Tutorial slides: PDF slides

Section 1: Motivation:

Section 2: Introduction to Reproducibility

 Section 3: The AAAI Reproduicbility Checklist

Section 4: The values and challenges of reproducible research

Section 5: Conclusion

Target audience

Everyone that submits papers to the top AI conferences, such as IJCAI and AAAI, is in the target audience.

The topic is interesting to the IJCAI researchers conducting empirical AI research that submit papers to IJCAI and AAAI. The guidelines will be implemented by these large AI conferences in one form or another the coming years. The tutorial will help authors to conduct better research and improve their papers so that they have an increased chance of being accepted at these conferences in the future.


Odd Erik Gundersen, odderik(at)ntnu.no

Adjunct Associate Professor, Norwegian University of Science and Technology

Chief AI Officer, Tr√łnderEnergi