January 7 2021, Yokohama, Japan
There is an ongoing reproducibility crisis. Claims supported by research that is irreproducible cannot be trusted. One could expect that experiments in artificial intelligence conducted fully on a computer were reproducible, and that the reproducibility crisis is a problem restricted to research in social sciences and psychology. Counter-intuitively, this is not the case, and also research conducted fully on a computer could be irreproducible. As if that was not enough, artificial intelligence research is among the least reproducible sciences according to Ioannidis in Why most published research findings are false from 2005.
Reproducible research is good research. Making your research reproducible can be achieved by following good research practices. We have proposed a general guideline for making empirical AI research reproducible and to promote good research practices. In this tutorial, we will present the guidelines and provide practical advice on how to implement them in your research.
The tutorial is based on the General Guidelines for Making Empirical AI Research Reproducible that I co-wrote with Yolanda Gil, past president of AAAI and Mausam, AAAI Program Co-Chair. It can be found here:
Tutorial slides: PDF slides
Section 1: Motivation:
Section 2: Introduction to Reproducibility
Section 3: The AAAI Reproduicbility Checklist
Section 4: The values and challenges of reproducible research
Section 5: Conclusion
Everyone that submits papers to the top AI conferences, such as IJCAI and AAAI, is in the target audience.
The topic is interesting to the IJCAI researchers conducting empirical AI research that submit papers to IJCAI and AAAI. The guidelines will be implemented by these large AI conferences in one form or another the coming years. The tutorial will help authors to conduct better research and improve their papers so that they have an increased chance of being accepted at these conferences in the future.
Odd Erik Gundersen, odderik(at)ntnu.no
Adjunct Associate Professor, Norwegian University of Science and Technology
Chief AI Officer, TrønderEnergi