IJCAI2020 Tutorial on Making Your AI Research Reproducible

January 7 2021, Yokohama, Japan

Making Research Reproducible

There is an ongoing reproducibility crisis. Claims supported by research that is irreproducible cannot be trusted. One could expect that experiments in artificial intelligence conducted fully on a computer were reproducible, and that the reproducibility crisis is a problem restricted to research in social sciences and psychology. Counter-intuitively, this is not the case, and also research conducted fully on a computer could be irreproducible. As if that was not enough, artificial intelligence research is among the least reproducible sciences according to Ioannidis in Why most published research findings are false from 2005.

Reproducible research is good research. Making your research reproducible can be achieved by following good research practices. We have proposed a general guideline for making empirical AI research reproducible and to promote good research practices. In this tutorial, we will present the guidelines and provide practical advice on how to implement them in your research.

Outline of the tutorial

The tutorial is based on the General Guidelines for Making Empirical AI Research Reproducible that I co-wrote with Yolanda Gil, past president of AAAI and Mausam, AAAI Program Co-Chair. It can be found here:

https://folk.idi.ntnu.no/odderik/reproducibility_guidelines.pdf

Tutorial slides: PDF slides

Section 1: Motivation:

• The challenges with reproducing AlphaGo.
• The reproducibility crisis (Nature study, ICLR reproducibility challenge).

Section 2: Introduction to Reproducibility

• The scientific method.
• Defining and understanding reproducibility.

 Section 3: The AAAI Reproduicbility Checklist

• Overview: The four sections.
• The paper: Research protocol, registered reports, structured abstracts, pseudo code
• Theoretical contributions
• Experiments relying on datasets: How well is data documented, Unbiased look at dataset bias, other issues, drawin conclusions from data.
• Computational Experiments: HW and Ancillary SW, code versioninbg, DRL that matters, Algorithm runs and variation

Section 4: The values and challenges of reproducible research

• The 10 year reproducibilty challenge
• The different incentives for the original and independent researchers.
• THe value of code and data.

Section 5: Conclusion

• What if you do not have time to do it all?
• Where should you start?
• What is the least amount of work to increase reproducibility?

Target audience

Everyone that submits papers to the top AI conferences, such as IJCAI and AAAI, is in the target audience.

The topic is interesting to the IJCAI researchers conducting empirical AI research that submit papers to IJCAI and AAAI. The guidelines will be implemented by these large AI conferences in one form or another the coming years. The tutorial will help authors to conduct better research and improve their papers so that they have an increased chance of being accepted at these conferences in the future.

Instructor

Odd Erik Gundersen, odderik(at)ntnu.no

Adjunct Associate Professor, Norwegian University of Science and Technology

Chief AI Officer, TrønderEnergi