Artificial Intelligence (AI), like any science, must rely on reproducible experiments to validate results. However, reproducing results from AI research publications is not easily accomplished. This may be because AI research has its own unique reproducibility challenges. For example, these include (1) the use of analytical methods that are still a focus of active investigation and (2) problems due to non-determinism in standard benchmark environments and variance intrinsic to AI methods. Acknowledging these difficulties, empirical AI research should be documented properly so that the experiments and results are clearly described.

AAAI does not provide any recommendations on how researchers can enhance the reproducibility of their work. In this AAAI-20 workshop, we aim to develop such recommendations, and to encourage future AAAI conferences to implement them. The goal is to finalize recommendations for AAAI 2021 and discuss how these should be evaluated. This year’s workshop is a continuation of the AAAI 2019 Workshop on Reproducible AI (RAI 2019) where we had several presentations and the attendees started discussing recommendations.

As input to the discussion, we organize the AAAI 2020 Reproducibility Challenge, where we encourage researchers to submit their experiences from reproducing paper accepted at previous AAAI conferences.

Program

Presentations

• Joelle Pinau: Machine Learning Reproducibility: An update from the NeurIPS 2019 Reproducibility Co-Chairs
• Jakub Kowalski: Experimental Studies in General Game Playing: An Experience Report
• Michael Gordon: Replication Markets in the Social and Behavioural Sciences
• Sweitze Roffel and George Tsatsaronis: Towards Reproducible Artificial Intelligence - Roles and Responsibilities of Researchers and Publishers

Papers

• Jakub Kowalski and Marek Szykula: Experimental Studies in General Game Playing: An Experience Report
• Michael Gordon, Thomas Pfeiffer, Domenico Viganola and Yang Liu: Replication Markets in the Social and Behavioural Sciences
• Anita de Waard, Georgios George Tsatsaronis, Rinke Hoekstra, Sweitze Roffet, Catriona Fennell, Sergios Petrides, Thom Pijnenburg and Efthymios Tsakonas: Towards Reproducible Artificial Intelligence: The Roles of Researchers and Publishers
• Yang Liu, Michael Gordon, Juntao Wang, Michael Bishop, Yiling Chen, Thomas Pfeiffer, Charles Twardy and Domenico Viganola: Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
• Vani Mandava: Reproducibility with Microsoft Research Open Data

Reproducibility Challenge

As input to the discussion on recommendations, we will emphasize the submission and acceptance of papers in which researchers describe their experiences from attempting to reproduce a paper(s) accepted at a previous AAAI conference(s) (i.e., try to reproduce the results from a previous AAAI conference paper and report your results).

Submissions should contain a description of the experiment, whether the results of the original paper were reproduced or not, a discussion on reproducibility challenges, lessons learned, and recommendations for best practices as well as a short note on each of the 24 variables presented in by Gundersen, Gil and Aha (2018).

Topics

Any topics related to reproducible AI are welcome, including position papers, surveys, recommendations, and comparisons of AI reproducibility with other fields of research. Our focus is especially on practical solutions for how to improve the reproducibility of research presented at AAAI.

Relevant reading

See suggested reading list here.

Format

This workshop will span a full day and will include invited talks, oral and poster presentations of submitted work, a panel and open discussion on how to make research results presented at AAAI reproducible.

Yolanda Gil and Joelle Pinau have accepted to share their experiences and provide input to the recommendations through invited talks.

Important dates

• Workshop submission deadline: Nov 15
• Notice of acceptance: Dec 4
• Final version: Jan 20
• Early registration deadline: Dec 13
• Workshop date: Feb 7

Submissions

Each submission will be in the form of a maximum 8-page paper including reference, using the main AAAI conference format. Authors can optionally anonymize their submissions. Papers should be submitted via EasyChair. Oral presentations and poster session participants will be selected from the submissions. Please send an email to the workshop chairs if you consider submitting a paper.

Please use this link to submit papers.

Workshop Chairs

• Odd Erik Gundersen, Norwegian University of Science and Technology, odderik_at_ntnu.no
• David W. Aha, Naval Research Laboratory, david.aha_at_nrl.navy.mil
• Daniel Garijo, University of Southern California, dgarijo_at_isi.edu