The AAAI 2020 Workshop on Reproducible AI - RAI2020
Future Direction and Reproducibility Challenge
February 7, full-day workshop
Venue: Hilton New York Midtown Hotel
Artificial Intelligence (AI), like any science, must rely on reproducible experiments to validate results. However, reproducing results from AI research publications is not easily accomplished. This may be because AI research has its own unique reproducibility challenges. For example, these include (1) the use of analytical methods that are still a focus of active investigation and (2) problems due to non-determinism in standard benchmark environments and variance intrinsic to AI methods. Acknowledging these difficulties, empirical AI research should be documented properly so that the experiments and results are clearly described.
AAAI does not provide any recommendations on how researchers can enhance the reproducibility of their work. In this AAAI-20 workshop, we aim to develop such recommendations, and to encourage future AAAI conferences to implement them. The goal is to finalize recommendations for AAAI 2021 and discuss how these should be evaluated. This year’s workshop is a continuation of the AAAI 2019 Workshop on Reproducible AI (RAI 2019) where we had several presentations and the attendees started discussing recommendations.
As input to the discussion, we organize the AAAI 2020 Reproducibility Challenge, where we encourage researchers to submit their experiences from reproducing paper accepted at previous AAAI conferences.
- Joelle Pinau: Machine Learning Reproducibility: An update from the NeurIPS 2019 Reproducibility Co-Chairs
- Jakub Kowalski: Experimental Studies in General Game Playing: An Experience Report
- Michael Gordon: Replication Markets in the Social and Behavioural Sciences
- Sweitze Roffel and George Tsatsaronis: Towards Reproducible Artificial Intelligence - Roles and Responsibilities of Researchers and Publishers
- Jakub Kowalski and Marek Szykula: Experimental Studies in General Game Playing: An Experience Report
- Michael Gordon, Thomas Pfeiffer, Domenico Viganola and Yang Liu: Replication Markets in the Social and Behavioural Sciences
- Anita de Waard, Georgios George Tsatsaronis, Rinke Hoekstra, Sweitze Roffet, Catriona Fennell, Sergios Petrides, Thom Pijnenburg and Efthymios Tsakonas: Towards Reproducible Artificial Intelligence: The Roles of Researchers and Publishers
- Yang Liu, Michael Gordon, Juntao Wang, Michael Bishop, Yiling Chen, Thomas Pfeiffer, Charles Twardy and Domenico Viganola: Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
- Vani Mandava: Reproducibility with Microsoft Research Open Data
As input to the discussion on recommendations, we will emphasize the submission and acceptance of papers in which researchers describe their experiences from attempting to reproduce a paper(s) accepted at a previous AAAI conference(s) (i.e., try to reproduce the results from a previous AAAI conference paper and report your results).
Submissions should contain a description of the experiment, whether the results of the original paper were reproduced or not, a discussion on reproducibility challenges, lessons learned, and recommendations for best practices as well as a short note on each of the 24 variables presented in by Gundersen, Gil and Aha (2018).
Any topics related to reproducible AI are welcome, including position papers, surveys, recommendations, and comparisons of AI reproducibility with other fields of research. Our focus is especially on practical solutions for how to improve the reproducibility of research presented at AAAI.
See suggested reading list here.
This workshop will span a full day and will include invited talks, oral and poster presentations of submitted work, a panel and open discussion on how to make research results presented at AAAI reproducible.
Workshop submission deadline: Nov 15 Notice of acceptance: Dec 4 Final version: Jan 20 Early registration deadline: Dec 13 Workshop date: Feb 7
Each submission will be in the form of a maximum 8-page paper including reference, using the main AAAI conference format. Authors can optionally anonymize their submissions. Papers should be submitted via EasyChair. Oral presentations and poster session participants will be selected from the submissions. Please send an email to the workshop chairs if you consider submitting a paper.
Please use this link to submit papers.