The AAAI 2020 Workshop on Reproducible AI - RAI2020

Future Direction and Reproducibility Challenge

February 7, full-day workshop

Venue: Hilton New York Midtown Hotel

Artificial Intelligence (AI), like any science, must rely on reproducible experiments to validate results. However, reproducing results from AI research publications is not easily accomplished. This may be because AI research has its own unique reproducibility challenges. For example, these include (1) the use of analytical methods that are still a focus of active investigation and (2) problems due to non-determinism in standard benchmark environments and variance intrinsic to AI methods. Acknowledging these difficulties, empirical AI research should be documented properly so that the experiments and results are clearly described.

AAAI does not provide any recommendations on how researchers can enhance the reproducibility of their work. In this AAAI-20 workshop, we aim to develop such recommendations, and to encourage future AAAI conferences to implement them. The goal is to finalize recommendations for AAAI 2021 and discuss how these should be evaluated. This year’s workshop is a continuation of the AAAI 2019 Workshop on Reproducible AI (RAI 2019) where we had several presentations and the attendees started discussing recommendations.

As input to the discussion, we organize the AAAI 2020 Reproducibility Challenge, where we encourage researchers to submit their experiences from reproducing paper accepted at previous AAAI conferences.

Reproducibility Challenge

As input to the discussion on recommendations, we will emphasize the submission and acceptance of papers in which researchers describe their experiences from attempting to reproduce a paper(s) accepted at a previous AAAI conference(s) (i.e., try to reproduce the results from a previous AAAI conference paper and report your results).

Submissions should contain a description of the experiment, whether the results of the original paper were reproduced or not, a discussion on reproducibility challenges, lessons learned, and recommendations for best practices as well as a short note on each of the 24 variables presented in by Gundersen, Gil and Aha (2018).

Topics

Any topics related to reproducible AI are welcome, including position papers, surveys, recommendations, and comparisons of AI reproducibility with other fields of research. Our focus is especially on practical solutions for how to improve the reproducibility of research presented at AAAI.

Relevant reading

See suggested reading list here.

Format

This workshop will span a full day and will include invited talks, oral and poster presentations of submitted work, a panel and open discussion on how to make research results presented at AAAI reproducible.

Yolanda Gil and Joelle Pinau have accepted to share their experiences and provide input to the recommendations through invited talks.

Important dates

Submissions

Each submission will be in the form of a maximum 8-page paper including reference, using the main AAAI conference format. Authors can optionally anonymize their submissions. Papers should be submitted via EasyChair. Oral presentations and poster session participants will be selected from the submissions. Please send an email to the workshop chairs if you consider submitting a paper.

Please use this link to submit papers.

Workshop Chairs