Conference web site: AAAI 2019
Artificial intelligence, like any science, must rely on reproducible experiments to validate results. Still, reproducing results from research in AI is not easily accomplished. This may be because AI research has its own unique reproducibility challenges. These are related to the use of analytical methods that are still a focus of active investigation and problems due to non-determinism in standard benchmark environments and variance intrinsic to AI methods. Acknowledging these difficulties, AI research should be documented properly so that the experiments and results are clearly described.
In this AAAI-19 workshop, we aim to create a forum to make a roadmap for improving the reproducibility of research results presented at future AAAI conferences and in other AAAI publications.
Before the workshop, we strongly suggest that everyone participating at the workshop take a look at the the recommendations published in AI Magazine, as we seek to make recommendations for AAAI regarding reproducibility.
Joelle Pineau has made a Machine Learning Reproducibility Checklist. Make sure to read through it as well.
Program will be posted here. List of confirmed speakers:
We are particularly interested in submissions that report on efforts to reproduce papers accepted for presentation at AAAI-19. Papers from earlier editions of the conference are also welcome.These submissions could be from the authors of the original papers but could also come from others who endeavoured to reproduce the work.These submissions should document how the results of the original paper were reproduced, and discuss reproducibility challenges, lessons learned, and recommendations for best practices. We suggest following the recommendations presented here
Any topics related to reproducible AI are welcome, including position papers, surveys, recommendations, and comparisons of AI reproducibility with other fields of research. Our focus is especially on practical solutions for how to improve the reproducibility of research presented at AAAI.
See suggested reading list here
The workshop will last a full day and will consist of both oral and poster presentations, as well as panel and open discussion regarding how to make research results presented at AAAI reproducible.
Each submission will be in the form of an up to 8-page paper, using the main AAAI conference format. We leave to the authors if they want to anonymize their submissions or not. Papers should be submitted via EasyChair. Oral presentations and poster session participants will be selected from the submissions.
Submit papers here.