How-to for AAAI Reproducibility Guidelines

This web page provides pointers to resources that can be used to implement the General Guidelines for Making Empirical AI Research Reproducible.

In order to improve the reproducibility of your experiment, you should properly describe the experiment and the claims that it supports. Also, research artifacts, such as data, the code that preprocesses data, sets up and runs the experiment and instructions for how to run the code, should be shared.

It is not mandatory to share research artifacts, but if you do, you should share and cite them in a way that removes any uncertainty about which code and data are used for producing the results presented in the paper.

There are many online resources that simplifies sharing research artifacts, such as data and code. The following three resources support the functionality that is required to follow the guidelines.

In alphabetical order:

Table 1: All three resources supports sharing data and code.

Data

Code

Media

Version

DOI

License

Github

BibTex

Word

View only

Free storage

Zenodo

Y

Y

Y

Y

Y

Y

Y

Y

Y

N

Unlimited

(50GB per data set)

Figshare

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Unlimited

(5GB per file)

Open Science Foundation

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

15GB

(Google Disk)

All three resources support uploading files directly or through Github by linking Github accounts. This is of course done in different ways for the three resources. A license can be specified as part of the process of uploading new research artifacts to these resources. Versioning is supported by all three and digital object identifiers (DOI) can be made for each uploaded research artifact.

Both Figshare and Zenodo provide unlimited storage. Storage is limited to 50 GB per data set for Zenodo and 5 GB per file for Figshare. Open Science Foundation provides different types of storage with different limits. Google Disk is one of these options, and it supports up to 15GB for free.

Both Figshare and Open Science Foundation support sharing view only or private links to the research artifacts. These are useful for double-blind peer review processes.

Although all three resources support Bibtex, the Bibtex entries must be edited in order to display the URL containing the DOI. You should add an entry "note" that contains the DOI in form of a URL, for exmaple: note = {https://doi.org/10.5281/zenodo.3519603}, where 10.5281/zenodo.3519603 is the actual DOI that will be provided by the resources (Zenodo in this example). In this way the URL will be displayed in the reference section for AAAI publications.

Feedback Form

Please provide feedback on the guidelines and sharing research artifacts. The feedback will help us improve the guidelines for future conferences.

Use this form to provide feedback: Feedback form

All questions can be directed to Odd Erik Gundersen, email: odderik [at] ntnu.no