Joelle Pineau, of Facebook AI Research and McGill University, is an AI researcher studying planning and learning algorithms applied to robotics, health care, games, and conversation. She put together this awesome, if exacting, reproducibility checklist and ran a reproducibility challenge at NeurIPS 2019. Read the Nature interview here: This AI researcher is trying to ward off a reproducibility crisis.

Reproducibility Checklist

The mission to encourage reproducibility gets at the question of how to do peer review in the age of arxiv. One frustrated researcher started papers without code as the naming-and-shaming mirror image of paperswithcode.

Laying bricks onto the temple of science assumes the ability to build on the work of others. Once we’ve mastered the art of fully specifying computational methods, including source code and dependencies, data availability, and data quality, then the frontier shifts to disentangling questions of same data, different conclusions.

Having worked toward reproducible research in life-science in a past life, it’s great to see this kind of effort in the AI community.