Careful Considerations in NLP - Datasets and Error Analysis

Notes from a couple of talks hosted by Rsqrd AI by two UW computational linguistics PhD students:

• On the topic of “Using Datasets Wisely”, Amandalynne Paullada who studies under Fei Xia and works on biomedical applications with Trevor Cohen.
• Angie McMillan-Major on “Operationalizing Error Analysis in NLP”

Resources

The Point of Collection

“Data sets are the results of their means of collection” – Mimi Onuoha

• The Point of Collection by Mimi Onuoha

100 Essentials books

by Emily Bender

• Linguistic Fundamentals for Natural Language Processing: 100 Essentials from Morphology and Syntax
• Linguistic Fundamentals for Natural Language Processing II: 100 Essentials from Semantics and Pragmatics

Papers

• Know thy corpus! Robust methods for digital curation of Web corpora
• Datasheets for Datasets
• Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models