Human Annotation

Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is important for applications that are sensitive to such information. …

Evaluation should model meaningful disagreement instead of reducing every task to a single gold label.