Concept - Inter-Annotator Agreement

Inter-Annotator Agreement

Used to validate annotations (s.374): how likely are two annotators to choose the same label for an instance, and how likely would that be if their labels were randomly shuffled?

Example (s.375-381) Asha (A) and Boris (B), 100 jokes:

	A = funny	A = not funny
B = funny	70	20
B = not funny	5	5

• Observed agreement = (70+5)/100 = 75%
• Chance agreement: $\frac{70+20}{100}\cdot \frac{70+5}{100}+\frac{5+5}{100}\cdot \frac{20+5}{100}=\frac{27}{40}+\frac{1}{40}=70\%$

Added context

Cohen’s kappa $=(p_o-p_e)/(1-p_e)=(0.75-0.70)/0.30\approx 0.17$ (not on the slides).

Related

Concept - Getting Labels

Appears in

• 03 Text Classification - Chunk 13 (s.361-381)