← Back to project catalogue
GP-DA-1NAH22QData and AIReady

Label disagreement and ambiguity laboratory

A controlled Data and AI study of annotator disagreement, ambiguous examples, consensus methods, soft labels, and uncertainty-based review.

Label disagreement and ambiguity laboratory project visual
GP-DA-1NAH22Q · Data and AI
  • Python 3.11
  • NumPy
  • pandas
  • SciPy
  • Matplotlib

Project definition

Problem statement

People may disagree because an example is difficult, two classes overlap, several interpretations are valid, the written guide is incomplete, or an annotator makes an error.

The engineering problem is to compare ways of combining these labels without hiding useful uncertainty, then measure when uncertain items should be sent for review.

Project objectives

  • Create controlled clear, boundary, multimodal, rare-class, and rubric-gap examples.
  • Generate repeated labels from generalist, specialist, systematically biased, and inattentive annotator profiles.
  • Compare majority vote, raw vote distributions, control weighting, and Dawid-Skene estimation.
  • Measure hard-label accuracy and full-distribution recovery separately.
  • Study label budgets and entropy-ranked human review.

Project structure

Project components

01

Scenario generator

Creates 2,400 four-class items with known latent distributions and five controlled disagreement sources.

02

Annotator simulator

Generates 24 synthetic annotators with declared strengths, specialisation, systematic confusion, and inattentive behavior.

03

Aggregation laboratory

Runs majority vote, raw vote distributions, control-weighted distributions, and a transparent Dawid-Skene EM method.

04

Evaluation engine

Measures hard accuracy, cross entropy, Brier score, Jensen-Shannon divergence, entropy error, and ambiguity AUC.

05

Review analyser

Ranks items by entropy and measures review precision and recall across available review budgets.

06

Evidence output

Writes CSV, JSON, eight figures, a self-contained dashboard, and reproducible experiment summaries.

Methodology

Project workflow

  1. 01
    Generate items

    Create the controlled class distributions and mark the synthetic control and ambiguity status.

  2. 02
    Collect repeated labels

    Sample 12 labels per item from the declared annotator processes.

  3. 03
    Aggregate

    Apply each method to the same retained labels at budgets of 3, 5, 8, and 12.

  4. 04
    Evaluate

    Compare modal-label recovery, distribution recovery, uncertainty alignment, and worker estimates.

  5. 05
    Prioritise review

    Rank uncertain items and measure the review workload and known ambiguity captured.

Demonstration scenario

The student selects one ambiguous item and inspects its 12 labels. Majority vote shows one class, while the soft methods preserve support for other classes. The dashboard then compares hard accuracy with distribution recovery and shows which items would be sent for review at a 20 percent budget.

Engineering

Tools and method

Runtime
Python 3.11 or later with an installable command-line package.
Data processing
NumPy and pandas for controlled sampling, retained annotations, aggregation tables, and reproducible outputs.
Statistical methods
SciPy and transparent project code for categorical metrics, ROC AUC, bootstrap intervals, and Dawid-Skene EM.
Presentation
Matplotlib figures and one local self-contained HTML dashboard generated from retained evidence.
Quality controls
Automated tests, Ruff, branch coverage, dependency audit, delivery validation, and document accessibility audits.

Testing

Evaluation

Evaluation measures

  • Hard modal-label accuracy for each aggregation method
  • Cross entropy, Brier score, and Jensen-Shannon divergence
  • Entropy mean absolute error and ambiguity-ranking AUC
  • Performance at 3, 5, 8, and 12 labels per item
  • Review precision and recall across review budgets
  • Five hundred seeded bootstrap samples

Project boundaries

  • All items and annotators are synthetic and contain no personal information.
  • The experiment does not estimate the quality, behavior, beliefs, or ambiguity rate of real people.
  • Entropy can prioritise review but cannot identify why real annotators disagreed.
  • A real annotation study requires consent, sampling, privacy protection, rubric versioning, and independent validation.

Included

  1. 01Complete Python source code and command-line tools
  2. 02Five controlled disagreement scenarios and four annotator groups
  3. 032,400 items and 28,800 retained annotations
  4. 04Four aggregation methods and four label budgets
  5. 05CSV and JSON results, eight labelled figures, and an offline dashboard
  6. 06Automated tests with 99.31 percent combined coverage
  7. 07Complete source code in a private GitHub repository
  8. 08114-page project documentation in PDF and editable Word formats
  9. 0920-page setup and usage guide in PDF and editable Word formats
  10. 1045 annotated references through August 2026

Project record

No information is collected on this page.

Permanent project ID
GP-DA-1NAH22Q
Catalogued
21 Aug 2026
Completed
24 Aug 2026
Verified
24 Aug 2026
Demonstration
Included in repository

Handover

After purchase

  1. 01
    Payment is confirmed

    The project is marked unavailable and cannot be purchased again.

  2. 02
    Repository access is granted

    The buyer's submitted GitHub account receives access to the private repository.

  3. 03
    The purchase record is delivered

    The certification sheet is prepared from the reviewed buyer details and sent privately by email.