← Back to project catalogue
GP-DA-094HG2JData and AIReady

Selective prediction and abstention laboratory

A reproducible machine-learning laboratory that measures when a classifier should decline uncertain cases instead of forcing every prediction.

Selective prediction and abstention laboratory project visual
GP-DA-094HG2J · Data and AI
  • Python 3.14
  • scikit-learn 1.9
  • NumPy
  • SciPy
  • pandas
  • Matplotlib

Project definition

Problem statement

A classifier can attach a high probability to an incorrect answer, especially after the input population changes.

The engineering problem is to compare practical rules that accept the most reliable predictions, decline the remaining cases, and show the error and review workload at each coverage level.

Project objectives

  • Build a reproducible three-class probability classifier with separate training, calibration, and test data.
  • Fit held-out temperature scaling and measure its effect using ECE, Brier score, and negative log loss.
  • Compare maximum-probability, entropy, margin, and class-conditional abstention policies.
  • Measure selective risk across seven accepted coverage targets.
  • Evaluate controlled feature shift, diagnostic subgroup gaps, and bootstrap uncertainty.

Project structure

Project components

01

Data generator

Creates a controlled imbalanced classification population and four isolated data splits.

02

Calibration

Fits one temperature on held-out logits and compares raw and calibrated probabilities.

03

Abstention policies

Ranks predictions using maximum probability, entropy, probability margin, or class-conditional allocation.

04

Evaluation

Calculates calibration metrics, selective risk, AURC, subgroup gaps, and bootstrap intervals.

05

Evidence outputs

Writes JSON, prediction-level CSV, eight figures, and a static result dashboard.

Methodology

Project workflow

  1. 01
    Generate the population

    Create 7,200 synthetic labelled observations with fixed class balance, noise, and random seed.

  2. 02
    Fit the classifier

    Train the scaler and logistic model only on the 3,600 training observations.

  3. 03
    Calibrate probabilities

    Fit one temperature using the separate 1,200-observation calibration split.

  4. 04
    Apply abstention

    Evaluate all four policies at accepted coverage from 50 to 100 percent.

  5. 05
    Stress and compare

    Repeat evaluation on corrupted features and compare calibration, risk, and subgroup gaps.

Demonstration scenario

The experiment fits the classifier and one temperature, displays the ordinary risk-coverage curves, and compares all four policies. A controlled feature shift then raises calibration error, selective risk, and subgroup gaps, showing why a fixed confidence threshold cannot be trusted after the input distribution changes.

Engineering

Tools and method

Runtime
Python 3.14 and an installable command-line package with pinned dependencies.
Machine learning
scikit-learn standardisation, logistic regression, and synthetic data generation.
Numerical methods
SciPy temperature optimisation and NumPy implementations of scores and metrics.
Experiments
Fixed split sizes, coverage grid, shift corruption, subgroup boundary, and bootstrap seed.
Quality controls
Forty-one tests, Ruff, dependency audit, repository checks, delivery checks, and document accessibility audits.

Testing

Evaluation

Evaluation measures

  • Accuracy, ECE, multiclass Brier score, and negative log loss
  • Selective risk across seven accepted coverage targets
  • Area under the risk-coverage curve for four policies
  • Ordinary and shifted diagnostic subgroup risk gaps
  • Three hundred bootstrap risk samples at 80 percent coverage
  • Determinism, automated tests, dependency audit, and retained evidence checks

Project boundaries

  • Every observation and label is synthetic and has no real-world class meaning.
  • The diagnostic subgroup is generated from one feature and is not a demographic fairness evaluation.
  • The controlled feature corruption represents one shift condition, not every possible distribution change.
  • The project is an engineering laboratory and not a clinical, financial, safety, educational, or eligibility decision system.

Included

  1. 01Python source for data generation, calibration, abstention policies, metrics, and experiments
  2. 02Synthetic three-class data with isolated training, calibration, test, and shifted splits
  3. 032,400 prediction records and 300 bootstrap samples for each final test condition
  4. 04Eight labelled result figures and a static dashboard
  5. 0541 automated tests with 99 percent statement coverage
  6. 06Complete source code in a private GitHub repository
  7. 0787-page project documentation in PDF and editable Word formats
  8. 0821-page setup and usage guide in PDF and editable Word formats

Project record

No information is collected on this page.

Permanent project ID
GP-DA-094HG2J
Catalogued
21 Aug 2026
Completed
24 Aug 2026
Verified
24 Aug 2026
Demonstration
Included in repository

Handover

After purchase

  1. 01
    Payment is confirmed

    The project is marked unavailable and cannot be purchased again.

  2. 02
    Repository access is granted

    The buyer's submitted GitHub account receives access to the private repository.

  3. 03
    The purchase record is delivered

    The certification sheet is prepared from the reviewed buyer details and sent privately by email.