← Back to project catalogue
GP-DA-1QRM1JOData and AIReady

Conformal prediction for small-data classification

A reproducible statistics laboratory that compares conformal prediction sets under limited, imbalanced, and shifted classification data.

Conformal prediction for small-data classification project visual
GP-DA-1QRM1JO · Data and AI
  • Python 3.14
  • scikit-learn 1.9
  • NumPy
  • SciPy
  • pandas
  • Matplotlib

Project definition

Problem statement

A classifier can appear accurate while expressing unjustified certainty, especially when training and calibration data are limited or imbalanced.

The engineering problem is to construct prediction sets with a measurable coverage target, compare their efficiency, and expose the conditions where aggregate coverage or exchangeability assumptions fail.

Project objectives

  • Implement transparent split-conformal classification with finite-sample corrected quantiles.
  • Compare point prediction, LAC, deterministic APS, and label-conditional LAC.
  • Measure coverage, set size, singleton rate, empty rate, point accuracy, and class coverage.
  • Evaluate sample-size variability, class imbalance, and controlled covariate shift.
  • Retain every trial and generated figure for independent reproduction.

Project structure

Project components

01

Conformal scores

Validates probability matrices and implements LAC and deterministic APS nonconformity scores.

02

Calibration

Calculates finite-sample corrected pooled or class-specific thresholds from held-out observations.

03

Data preparation

Loads three prepared scikit-learn datasets and generates controlled small, imbalanced, and shifted cases.

04

Evaluation

Fits a scaled logistic model and calculates marginal, class, set-size, singleton, empty, and accuracy measures.

05

Experiment runner

Executes the fixed 1,350-fit matrix across thirty deterministic split seeds per condition.

06

Evidence outputs

Stores trial-level JSON, aggregate results, eight figures, and a static result dashboard.

Methodology

Project workflow

  1. 01
    Prepare data

    Load a documented dataset or create a controlled synthetic population.

  2. 02
    Split observations

    Reserve separate stratified training, calibration, and test partitions.

  3. 03
    Fit probabilities

    Train the scaled logistic classifier only on the training partition.

  4. 04
    Calibrate sets

    Calculate the selected pooled or class-specific nonconformity threshold.

  5. 05
    Measure and compare

    Build test prediction sets and retain coverage, efficiency, class, and variability evidence.

Demonstration scenario

A Wine dataset run produces an APS prediction set and its coverage metrics. The complete result dashboard then compares methods, shows increasing small-sample variability, reveals severe worst-class undercoverage under imbalance, and demonstrates the LAC coverage decline under controlled shift.

Engineering

Tools and method

Language and runtime
Python 3.14 with an installable local package and command-line interface.
Machine learning
scikit-learn pipelines for stratified splitting, standardisation, and logistic probability estimation.
Statistics
NumPy implementations of LAC, APS, finite-sample quantiles, prediction sets, and descriptive summaries.
Experiments
Fixed methods, seeds, sample sizes, imbalance ratios, and shift magnitudes declared before analysis.
Visualisation
Matplotlib figures generated directly from the retained JSON evidence.
Quality controls
Thirty tests, Ruff, pip-audit, repository checks, delivery checks, and document accessibility audits.

Testing

Evaluation

Evaluation measures

  • Marginal and worst-class empirical coverage
  • Mean prediction-set size, singleton rate, and empty rate
  • Coverage mean and standard deviation across thirty splits
  • Minority-class behavior across four imbalance conditions
  • LAC and APS behavior across five controlled shift magnitudes
  • Determinism, automated tests, dependency audit, and retained evidence checks

Project boundaries

  • The project is an educational engineering benchmark and not a high-impact decision system.
  • Coverage is marginal unless a class-conditional method and its assumptions are explicitly used.
  • The controlled synthetic shift does not represent every real distribution change.
  • No buyer, student, university, payment, clinical, or other personal information is collected.

Included

  1. 01Python source for conformal scores, datasets, evaluation, and command-line experiments
  2. 02Three prepared datasets and controlled synthetic data generators
  3. 03Results from 1,350 deterministic model fits
  4. 04Eight labelled result figures and a static dashboard
  5. 0530 automated tests and dependency audit instructions
  6. 06Complete source code in a private GitHub repository
  7. 0786-page project documentation in PDF and editable Word formats
  8. 0820-page setup and usage guide in PDF and editable Word formats

Project record

No information is collected on this page.

Permanent project ID
GP-DA-1QRM1JO
Catalogued
21 Aug 2026
Completed
24 Aug 2026
Verified
24 Aug 2026
Demonstration
Included in repository

Handover

After purchase

  1. 01
    Payment is confirmed

    The project is marked unavailable and cannot be purchased again.

  2. 02
    Repository access is granted

    The buyer's submitted GitHub account receives access to the private repository.

  3. 03
    The purchase record is delivered

    The certification sheet is prepared from the reviewed buyer details and sent privately by email.