Indic classifier fairness and drift monitor
A reproducible study of accuracy, probability calibration, language-script reliability, selective prediction, and controlled drift in Indic language classifiers.
Project definition
Problem statement
An Indic language classifier can have strong overall accuracy while making more errors for a particular language, romanized writing, short input, or a changed input distribution.
This project measures those differences, improves probability calibration, tests confidence-based review, and checks whether fixed drift signals identify six controlled changes.
Project objectives
- Prepare balanced native and romanized examples without leaking paired records across partitions.
- Compare logistic regression and Complement Naive Bayes on one character TF-IDF representation.
- Calibrate both models with a separate calibration partition.
- Measure performance by language, script, source, and text length with uncertainty.
- Evaluate confidence thresholds and six controlled distribution shifts.
Project structure
Project components
Dataset preparation
Downloads the official source, verifies its SHA-256 digest, selects paired records, and creates group-safe partitions.
Classifier pipelines
Fits two transparent character TF-IDF models on the same training data.
Probability calibration
Fits one temperature per model on held-out calibration data and compares probability quality.
Slice analysis
Measures accuracy, confidence, errors, and bootstrap intervals across declared reliability slices.
Selective prediction
Measures accepted accuracy, coverage, and review count across confidence thresholds.
Drift monitor
Calculates categorical, length, confidence, and performance signals for seven evaluation windows.
Methodology
Project workflow
- 01Verify and prepare
Check the archive digest and create 5,000 source groups with native and romanized representations.
- 02Split by group
Keep paired representations together across training, calibration, and evaluation partitions.
- 03Train and calibrate
Fit both classifiers and estimate one independent temperature for each model.
- 04Evaluate reliability
Write overall, probability, threshold, slice, and per-example evidence without retaining source sentences.
- 05Test controlled drift
Run six shifted windows, calculate fixed monitoring signals, and compare expected and observed alerts.
Demonstration scenario
The experiment prepares the paired Indic dataset, trains and calibrates both models, then shows that native-script accuracy is stronger than romanized accuracy. A compound window adds romanized emphasis, short inputs, and typing noise. Macro F1 falls to about 0.78, several drift signals cross their fixed thresholds, and both models alert.
Engineering
Tools and method
- Runtime
- Python 3.12 with an installable command-line package and exact-pinned dependencies.
- Machine learning
- scikit-learn character TF-IDF, multinomial logistic regression, and Complement Naive Bayes.
- Statistics
- SciPy and NumPy for temperature fitting, bootstrap intervals, distribution distance, PSI, and KS calculations.
- Evidence
- pandas CSV and JSON outputs with Matplotlib figures generated from retained results.
- Privacy
- Hash-derived audit identifiers and no benchmark sentences in committed prediction evidence.
- Quality controls
- Tests, coverage, Ruff, dependency audit, document accessibility audit, container checks, and repository acceptance.
Testing
Evaluation
Evaluation measures
- Accuracy, macro F1, macro recall, log loss, multiclass Brier score, and calibration error
- Language, script, collection-source, and input-length reliability slices
- Coverage, accepted accuracy, and review count across confidence thresholds
- Language, script, prediction, length, confidence, and performance drift signals
- Alert agreement across two baseline cases and twelve shifted model-window cases
- Automated tests, source coverage, dependency audit, and retained evidence checks
Project boundaries
- Language and script slices are reliability diagnostics, not a social fairness certification.
- The ten-language closed set does not support unknown-language rejection or all forms of natural code mixing.
- Controlled drift results apply to the declared scenarios and thresholds, not every production change.
- The project must not be used to infer identity or make moderation, surveillance, authentication, or eligibility decisions.
Included
- 01Installable Python package and command-line workflow
- 0210,000 balanced native and romanized examples across ten Indic languages
- 03Two character TF-IDF classifiers with independent temperature scaling
- 04Seven monitoring windows, 876 slice rows, and 67,500 prediction audit rows
- 05Twelve result figures in PNG and SVG formats
- 0629 automated tests with 99 percent source coverage
- 07Complete source code in a private GitHub repository
- 0884-page project report in PDF and editable Word formats
- 0915-page setup and demonstration guide in PDF and editable Word formats
- 1050 annotated references and three attributed literature images
Project record
No information is collected on this page.
- Permanent project ID
- GP-DA-0DMXAEX
- Catalogued
- 21 Aug 2026
- Completed
- 26 Aug 2026
- Verified
- 26 Aug 2026
- Demonstration
- Included in repository
Handover
After purchase
- 01Payment is confirmed
The project is marked unavailable and cannot be purchased again.
- 02Repository access is granted
The buyer's submitted GitHub account receives access to the private repository.
- 03The purchase record is delivered
The certification sheet is prepared from the reviewed buyer details and sent privately by email.