← Back to project catalogue
GP-CY-0BGA2QICybersecurityReady

Adversarial ML Robustness Benchmark

An offline Python benchmark that measures bounded adversarial evasion, calibration, confidence rejection, and transfer behaviour across controlled classifiers and datasets.

Adversarial ML Robustness Benchmark project visual
GP-CY-0BGA2QI · Cybersecurity
  • Python 3.11
  • NumPy
  • scikit-learn
  • pandas
  • Matplotlib
  • Docker

Project definition

Problem statement

A classifier can be accurate on ordinary test data while changing its prediction after a small, bounded input perturbation. A single attack result does not reveal whether the weakness is consistent across models, budgets, datasets, or evaluation methods.

This project provides a controlled and reproducible laboratory for measuring robustness together with clean accuracy, confidence calibration, rejection behaviour, and transfer between models.

Project objectives

  • Train and compare a transparent linear classifier, a standard neural network, and a robust neural network.
  • Evaluate handwritten digits and synthetic two-moons data using fixed train, validation, and test procedures.
  • Apply FGSM, multi-step PGD, and bounded random search at five perturbation budgets.
  • Measure clean accuracy, robust accuracy, attack success, perturbation size, calibration, and confidence rejection.
  • Study whether examples produced against one model transfer to another model.
  • Reproduce all retained tables, figures, and dashboard outputs from the command line.

Project structure

Project components

01

Dataset pipeline

Prepares deterministic digits and two-moons splits without using personal or externally submitted data.

02

Model laboratory

Implements a linear softmax model, a one-hidden-layer neural network, and a neural network trained with bounded adversarial examples.

03

Attack laboratory

Generates bounded untargeted examples with FGSM, multi-step PGD, and random search.

04

Metric engine

Calculates accuracy, negative log likelihood, Brier score, calibration error, attack success, perturbation magnitude, and selective accuracy.

05

Experiment runner

Runs three seeds across both datasets, three models, three attacks, and five budgets.

06

Evidence outputs

Writes machine-readable results, nine labelled figures, and an offline results dashboard.

Methodology

Project workflow

  1. 01
    Prepare datasets

    The benchmark creates deterministic data splits and records the experimental seed.

  2. 02
    Train models

    All three model families are trained under documented settings, including the robust training variant.

  3. 03
    Generate bounded inputs

    Each attack is run under the same five perturbation budgets and constraint checks.

  4. 04
    Measure behaviour

    Clean, robustness, calibration, rejection, and transfer metrics are retained for analysis.

  5. 05
    Review evidence

    CSV results, labelled figures, and the offline dashboard support comparison and discussion.

Demonstration scenario

The student runs the complete benchmark and compares the standard and robust neural networks on handwritten digits. At the largest tested PGD budget, the robust model retains substantially more accuracy. The calibration, rejection, and transfer results show where that improvement holds and where limitations remain.

Engineering

Tools and method

Core package
Typed Python modules for models, gradients, attacks, metrics, experiments, plotting, and commands.
Transparent models
Small model implementations expose the gradients and prediction calculations used by the benchmark.
Controlled experiment
Three seeds, two datasets, three models, three attacks, and five budgets produce 900 retained result rows.
Quality controls
54 tests, 98.03 percent statement coverage, Ruff checks, dependency audit, repository validation, and a non-root Docker workflow.

Testing

Evaluation

Evaluation measures

  • Clean and adversarial accuracy across models, datasets, attacks, and perturbation budgets
  • Attack success rate and measured perturbation magnitude
  • Negative log likelihood, Brier score, and expected calibration error
  • Coverage and accuracy after confidence-based rejection
  • Transfer accuracy when adversarial examples are evaluated by a different model
  • Repeatability across three controlled random seeds

Project boundaries

  • The benchmark uses local controlled datasets and does not accept a remote model or service as a target.
  • The included attacks are bounded research methods for defensive evaluation, not an intrusion workflow.
  • Results apply only to the documented models, datasets, seeds, attacks, and perturbation budgets.
  • Adversarial training improves the prepared evaluation but does not prove universal robustness or security.
  • No information is collected.

Included

  1. 01Complete Python source code and command-line tools
  2. 02Linear, standard neural-network, and robust neural-network models
  3. 03FGSM, multi-step PGD, and bounded random-search experiments
  4. 04900 retained result rows and nine generated figures
  5. 0554 automated tests with 98.03 percent statement coverage
  6. 0680-page project report in PDF and editable Word formats
  7. 0721-page setup and usage guide in PDF and editable Word formats
  8. 0855 annotated academic and standards references

Project record

No information is collected on this page.

Permanent project ID
GP-CY-0BGA2QI
Catalogued
21 Aug 2026
Completed
25 Aug 2026
Verified
25 Aug 2026
Demonstration
Included in repository

Handover

After purchase

  1. 01
    Payment is confirmed

    The project is marked unavailable and cannot be purchased again.

  2. 02
    Repository access is granted

    The buyer's submitted GitHub account receives access to the private repository.

  3. 03
    The purchase record is delivered

    The certification sheet is prepared from the reviewed buyer details and sent privately by email.