Benchmarking Uncertainty Quantification (UQ) for Deep Learning

A comprehensive, modular benchmarking framework for evaluating state-of-the-art Uncertainty Quantification (UQ) methods across multiple datasets and backbone architectures. Designed for reproducibility, extensibility, and large-scale benchmarking on HPC clusters.

Table of Contents

• About the Project
  • Motivation
  • Problem Statement
  • Solution
  • Key Accomplishments
  • Supported Components
• Download
• Preparing Datasets
• Installation
• Usage
• Core Scripts
• Configuration
• Extending the Framework
• References
• Author & Team
• Funding & Support

About the Project

Motivation

Deep learning models have achieved impressive accuracy on image classification tasks, but they often make highly confident predictions even when they are incorrect. This overconfidence can lead to catastrophic failures in safety-critical domains such as healthcare, autonomous driving, and security. Reliable Uncertainty Quantification (UQ) is essential for detecting such failures and building trustworthy AI systems.

Problem Statement

Despite the growing number of UQ methods in the literature, there is a lack of standardized benchmarks for comparing their effectiveness in real-world scenarios—especially for detecting misclassifications. Existing results are often not directly comparable due to differences in datasets, architectures, or evaluation protocols.

Solution

This project provides a comprehensive, modular benchmarking framework to evaluate and compare state-of-the-art UQ methods on standardized image classification benchmarks (CIFAR-10 and Tiny ImageNet) using popular deep learning backbones (WideResNet, DenseNet). The framework covers training, evaluation, and analysis of UQ techniques, making it easy to reproduce experiments and extend to new methods or architectures.

Key Accomplishments

Core Framework

• Implemented and benchmarked seven UQ methods: Deep Ensembles, MC Dropout, Latent Posterior Bayesian Neural Network (LP-BNN), Dirichlet Meta-Model, MAPLE, Deterministic Uncertainty Estimation (DUE), and Deep Deterministic Uncertainty (DDU).
• Supported multiple datasets: CIFAR-10 and Tiny ImageNet with automatic download and preprocessing using a modular dataset registry for easy extension.
• Multiple backbone architectures: WideResNet and DenseNet with method-specific implementations (LP-BNN, MAPLE variants) for optimized performance.

Production-Grade Quality

• Comprehensive test suite with 36+ passing tests covering all UQ methods, configurations, and data utilities.
• Standardized logging format across all scripts: YYYY-MM-DD HH:MM:SS - INFO/WARNING/ERROR for consistency and debugging.
• Modular configuration-driven experiments for full reproducibility with YAML-based hyperparameter management.
• Standardized evaluation metrics for uncertainty and misclassification detection including:
  • Accuracy: Classification accuracy
  • AUGRC: Area Under Risk-Coverage curve for selective prediction
  • ECE: Expected Calibration Error (confidence-accuracy alignment)
  • Brier Score: Mean squared error between predictions and labels
  • NLL: Negative Log-Likelihood (probability of true class)
  • AUROC: Area Under ROC Curve for error detection
  • AUPRC: Area Under Precision-Recall Curve for imbalanced error detection

Automation & Tooling

• Production-grade combine_results.py with comprehensive error handling, input validation, and detailed progress reporting.
• Batch processing scripts for efficient SLURM cluster support enabling large-scale benchmarks.
• Feature extraction utilities for model validation and layer analysis.
• Method registry system for dynamic method instantiation and parameter management.

Extensibility

• Modular architecture enabling easy addition of new UQ methods (inherit from UQMethodBase).
• Backbone registry with support for method-specific customizations (e.g., LP-BNN layers, MAPLE memory modules).
• Batch unpacking utilities for flexible data handling across different scenarios.

Supported Components

Datasets

• Supported datasets: CIFAR-10, Tiny ImageNet, MNIST, Fashion-MNIST, ImageNet-100

Backbone Models

• WideResNet (28-10): Deep residual networks with configurable width
• DenseNet: Densely connected networks with feature reuse
• Method-specific variants:
  • DenseNet for LP-BNN
  • DenseNet for MAPLE
  • WideResNet for LP-BNN
  • WideResNet for MAPLE

Download

To get started, clone this repository to your local machine:

git clone https://github.com/dristanta-silwal/benchmarking-UQ.git
cd benchmarking-UQ

Preparing Datasets

Datasets are automatically downloaded and prepared by running:

bash scripts/batch/prepare_data.sh          # Default: MNIST, CIFAR-10
bash scripts/batch/prepare_data.sh cifar10 tinyimagenet mnist fashionmnist  # Specific datasets

Installation

Requirements: Python >= 3.10

For comprehensive setup instructions, GPU configuration, troubleshooting, and dependency management, see INSTALL.md.

Quick start (uv - recommended):

curl -LsSf https://astral.sh/uv/install.sh | sh  # Install uv
git clone https://github.com/dristanta-silwal/benchmarking-UQ.git
cd benchmarking-UQ
uv sync && uv run python -m pytest tests/ -v --tb=short  # Install & verify

Alternative: pip or conda installs available in INSTALL.md

Usage

Typical workflow:

1. Prepare datasets: bash scripts/batch/prepare_data.sh
2. Train: uv run python scripts/train.py --config configs/de_wrn.yaml
3. Evaluate: uv run python scripts/evaluate.py --config configs/de_wrn.yaml
4. Combine results (multiple runs): uv run python scripts/combine_results.py CIFAR10

For clusters: sbatch scripts/jobs/jobs.slurm (edit SLURM parameters in the file)

Core Scripts

Script	Purpose
train.py	Train UQ method: uv run python scripts/train.py --config configs/{method}_{backbone}.yaml
evaluate.py	Evaluate & compute metrics: uv run python scripts/evaluate.py --config configs/{method}_{backbone}.yaml
combine_results.py	Aggregate multiple runs: uv run python scripts/combine_results.py CIFAR10
test.py	Test set inference
misclassified_img.py	Analyze misclassified images

Configuration

Experiments are controlled by YAML configs in configs/. Naming convention: {method}_{backbone}{_variant}{_scale}.yaml

Methods (lowercase): de (Deep Ensembles), mc (MC Dropout), lp-bnn, maple, dirichlet, due, ddu
Backbones: wrn (WideResNet), dn (DenseNet)
Variants (optional): custom (optimized layers), sngp (special variant)
Datasets (optional): tiny (tinyimagenet)

Examples: de_wrn_tiny.yaml (quick test), lp-bnn_dn_custom.yaml (optimized), mc_wrn.yaml (production)

See CONFIG_TEMPLATE.yaml for full structure and parameters.

Extending the Framework

For detailed instructions on adding new UQ methods, backbone architectures, or datasets, see CONTRIBUTING.md.

Quick reference:

• New UQ Method: Inherit from UQMethodBase, register in models/uq_methods/__init__.py, create config
• New Backbone: Implement in models/backbones/, register in factory, test with existing methods
• New Dataset: Add handler to scripts/batch/prepare_data.sh, register in dataset registry

Running Tests: pytest tests/ -v (use -m "not slow" to skip GPU tests)

References

For comprehensive references including papers, GitHub repositories, datasets, tools, and frameworks, please see references.md.

---

Author & Team

Primary Author

Dristanta Silwal
Undergraduate Researcher
University of Idaho

Project Advisors & Supervision

• Supervised by: Dr. Min Xian
• Faculty Advisor: Dr. Terence (Terry) Soule

Contributors

• Patrick Du
• Prabesh Tamang

Collaborators

• Kyle Lucke

Funding & Support

This project is supported by:

• Summer Undergraduate Research Fellowship (SURF) 2025 – The Office of Undergraduate Research, University of Idaho (UI)
• Machine Intelligence and Data Analytics (MIDA) Lab
• University of Idaho College of Engineering – Capstone Project 2025/26