When Common Sense Overrides Vision: Diagnosing and Correcting Directed Prior Attraction

Official research artifacts for the AAAI-27 submission "When Common Sense Overrides Vision: Diagnosing and Correcting Directed Prior Attraction".

Commonsense-driven hallucination (CDH) occurs when a vision-language model replaces atypical visual evidence with the ordinary state it expects. This repository diagnoses the resulting directed prior attraction and implements Selective Prior Calibration (SPC): SPC scores the native answer candidates with and without the image, learns a context-conditioned coefficient that removes candidate-aligned prior preference from the image-conditioned scores, and proposes a correction only when a learned Selective Support Gate fires; otherwise it retains the native answer. An optional SPC+NoLan analysis is retained for completeness, but it is not the primary method.

The release contains the core inference and analysis code, frozen experiment configurations, and regression tests. It also includes equation-preserving ports of the published NoLan, VCD, MFCD, and PAI token-level rules for Qwen3-VL and LLaVA-NeXT. Their repositories, revisions, equations, and fixed parameters are recorded in configs/official_baseline_sources.json. Benchmark images, annotations, frozen score files, and the paper itself are distributed separately (see Data below).

Main results

All changes are accuracy points, reported as Delta CF / Delta CS where CF is the counterfactual side and CS is the matched commonsense side. The setting is selected on 70 development pairs and evaluated on 230 held-out pairs.

Held-out CDH-Bench (RQ1-RQ2). Among native CF errors, the wrong answer equals the prior-estimate winner in 70.1% (Qwen MC) to 91.6% (LLaVA QA) of cases, and in 80.5-91.6% of those errors the winner is also the correct answer on the paired commonsense image: the attraction is directed at a concrete normal-world answer. The frozen SPC route repairs these errors while retaining commonsense accuracy:

Model	MC (Delta CF / Delta CS)	QA (Delta CF / Delta CS)
Qwen3-VL-32B-Instruct	+7.8 / +0.4	+22.2 / -1.7
LLaVA-1.6-34B	+7.0 / -0.4	+23.9 / -4.8

Generality (RQ3). The same selected setting, with no refitting, transfers to held-out semantic families, repeated re-splits, MC option permutations, and external conflict benchmarks: Visual CounterFact (+3.1/-0.2 Qwen, +4.5/-0.1 LLaVA), HallusionBench (+5.2/-0.9), and ConflictVIS (+2.1 MC accuracy, +25.7 QA accuracy).

Scope (RQ4). One selected setting is applied across six benchmarks. SPC intervenes and repairs where a directed normal-world competitor exists (CDH-Bench, Visual CounterFact, HallusionBench, ConflictVIS), and changes almost nothing where that structure is absent - 17 of 9,000 POPE questions and 1 of 140 POPEv2 questions, with no significant accuracy change. The full spectrum table is regenerated by analyze_benchmark_spectrum.py into result/paper_revision_stats/benchmark_spectrum.json.

Repository layout

Path	Contents
evaluate_cdh_bench.py	Main CDH inference and mitigation entry point
evaluate_*_candidate_scores.py	Candidate-score evaluators (CDH MC, paired QA, VCF, POPE, ConflictVIS)
spc_vlm_runtime.py	Compact Qwen3-VL/LLaVA candidate-scoring runtime
official_nolan_qwen3vl.py	Published NoLan dynamic token-level rule with synchronized image/text streams
official_token_baselines.py	Published VCD, MFCD, and PAI image transforms and token equations
evaluate_official_*.py	Full-vocabulary and task-constrained baseline evaluators
prepare_*.py	Builders for the derived stress-test and transfer annotations
analyze_*.py	SPC fitting, controls, robustness, transfer, and revision analyses
configs/	Frozen model, split, baseline, and method protocols
tests/	Unit and regression tests for the released pipeline

Benchmark images, annotations, and frozen candidate-score files are not tracked in this repository; they are downloaded from the Hugging Face dataset (see Data) and written to images/, data/, and result/ locally.

Every table-level artifact is indexed in ARTIFACT_MAP.md. Exact commands for the principal experiments are in REPRODUCE.md.

Quick start

Python 3.10 or newer is recommended. The reported environment used the pinned versions in requirements-paper.txt.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements-paper.txt

Reproduce the analyses (after downloading the data)

All statistical analyses replay frozen candidate-score files and verify themselves against the frozen reference artifacts. First download the benchmark and score files from the Hugging Face dataset (see Data) so that images/, data/, and result/ are populated, then run from the repository root:

# Main-table composition: proposal decomposition, MC-only ablation, stability radius
python analyze_proposal_decomposition.py --task all

# Lambda stable intervals (Figure: per-row stable intervals of the learned coefficient)
python analyze_lambda_stable_intervals.py

# Six-benchmark spectrum table (RQ4)
python analyze_benchmark_spectrum.py

# Robustness and generality analyses (gate Pareto, family holdout, repeated splits,
# option permutation, external transfers) - see REPRODUCE.md
python analyze_spc_gate_pareto.py \
  --config configs/spc_gate_pareto_v1.json \
  --output result/cprc_robustness/gate_pareto_qwen_llava_v1.json

Run model inference from scratch (GPU + weights required)

Place model weights at the paths expected by the frozen configs:

models/Qwen3-VL-32B-Instruct/
models/llava-v1.6-34b-hf/
models/Qwen3-VL-30B-A3B-Instruct/

Download the complete CDH-Bench image archive from the Hugging Face dataset into images/ (see Data). A frozen split can then be re-scored with:

CUDA_VISIBLE_DEVICES=0,1,2,3 python evaluate_cdh_mc_candidate_scores.py \
  --dataset CDH-Bench.revised.strict.jsonl \
  --images-root images \
  --output result/reproduced_qwen32b/results.jsonl \
  --models configs/qwen_32b_instruct_bf16_answer1.json \
  --pair-manifest runtime/paper/cdh_exposure_ledger.json \
  --pair-split cpr_unseen \
  --retry 2

Data

• CDH-Bench (images, paired annotations, split ledger, and the frozen candidate-score files used by all analyses): download the dataset release from Hugging Face at https://huggingface.co/datasets/cks19999/CDH-Bench and unpack it at the repository root so that images/, data/, and result/ are populated. The frozen score files under result/ let every table be reproduced without a GPU; they can also be regenerated from the images with the evaluate_* scripts.
• POPEv2: annotations and the derived 140 images are built from the official POPEv2 release with prepare_popev2_qa_transfer.py; the upstream source repository and COCO image URLs are recorded in the generated manifest.json.
• Other third-party benchmarks (POPE, Visual CounterFact, HallusionBench, ConflictVIS): images are not redistributed. REPRODUCE.md and configs/spc_external_v1.json record the pinned upstream revisions and the preparation commands needed to fetch the images yourself. Please follow each upstream benchmark's license and terms of use.

Tests

python -m pytest tests/ -q

The suite covers the candidate-scoring runtime, the frozen-replay and selection protocols, the transfer evaluators, and the statistical utilities used by the paper analyses.

Reproducibility notes

• CF and CS members of a pair always remain in the same partition.
• The primary split uses 70 development pairs and 230 unseen pairs across 14 subcategories.
• SPC settings are selected only from development data; held-out outcomes are retained for final auditing.
• The following pre-rename names are frozen data identifiers retained for compatibility with the released dataset: the result/cprc_* directory layout (e.g. result/cprc_robustness/, result/cprc_external/, result/cprc_qwen30b_a3b_*, result/anchored_cprc/, and the result/hierarchical_eb_lambda_32b/cprc_* files), and the machine-readable keys cprc_map, bprc_full, map_no_laplace_or_density, and eb_cprc. cprc_map and eb_cprc are stored in and read back from the frozen artifacts; bprc_full and map_no_laplace_or_density refer to the SPC learned route, cprc_map to the SPC+NoLan composition.
• No API keys, local model weights, runtime logs, or generated caches are part of this release.

Citation

The repository currently accompanies an anonymous conference submission. Final citation metadata will be added after the review process.