Layer-Time Geometry of Transformer Language Models

Companion code and documents for "Layer-Time Geometry of Transformer Language Models" by Agus Sudjianto (H2O.ai) and Aijun Zhang (Wells Fargo).

This repository contains two documents and a full experimental pipeline:

• paper.pdf (74 pages) — Pedagogical shortcourse version for statistics audiences
• monograph.pdf (204 pages) — Full monograph with proofs, worked examples, and extended applications

Paper Overview

Transformer language models process text through a sequence of layers, producing a hidden-state vector at every (layer, token) position. This paper develops a geometric framework for analyzing these hidden states as a field on a discrete two-dimensional grid — with layer depth on one axis and token position on the other.

Core Ideas

1. Metric-aware representation. Raw hidden states live in an anisotropic space where Euclidean distances are misleading. We apply PCA-based whitening to project into an isotropic space where geometric operations (angles, norms, rotations) are valid.

2. Directional-radial decomposition. Each whitened hidden state is factored into a direction (unit vector on the sphere) and a magnitude (scalar energy). This separates what the model represents from how strongly it represents it.

3. Transport operators and curvature. We define local transport operators (rotation + scaling) at every edge of the layer-time grid using Rodrigues' rotation formula. The holonomy — the discrepancy when transporting around a unit cell (plaquette) via two different paths — measures curvature. Non-zero curvature means the model's layer processing depends on token context: a geometric signature of compositional reasoning.

4. Operator decomposition via polar factorisation. Layer-wise transition operators are decomposed into rotation U (orthogonal) and metric factor P (symmetric positive-definite) via polar decomposition. The rotation side captures directional computation; the metric side captures stretching, compression, and spectral structure.

5. Metric-side diagnostics. The symmetric factor P yields condition number (anisotropy), effective spectral rank (dimensionality of active computation), stretching field (local expansion/contraction), and Lyapunov exponents (trajectory stability). These complement the rotation-side curvature diagnostics.

6. Length-robust metrics. Total curvature scales with sequence length. We introduce refined metrics — curvature concentration (fraction of curvature in final layers), scale-rotation ratio, and windowed directionality — that decouple geometric signal from sequence length.

7. Geometric algebra. Bivectors from geometric (Clifford) algebra provide a coordinate-free way to describe rotation planes, enabling token-level attribution of layer-wise rotations and ambiguity detection via bivector energy.

Key Findings

• Three-phase computation: Transformers exhibit (i) aggressive rescaling at the embedding layer, (ii) gradual linear accumulation through middle layers, and (iii) concentrated nonlinear transformation at final layers.
• Curvature localizes reasoning: Curvature maps reveal where in the layer-token grid the model performs compositional computation. Retrieval tasks show curvature concentrated in final layers; reasoning tasks show curvature distributed broadly.
• Rotation-metric complementarity: Rotation-side metrics (curvature concentration, SRR) respond most strongly to robustness condition (adversarial vs clean). Metric-side metrics (condition number, stretching) respond to cognitive task type and reasoning depth. A complete diagnostic requires both sides of the polar decomposition.
• Curvature concentration is the best single rotation-side metric: low length correlation (r=-0.117), high discriminative power (eta^2=0.500 after length control), and high paraphrase stability (r=0.856).
• Condition number is the strongest metric-side signal (eta^2=0.089), driven by cognitive task type. Linguistic analysis produces the most anisotropic operators; factual retrieval the least.
• DOE validation: A 298-prompt Sobol quasi-Monte Carlo design with 6 factors confirms that robustness condition is the top factor for rotation metrics, while cognitive task type and reasoning depth drive metric-side variation.

Paper Structure (shortcourse version)

Sections	Topic
1-3	Introduction, self-attention mechanics, layer-time representation
4-5	Metric structure, whitening, directional-radial decomposition
6-8	Kernels, directed interaction operators, S+A decomposition
9-10	Operator decomposition, metric spectral analysis (kappa, erank, stretching, Lyapunov)
11	Discrete geometric flow and curvature via transport holonomy
12	Geometric algebra interpretation
13-15	Sample-level geometry, reasoning metrics, failure diagnostics
16-17	Experimental results (8 experiments including DOE metric-side analysis)
18	Related work
19	Geometric steering for controlled generation
20	Worked examples: GRASP, hallucination detection, BLADE, NLI alignment, CoT reasoning
21	Conclusion
Appendices	Statistics bridge reference, suggested further reading

The paper is written as a shortcourse, with intuition boxes, exercises, statistical bridge explanations (connecting geometric concepts to familiar statistical methods), and a terminology guide for statisticians.

Monograph Structure (full version)

The 204-page monograph expands the shortcourse into 14 chapters with full proofs, extended theory (metric variation, Riemann curvature tensor, fibre bundle structure), 39 references, and detailed worked examples including GRASP, hallucination taxonomy, BLADE, geometric steering, and NLI alignment.

Repository Structure

paper.tex                    LaTeX source — pedagogical shortcourse (74 pages)
paper.pdf                    Compiled PDF
monograph.tex                LaTeX source — full monograph (204 pages)
monograph.pdf                Compiled PDF
layer_time_geometry.py       Core library — all geometric computations (CPU + GPU)
prompt_dataset.py            260 structured prompts (4 families x 65)
doe_design.py                DOE factor definitions and Sobol design generation
doe_prompt_dataset.py        298-prompt DOE dataset with 6 controlled factors
run_analysis.py              Basic 8-prompt demonstration
run_validation_experiments.py  4 statistical validation experiments
run_scaled_experiments.py    Full 260-prompt reproducible pipeline
run_doe_experiment.py        DOE experiment (5-phase pipeline)
run_steering_experiment.py   Geometric steering experiment
results/                     Experimental outputs (DOE, steering)
requirements.txt             Python dependencies

Quick Start

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Run the basic 8-prompt demo (fastest)
python run_analysis.py

# Run the 4 validation experiments
python run_validation_experiments.py

# Run the full scaled experiment (260 prompts, ~30 min on GPU)
python run_scaled_experiments.py

# Run the DOE experiment (298 prompts, Sobol design)
python run_doe_experiment.py

# Run the steering experiment
python run_steering_experiment.py

GPU recommended. The experiments use Qwen2.5-7B and require ~16GB VRAM. Geometric computations use batched PyTorch GPU operations (~200x faster than CPU).

Reproducibility

run_scaled_experiments.py is a 6-phase pipeline that saves all intermediate artifacts:

Phase	Output
1. Hidden-state extraction	results/scaled/hidden_states/*.npz
2. Whitening metric	results/scaled/whitening_metric.npz
3. Geometric computation	results/scaled/geometric_metrics.csv
4. Statistical analysis	group_summary.csv, anova_results.csv, pairwise_comparisons.csv, paraphrase_stability.csv, within_bin_analysis.csv
5. Visualization	results/scaled/figures/fig*.png
6. Configuration snapshot	results/scaled/experiment_config.json

run_doe_experiment.py follows the same phase structure with DOE-specific outputs in results/doe/.

Each phase checks for existing artifacts and skips if already computed.

Prompt Datasets

Scaled experiment (260 prompts)

The prompts in prompt_dataset.py are organized as:

• Retrieval (65): geography capitals, science facts, historical dates, language translations, definitions
• Arithmetic (65): addition, multiplication, subtraction, division, mixed operations
• Reasoning (65): syllogisms, conditional logic, spatial reasoning, temporal reasoning, causal reasoning
• Adversarial (65): letter counting, negation traps, garden path, false presupposition, trick questions

DOE experiment (298 prompts)

The prompts in doe_prompt_dataset.py use a Sobol quasi-Monte Carlo design with 6 controlled factors:

• Cognitive task (6 levels): factual retrieval, definition, arithmetic, deductive reasoning, causal reasoning, linguistic analysis
• Reasoning depth (3 levels): single-step, two-step, multi-step
• Domain (4 levels): science, history, everyday, abstract
• Answer format (3 levels): single word/number, short phrase, full sentence
• Robustness condition (5 levels): clean, negation trap, false presupposition, garden path, trick question
• Prompt length (3 levels): short, medium, long

Model

All experiments use Qwen/Qwen2.5-7B (7B parameters, 28 layers, hidden dim 3584).

Citation

@article{sudjianto2026layertime,
  title={Layer-Time Geometry of Transformer Language Models},
  author={Sudjianto, Agus and Zhang, Aijun},
  year={2026}
}

License

This code is provided for educational and research purposes.