Navette - Weaving thin-film systems that perform

Navette is a high-performance, physically rigorous 1D optical engine designed for the simulation of light propagation in stratified media. Built on a modern Scattering Matrix (S-matrix) architecture, it offers a numerically stable and vectorized alternative to traditional Transfer Matrix Methods (TMM).

1. Unconditional Numerical Stability

Traditional TMM suffers from numerical divergence (exponentially growing evanescent waves) when dealing with thick layers or highly absorbing materials. Navette utilizes the Redheffer Star Product to propagate scattering matrices, ensuring that all matrix elements remain bounded and physically meaningful, regardless of layer thickness.

2. High-Concurrency Performance

As a Principal Performance Engineer, you need tools that scale. Navette is built for speed:

• Parallel Execution: Utilizes Rust + rayon data-parallelism across wavelengths/angles to saturate all available CPU cores.

• Vectorized Engine: Operations are performed across the entire (wavelength × angle) coordinate space in a single pass, eliminating Python's loop overhead.

• Memory Efficiency: Collapses multi-layer stacks into a compact global S-matrix to minimize cache misses.

Measured, not claimed. All numbers below are --release builds on a 32-core Windows box, taken by the scripts named beside them under validation/benches/; each of those scripts refuses to run against a debug build (see the profile note under Getting started). "vs numba" compares against the numba reference implementation this engine replaced.

What	Measured	Script
Full observable mask, 40 lambda x 3 theta, 5 layers (complex amplitudes + dispersion)	0.164 ms median -- ~1.4 us per point for every channel	bench_backside_speed
Rigorous 12-channel request, 6 layers, 20 000 / 60 000 points	1.68 ms / 4.02 ms, i.e. 1.1-1.8x the numba kernel	bench_core_engine_scaling.py
Photometric 4-channel request, same grid sizes	1.50 ms / 3.70 ms	bench_core_engine_scaling.py
pchip interpolation, 1 M points	1.22 ms (1.2 ns/pt) vs 1.59 ms numba; accuracy identical (~1e-14 vs analytic) on both sides	1dinterpol_test_bench
dE76 / dE94 / CMC / DIN99 / dE2000 batches	23-36x faster than the reference, at exact parity with colour-science including black/white/near-black rows	bench_validate_color
Weaver set_data / get_weaved / unweave_cached	1.5-7.8x the Python reference across small and mid grids	navette_spectral_bench
Batch unweave_collection	1.15-2.29x faster than before R5.1; still the one path that can trail the reference at extreme key counts	navette_spectral_bench
One LM thickness optimize (synthesis), with needle re-fold	2.1 ms, re-fold 9.1 % overhead	bench_refold
Structure grid assert	0.9 us	bench_grid_assert

Two caveats kept deliberately visible: small grids (under ~2 000 points) are dispatch-bound and still behind the numba kernel, and bench_refold does not exercise the needle insertion path, which is the expensive part of synthesis.

3. Partial Coherence Support

Real-world systems involve thick substrates — a 1 mm glass slide — where the path-length spread across the layer exceeds the source coherence length and the phase relation between its two surfaces is destroyed. Navette cuts the stack into maximal coherent runs at the layers you flag, uses the complex Redheffer star product inside a block and a real intensity star product across blocks (Katsidis & Siapkas 2002; Byrnes, arXiv:1603.02720). A flagged layer contributes only its attenuation τ = exp(−2·Im β).

• Coherent blocks preserve phase for thin-film interference; incoherent joins switch to intensity propagation, so a macroscopic substrate stops producing the unphysical ringing that assuming perfect coherence gives it.

• The flag goes on an interior row, and thickness is part of what it claims. Rows 0 and last are half-spaces — the block sweep never reads their flags, so flagging "the substrate" as the exit medium is a no-op and now says so. A thick substrate is an interior layer with a real thickness, flagged, between the films and the exit medium. The flag asserts decoherence, which needs the spread to exceed L_C = λ²/Δλ; in the UV–VIS–NIR that is roughly 50–100 µm. Flag something thinner and you get a warning quoting the source bandwidth that layer would need — a number you can check against your own source rather than a threshold somebody picked. The flag is still honoured either way: the thin incoherent limit is a legitimate thing to ask for, as long as it is what you meant.

• Three coherence modes, and one of them refuses rather than guesses. FRONT_BLOCK (the default) is exact for intensities — Rs, Rp, Ts, Tp, A are correct totals over every incoherent echo — but its p–s cross channel comes from the first coherent block alone. On a flagged stack, the twelve observables that need that channel (Δ, DOP, S2/S3, the raw cross terms and the retardance, in reflection and transmission) would be a ratio of two different stacks, so the doors refuse them and name COHERENCY_MATRIX, which cascades the cross channel with the echoes. FULLY_COHERENT treats the whole stack as one block. With nothing flagged all modes agree bit for bit and nothing is refused.

• Synthesis refuses what it cannot honour. The needle pipeline solves a design as one coherent block, so handing it a coherent: false film used to silently produce a coherent answer. It now refuses at DesignStack construction, names the film and points at solve_structure / ScatterMatrix — the doors that do honour the flag. Multiblock synthesis is a phase of its own; refusing is not it, and the message says which is which.

• Checked against something that is not itself. The parity suite compares the engine to a port of the same algorithm, which pins the port rather than the physics. validation/review/incoherent_check.py uses three oracles that do not know how Navette works: the lossless-slab closed form R = 2R₁/(1+R₁) (1e-12), bit-identical reflectance from 10 µm to 3.7 mm, and the definition itself — a flagged answer IS the coherent answer averaged over one round-trip phase period (8e-17). The same average on the Stokes vector is what put numbers on the cross-channel refusal above.

4. Advanced Physics Modeling

Navette goes beyond simple Fresnel equations to provide research-grade accuracy:

• Interface Roughness: Implements the Névot-Croce model, providing superior accuracy for high-frequency or X-ray reflectometry compared to standard Gaussian approximations.

• Ellipsometric Rigor: Outputs (Ψ,Δ) parameters that strictly follow the Azzam & Bashara convention, ensuring direct compatibility with commercial ellipsometers (e.g., Woollam, Horiba).

• Dispersion Models as Data: A material is a model name plus plain params — sixteen of them, from Konstant and Table (measured nk) through Cauchy / Sellmeier (each with an Urbach tail), Lorentz, Drude, DrudeLorentz, CodyLorentz, four ForouhiBloomer variants, TaucLorentz and UBF. Six EMA mixing rules (Bruggeman, Maxwell-Garnett, Looyenga, Lichtenecker, Mori-Tanaka, power-law) compose them, and EMA specs nest, so a rough oxide on a mixed host is one spec rather than a preprocessing step. Oscillator models derive ε₁ from ε₂ by Kramers-Kronig, validated in validation/review/ against an analytic Lorentz oscillator and an independent pair-sampled principal-value quadrature — with the residual near resonance attributed, by an h-refinement study, to the FFT-KK grid rather than the quadrature. Optical gain is refused at every door: Im(n) < 0 is not representable on either path — the coherent one conjugates the propagation phase back to decay, so a gain layer returns the loss layer's answer with a positive absorptance — and the message says what to do if the sign is a provider convention rather than a mistake.

• Guided Modes and Field Profiles: eigenmode_landscape scans |1/r(n_eff)|² over a complex effective-index box, find_eigenmodes locates the minima and Nelder-Mead refines them (a converged surface-plasmon mode reaches 1e-24 on the pinned stack), and field_profile returns normalised |E(z)| through the stack with per-layer boundaries. These kernels solve the whole stack as one coherent block by construction and do not consult incoherent_flags — a guided mode is a coherent-stack concept and there is no eigenmode to find across a partition — which every one of their docstrings now says, because every other method on the same object does honour the flags.

• Differential Phase Observables: PDts/PDtp are first-class compute() keys — the transmitted phase with the equivalent incidence-medium layer subtracted (arg(t) − 2π·n(λ)·D·cosθ/λ), so a target on Δφ constrains what the coating adds, not the ambient's round trip. The reference index follows the wavelength (per-λ columns), keys emit the wrapped principal value in (−π, π], and a differential_phase(*, s_pol=True, p_pol=True) view returns both. Group delay over Δφ (GD/GDD with a differential target) carries the reference's own dispersion, and the synthesis merit's phase demands share the same derivation — the merit op point and the compute key agree bit for bit. Coherent stacks only, as with the dispersion keys.

5. Automated Coating Design

Navette doesn't just simulate — it synthesizes, with the classic needle method running natively on the same engine:

• Needle Insertion: Probes every candidate position with an infinitesimal test layer and inserts real material where the merit function improves most — the Tikhonravov needle algorithm, merit-driven and target-aware.

• Thickness Optimization: Levenberg-Marquardt refinement over free layers with bounds and clamping, interleaved with insertion passes and impact-ranked cleanup (merge, thin-layer removal, re-optimization).

• Multi-domain Targets: One joint merit over spectral, angular, and CIE color demands — multiple angles, illuminants with own-white metamerism control, and per-target wavelength windows — all folded into the needle gradient with analytic chain-rule terms, so a single run designs for daylight and showroom light at once.

• One Design, Several Surroundings: A coating measured bare and then laminated is two optical problems with one set of films. run_needle(design={...}, environments=[...]) defines the films once, as a named design segment, and lists what sits around them in each environment; one optimization moves one set of thicknesses against every environment's demands at the same time, so the films cannot drift apart. Demands carry an environment= tag; an unknown name refuses, naming the roster, because scoring against the wrong surroundings looks like a physics result rather than a mistake. The needle sees all of them at once — each environment scans its own assembly and a site in the shared design is routed by parameter into a shared bucket, so the candidate that lands is the one that helps the joint merit, not the first environment. Cost is honest and linear: K environments means K assemblies and K solves per evaluation. The same design writes as a program document (design: / environments: sections) and compiles through the same compiler — the two spellings land on identical merit bits.

• Graded Media: Mixture gradients are a first-class layer property, not a hand-built stack of sublayers. A gradient spec names the two endpoint materials, the EMA kernel (any of the six in-tree mixing rules) and the profile mode — FixedSpan, where composition runs f_start → f_end across the film, or RateCapped, a thickness-relative slope clamped to [f_min, f_max] so a thick film saturates into a pure-material tail. Single-material drift (InhMode) has the same two modes. Graded layers still serve as pinned background (substrate diffusion gradients, rugate foundations) while the needle designs around them — what is new is that a graded span's thickness can itself be an optimizer parameter. Two things about graded spans are easy to get wrong and are written down in docs/graded-media.md: a span expands into sublayer rows (16 authored spans can reach the solver as 80 rows, and the row count is what a solve costs), and Névot-Croce interface roughness on a graded span applies at every sublayer boundary — where the model's σ ≪ thickness condition is the first thing to fail.

• One Physical Layer, One Parameter: A profiled film is one layer, so the optimizer moves its total thickness as a single number rather than one parameter per sublayer, and a thickness-relative profile is re-derived at the construction points when its span rescales instead of carrying stale nk. For the same reason the thin-layer floor, the thickness ceiling and the layer budget are span quantities: they count design layers, not solver rows.

• Thin-Layer Policy: A film driven below the minimum thickness you can actually deposit need not be deleted. thin_layer_policy decides: 'remove' (the default, and bit-for-bit the historical behaviour), 'clamp_up_final' — the search runs exactly as before and only the final pass lifts a surviving sub-minimum film to the floor — or 'clamp_up_always'. A film that carries an interface slice clamps up like any other single layer.

• Saved Designs Stay Readable: state files carry a schema version and the reader accepts a range ([1, 2]), not a point. A state written by an older build is reconstructed from its defaults rather than refused; one written by a newer build is refused with a message that says so, because the remedy there is upgrading, not hand-editing the file.

Technical Specifications

Feature	Implementation & Engineering Benefit
Core Algorithm	1D Scattering Matrix ($S$-matrix): Utilizes the Redheffer Star Product to eliminate numerical divergence and precision loss in thick or highly absorbing layers.
Propagation Logic	Hybrid Mixed Coherence: Sophisticated dual-stage engine supporting phase-accurate (coherent) and intensity-only (incoherent) layers within a single pass.
Coherent Blocks	$2 \times 2$ Complex Field Matrices: Maintains full phase and amplitude information, ensuring rigorous calculation of thin-film interference and ellipsometric parameters.
Incoherent Blocks	Stokes-Mueller / Intensity Redheffer: Prevents unphysical interference artifacts in macroscopic substrates by utilizing intensity-based propagation.
Coherence Modes	Three, with a refusal instead of a guess: FRONT_BLOCK (default, exact intensities), COHERENCY_MATRIX (adds the cascaded p–s coherency channel), FULLY_COHERENT. The twelve cross-channel observables are refused under the default on a flagged stack rather than answered from the front block alone.
Multi-Environment Design	One set of films, K surroundings: a named design segment is defined once and optimized against every environment's demands at the same time, so a coating measured bare and then laminated cannot drift into two designs. K environments means K assemblies and K solves per evaluation — honest and linear.
Material Models	Sixteen dispersion models + six EMA mixing rules, composable and nestable, with Kramers-Kronig ε₁ for the oscillator families and independently validated quadrature.
Guided Modes	Complex eigenmode search: landscape scan → coarse minima → Nelder-Mead refinement over complex n_eff, plus normalised |E(z)| field profiles. Solves the stack as one coherent block by construction.
Roughness Model	Névot-Croce (Exact Wavevector): Achieves research-grade accuracy for X-ray and UV interfaces by modeling exact wavevector correlations across boundaries.
Optimization	Rust / rayon + PyO3: Native multi-threaded kernels (GIL released) with a thin Python API, optimized for high-concurrency simulation and real-time GUI responsiveness.
Polarization	Full $s$ and $p$ Support: Comprehensive Jones and Stokes calculus integration, following standard commercial ellipsometry conventions (Azzam & Bashara). The cross-polarization observables (Δ, DOP, S2/S3, retardance) need COHERENCY_MATRIX on a stack with an incoherent flag; see Partial Coherence Support.
Complexity	$O(N)$ Scaling: Optimized linear time complexity relative to the number of layers, ensuring stable performance for complex multi-stack architectures.

Project layout

Navette/
├── Cargo.toml                # Rust workspace (cargo check/test --workspace)
├── pyproject.toml            # maturin project: builds the `navette` wheel (src layout)
├── src/navette/              # unified Python package
│   ├── __init__.py           # version + public surface
│   ├── color/                # wrapper over native `navette._color`
│   ├── interpolate/          # wrapper over native `navette._interpolate`
│   ├── smatrix/              # ScatterMatrix + needle (native `navette._smatrix`)
│   ├── spectralweave/        # weavers + merit (native `navette._spectralweave`)
│   ├── materials/            # dispersion models (native `navette._materials`)
│   ├── _*.py                 # shims re-exporting the `navette._navette` submodules
│   ├── structure/            # stacks, architect (native model + thin wrappers)
│   ├── synthesis/            # needle pipeline driver (native DesignStack)
│   ├── config/               # native-validated holders, program documents
│   └── data/CIE/             # bundled reference spectra
├── rust/                     # Rust sources: one engine crate + bindings
│   ├── navette/              # pure-Rust engine (color/interpolate/materials/
│   │                         # smatrix/spectralweave/structure modules;
│   │                         # published as `navette` on crates.io)
│   └── navette-py/           # PyO3 aggregator -> navette._navette (one wheel)
├── validation/               # tests, parity, benches, goldens + references (see validation/README.md)
├── tools/check_exposure.py   # bidirectional exposure lint (CI)
├── examples/  docs/plans/  benchmarks/

Install & build

# Single aggregated native extension (navette._navette, all engines):
maturin develop --release
# checks
cargo check --workspace
cargo test --workspace     # everything (needs Python for binding crates)
cargo test-pure            # pure-Rust gate (no Python needed)
cargo fmt --all            # rustfmt defaults; CI fails on any diff
python tools/check_toolchain.py   # is your clippy as new as CI's?
pytest validation

Lint on the toolchain CI uses. cargo clippy only reports the lints its own version knows. Between 0.6.13 and 0.6.30 the local toolchain was one minor version behind CI's stable, the local run was clean, and CI was red for 17 consecutive pushes on a lint the local clippy did not have. tools/check_toolchain.py fails when that gap reopens.

Run this once per clone so git blame skips the tree-wide reformat commit (0.6.32) and points at whoever actually wrote each line:

git config blame.ignoreRevsFile .git-blame-ignore-revs

Always pass --release. Plain maturin develop builds with the dev profile: the extension imports and computes correctly, but runs several times slower, so every timing taken against it is meaningless. This is not hypothetical — a whole round of committed benchmark results (and the conclusions drawn from them) had to be discarded for exactly this reason. navette.build_profile() reports which profile is installed, and the benches under validation/benches/ exit rather than time a "debug" one.

Architecture: Rust core, Python addon

All logic and all validation live in the navette Rust crate — it runs fully standalone (file → design → solve → report, no interpreter). The Python package is a thin addon: validated config holders, YAML→dict parsing, result reshapes, and re-exports. Conversely every feature-level Rust function is exposed via PyO3, so Python can drive the whole engine. tools/check_exposure.py enforces this both ways in CI (see docs/plans/exposure_audit.md).

CI

.github/workflows/ci.yml runs on every push and pull request: cargo test --workspace, a zero-compiler-warnings check (-D warnings), pytest validation on Windows and Linux (pinned runner images, so the recorded fingerprints stay platform-stable), five blocking lints — exposure, CIE sync, .pyi surface sync, message hygiene (tools/check_message_whitespace.py: space runs and console-unencodable characters inside message literals) and review-harness documentation (tools/check_review_docs.py: every harness in validation/review/ is listed in that directory's README with the section or plan item that asked for it, and none imports the parity reference they exist to be independent of) — and an assertion that the installed extension is a release build. cargo clippy -D warnings (since 0.6.6) and cargo fmt --all --check (since 0.6.32) are blocking; nothing in the workflow is advisory any more.

Layout notes

• rust/ holds the Cargo workspace (the single navette engine crate plus the navette-py PyO3 aggregator) — the idiomatic Rust layout, publishable to crates.io.
• src/navette/ is the Python package in src-layout — the idiomatic Python layout, which maturin detects automatically for mixed projects.

Release & publish

Release automation: tag vX.Y.Z (must match pyproject.toml, workspace Cargo.toml, its internal navette dependency, __about__.py and both Cargo.lock entries — all six enforced by CI) → .github/workflows/release.yml builds wheels (Linux/Windows/macOS) and publishes to PyPI (trusted publisher) + crates.io (token), leaf crates first.

maturin build --release   # -> target/wheels/navette-0.7.14-*.whl (single wheel, all engines)

Optimizer backends

LmConfig(optimizer=...) chooses which least-squares solver runs. navette._smatrix.available_optimizers() reports what the installed wheel actually has; a name it lacks is refused with the rebuild command, never quietly replaced by a different solver.

Name	What it is
"builtin" (default)	This crate's bounded Levenberg-Marquardt: QR step solve, gain-ratio damping, analytic Jacobian. Bounds are enforced by vetoing and clamping the solved step, so a thickness may finish exactly on a bound — which is how the synthesis loop learns a film wants removing.
"trf"	Trust-region reflective (Branch-Coleman-Li), the reference method for bounded least squares and the same algorithm as scipy.optimize.least_squares(method="trf"). Hand-rolled, no dependency, always available. Bounds enter the subproblem rather than clipping its answer, so a boundary optimum is handled by construction — but its iterates are strictly interior, so it stops one ULP short of a bound instead of on it. lambda_* and damping do nothing here.

Optional cargo features

Off by default, so a standard wheel pulls no extra dependencies.

Feature	What it adds
opt-minpack-lm	LmConfig(optimizer="minpack_lm") — the levenberg-marquardt crate (MINPACK lmdif-derived, MIT), as a reference to compare the built-in LM against. Unbounded, so it runs on an interior reparametrization: its optima are strictly inside the thickness box, where the built-in's may sit exactly on it.
opt-argmin	LmConfig(optimizer="argmin_gauss_newton") and "argmin_trust_region" — two solvers from the argmin ecosystem (MIT/Apache-2.0), as baselines, not as candidates. Both unbounded. The Gauss-Newton one is undamped, so it raises as soon as JᵀJ is singular — a film driven toward zero thickness is enough — and it refuses two of the three refold starts in validation/review/lm_check.py. The trust region finds the right optimum but has no convergence test of its own, so it always runs the full max_iterations: 218–393 residual evaluations where "trf" takes 8–32. Shares nalgebra with opt-minpack-lm.

maturin develop --release --features opt-minpack-lm
maturin develop --release --features opt-argmin

Manual fallback: cargo publish -p navette; maturin upload target/wheels/navette-0.5.0-*.whl.