perf: enable C-accelerated YAML parser/emitter via ruamel.yaml.clib

[oboehmer]

Description

Enables C-accelerated YAML parsing and dumping by switching ruamel.yaml.YAML(typ="safe") from pure=True to pure=False in nac_test/utils/yaml.py, and adds ruamel.yaml.clib as an explicit dependency for CPython in pyproject.toml.

ruamel.yaml.clib was already installed on macOS and Linux before this PR, pulled in unconditionally by genie → genie-libs → genie-libs-sdk. Two consequences:

• nac_yaml was already C-accelerated on macOS/Linux. It leaves pure=False at its default, so it picked up clib transitively. This PR does not newly enable that — it guarantees and version-pins the dependency explicitly instead of relying on a transitive one.
• Windows is the only new install surface. pyats/genie are excluded there (sys_platform != 'win32'), so clib was absent. Windows now gains C-acceleration for both nac_test.utils.yaml and nac_yaml.

The measurable delta from this PR: nac_test.utils.yaml stops opting out via pure=True on all platforms (~3.1× faster YAML loading), and Windows gains acceleration across the board.

Closes

• Fixes perf: use C-accelerated YAML parser (ruamel.yaml.clib / pure=False) in yaml utils #943

Related Issue(s)

• Related to perf: slow data model loading - YAML re-parsed by every test subprocess (~1.9s per test) #931

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactoring / Technical debt (internal improvements with no user-facing changes)
• Documentation update
• Chore (build process, CI, tooling, dependencies)
• Other (please describe):

Test Framework Affected

• PyATS
• Robot Framework
• Both
• N/A (not test-framework specific)

Network as Code (NaC) Architecture Affected

• All architectures
• N/A (architecture-agnostic)

Platform Tested

• macOS (version tested: Darwin arm64, Python 3.12.10)
• Linux (distro/version tested: )

Key Changes

• nac_test/utils/yaml.py:
  • Changed YAML(typ="safe", pure=True) to YAML(typ="safe", pure=False) in _create_yaml_dumper() and safe_load().
  • Added documentation explaining C-accelerated parsing/dumping behavior and transparent fallback.
• pyproject.toml:
  • Added "ruamel.yaml.clib>=0.2.12; platform_python_implementation == 'CPython'" under dependencies.
• uv.lock:
  • Locked ruamel-yaml-clib with pre-compiled wheels for macOS, Linux, and Windows (win_amd64/win32).
• CHANGELOG.md:
  • Added performance changelog entry under # unreleased.

Testing Done

• Unit tests added/updated
• Integration tests performed
• Manual testing performed:
  • PyATS tests executed successfully
  • Robot Framework tests executed successfully
  • D2D/SSH tests executed successfully (if applicable)
  • HTML reports generated correctly
• All existing tests pass (pytest / pre-commit run -a)

Test Commands Used

uv run pytest -n auto --dist loadscope
uv run ruff check nac_test/ pyproject.toml
uv run mypy nac_test/utils/yaml.py

Checklist

• Code follows project style guidelines (pre-commit run -a passes)
• Self-review of code completed
• Code is commented where necessary (especially complex logic)
• Documentation updated (if applicable)
• No new warnings introduced
• Changes work on both macOS and Linux
• CHANGELOG.md updated (if applicable)

Additional Notes

• Windows Compatibility: ruamel.yaml.clib distributes pre-compiled binary wheels (.whl) on PyPI for Windows across all supported Python versions (3.10-3.14), so no local C compiler is required on Windows.
• Data Equality: Verified that YAML(typ="safe", pure=False) parses to identical Python data structures as pure=True, and raises identical exception types. Emitted YAML is semantically identical but not byte-identical — the C emitter wraps long scalars and quotes multi-line strings differently. Only affects the generated pyATS testbed file, which pyATS re-parses.

[oboehmer]

TL;DR

@danischm asked why ruamel.yaml.clib was split out of ruamel.yaml, and whether adding it as a direct dependency risks breaking our user base. Short answer: low risk, recommend keeping it.

• The split was for packaging convenience, not stability — it lets ruamel.yaml ship as a universal wheel and lets distros place the .so separately. It is not a signal about the C extension's quality.
• It is already installed for every macOS/Linux user today — genie → genie-libs → genie-libs-sdk requires it unconditionally, and it was already in uv.lock on main before this PR. The only new install surface is Windows.
• Install-time risk is the real surface, and it is covered — pre-built wheels exist for every supported interpreter and platform, so no user needs a C compiler.
• No version conflict — genie caps <0.2.15, we floor >=0.2.12, resolves to 0.2.14.
• One thing to watch: ruamel.yaml 0.19 introduced a successor, ruamel.yaml.clibz, and relabelled our package's extra as oldlibyaml. Not a problem today, but we should follow genie when it migrates. Details in §5.

Note: nac_yaml was already C-accelerated on macOS/Linux — the PR description has been updated to reflect this. See §2.

---

Risk assessment: adding ruamel.yaml.clib as a direct dependency

1. Why upstream split it out

Per the ruamel.yaml.clib README, the split was for packaging convenience:

This package was split of from ruamel.yaml, so that ruamel.yaml can be build as a universal wheel. Apart from the C code seldom changing, and taking a long time to compile for all platforms, this allows installation of the .so on Linux systems under /usr/lib64/pythonX.Y (without a .pth file or a ruamel directory) and the Python code for ruamel.yaml under /usr/lib/pythonX.Y.

Three motivations, all build/distribution concerns of the maintainer and Linux distro packagers:

• ruamel.yaml ships as a single pure-Python universal wheel instead of maintaining a per-platform build matrix
• The rarely-changing C code is not recompiled on every ruamel.yaml release
• Distros can split the .so (/usr/lib64) from the Python code (/usr/lib)

Nothing in that rationale marks the C extension as unstable or deprecated — it remains the accelerator ruamel.yaml imports and prefers automatically. Worth noting that pure=False is ruamel.yaml's own default; our previous pure=True was the deviation.

2. It is already installed for every macOS/Linux user today

ruamel-yaml-clib was already present in uv.lock on main, before this PR:

genie → genie-libs → genie-libs-sdk → ruamel-yaml-clib   (unconditional, no marker)

genie>=26.5; sys_platform != 'win32' is a core nac-test dependency.

Platform	Before this PR	After this PR	Net change
macOS / Linux	already installed (via genie)	installed	none — now explicit and floor-pinned
Windows	absent (pyats/genie excluded)	installed	new, and the only new install surface

On macOS/Linux we introduce nothing new into the user base; we make explicit what genie already forces. To be precise about what changes: nac_yaml was already C-accelerated there, since it leaves pure=False as the default and clib was already present. The real deltas in this PR are (a) nac_test.utils.yaml no longer opts out via pure=True, and (b) Windows gains acceleration.

3. Install-time risk

This is the failure mode that would actually affect users: if no wheel matches a user's platform, pip falls back to the sdist and requires a local C compiler — which would break pip install nac-test for anyone without a toolchain.

Current coverage is 46 wheels spanning cp310–cp313 (our entire supported range): macosx_universal2 / arm64, manylinux2014 (x86_64 / aarch64 / i686), musllinux_1_2, win32, win_amd64. Every platform we support has a pre-built wheel, including the Windows case that is new here.

Two items to track:

• cp314 currently has no manylinux wheels (only macOS, musl, Windows). If we add Python 3.14, glibc Linux would fall back to compiling from sdist. This arrives via genie regardless of this PR, but it is worth watching before we bump supported versions.
• Exotic platforms outside the wheel matrix would hit the sdist path. Same caveat already applies today on macOS/Linux via genie, so this PR only extends it to Windows.

4. Version constraint check

Package	Constraint on clib
genie.libs.sdk	ruamel.yaml.clib<0.2.15
nac-test (this PR)	ruamel-yaml-clib>=0.2.12; platform_python_implementation == 'CPython'

No conflict. The effective window on macOS/Linux is >=0.2.12,<0.2.15, resolving to 0.2.14. Genie owns the ceiling, we own the floor. One nuance: on Windows genie is absent, so no ceiling applies and Windows may resolve to a newer clib than macOS/Linux once 0.2.15 ships.

The platform_python_implementation == 'CPython' marker correctly excludes PyPy, where the C extension is neither available nor desirable.

5. Forward-looking note: clib → clibz

Flagging for a future change, not this PR. ruamel.yaml 0.19.1 declares two accelerator extras:

ruamel.yaml.clib;         extra == "oldlibyaml"
ruamel.yaml.clibz>=0.3.7; extra == "libyaml"

and cyaml.py now prefers _ruamel_yaml with a fallback to _ruamel_yaml_clibz. The naming ("oldlibyaml") signals upstream is migrating toward ruamel.yaml.clibz.

We should stay on ruamel.yaml.clib for now: it is what genie pins and installs, and it is what is actually active today (cyaml.__yaml_lib == 'clib'). But when genie moves, we should follow — possibly by depending on the extra (ruamel.yaml[oldlibyaml]) rather than the package directly, so upstream chooses the accelerator for us.

One behavioural caveat worth disclosing

The C emitter produces textually different output from the pure-Python emitter: long scalars wrap differently and multi-line strings are quoted differently. Round-trip data is identical, but the bytes are not.

Blast radius is small — dump() is used only in testbed_generator.py to produce pyATS testbed YAML, which pyATS re-parses (machine-consumed, never diffed or committed). But if we ever snapshot-test emitted YAML, that would need attention.

Recommendation

Keep it as a direct dependency. It is already unconditionally installed on macOS/Linux via genie, wheels cover every supported interpreter and platform, and there is no version conflict. Making it optional would add a user-facing knob, a second CI configuration, and support ambiguity — without removing any risk that actually exists.