Skip to content

test: pin Rich colour and width so CLI-output tests stop failing outside CI - #1332

Open
saltas888 wants to merge 3 commits into
developfrom
pha/INBOX-166
Open

test: pin Rich colour and width so CLI-output tests stop failing outside CI#1332
saltas888 wants to merge 3 commits into
developfrom
pha/INBOX-166

Conversation

@saltas888

@saltas888 saltas888 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Why

Tests that assert on CLI text compare against output whose colour and width Rich decides from the ambient environment. On a developer machine that exports FORCE_COLOR (or has a narrow terminal), Rich emits ANSI escapes and truncates its tables, so a large block of tests fails locally while staying green in CI — which trains contributors to explain away red test output.

This was reported independently by two contributors in the same week, in their PR descriptions: #1227 recorded 3 pre-existing failures in tests/unit/sdk/test_schema.py that "assert on plain strings while Rich emits ANSI colour codes in this environment", and #1233 recorded further unrelated-to-that-PR failures on stable.

Goal: make the suite's rendering environment deterministic, so CLI-output assertions give the same answer everywhere.

Non-goals: no production code changes, and no loosening of any assertion — the exact-output tests keep their value.

Closes INBOX-166

What changed

  • tests/conftest.py — a pytest_configure hook pins the rendering environment for the whole suite: unset FORCE_COLOR, set NO_COLOR=1 and COLUMNS=200.
  • tests/unit/sdk/test_schema.py — the three consoles the tests build themselves are now explicit: Console(file=StringIO(), width=1000, no_color=True, force_terminal=False), so they do not depend on the environment at all.
  • tests/unit/test_render_env.py (new) — pins the pinning: that the hook ran before this module was imported, that the width survives any TERM, and that FORCE_COLOR is what would actually clamp it. A future edit that loosens the hook fails here with an explanation, instead of as a scatter of puzzling output-comparison failures across the ctl tests.
  • tests/AGENTS.md — documents the invariant and the two traps below, so the next contributor does not rediscover them.

Implementation notes

Three details drove the design, all verified against rich 13.9.4 / click 8.3.3 rather than assumed:

  1. It has to be pytest_configure, not a fixture. Rich snapshots no_color in Console.__init__, and many infrahub_sdk.ctl modules build a module-level Console() at import time — which happens while pytest collects the test modules, before any fixture can run. A session-scoped autouse fixture is already too late for those consoles.

  2. FORCE_COLOR must be removed, not blanked. Rich treats any FORCE_COLOR value as proof it is writing to a terminal, empty string included (if force_color is not None: self._force_terminal = True).

  3. TERM=dumb is deliberately not set. It looks like the obvious way to disable colour, but it is the trigger for Rich's dumb-terminal path, which pins the width to 80 and ignores COLUMNS — truncating the very wide tables the CLI-output fixtures record (e.g. the repository_list fixture is 139 columns).

    To be precise about the mechanism, since it was raised in review: TERM=dumb alone is harmless here, because is_dumb_terminal is is_terminal and TERM in ("dumb", "unknown") and captured test output is never a terminal. It is removing FORCE_COLOR (point 2) that keeps is_terminal False and so defuses the dumb path. Pinning TERM to a non-dumb value would therefore be a no-op for width while implying a knob that does nothing — whereas adopting TERM=dumb, as the card originally suggested, is what creates the failure if anything ever does force a terminal. test_render_env.py pins both halves of this.

One central hook covers all 24 CliRunner() call sites, so no per-test env plumbing was needed — passing env= to each runner would have been 24 edits that drift, and (per point 1) would not have fixed the module-level consoles anyway.

What stayed the same

No production code touched — the diff is tests/ plus a changelog fragment. No API, schema, dependency, or CI-workflow changes.

How to review

tests/conftest.py is the whole change; the rest follows from it. The comment block there records the three findings above.

How to test

The card's own acceptance gate, plus the environments that reproduce the original reports:

uv run pytest tests/unit/                                   # bare environment
FORCE_COLOR=1 COLUMNS=40 uv run pytest tests/unit/          # forced colour, narrow
NO_COLOR=1 uv run pytest tests/unit/                        # colour disabled
TERM=dumb uv run pytest tests/unit/                         # dumb terminal
FORCE_COLOR=3 TERM=xterm-256color COLUMNS=40 uv run pytest tests/unit/
FORCE_COLOR= TERM=dumb uv run pytest tests/unit/            # empty FORCE_COLOR + dumb
uv run ruff check . && uv run ruff format --check .

Before (on develop, with FORCE_COLOR=3 exported — the reported dev-machine case): 37 failed, 1781 passed, spread across tests/unit/ctl/test_marketplace_app.py (31), tests/unit/sdk/test_schema.py (3), tests/unit/ctl/test_repository_app.py (2) and tests/unit/ctl/test_task_app.py (1).

After: 1827 passed, 1 xfailed — identical under all six environments above. ruff check, ruff format --check, ty, mypy (158 files) and rumdl (129 files) are all clean; invoke lint reports only vale is not installed locally.

Two notes for the reviewer:

  • The card also named test_gitrepo_init in tests/unit/sdk/test_repository.py. It does not fail on current develop in any of the environments tested, so nothing was changed for it.
  • test_marketplace_app.py's 31 failures were not in the card's list but share the exact same root cause and are fixed by the same hook.

Impact & rollout

  • Backward compatibility: test-only; no runtime behaviour changes.
  • Config/env changes: the test process now sets NO_COLOR/COLUMNS and unsets FORCE_COLOR for itself. A side effect is that pytest's own report renders at 200 columns.
  • Deployment notes: safe to merge; nothing to coordinate.

Checklist

  • Tests added/updated
  • Changelog entry added
  • External docs updated (if user-facing or ops-facing change) — n/a, test-only
  • Internal .md docs updated (tests/AGENTS.md)

Filed from Engineering Inbox card INBOX-166 by the platform-health agent. Opened for human review — the agent does not merge.

🤖 Generated with Claude Code


Summary by cubic

Pins Rich's colour and width for the test suite so CLI-output assertions behave the same everywhere, fixing 37 tests that failed on developer machines but passed in CI. Closes INBOX-166.

  • tests/conftest.py now unsets FORCE_COLOR and sets NO_COLOR=1 and COLUMNS=200 from pytest_configure before any test module is imported.
  • The three consoles in tests/unit/sdk/test_schema.py are constructed with no_color=True and force_terminal=False so they don't depend on the environment.
  • tests/unit/test_render_env.py guards the pinning: the env is applied, the width survives any TERM, and FORCE_COLOR set with TERM=dumb is pinned as the failure mode the hook prevents, for every value including the empty string.

Written for commit caada89. Summary will update on new commits.

Review in cubic

Tests that assert on CLI text let Rich decide colour and width from the
ambient environment, so a developer whose shell exports FORCE_COLOR got
ANSI escapes and truncated Rich tables in captured output. 37 tests across
test_marketplace_app.py, test_repository_app.py, test_task_app.py and
test_schema.py failed locally while staying green in CI, which trains
contributors to ignore red test output.

Pin the rendering environment from pytest_configure in tests/conftest.py:
unset FORCE_COLOR, set NO_COLOR=1 and COLUMNS=200. The hook has to run
there rather than in a fixture: Rich snapshots no_color when a Console is
constructed, and many infrahub_sdk.ctl modules build a module-level
Console() during collection, before any fixture runs. Rich also treats any
FORCE_COLOR value, empty string included, as proof it is writing to a
terminal, so the variable has to be removed rather than blanked.

TERM is deliberately left alone. TERM=dumb sends Rich down its
dumb-terminal path, which pins the width to 80 and ignores COLUMNS, which
would truncate the wide tables the CLI-output fixtures record.

One central hook covers all 24 CliRunner call sites, so no per-test env
plumbing is needed. The three consoles that tests/unit/sdk/test_schema.py
builds itself are made explicit with no_color=True and
force_terminal=False so they do not depend on the environment at all.

The suite is now green under FORCE_COLOR=1 COLUMNS=40, NO_COLOR=1,
TERM=dumb and a bare environment alike.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@saltas888
saltas888 requested a review from a team as a code owner September 8, 2026 11:19
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 8, 2026

Copy link
Copy Markdown

Deploying infrahub-sdk-python with  Cloudflare Pages  Cloudflare Pages

Latest commit: caada89
Status: ✅  Deploy successful!
Preview URL: https://1ea6d260.infrahub-sdk-python.pages.dev
Branch Preview URL: https://pha-inbox-166.infrahub-sdk-python.pages.dev

View logs

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@             Coverage Diff             @@
##           develop    #1332      +/-   ##
===========================================
- Coverage    85.54%   84.24%   -1.31%     
===========================================
  Files          148      147       -1     
  Lines        14271    13047    -1224     
  Branches      1953     1930      -23     
===========================================
- Hits         12208    10991    -1217     
+ Misses        1496     1493       -3     
+ Partials       567      563       -4     
Flag Coverage Δ
integration-tests 39.10% <ø> (-4.86%) ⬇️
python-3.10 57.07% <ø> (-3.62%) ⬇️
python-3.11 57.06% <ø> (-3.65%) ⬇️
python-3.12 57.06% <ø> (-3.65%) ⬇️
python-3.13 57.07% <ø> (-3.64%) ⬇️
python-3.14 57.07% <ø> (-3.62%) ⬇️
python-filler-3.12 23.68% <ø> (+2.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread tests/conftest.py
Review on #1332 asked whether TERM=dumb defeats the COLUMNS=200 pin, since
pytest_configure leaves TERM alone. It does not, but nothing in the suite
said so, so the question was fair.

Rich clamps to width 80 and ignores COLUMNS only on a dumb terminal, and
is_dumb_terminal is `is_terminal and TERM in ("dumb", "unknown")`. Captured
test output is never a terminal, and the hook removes the one variable that
would make Rich claim otherwise -- FORCE_COLOR, which Rich reads as proof of
a terminal for any value. So removing FORCE_COLOR is what defuses the dumb
path; pinning TERM would be a no-op.

Add tests/unit/test_render_env.py to hold that: the hook's env is applied,
the width survives TERM in dumb/unknown/xterm-256color/screen/empty, and
the one combination that would clamp to 80 (FORCE_COLOR set with TERM=dumb)
is pinned as the failure mode the hook exists to prevent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread tests/unit/test_render_env.py Outdated
The regression test pinned the FORCE_COLOR + TERM=dumb clamp with
FORCE_COLOR=1 only, while the invariant it exists to document is that Rich
tests `FORCE_COLOR is not None` -- so `export FORCE_COLOR=` forces a
terminal exactly as `1` does. That empty form is the case that makes
removing the variable the only correct fix rather than overriding it with
a falsy value, and it was the one form the test never exercised.

Parameterize over "1", "3" and "" ("3" being what a real shell exports).
All three clamp to width 80, so the documented claim is now pinned instead
of asserted in a comment.

Raised by cubic-dev-ai on PR #1332.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant