feat(pypi): derive Metaflow @pypi environments from uv.lock

[Abhishek Patil (abhishek-pattern)]

What

Adds @uv_pypi_base / @uv_pypi — Metaflow's @pypi_base / @pypi with the Python version and packages filled in from the repo's own uv.lock, so a flow's environment cannot drift from what uv sync installs.

from metaflow import FlowSpec, step

from ds_platform_utils.metaflow import uv_pypi_base


@uv_pypi_base
class MyFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.end)

    @step
    def end(self):
        pass

No python=, no packages=, nothing to keep in sync. Both decorators work bare or called: @uv_pypi_base(dependency_groups=["dev"]), python=, project_root=, disabled=.

Why versions come from uv.lock

pyproject.toml holds constraints, not versions. A >= bound, or a git dependency pinned to rev = "main", means two bakes a week apart can produce different images. uv.lock records what uv actually resolved, including the commit SHA for git dependencies, so a bake is reproducible.

Resolving a universal lockfile

uv.lock is a universal resolution — every Python version and platform in range at once, each tagged with a marker. @pypi takes a flat {name: version} map with nowhere to put a marker, so markers are resolved at decoration time:

Lock state	Emitted
One entry, or several with a marker that selects one	that exact version
Marker excludes this environment (sys_platform == 'darwin' on a Linux bake)	omitted
Several entries, nothing to tell them apart	"", left for @pypi

Concretely, from one unchanged lockfile:

python=3.10  ->  pandas 2.3.3
python=3.11  ->  pandas 3.0.5

The Python version is resolved first and handed down, so the interpreter and the packages can't disagree. Markers evaluate against Linux, which is what Metaflow builds for a remote task.

Only the root project's direct dependencies are emitted, not the transitive closure — @pypi resolves transitives itself, and per-platform wheel availability is its job.

Where the Python version comes from

1. .python-version — the interpreter uv pinned
2. requires-python in uv.lock, else pyproject.toml — a range, so its floor
3. The running interpreter

An installed package can't find the flow repo from __file__, so project files are located by walking up from the launch directory; project_root= overrides.

Reviewer notes

• The second commit changes what flows bake. Moving the floor to 3.10 is what selects pandas 2.3.3 over 3.0.5. Worth a deliberate look — requires-python is a published constraint, so raising it later would stop 3.10 consumers installing this library.
• packaging is now a declared runtime dependency; it was only present transitively and is used for PEP 508 marker and specifier parsing.
• Only uv_pypi and uv_pypi_base are exported. The helpers that build the map are private.
• Pre-existing, not from this PR: the 3.10 floor moves ruff's inferred target, so ruff 0.16 now wants PEP 604 annotations across ~158 sites repo-wide. This PR keeps the existing Optional/Union style rather than mixing. Either declare requires-python and modernize in one dedicated pass, or pin target-version in [tool.ruff].

Testing

24 unit tests in tests/unit_tests/metaflow/test__pypi_packages.py; 80 pass overall. Coverage includes version pinning, git SHA references, marker resolution by Python version, platform-gated dependencies being dropped, the unpinned fallback, dependency groups, the upward project-root search, and the empty map that lets remote tasks re-import the flow module.

Decorator tests assert against a spy on Metaflow's pypi_base — the contract is "call Metaflow with this environment", and Metaflow 2.19 already moved where decorators are recorded. One test exercises the real decorator end to end.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Introduces Metaflow decorators that derive @pypi_base / @pypi configuration (Python version + direct dependency pins) from the repository’s uv.lock, aiming to prevent environment drift between local uv sync and Metaflow-baked images.

Changes:

• Added uv_pypi_base and uv_pypi decorators that compute python= and packages= from .python-version / requires-python and uv.lock.
• Implemented uv.lock parsing + PEP 508 marker evaluation to emit a flat name -> version/direct-reference map for Metaflow.
• Added docs and unit tests covering version selection, marker handling, git SHA pinning, dependency groups, and project-root discovery; updated runtime deps (packaging) and bumped default Python pin to 3.10.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit_tests/metaflow/test__pypi_packages.py	New unit tests exercising uv.lock parsing and decorator behavior.
src/ds_platform_utils/metaflow/pypi_packages.py	New implementation of uv.lock → Metaflow @pypi(_base) environment derivation and wrappers.
src/ds_platform_utils/metaflow/__init__.py	Exports uv_pypi and uv_pypi_base as the public API.
README.md	Adds docs link for the new decorators.
pyproject.toml	Adds runtime dependency on packaging and bumps the polars lower bound.
docs/metaflow/pypi_packages.md	New documentation page for uv_pypi_base / uv_pypi.
.python-version	Updates the repo’s pinned Python version to 3.10.

Suppressed comments (1)

src/ds_platform_utils/metaflow/pypi_packages.py:295

• _split_lock_packages overwrites root every time it sees any local-source package (virtual/editable/directory). If a lock contains additional local directory/editable dependencies, the last one wins and the derived dependency list can come from the wrong project entry. Avoid overwriting root once it has been found.

        if any(key in source for key in _LOCAL_SOURCE_KEYS):
            # the repo itself -- it is the thing depending on everything else, not a dep.
            root = package
            continue

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.