Skip to content

[deckhouse-cli] Add plugin mirroring to d8 mirror pull and push - #445

Merged
ldmonster merged 20 commits into
mainfrom
feat/mirror-plugins
Aug 19, 2026
Merged

[deckhouse-cli] Add plugin mirroring to d8 mirror pull and push#445
ldmonster merged 20 commits into
mainfrom
feat/mirror-plugins

Conversation

@Glitchy-Sheep

@Glitchy-Sheep Glitchy-Sheep commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

d8 mirror now carries CLI plugins into air-gapped installs.

Pull picks the plugin versions the mirrored modules need, packs each plugin into its own tar, and push publishes them so d8 inside the closed network can install them.

How it works

  • Plugins live at deckhouse-cli/plugins/<name>, next to the installer, outside the edition path.
  • Each plugin version carries a contract: a base64 JSON annotation with its requirements (modules, other plugins, platform version).
  • The plugins phase runs last in pull. It sees which modules and platform versions actually made it into the bundle.
  • Selection is "nothing extra": a plugin enters the bundle only for a module that needs it, as a dependency of a selected plugin, or via --include-plugin.
  • Dependencies resolve recursively: built-in d8 commands count as present, already picked versions are reused.
  • A candidate with a dependency cycle is rejected, and its half-resolved dependencies roll back.
  • Content problems (no compatible version, broken contract) skip the plugin with a reason in the summary. Network problems fail the whole pull.
  • Multi-platform plugin images stay whole: the OCI index survives the bundle unchanged.

Before / After

Before: d8 mirror carried the platform, modules, packages and security databases; plugins could not reach closed networks at all.

After: the bundle also holds plugin-<name>.tar for every plugin the modules need; push uploads them to the same paths and adds a name tag to the catalog.

Tests

  • Resolver unit suite (31 scenarios): selection gates, dependencies, explicit includes, error policy.
  • TestPullE2E_* (9 tests): the whole PullService.Pull on a fake registry, plus a pull-to-push roundtrip into a target registry.
  • The e2e suite checks that module versions reach the plugin resolver and every picked version lands in the summary with its reason.
  • Command level: a real Puller.Execute with the registry stub produces plugin tars next to module tars; dry-run writes nothing.
  • Multi-platform round-trip against a real in-memory HTTP registry.

Notes

  • --include-plugin accepts ranges and exact pins (name@=vX.Y.Z); only pins reach pre-releases. With --proxy-registry, only exact pins are allowed.

@Glitchy-Sheep
Add plugins registry service under deckhouse-cli/plugins
d2e3f70 
- `PluginsService` lists plugin names from the catalog path, `Plugin(name)` scopes to one plugin repo with its version tags.
- `ContractAnnotation` reads the base64 contract from the manifest: index annotation first, first child as fallback.
- The catalog hangs off the bare registry root, outside the edition segment - same asymmetry as the installer, pinned by tests.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Store multi-platform OCI indexes in image layouts
ed3d1c8 
- `AddIndex` writes a whole image index into the layout: platform children, their descriptors, and index annotations stay as published.
- Same (tag, digest) idempotency guard as `AddImage`, so retried pulls do not duplicate descriptors.
- Needed for CLI plugins: their images are multi-platform and flattening to one platform would lose the rest.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Push multi-platform OCI indexes from layouts
ceca572 
- `pushManifest` picks the push call by descriptor media type: `PushIndex` for a nested index, `PushImage` for a single image.
- Before, push failed on any layout with an index: `index.Image()` rejects index media types.
- CLI plugin images are such indexes: platform children and the contract annotation must reach the target registry whole.
- Round-trip test against an in-memory registry: index digest, both platform children, and the contract annotation survive the push.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Add plugin catalog access for mirror
7f6d98b 
- `Catalog` reads the plugins registry: names from the directory-as-tags index, stable semver versions newest first, contracts decoded from the manifest annotation.
- Results are memoized per pull: one tags listing per plugin, one manifest fetch per version.
- `ErrInvalidContract` marks broken published contracts, so the resolver can skip that version; transport errors stay plain and fail the pull.
- Resolver input/output types: typed module versions in, selected plugin versions with provenance out.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Resolve plugin versions against bundled module versions
1bc3ad8 
- For each bundled version of a mirrored module, the newest plugin version the bundle satisfies is picked; picks are deduplicated.
- Mandatory plugin dependencies resolve recursively: shared versions are reused, disjoint constraints get two versions, cycles and depth are guarded.
- A plugin with no compatible version is skipped with the reason recorded; an unmet `--include-plugin` fails the pull.
- Module and Deckhouse constraints go through `requirements.NormalizedForConstraint` (now exported), so mirror and install compare versions the same way.
- Exact pins bypass the stable-version list, so pre-releases stay reachable.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Add plugins phase to d8 mirror pull
fb00666 
- `PullPlugins` resolves plugin versions for the bundle, pulls them, and packs one `plugin-<name>.tar` per plugin.
- Multi-platform indexes are pulled whole: children fetched by digest, so platform binaries and the contract annotation stay as published.
- Tars carry the `deckhouse-cli/plugins/<name>` prefix - the path mirror push uploads verbatim and registry-packages-proxy serves.
- A registry without a plugins catalog skips auto-selection quietly; dry-run prints the plan and fills stats without downloads.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Wire plugins phase into mirror pull pipeline
2ceed48 
- The plugins phase runs last: it resolves against the module and platform versions the earlier phases actually selected.
- New `--include-plugin name[@constraint]` flag adds plugins on top of the automatic selection; with `--proxy-registry` an exact pin is required.
- `PullSummary.Plugins` carries per-version provenance, skips, and advisories for the summary renderer.
- Dependencies on built-in d8 commands (`delivery-kit`, `package`) are never mirrored.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Publish plugin name index on mirror push
959123c 
- `createPluginsIndex` tags the deckhouse-cli/plugins path with one tag per plugin, so `ListTags` discovery works on the target registry.
- The same directory-as-tags convention modules and packages use; matches what registry-bundle serves for bundle-backed registries.
- `PushSummary.Plugins` counts pushed plugin repositories; the push summary prints a Plugins line.
- Plugin layouts are not touched by `--modules-path-suffix`: their registry path is fixed, pinned by a test.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Render plugins in mirror pull summary
615f589 
- The Plugins line shows the count with a provenance breakdown: how many serve modules, how many are dependencies, how many were explicit.
- Verbose mode draws the tree: plugins grouped by the module they serve, dependencies nested under their dependents.
- Skipped plugins with their reasons are always visible - losing a plugin in an air-gapped bundle should not hide behind a flag.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Document plugin mirroring in d8 mirror
886116c 
- Plugin Mirroring section in the mirror README: selection rules, registry layout, what mirror checks vs what install checks.
- Air-gapped flow in docs/plugins.md: pull -> push -> `d8 plugins install` through the proxy.
- Proxy-registry doc: auto-selection needs a catalog, so only exact `--include-plugin` pins work there.
- The push layout doc-tree gains the deckhouse-cli/plugins branch.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep Glitchy-Sheep self-assigned this Aug 14, 2026
@Glitchy-Sheep Glitchy-Sheep added the enhancement New feature or request label Aug 14, 2026
@Glitchy-Sheep Glitchy-Sheep changed the title [deckhouse-cli] Add plugins mirroring [deckhouse-cli] Add plugin mirroring to d8 mirror pull and push Aug 14, 2026
@Glitchy-Sheep
Add e2e coverage for module-driven plugin mirroring
9737b66 
- The e2e suite runs the whole `PullService.Pull`: module versions from release channels reach the plugin resolver and pick compatible plugin versions.
- Checks summary reasons for picked and skipped plugins, the `deckhouse` constraint, exact pins and the `--only-extra-images` skip.
- A pull-to-push roundtrip carries plugin tars into a target registry and checks the plugin name index.
- `--modules-path-suffix` moves modules only - the roundtrip checks that plugins stay put.
- Test stubs now carry a pullable module and a plugin catalog: the plugins phase is no longer a no-op in orchestration and command-level tests.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Drop the unused version parameter from the stub moduleImage
474d565 
- unparam: every call passed stubModuleVersion, so the helper now reads the constant itself.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@ldmonster
ldmonster requested a balanced review from Copilot August 17, 2026 09:14
Copilot AI reviewed Aug 17, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds automatic CLI plugin mirroring to air-gapped pull/push workflows.

Changes:

  • Resolves plugins from mirrored modules, explicit includes, and dependencies.
  • Preserves and pushes multi-platform plugin indexes.
  • Adds plugin summaries, validation, documentation, and extensive tests.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
pkg/registry/service/service.go Wires plugin registry service.
pkg/registry/service/plugin_service.go Adds plugin catalog operations.
pkg/registry/service/plugin_service_test.go Tests catalog scoping and contracts.
pkg/registry/image/layout.go Adds OCI index storage.
pkg/registry/image/layout_test.go Tests index layout behavior.
pkg/fake/deckhouse_stub.go Adds module/plugin fixtures.
pkg/fake/deckhouse_stub_test.go Tests new fixtures.
internal/plugins/requirements/checks.go Exports version normalization.
internal/plugins/requirements/checks_test.go Updates normalization tests.
internal/plugins/README.md Documents air-gapped delivery.
internal/mirror/summary.go Adds plugin statistics.
internal/mirror/README.MD Documents plugin mirroring.
internal/mirror/pusher/pusher.go Pushes nested OCI indexes.
internal/mirror/pusher/pusher_test.go Tests multi-platform pushes.
internal/mirror/push.go Pushes plugins and catalog tags.
internal/mirror/push_test.go Tests plugin push layout.
internal/mirror/pull.go Integrates plugin pull phase.
internal/mirror/pull_test.go Extends pull coverage.
internal/mirror/pull_plugins_wiring_test.go Tests phase handoff.
internal/mirror/pull_plugins_e2e_test.go Tests pull/push plugin workflows.
internal/mirror/PROXY-REGISTRY.md Documents proxy restrictions.
internal/mirror/plugins/types.go Defines resolver types.
internal/mirror/plugins/stats.go Implements plugin accounting.
internal/mirror/plugins/resolver.go Implements plugin resolution.
internal/mirror/plugins/resolver_test.go Tests resolver scenarios.
internal/mirror/plugins/pull_plugins_test.go Tests plugin pulling and packing.
internal/mirror/plugins/plugins.go Implements plugin mirror service.
internal/mirror/plugins/doc.go Documents the package.
internal/mirror/plugins/catalog.go Implements registry catalog access.
internal/mirror/plugins/catalog_test.go Tests catalog behavior.
internal/mirror/cmd/push/summary.go Displays pushed plugin count.
internal/mirror/cmd/push/summary_test.go Tests push summary output.
internal/mirror/cmd/pull/validation.go Validates proxy plugin pins.
internal/mirror/cmd/pull/validation_test.go Tests plugin pin validation.
internal/mirror/cmd/pull/summary.go Renders plugin pull summary.
internal/mirror/cmd/pull/summary_test.go Tests plugin summary output.
internal/mirror/cmd/pull/pull.go Wires plugin CLI options.
internal/mirror/cmd/pull/pull_plugins_stub_test.go Tests command-level plugin pulls.
internal/mirror/cmd/pull/flags/flags.go Adds --include-plugin.
internal/layout.go Defines plugin registry segments.
docs/plugins.md Documents air-gapped installs.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/mirror/plugins/plugins.go
Comment thread internal/mirror/plugins/plugins.go Outdated
Comment thread internal/mirror/cmd/pull/validation.go
Comment thread internal/mirror/cmd/pull/summary.go
@Glitchy-Sheep
Show plugin resolver warnings in pull summary
467081d 
PluginsStats.Warnings was filled by the resolver but never rendered,
so unmet gates and co-installation conflicts only appeared in the live
log. Warnings now follow the skipped plugins, always visible: a plugin
the target cluster cannot run is as much a surprise as a missing one.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Allow plugins-only proxy pull with exact pins
b55c8c7 
validateProxyRegistryFlag rejected --no-platform --no-modules as
"nothing to do" before it reached the --include-plugin check, so exact
plugin pins could never be mirrored on their own through a proxy
registry. Plugins are now counted as a component in that guard.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Validate plugin names before filesystem and registry use
9551e5f 
Mirror built layout directories, bundle tar names and registry routes
from plugin names taken verbatim from the catalog, --include-plugin and
contract dependencies. The registry client normalizes "../x" away while
filepath.Join keeps it, so a hostile source registry could write outside
the working directory.

The single-component rule from internal/plugins now lives in
internal/plugins/layout and is applied at every name source: catalog
entries are dropped, explicit includes error out, dependency names
reject the dependent. layoutFor guards the join as a last line.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Carry subject and inline data through rebuilt plugin index
18a1be4 
rebuildIndex re-marshals the top-level index locally and dropped the
OCI 1.1 subject and per-child inline data, silently changing the digest
of indexes that carry them. Both fields now travel with the index;
per-child artifactType stays out because ggcr's mutate cannot set it.

The multi-platform round-trip test now asserts the bundled index keeps
the published digest, which its doc comment promised but never checked.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Reject --include-plugin together with --only-extra-images
508bc3f 
--only-extra-images skips the plugins phase, so an explicit
--include-plugin was accepted by validation and then dropped without a
word: exit 0, no plugin tar in the bundle. Fail up front instead - an
explicit include is either mirrored or an error.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Resolve explicit plugin pins before their dependencies
75415b5 
Explicit pins were resolved one by one in name order, so a dependency
the user pinned was reused only when its name sorted first; otherwise a
newer version was pulled beside it, and with --proxy-registry the pull
failed outright because dependencies were looked up in a catalog the
proxy does not serve. Pins are now all committed first, then their
dependencies resolved; without a catalog an unpinned dependency is an
error that names the pin to add.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Document that plugin contracts depend on external modules only
6c7e1c8 
d8 mirror auto-selects plugins from the mirrored external modules;
modules embedded in the platform image are not a supported dependency.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Drop auto-selection skips for plugins added by --include-plugin
c5b82a7 
A plugin the auto path passed over but the user then included
explicitly was reported both as mirrored and as skipped in the summary.
A skip is dropped once the plugin is in the final selection; the
explicit path's warning about the unmet module requirement stays.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Glitchy-Sheep marked this pull request as ready for review August 18, 2026 07:14
@ldmonster
ldmonster merged commit 18d0ef8 into main Aug 19, 2026
11 of 17 checks passed
@ldmonster
ldmonster deleted the feat/mirror-plugins branch August 19, 2026 11:21
@Glitchy-Sheep Glitchy-Sheep mentioned this pull request Aug 19, 2026
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ldmonster ldmonster ldmonster approved these changes

Assignees

@Glitchy-Sheep Glitchy-Sheep

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Glitchy-Sheep @ldmonster