Skip to content

fix: propagate deploy tags on the model-customization endpoint paths - #6179

Closed
rohangujarathi wants to merge 1 commit into
aws:masterfrom
rohangujarathi:fix-modelbuilder-deploy-tags
Closed

fix: propagate deploy tags on the model-customization endpoint paths#6179
rohangujarathi wants to merge 1 commit into
aws:masterfrom
rohangujarathi:fix-modelbuilder-deploy-tags

Conversation

@rohangujarathi

Copy link
Copy Markdown
Member

Tags passed to ModelBuilder.deploy() were accepted but never reached the endpoints created on the model-customization paths, so they could not be used for tag-based resource association.

deploy() branches into _deploy_model_customization() for fine-tuned and Nova models and returns before reaching _deploy_core_endpoint(), which is the only place that merges the deploy-time tags kwarg into self._tags. Those paths build their endpoint tags from self._tags, so they saw only the tags ModelBuilder had added itself. Merge the kwarg into the accumulator on entry so all three endpoint-creation paths below pick it up.

The restricted-model-package path created its endpoint with no tags at all; it now receives them like the other two.

Adds _tags_as_key_value_list() to normalize tags into the lowercase key/value list the sagemaker.core.resources create() calls are typed for. self._tags can hold either casing or a bare {key: value} mapping, and the previous inline reads indexed tag["Key"] directly after format_tags(), which returns a list unchanged whatever its casing -- so a lowercase list raised KeyError. That was already reachable through build(tags=...) and add_tags().

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

Claude finished @rohangujarathi's task in 2m 40s —— View job


PR Review: propagate deploy tags on model-customization endpoint paths

  • Read PR diff (/tmp/pr.diff)
  • Analyze _tags_as_key_value_list and the three endpoint-creation paths
  • Trace the add_tags / _validate_new_tags merge path
  • Review tests
  • Post review

(The inline-comment tool wasn't available in this run, so findings are consolidated here with file:line references.)


The overall approach is sound — centralizing tag normalization in _tags_as_key_value_list() and merging the deploy-time tags kwarg on entry so all three endpoint paths pick it up is the right shape, and handling both casings + object tags + bare mappings is a nice cleanup. I found one correctness issue that I believe defeats the fix in its main scenario, plus a test-coverage gap.

🔴 High — add_tags() will raise KeyError when self._tags is already populated

model_builder.py:6230-6231:

if kwargs.get("tags"):
    self.add_tags(_tags_as_key_value_list(kwargs["tags"]))

_tags_as_key_value_list() returns lowercase {"key":.., "value":..} dicts. That value flows into add_tags() (model_builder_utils.py:2792) → _validate_new_tags()tag_exists() (sagemaker-core/.../common_utils.py:2100-2102):

for curr_tag in curr_tags:
    if tag["Key"] == curr_tag["Key"]:   # tag is the NEW lowercase dict → KeyError: 'Key'

tag_exists indexes the new tag with the capitalized "Key". As soon as curr_tags (i.e. self._tags) is a non-empty list, the loop body runs and tag["Key"] raises KeyError because the PR passes lowercase-keyed dicts.

Is self._tags non-empty here in practice? Yes, exactly in the case this PR targets:

  • build() on a JumpStart string model calls add_jumpstart_model_info_tags(...), which appends capitalized {"Key":.., "Value":..} dicts (add_single_jumpstart_tag, jumpstart/utils.py:418-422).
  • _extract_and_extend_tags_from_model_trainer() (model_builder.py:3642) extends self._tags with CoreTag objects for ModelTrainer-based (fine-tuned) models.

So a user who does build(...) then deploy(tags=[...]) on a fine-tuned/Nova path — the whole point of this change — will hit self.add_tags([{"key":..,"value":..}]) with a pre-populated self._tags and crash before reaching Endpoint.create.

Note the standard path (_deploy_core_endpoint, model_builder.py:3857-3858) sidesteps this by passing the raw tags kwarg into add_tags rather than the lowercased form. Suggested fixes, either of:

  • Pass the raw kwarg to the accumulator (mirror _deploy_core_endpoint): self.add_tags(kwargs["tags"]), and rely on the existing _tags_as_key_value_list(self._tags) calls at each Endpoint.create to normalize — which they already do.
  • Or make tag_exists / the merge tolerant of both casings.

🟡 Medium — Test gap: the merge-with-existing-tags case is never exercised

In test_model_builder_deploy.py, every builder in TestDeployNovaModelTags starts from builder._tags = None (_make_builder, line 186), and test_deploy_tags_reach_the_endpoint_through_model_customization calls _deploy_model_customization on a builder whose _tags is None. With curr_tags=None, _validate_new_tags returns early and never calls tag_exists, so the KeyError above is masked.

Please add a case where self._tags is pre-populated (e.g. a capitalized {"Key":..,"Value":..} jumpstart tag, or a CoreTag object) and then deploy(tags=[...]) is passed, asserting both the pre-existing and the user tag reach Endpoint.create. That is the realistic build-then-deploy flow and would have caught this.

Minor (non-blocking)

  • _tags_as_key_value_list bare-mapping branch (line 40) doesn't str()-coerce keys/values the way format_tags does (common_utils.py:1696). Pydantic coercion at Endpoint.create likely covers it, so this is only a consistency nit.

Nice touches worth keeping: routing the _deploy_nova_model dedup through _tags_as_key_value_list (model_builder.py:6316, 6572-6575) correctly handles CoreTag objects that the old tag["Key"] indexing would have broken on, and giving the restricted-model-package path tags at all (6276) is a real fix.
· branch fix-modelbuilder-deploy-tags

Tags passed to ModelBuilder.deploy() were accepted but never reached the
endpoints created on the model-customization paths, so they could not be used
for tag-based resource association.

deploy() branches into _deploy_model_customization() for fine-tuned and Nova
models and returns before reaching _deploy_core_endpoint(), which is the only
place that applies the deploy-time tags kwarg. Those paths built their endpoint
tags from self._tags alone, so they saw only the tags ModelBuilder had added
itself. Resolve the endpoint tags once on entry from both sources and use them
for whichever path creates the endpoint.

The restricted-model-package path created its endpoint with no tags at all; it
now receives them like the other two.

Adds _tags_as_key_value_list() to normalize tags into the lowercase key/value
list the sagemaker.core.resources create() calls are typed for. self._tags can
hold either casing, Tag objects, or a bare {key: value} mapping, and the
previous inline reads indexed tag["Key"] directly after format_tags(), which
returns a list unchanged whatever its casing -- so a lowercase list raised
KeyError. That was already reachable through build(tags=...) and add_tags().

The merge deliberately does not go through add_tags(): tag_exists() compares
tags as tag["Key"] == curr_tag["Key"], so writing the lowercase form these
create() calls require back onto the accumulator raises KeyError as soon as
self._tags is non-empty, which is the normal state after build(). Resolving
into a local list instead also keeps deploy() from mutating builder state as a
side effect. SageMaker rejects duplicate tag keys, so on a collision the tag
ModelBuilder manages is kept and the caller's duplicate is dropped rather than
displacing a tag the SDK relies on.
@rohangujarathi
rohangujarathi force-pushed the fix-modelbuilder-deploy-tags branch from 23cab69 to 37d2b09 Compare August 13, 2026 00:46
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

Claude finished @rohangujarathi's task in 1m 27s —— View job


PR Review: propagate deploy tags on model-customization endpoint paths (updated revision)

  • Read PR diff
  • Verify the add_tags/KeyError fix from the prior review
  • Check _merge_tags and all endpoint-creation call sites
  • Review test coverage
  • Post review

This revision resolves the 🔴 blocker from the previous review. The deploy-time merge no longer routes through add_tags()/tag_exists(), so the KeyError on a pre-populated self._tags is gone.

What I verified

  • _deploy_model_customization now resolves endpoint_tags = _merge_tags(self._tags, kwargs.get("tags")) on entry (model_builder.py:6250) and threads it into all three endpoint-creation branches: the Nova direct path (tags=endpoint_tags, :6268), the restricted-model-package path (tags=endpoint_tags or None, :6296), and the single-IC path (:6339). The restricted path previously got no tags at all — now fixed.
  • _merge_tags normalizes via _tags_as_key_value_list and dedups by key keeping the first entry, with self._tags passed first — so an SDK-managed tag (e.g. the model-source reuse tag) can't be displaced by a colliding deploy-time tag. Confirmed against test_a_managed_tag_is_not_displaced_by_a_colliding_deploy_tag.
  • deploy() passes tags through **kwargs into _deploy_model_customization (:6076-6084), so kwargs.get("tags") is the right accessor.
  • _deploy_nova_model prepends its own jumpstart/recipe tags, then merges the caller's endpoint_tags, falling back to self._tags when called directly. CoreTag objects are handled via the getattr(tag, "key"/"value") branch, which the old tag["Key"] indexing would have crashed on.
  • Test coverage now includes the realistic build-then-deploy flow with a pre-populated, mixed-form self._tags (test_deploy_tags_merge_with_tags_already_on_the_builder) — the exact gap flagged last time. 👍

Minor (non-blocking)

  • _tags_as_key_value_list(tags: Optional[Tags]): the Tags type alias is Union[List[TagsDict], TagsDict] and doesn't cover the CoreTag/object entries the function's else branch handles (and that the trainer path puts on self._tags). The runtime behavior is correct; the annotation is just narrower than reality. Consider widening it or a comment, purely for accuracy.
  • The bare-mapping branch (:39-40) still shares format_tags's ambiguity: a single {"Key": .., "Value": ..} dict passed as a mapping (not wrapped in a list) expands into two junk tags. The codebase already avoids this by list-wrapping at the one relevant call site (:6058), so it's not newly introduced — just worth keeping in mind.

Looks good to me — the correctness fix is sound and well-tested. Nice work addressing the earlier feedback.
· branch fix-modelbuilder-deploy-tags

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant