Skip to content

Fix the refs to targetRefs migration to not OOM - #1067

Merged
pmachapman merged 1 commit into
mainfrom
fix_migration
Oct 11, 2026
Merged

pmachapman merged 1 commit into
mainfrom
fix_migration

Conversation

@pmachapman

@pmachapman pmachapman commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1063.

This PR fixes an OOM error with the refs to targetRefs migration that occurred due to each pretranslation being loaded into memory via ToListAsync().

Execution times on a local copy of production were comparable to running the script below (~7 minutes), with QA around 5 minutes. This will likely be slower on MongoDB Atlas.

If you want to manually migrate the collection, run:

db.translation.pretranslations.aggregate([
  {
    $match: {
      refs: { $exists: true }
    }
  },
  {
    $set: {
      targetRefs: {
        $ifNull: ["$targetRefs", "$refs"]
      }
    }
  },
  {
    $unset: "refs"
  },
  {
    $merge: {
      into: "translation.pretranslations",
      on: "_id",
      whenMatched: "replace"
    }
  }
])

I reverted QA by modifying the schema_version, then running this script:

db.translation.pretranslations.aggregate([
  {
    $match: {
      sourceRefs: { $exists: false },
      targetRefs: { $exists: true }
    }
  },
  {
    $set: {
      refs: "$targetRefs"
    }
  },
  {
    $unset: "targetRefs"
  },
  {
    $merge: {
      into: "translation.pretranslations",
      on: "_id",
      whenMatched: "replace"
    }
  }
])

This change is Reviewable

@pmachapman
pmachapman requested a review from Enkidu93 October 8, 2026 02:43

@ddaspit ddaspit left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@ddaspit reviewed 1 file and all commit messages, and made 1 comment.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on Enkidu93).

@Enkidu93 Enkidu93 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Enkidu93 reviewed 1 file and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on pmachapman).

@pmachapman
pmachapman merged commit 4bd83a7 into main Oct 11, 2026
2 checks passed
@pmachapman
pmachapman deleted the fix_migration branch October 11, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schema version not updated for long running migration

3 participants