feat: upgrade sarama from Shopify v1.30.1 to IBM v1.48.0 - #274
Open
Danish-sarwar1 wants to merge 8 commits into
Open
feat: upgrade sarama from Shopify v1.30.1 to IBM v1.48.0#274Danish-sarwar1 wants to merge 8 commits into
Danish-sarwar1 wants to merge 8 commits into
Conversation
|
Important Review skippedToo many files! This PR contains 1022 files, which is 922 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. Usage-priced reviews support at most 300 files. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1022)
You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Replace all github.com/Shopify/sarama imports with github.com/IBM/sarama - Fix error comparisons to use errors.Is() pattern - Adds Kafka 4.x protocol support (fixes OffsetCommit v1 rejection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Danish-sarwar1
force-pushed
the
feat/upgrade-sarama-ibm
branch
from
May 7, 2026 18:10
7561b8e to
f453c44
Compare
go.mod requires go 1.25.0 after the IBM/sarama upgrade. Bump the build-container image and Travis host Go version to match, and reformat two doc comments to satisfy Go 1.19+ gofmt rules.
- LeaderElectionID '854ae6e3.' -> 'redshiftsink-operator-leader' for readability. Note: rolling deploys across this change will briefly hold two distinct leases. - realtime_calculator.go: drop unused SQL fragment accidentally appended after the package's last function.
When ConsumeClaim exits (e.g. during reload from offset 0), the defer blocks on wg.Wait() waiting for the processor goroutine. But the processor is blocked on session.Context().Done() which never fires because sarama is waiting for the claim handler to finish — a deadlock. Fix: close processChan in the defer before wg.Wait(), and handle the closed channel in the processor. This lets the processor exit cleanly, unblocking wg.Wait() and allowing the manager to retry ConsumeClaim.
# Conflicts: # .travis.yml
convertDebeziumDate, convertDebeziumMilliseconds and convertDebeziumMicroseconds printed the year with %d, so a MySQL value like 0202-04-17 was serialized to S3 as "202-04-17" (9 chars). Redshift COPY (json 'auto') rejects such values with err 1205 "Invalid Date Format - length must be 10 or more" and the loader retries the same batch forever, stalling the sink and burning cluster cycles. Pad the year with %04d so years below 1000 serialize as valid YYYY-MM-DD and load faithfully. Years 1000-9999 are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: zero-pad years below 1000 in date/datetime formatting
migrateTable returned early after GrantSchemaAccess without ever reaching tx.Commit(), leaving the table-replacement transaction (RENAME/CREATE/UNLOAD/COPY/DROP/GRANT) open forever. The abandoned transaction holds AccessExclusiveLock on the migrated table, blocking every query on it (including the loader's own merge on a second pooled connection) until the pod dies, after which the migration retries and hangs again. Bug present since #172; fires for any sink with loader.redshiftGroup set whenever a migration-style schema change occurs. Also rollback the tx if the grant itself fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrade Kafka client library for Kafka 4.x support. Fixes OffsetCommit v1 rejection (EOF errors) on Kafka 4.1.1. Import path: Shopify/sarama -> IBM/sarama. Go 1.25+.