Skip to content

feat(write): route postpone batch writes to fixed buckets - #659

Open
XiaoHongbo-Hope wants to merge 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/postpone-fixed-bucket-write
Open

feat(write): route postpone batch writes to fixed buckets#659
XiaoHongbo-Hope wants to merge 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/postpone-fixed-bucket-write

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Align Rust batch writes for postpone-bucket primary-key tables with the fixed-bucket behavior proposed in apache/paimon#8985. This extracts the core writer change from #658; Go binding and release changes remain in the draft PR.

Changes

  • route batch writes to non-negative fixed buckets by default, while preserving legacy bucket -2 behavior when postpone.batch-write-fixed-bucket=false
  • infer bucket counts for new partitions and reuse persisted per-partition counts on later writes
  • stream writes for partitions whose bucket counts are already known, avoiding accumulation of every Arrow batch
  • persist total_buckets in commit messages and reject concurrent conflicting bucket-count decisions
  • reject the unsupported combination with deletion vectors, whose scans skip the level-0 files produced by batch writers
  • cover visibility, repeated writes, partitioning, conflict detection, option validation, and DataFusion reads

Verification

  • cargo fmt --all -- --check
  • cargo check --locked --offline -p paimon
  • cargo test --locked --offline -p paimon test_postpone_ --lib
  • cargo test --locked --offline -p paimon-datafusion test_postpone_batch_write_uses_visible_fixed_bucket --test pk_tables -- --exact
  • DLF regression REST catalog E2E: write, commit, and immediate read from real buckets

@JingsongLi

Copy link
Copy Markdown
Contributor

TableWrite::prepare_commit declares that the writer is reusable, but the fixed-bucket state is cleared after each prepare. If the writer is reused before the first message has been committed, the second batch will re-derive the number of buckets based on the old snapshot, and subsequent commits may conflict due to inconsistencies in total_buckets for the same partition. The reference implementation restricts this pattern to a one-shot operation.

@XiaoHongbo-Hope

XiaoHongbo-Hope commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @JingsongLi, fixed in a18eede.

@JingsongLi

Copy link
Copy Markdown
Contributor
  • write_arrow_batch appends the internal _VALUE_KIND column via enrich_rowkind_batch.
  • prepare_commit calls binary_row_batch_size(batch, table.schema().fields()) when planning the bucket, which requires the number of batch columns to be strictly equal to the number of fields in the user table.
  • I added a temporary regression test: bucket=-2, PK table, rowkind.field=op, the write succeeded but prepare_commit consistently failed: BinaryRow size planning expected 3 columns, got 4.
  • It is recommended to exclude the internal _VALUE_KIND during sizing and add a fixed-bucket + rowkind regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants