GH-1194: Preserve empty list offset buffers - #1229
Conversation
This comment has been minimized.
This comment has been minimized.
|
Could a maintainer please add the |
|
could you please take a look at it? thanks! |
| ArrowBuf oldOffsetBuffer = offsetBuffer; | ||
| offsetBuffer = allocateOffsetBuffer(requiredCapacity); | ||
| offsetBuffer.setBytes( | ||
| 0, oldOffsetBuffer, 0, Math.min(oldOffsetBuffer.capacity(), requiredCapacity)); |
There was a problem hiding this comment.
If requiredCapacity exceeds oldOffsetBuffer.capacity(), the remaining offset entries cannot be left uninitialized or zeroed — per the Arrow spec the offset buffer must be monotonically non-decreasing, so unfilled entries must be set to the last copied offset value to represent empty lists at those positions.
There was a problem hiding this comment.
Addressed in the latest push. The helper now copies existing complete offset entries and fills any newly allocated offset entries with the last copied offset value, for both ListVector and LargeListVector.
Validation: mvn -pl vector -am -P=-error-prone -Dmaven.gitcommitid.skip=true -Dsurefire.failIfNoSpecifiedTests=false -Dtest=TestSplitAndTransfer,TestListVector,TestLargeListVector test
b16b4f5 to
b93341f
Compare
3 participants
What's Changed
Preserve the required one-entry offset buffer for empty
ListVectorandLargeListVectorinstances when setting value count and when splitting/transferring zero values. This keeps empty list vectors aligned with the Arrow offset-buffer invariant and avoids exposing buffers whose writer index exceeds capacity.Adds regression coverage for allocated, unallocated, split-and-transfer, and nested empty list vectors.
Closes #1194.
Tests:
mvn -pl vector -am -P=-error-prone -Dmaven.gitcommitid.skip=true -Dsurefire.failIfNoSpecifiedTests=false -Dtest=TestSplitAndTransfer,TestListVector,TestLargeListVector testmvn -P=-error-prone -Dmaven.gitcommitid.skip=true test