Skip to content

Add Deserializer::overlapped_lists() to disable event buffering at runtime - #1020

Open
chiliec wants to merge 2 commits into
tafia:masterfrom
chiliec:de-overlapped-lists-toggle
Open

chiliec wants to merge 2 commits into
tafia:masterfrom
chiliec:de-overlapped-lists-toggle

Conversation

@chiliec

@chiliec chiliec commented Sep 20, 2026

Copy link
Copy Markdown

Closes #885

What

The overlapped-lists feature is unified across the dependency tree, so a crate that never enabled it can still end up buffering and rescanning events on every sequence field. As reported in #885 this turns a sub-second parse of a tens-of-MB document into hours.

This adds the option suggested in the issue thread: a per-deserializer switch Deserializer::overlapped_lists(bool) (only present with the feature, default true). When set to false, MapValueSeqAccess::next_element_seed returns None at the first element that does not belong to the sequence instead of calling skip(), i.e. it behaves exactly like a build without the feature. Nothing is buffered, so event_buffer_size becomes irrelevant for that deserializer.

Changes

  • src/de/mod.rs: new skip_unknown field, public overlapped_lists() setter with a doctest, crate-private can_skip(); unit test de::tests::skip::disabled asserting the duplicate-field error and an empty skip buffer.
  • src/de/map.rs: the two skip() call sites check can_skip().
  • Cargo.toml: feature docs mention the new switch.
  • Changelog.md: entry under Unreleased / New Features.

Verification

cargo test --all-features            # all suites pass, 0 failures
cargo fmt --check                    # clean
RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps   # clean
cargo clippy --all-features --lib    # 42 warnings, identical count before and after this change

RED→GREEN: with the map.rs change reverted, de::tests::skip::disabled fails with Expected `Err(Custom("duplicate field `item`"))`, but got `Ok(List { item: [(), (), ()] })` ; with it, it passes.

…ntime

The `overlapped-lists` feature is unified across the dependency tree, so a
crate that never asked for it can still be forced to buffer and rescan
events, which can turn a sub-second parse of a large document into hours.

Add a per-deserializer switch that makes the sequence access stop at the
first unrelated element instead of skipping and buffering it, restoring the
behaviour of a build without the feature.

closes tafia#885

@Mingun Mingun left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will be dangerous to change the mode during parse. When mode is disabled, you still need to reach checkpoint and drain already buffered events, otherwise you will get wrong results.

Comment thread src/de/mod.rs Outdated
Comment on lines +2787 to +2791
#[cfg(feature = "overlapped-lists")]
pub fn overlapped_lists(&mut self, enable: bool) -> &mut Self {
self.skip_unknown = enable;
self
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wouldn't work for your use-case if would be gated. If you does not enable overlapped-lists feature, you will not able to call this method.

Suggested change
#[cfg(feature = "overlapped-lists")]
pub fn overlapped_lists(&mut self, enable: bool) -> &mut Self {
self.skip_unknown = enable;
self
}
pub fn overlapped_lists(&mut self, enable: bool) -> &mut Self {
#[cfg(feature = "overlapped-lists")]
self.skip_unknown = enable;
self
}

Also, what if you change it in-flight? It seems that it should be configurable only at creation time.

Comment thread src/de/map.rs Outdated
Comment on lines +957 to +961
if self.map.de.can_skip() {
self.map.de.skip()?;
continue;
}
Ok(None)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would beter to have

Suggested change
if self.map.de.can_skip() {
self.map.de.skip()?;
continue;
}
Ok(None)
#[cfg(feature = "overlapped-lists")]
if self.map.de.skip()? {
continue;
}
Ok(None)

Comment thread src/de/mod.rs Outdated
Comment on lines +2793 to +2798
#[cfg(feature = "overlapped-lists")]
#[inline]
pub(crate) const fn can_skip(&self) -> bool {
self.skip_unknown
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline this check into skip()

Suggested change
#[cfg(feature = "overlapped-lists")]
#[inline]
pub(crate) const fn can_skip(&self) -> bool {
self.skip_unknown
}

@Mingun Mingun added enhancement serde Issues related to mapping from Rust types to XML labels Sep 21, 2026
…nd inline the check into skip()

The method is now always present so callers can use it unconditionally;
without the overlapped-lists feature it is a no-op. skip() itself returns
whether events were buffered and the seq access stops iteration when it
reports false, which removes the separate can_skip() accessor. Docs say
to set the mode before deserialization starts.
@chiliec

chiliec commented Sep 22, 2026

Copy link
Copy Markdown
Author

Thanks for the review, pushed e56045e with all three points:

  • overlapped_lists() is no longer gated: it always exists and is a no-op without the feature, so a downstream crate can call it unconditionally. The doctest now runs under both feature configurations (the "supported by default" half is behind a hidden cfg).
  • The check is inlined into skip(), which now returns Result<bool> (Ok(false) = disabled, nothing consumed) and MapValueSeqAccess does if self.map.de.skip()? { continue; } Ok(None). can_skip() is gone; the existing skip() unit tests assert the true return.
  • On changing the mode in flight: I agree it should be set up front, and I've documented that in the method docs ("call before deserialization starts"). I kept it as a &mut self setter rather than a constructor argument to match event_buffer_size() next to it, which has the same property. If you'd rather it be creation-time only, I can move it into the constructors instead — just say which shape you prefer.

cargo test --all-features and --features serialize pass, cargo fmt --check clean, no new clippy warnings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement serde Issues related to mapping from Rust types to XML

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problem with overlapped-lists that is enabled by the dependency

2 participants