Skip to content

Fix TOML key and table header escaping - #1956

Merged
asolntsev merged 2 commits into
datafaker-net:mainfrom
jump-oss:fix/toml-key-escaping
Nov 8, 2026
Merged

asolntsev merged 2 commits into
datafaker-net:mainfrom
jump-oss:fix/toml-key-escaping

Conversation

@jump-oss

@jump-oss jump-oss commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #1954.

  • Quote and escape field names when they cannot be written as bare TOML keys, while preserving existing output for valid bare keys.
  • Apply key formatting to scalar, array, and collection assignments, including empty containers, as well as table and array-of-tables headers.
  • Treat dots within individual field names as literal characters, while preserving separators between genuinely nested schema components.
  • Reuse the existing string escaping logic.

Null-value handling and the separate test cleanup discussed in #1949 are not included.

Tests

Added parameterized regression tests for key names containing spaces, dots, quotes, backslashes, Unicode, and control characters, including nested tables and arrays of tables.

Verified locally with Java 17:

  • .\mvnw.cmd -B "-Dtest=TomlTest" test -PnoGpg — 335 tests passed.
  • .\mvnw.cmd -B test -PnoGpg — 21,811 tests passed.

@what-the-diff

what-the-diff Bot commented Nov 4, 2026

Copy link
Copy Markdown

PR Summary

  • TOML keys are written safely and correctly: Keys with spaces, punctuation, Unicode, or other special characters are now quoted and escaped as needed, including keys in nested tables. This helps ensure generated TOML remains valid.
  • Added tests for key formatting: Tests cover keys in different TOML structures and verify handling of special characters.

@codecov-commenter

codecov-commenter commented Nov 4, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.43%. Comparing base (ca770ef) to head (89e4936).
⚠️ Report is 4 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1956      +/-   ##
============================================
+ Coverage     92.41%   92.43%   +0.01%     
- Complexity     3579     3583       +4     
============================================
  Files           348      348              
  Lines          7081     7084       +3     
  Branches        694      695       +1     
============================================
+ Hits           6544     6548       +4     
+ Misses          371      370       -1     
  Partials        166      166              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@asolntsev asolntsev added this to the 3.0.0 milestone Nov 7, 2026

@asolntsev asolntsev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! The key quoting looks correct: the bare-key check matches exactly the TOML 1.0 bare-key set, and the tests cover it well. Two small suggestions inline.

While reviewing, I noticed some older bugs in TomlTransformer. This PR doesn't introduce them, so they can go in separate issues or a follow-up:

  1. Fields after a nested table end up inside it. Schema.of(field("a", () -> Schema.of(field("x", () -> 1))), field("b", () -> 2)) produces [a]\nx = 1\nb = 2, which parses as a.b = 2. Plain fields need to be written before tables.
  2. null values are written as key = null, which is not valid TOML.
  3. A null field name becomes the key null and clashes with a field literally named "null". The duplicate check holds raw names, so it keeps both, and the output has a duplicate key.
  4. Mixed arrays: array-of-tables detection only checks the first element. List.of(Schema.of(...), "x") throws ClassCastException, and List.of("x", Schema.of(...)) writes the schema's toString() as a string.
  5. NaN / Infinity are written in Java form. TOML requires nan / inf.
  6. Nested lists come out as quoted strings: List.of(List.of(1, 2)) produces [ "[1, 2]" ] instead of [ [ 1, 2 ] ].

private void writeValue(StringBuilder sb, @Nullable String key, String pathPrefix, @Nullable Object val) {
private String formatKey(@Nullable String key) {
String name = String.valueOf(key);
return name.matches("[A-Za-z0-9_-]+") ? name : value2String(name);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two suggestions:

  1. String.matches compiles the regex again on every call, so it runs for every field of every record. A precompiled constant avoids that:
private static final Pattern BARE_KEY = Pattern.compile("[A-Za-z0-9_-]+");
...
return BARE_KEY.matcher(name).matches() ? name : value2String(name);
  1. Quoting keys through value2String, the general value serializer, ties key output to value formatting. A later change there, for example null handling, would silently change keys too. Consider extracting a shared escapeString(String) helper that both formatKey and value2String call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review! I've addressed both suggestions in 89e4936:

  • Precompiled the bare-key regex as a static final Pattern.
  • Extracted quoteAndEscapeString(String), shared by formatKey and value2String, so key formatting no longer depends on the general value serializer.

All 335 TOML tests and the full suite of 21,811 tests pass locally.

I'll address the six pre-existing issues in separate follow-up PRs. Null-value handling is already tracked in #1955, where the intended behavior still needs to be agreed on.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For null handling, I plan to throw an IllegalArgumentException for null field names, null field values, and null elements in arrays or collections, including nested containers. This avoids silently dropping data or changing its meaning. Literal "null" strings would remain valid as keys and values.

I'm happy to adjust this policy if you prefer a different approach.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update, 89e4936 looks good.

About null handling: I agree with throwing for null field names and for null elements inside arrays/collections, because TOML can't represent either of them.

For null field values, though, I'd rather not throw. Datafaker schemas often have optional or nullable fields. If TOML throws on them, the same schema works with JSON/CSV but fails with TOML. TOML has no null, and the usual way to say "no value" is to leave the key out. So I suggest:

  • field("a", () -> null) → the a = ... line isn't written at all
  • a nested Schema or array-of-tables that is null → its [a] / [[a]] header isn't written either
  • null field name → IllegalArgumentException
  • null element in an array/collection (including nested ones) → IllegalArgumentException
  • the literal string "null" stays a normal key/value, as you said

Let's continue this in #1955 and keep this PR as it is.

@jump-oss

jump-oss commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

The JDK 21 / macOS job passed the Maven build, but failed while uploading test results with Failed to CreateArtifact: Unable to make request: ENOTFOUND.

Could you please rerun the failed job? I don't have the rerun option available. Thanks!

@asolntsev
asolntsev merged commit 4688ecf into datafaker-net:main Nov 8, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TomlTransformer does not quote or escape keys and table header names

3 participants