fs: implement glob natively

[avivkeller]

Ref: https://openjs-foundation.slack.com/archives/C019Y2T6STH/p1787069174254509

(AI assistance used to help port minimatch to C++ and take improvements from other implementations, e.g. Rust's fast-glob [oxc] and picomatch, etc).

I'd love some pretty pedantic reviews since this is substantial. It's 1:1 with the existing glob, so it's non-breaking, but please tear this apart just in case. This implementation is over 2x faster than the old one.

Benchmarking Result

                                                                                                       confidence improvement accuracy (*)    (**)   (***)
fs/bench-glob-options.js options='exclude-callback' mode='callback' pattern='**/*.js' dir='lib' n=1000        ***    115.93 %      ±19.24% ±26.59% ±36.73%
fs/bench-glob-options.js options='exclude-callback' mode='promise' pattern='**/*.js' dir='lib' n=1000         ***    115.54 %      ±20.02% ±27.26% ±36.78%
fs/bench-glob-options.js options='exclude-callback' mode='sync' pattern='**/*.js' dir='lib' n=1000            ***     69.66 %      ±14.00% ±18.97% ±25.42%
fs/bench-glob-options.js options='exclude-pattern' mode='callback' pattern='**/*.js' dir='lib' n=1000         ***     37.86 %      ±12.27% ±16.90% ±23.19%
fs/bench-glob-options.js options='exclude-pattern' mode='promise' pattern='**/*.js' dir='lib' n=1000          ***     41.06 %      ±11.48% ±15.49% ±20.60%
fs/bench-glob-options.js options='exclude-pattern' mode='sync' pattern='**/*.js' dir='lib' n=1000                      3.81 %      ±11.93% ±16.14% ±21.56%
fs/bench-glob-options.js options='none' mode='callback' pattern='**/*.js' dir='lib' n=1000                    ***    153.86 %      ±20.91% ±28.39% ±38.12%
fs/bench-glob-options.js options='none' mode='promise' pattern='**/*.js' dir='lib' n=1000                     ***    150.49 %      ±18.20% ±24.64% ±32.95%
fs/bench-glob-options.js options='none' mode='sync' pattern='**/*.js' dir='lib' n=1000                        ***     98.16 %      ±11.23% ±15.15% ±20.14%
fs/bench-glob-options.js options='withFileTypes' mode='callback' pattern='**/*.js' dir='lib' n=1000           ***    111.98 %      ±19.57% ±26.99% ±37.14%
fs/bench-glob-options.js options='withFileTypes' mode='promise' pattern='**/*.js' dir='lib' n=1000            ***    111.56 %      ±17.41% ±23.58% ±31.53%
fs/bench-glob-options.js options='withFileTypes' mode='sync' pattern='**/*.js' dir='lib' n=1000               ***     76.30 %      ±15.82% ±21.46% ±28.80%
fs/bench-glob.js mode='callback' pattern='*.js' dir='lib' n=1000                                              ***     64.04 %       ±6.85%  ±9.31% ±12.51%
fs/bench-glob.js mode='callback' pattern='**/*.{js,json}' dir='lib' n=1000                                    ***    181.99 %       ±5.73%  ±7.78% ±10.46%
fs/bench-glob.js mode='callback' pattern='**/*.js' dir='lib' n=1000                                           ***    162.05 %       ±7.08%  ±9.77% ±13.47%
fs/bench-glob.js mode='callback' pattern='**/*' dir='lib' n=1000                                              ***    167.88 %       ±9.26% ±12.60% ±16.98%
fs/bench-glob.js mode='promise' pattern='*.js' dir='lib' n=1000                                               ***     46.39 %       ±2.77%  ±3.78%  ±5.12%
fs/bench-glob.js mode='promise' pattern='**/*.{js,json}' dir='lib' n=1000                                     ***    167.71 %       ±7.07%  ±9.74% ±13.39%
fs/bench-glob.js mode='promise' pattern='**/*.js' dir='lib' n=1000                                            ***    158.22 %      ±13.81% ±19.17% ±26.64%
fs/bench-glob.js mode='promise' pattern='**/*' dir='lib' n=1000                                               ***    156.03 %      ±11.17% ±15.48% ±21.47%
fs/bench-glob.js mode='sync' pattern='*.js' dir='lib' n=1000                                                  ***     29.58 %       ±4.99%  ±6.92%  ±9.59%
fs/bench-glob.js mode='sync' pattern='**/*.{js,json}' dir='lib' n=1000                                        ***    112.35 %       ±6.09%  ±8.24% ±11.01%
fs/bench-glob.js mode='sync' pattern='**/*.js' dir='lib' n=1000                                               ***     93.18 %       ±5.53%  ±7.60% ±10.38%
fs/bench-glob.js mode='sync' pattern='**/*' dir='lib' n=1000                                                  ***     85.16 %      ±13.12% ±18.20% ±25.31%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 24 comparisons, you can thus
expect the following amount of false-positive results:
  1.20 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.24 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

AI-Assisted-By: Claude Fable 5, Claude Opus 5

[nodejs-github-bot]

Review requested:

• @nodejs/actions
• @nodejs/performance
• @nodejs/security-wg
• @nodejs/tsc

[avivkeller]

cc @isaacs @nodejs/fs

[codecov]

Codecov Report

❌ Patch coverage is 76.21984% with 887 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.90%. Comparing base (55e4ca3) to head (fd43e85).
⚠️ Report is 38 commits behind head on main.

Files with missing lines	Patch %	Lines
src/glob/glob_parser.cc	73.89%	149 Missing and 106 partials ⚠️
src/glob/glob_matcher.cc	68.30%	71 Missing and 84 partials ⚠️
src/glob/glob_walker.cc	84.66%	41 Missing and 95 partials ⚠️
src/glob/node_glob.cc	67.40%	87 Missing and 45 partials ⚠️
src/glob/glob_program.cc	80.73%	78 Missing and 42 partials ⚠️
src/glob/glob_unicode.cc	40.17%	64 Missing and 3 partials ⚠️
src/glob/glob_ast.h	84.84%	2 Missing and 3 partials ⚠️
src/glob/glob_unicode.h	33.33%	4 Missing ⚠️
src/glob/node_glob.h	42.85%	4 Missing ⚠️
src/glob/glob_parser.h	82.35%	2 Missing and 1 partial ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65392      +/-   ##
==========================================
- Coverage   90.14%   89.90%   -0.24%     
==========================================
  Files         752      765      +13     
  Lines      251870   255172    +3302     
  Branches    47365    48094     +729     
==========================================
+ Hits       227037   229411    +2374     
- Misses      16177    16684     +507     
- Partials     8656     9077     +421     

Files with missing lines	Coverage Δ
lib/fs.js	98.39% <100.00%> (+0.02%)	⬆️
lib/internal/fs/watchers.js	88.06% <100.00%> (-0.22%)	⬇️
lib/internal/test_runner/runner.js	94.80% <100.00%> (-0.01%)	⬇️
src/glob/glob_program.h	100.00% <100.00%> (ø)
src/glob/glob_walker.h	100.00% <100.00%> (ø)
src/lru_cache-inl.h	90.47% <100.00%> (+1.58%)	⬆️
src/node_binding.cc	82.42% <ø> (ø)
src/node_external_reference.h	100.00% <ø> (ø)
src/node_snapshotable.cc	73.53% <ø> (ø)
src/node_util.cc	81.39% <ø> (ø)
... and 15 more

... and 76 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[aduh95]

take improvements from other implementations, e.g. Rust's fast-glob [oxc] and picomatch, etc

I would feel more comfortable adopting one of those instead of having to maintain our own, have you explored that route?

[avivkeller]

I would feel more comfortable adopting one of those instead of having to maintain our own, have you explored that route?

I have, but the perf wins here are dependent on this being in native, and not in JS:

1. (Minor) We are calling libuv directly, so we don't need to flip-flop between C++ and JS

2. (Major) We are writing code highly optimized for Node.js's usage. Existing JS implementations compile to a RegEx and rely on V8 to optimize it, however, we can skip that whole step by not relying on V8 at all, and manually compiling the glob to an AST.

   If a directory has 100k entries and 500 match, existing implementations are forced to create and discard ~99.5k strings to get there (since they are written in JS), in a native implementation, we don't need to create, allocate, or garbage collect anything inside of V8, we can handle it all natively.

I also think this can bring some improvements to things like the test runner. For instance, --test-coverage-include/exclude is globbed on the native side (like a supercharged version of #65371), we should be able to bring this performance improvement elsewhere. Maybe this can help in similar ways in other places to?

[avivkeller]

(Whoops!)

[isaacs]

Here's the results of node-glob's benchmark script with this branch, as of fd617c2: https://gist.github.com/isaacs/abf2714edd59cb6803d36ff6b3407d22

Notes:

1. Correctness does not seem to be impacted in any way. That's great! Agents often take shortcuts in tasks like this, so it's good to see that the tests kept it well behaved ;)
2. The fs.globSync performance is a bit faster than node-glob. That's not unexpected, since it was already comparable, but it does seem to have improved.
3. The fs.glob performance (async) is still considerably slower. The only explanation I can imagine for this is that it's still stacking waits serially rather than parallelizing, or the more aggressive caching with path-scurry is saving node-glob from a lot of repetitive computations. (I know the node impl does cache, but it might not be caching as much or as aggressively; path-scurry effectively caches every path and fs operation for the duration of the walk.)
4. Performance of fast-glob and globby are there for scale, but they can be safely ignored; these implementations sacrifice correctness for speed, and thus are not apples-to-apples comparable.

Given the deep cuts made here, I'd expect to see the native implementation get at least 5 to 10 times faster than node-glob. No longer being 6x slower is indeed a significant improvement, perhaps enough to make it worth landing this as-is and making further improvements later. But it's definitely still not what I'd call "done", by quite a bit.

[aduh95]

I have, but the perf wins here are dependent on this being in native, and not in JS:

It's not what I meant though, my question is whether you explored adopting one of the native alternatives your AI based itself on (you mentioned Rust, surely that is not JS)

[avivkeller]

It's not what I meant though, my question is whether you explored adopting one of the native alternatives your AI based itself on (you mentioned Rust, surely that is not JS)

https://github.com/oxc-project/fast-glob, the Rust implementation I mentioned, isn't compliant with minimatch (e.g. negated patterns don't work). A lot of the really fast implementations aren't compatible with our current set up (as they match different globs). As @isaacs said about a different fast-glob (a different library of the same name), "these implementations sacrifice correctness for speed, and thus are not apples-to-apples comparable." That can be said about several implementations, so working from scratch was the best way to both match the existing behavior and take the performance benefits from the existing implementations.

[mcollina]

lgtm

[mcollina]

Minimatch had its own tests that covered it extensively. Those tests have not been ported over to this codebase, which I think should be to guarantee backward-compatibility and no regressions.

[jasnell]

Hmm... we really ought to make sure that things work with non-UTF8 characters in the file path. Not necessarily critical for this version as long as it matches what the js version does, but we should check on it.

[jasnell]

Minimatch also had documentation for the glob syntax that does not appear to be preserved. The source itself could use code comments and a description of the glob syntax and rules.

[jasnell]

I plan on taking an additional detailed review pass on this over the next day or two. I know you have two sign offs already. I'd appreciate if you let it sit for a few days for review before landing.

[avivkeller]

Of course!

[jasnell]

The double nesting here is rather awkward. Might be worthwhile splitting these out into two separate functions.

[jasnell]

If I'm reading this function correctly, you've got a minor correctness issue here. Someone could theoretically craft a patterns ≥ 65KB containing every code unit U+0080..U+FFFF ... consuming all possible sentinels.

[avivkeller]

In the unlikely scenario that a pattern is both > 65412 characters (the sentinel search length) and < 65536 characters (the max pattern length), yes, this would collide.

Let me see if I can think of a way around this without lowering the max, if I can't, I can lower the max by 124 characters and call this a breaking change (although, the percent of clients being broken would be infinitesimal)

[avivkeller]

Okay, I think the best solution would be if the string contains every possible unicode value, while being less than the max length, we'd throw a kPatternTooLong. That makes this a breaking change

[jasnell]

ok, handful of additional comments. Main issue is that test coverage needs to be expanded. couple of bugs but overall solid.

[avivkeller]

Breaking change since a string that contains every unicode character will now throw an error (that being said, this should never occur in pretty much any case). But, it's still a change.