[Spark] List _change_data separately to speed up VACUUM listing

[sezruby]

What

VACUUM now lists the table's _change_data directory as its own listing
branch instead of recursing into it inline. Gated by
spark.databricks.delta.vacuum.listing.changeDataDirSeparately.enabled
(default true).

Relates to delta-io#2201 (improving VACUUM listing parallelism).

Why

VACUUM discovers files by:

1. shallow-listing the table root,
2. distributing that first level across the cluster, then
3. recursing each first-level directory in a single task.

_change_data is one of those first-level directories, and it mirrors
the table's partitioning (or is a single large flat directory), so it can
hold a large fraction of the table's files. Recursing it inline leaves
one task listing that entire subtree while the rest of the cluster is
idle — typically the tail of VACUUM on tables with Change Data Feed
enabled. Simply deepening the initial listing does not help the common
case where _change_data is a flat directory: there are no
sub-directories to spread, so it is still a single listStatus.

What changed

• VacuumCommand.getFilesFromFilesystem excludes _change_data from the
  main listing (via the dir-name filter) and lists it through a separate
  recursiveListDirs call. Its sub-directories then get their own
  repartition + parallel recursion, and because the two branches are
  unioned Spark schedules them concurrently.
• The _change_data directory entry itself is emitted explicitly (the
  listing helper emits the contents of its roots, not the roots), so
  empty-directory cleanup behaves exactly as before.
• If the table has no _change_data directory, or the config is off, the
  original single-call listing path is used unchanged.
• recursiveListDirs is left as-is (no signature change).

The set of files considered by VACUUM is identical; only the listing
parallelism differs.

Tests

• listing _change_data as a separate branch yields the same files and directories — builds a table-shaped tree (data partitions + a
  _change_data dir that mirrors partitioning and nests deeper) and
  asserts the separate-branch decomposition returns exactly the same set
  (paths + isDir) as the inline listing.
• VACUUM removes the same untracked _change_data files whether listed separately or inline — end-to-end SQL VACUUM run with the config both
  on and off, confirming the same untracked files (including one nested
  deep under _change_data) are removed and the tracked file retained.

Both pass locally (spark/testOnly ... DeltaVacuumSuite).

🤖 Generated with Claude Code