[Spark] Add vacuum.listing.initialDepth to improve VACUUM listing parallelism

[sezruby]

What

Adds an opt-in spark.databricks.delta.vacuum.listing.initialDepth config
(default 1) that controls how many directory levels VACUUM lists
breadth-first before fanning out to the parallel recursive listing.

Addresses delta-io#2201.

Why

VACUUM discovers files by:

1. shallow-listing the table root,
2. re-distributing that first level across the cluster, then
3. recursing each first-level directory in a single task.

When the first level is skewed, step 3 dominates: one task lists an entire
subtree while the rest of the cluster is idle. Common causes are a
low-cardinality partition column (a handful of top-level partition dirs,
each huge) and a large _change_data directory. On big tables this makes
the listing phase the bottleneck of VACUUM.

What changed

• recursiveListDirs takes a new initialListingDepth parameter. For
  depth > 1 it descends that many levels breadth-first, re-distributing
  the frontier after each level, before handing the remaining directories
  to the existing parallel subtree recursion. Files and interior
  directories found along the way are emitted as they are discovered.
• depth = 1 (the default) reduces to exactly the previous code path, so
  behavior is unchanged unless the config is set.
• VacuumCommand reads the new config and passes it through.

The set of files considered by VACUUM is identical for any depth; only
the listing parallelism differs.

Tests

• recursiveListDirs returns the same files and directories for any initialListingDepth — builds a tree with files at multiple depths and
  nested empty directories, and asserts depths 2–5 produce the exact same
  set as depth 1 (depth-invariance).
• recursiveListDirs rejects a non-positive initialListingDepth.
• VACUUM with a larger vacuum.listing.initialDepth deletes the same untracked files — end-to-end SQL VACUUM with initialDepth=3, confirming
  an untracked file nested deeper than the configured depth is still
  discovered and removed while the tracked file is retained.

All three pass locally (spark/testOnly ... DeltaVacuumSuite).

🤖 Generated with Claude Code

[sezruby]

Superseded by #14. Switched approach from a whole-tree initialListingDepth (which adds a breadth-first barrier over the entire tree and doesn't help a flat _change_data directory) to listing _change_data as a separate, concurrently-scheduled listing branch — targets the actual straggler and works for both flat and partitioned CDF layouts.