feat(datafusion): allow configuring SQLContext runtime resources

[shyjsarah]

Purpose

Linked issue: close #673

SQLContext currently always creates a DataFusion session with the default RuntimeEnv. Applications embedding Paimon Rust cannot provide a bounded execution memory pool, choose a spill policy, select a temporary directory, or cap temporary disk usage without replacing Paimon's SQL context and its custom planners and functions.

This change makes DataFusion runtime resources configurable while preserving the existing zero-configuration behavior.

Brief change log

• Add a public, backward-compatible SQLContextBuilder that accepts an Arc<RuntimeEnv>.
• Keep SQLContext::new() unchanged and delegate its construction through the builder.
• Add optional keyword-only Python SQLContext arguments for:
  • memory_pool_type (fair or greedy)
  • memory_pool_bytes
  • temp_directory
  • max_temp_directory_size_bytes
• Update Python type stubs and user documentation.
• Add Rust coverage for custom RuntimeEnv injection.
• Add a Python regression test that executes an external sort and verifies that DataFusion reports spilled rows and bytes.

Tests

• cargo fmt --all -- --check
• PYO3_PYTHON=bindings/python/.venv/bin/python cargo clippy -p paimon-datafusion -p pypaimon_rust --all-targets --features fulltext -- -D warnings
• cargo test -p paimon-datafusion test_sql_context_builder_uses_custom_runtime_env --lib
• make build
• uv run --no-sync pytest tests/test_datafusion.py -k 'runtime_resource_configuration' -q

API and Format

This adds public Rust and Python APIs. Existing constructors remain backward compatible. There is no storage-format change.

Documentation

The Python binding documentation now includes a bounded-memory and local-spill example and clarifies that DataFusion's memory pool does not account for every host or external-library allocation.

AI assistance

AI tooling was used to help implement and test this change. I reviewed the builder, Python binding, DataFusion memory-pool selection, temporary-directory behavior, and compatibility path end-to-end. The primary known limitation is the one documented above: DataFusion memory pools only account for allocations registered with the pool, and only spill-capable operators can move intermediate state to disk.