fix: upgrade native LightGBM to 4.7.0

[Rana Singh (ranadeepsingh)]

Related Issues/PRs

Related to #2697 and delivery tracker #2699.
The contributor's reproduction and proposed upgrade route
informed this change. It consumes the upstream correction in
lightgbm-org/LightGBM#6738 through the official 4.7.0 release.

This does not close the delivery issue or attribute every report in
#2310 to this defect. The separate streaming/OpenMP fix in
#2751 remains independent; it overlaps the LightGBM overview
and parameter file, so either PR must be rebased and revalidated if the other
lands first.

What changes are proposed in this pull request?

Upgrade SynapseML's native LightGBM dependency from 3.3.510 to 4.7.0 to
address the reproduced distributed categorical-training crash.

Draft, blocked on native publication. The proposed
com.microsoft.ml.lightgbm:lightgbmlib:4.7.0 coordinate is not yet on Maven
Central. Standard CI cannot resolve it until an authorized publisher completes
publication. This PR does not publish artifacts, add a CI cache fallback, or
claim that an existing Fabric runtime already contains the fix.

• Prepare a deterministic local Maven package from checksum-pinned official
  Linux, macOS and Windows 4.7.0 JARs. Preserve all six native binaries and
  record their source/asset provenance and license.
• Use the official Java 8 wrappers. The macOS release JAR contains Java 21
  classes, so verify matching Java/JNI declarations and use the Linux wrappers
  instead. Native macOS qualification remains required.
• Add public-estimator regression coverage for the categorical crash,
  dense/sparse bulk and streaming paths, repeat fitting, persistence,
  classifier probabilities, and old native models/reference datasets.
• Keep CUDA unsupported and update its wording without changing public
  SynapseML JVM signatures or serialized parameter shapes.

The packaging script hash normalizes LF/CRLF checkout differences. Its unit
tests run in the existing CIHelpers job, without changing pipeline YAML.

How is this patch tested?

• I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Current-target validation uses JDK 11, Scala 2.12.17 and Spark 3.5.0 through
.github/skills/synapseml-local-setup/scripts/synapseml-sbt.sh --repo "$PWD" --.
The candidate resolver is added only to the SBT session:

set ThisBuild / resolvers += "lightgbm470-pr-candidate-maven" at file(sys.env("LIGHTGBM_CANDIDATE_REPO")).toURI.toString
set lightgbm / Compile / bgCopyClasspath := false

The second setting avoids duplicating the classpath on the nearly full local
drive; it does not change the generated code or repository configuration.

Command	Result
sbt lightgbm/compile lightgbm/Test/compile	Passed on the current target.
sbt lightgbm/scalastyle lightgbm/Test/scalastyle	Passed on the current target.
sbt "lightgbm/testOnly com.microsoft.azure.synapse.ml.lightgbm.split1.LightGBMNativeCompatibilitySuite com.microsoft.azure.synapse.ml.lightgbm.split1.GroupIdManagerSuite com.microsoft.azure.synapse.ml.lightgbm.split1.LightGBMIpv6NetworkE2ESuite"	Passed: 18 tests on the current target.
sbt "lightgbm/testOnly com.microsoft.azure.synapse.ml.lightgbm.split1.VerifyLightGBMCommon -- -z device -z CUDA -z GPU"	Passed: 18 tests on the current target.
sbt "lightgbm/testOnly com.microsoft.azure.synapse.ml.lightgbm.split1.LightGBMValidationDataSuite"	Passed: 6 tests on the current target.
sbt lightgbm/codegen	Blocked on this rerun by host I/O errors after all 42 Scala tests passed. The earlier-base run passed, including syntax checks of 18 generated Python/stub files.
python -m pytest tools/ci/tests/test_prepare_native_package.py -q -p no:cacheprovider	Passed: 14 tests.
python3 -B -m pytest tools/ci/tests/ -q -p no:cacheprovider	Passed on the final tree: 282 tests and 63 subtests.
python3 -m black --check tools/lightgbm tools/ci/tests/test_prepare_native_package.py	Passed with pinned Black 22.3.0.

The Scala run started at 68b185daae against target 861c3a1e14. The final
commit, 4691a1e6e3, only relocates the Python packaging tests into CI and
updates their README commands. Scala sources and the native candidate are
unchanged. The full CI helper selection above ran after that final commit.

The exported classpath shows SBT reused the earlier local repository-final
JAR, SHA-256
ef2ae1cd2a2c26474a824c96ef5b6dd776dbe70dee3aed920065f2d4de35b9f3.
The final prepared JAR is
425352bdb4614d9c45b909b7edd54199e911fe7074acab8c8c9899a92b7a9efc.
Every Java class and all six native binaries match byte-for-byte. The package
provenance changed after normalizing the script hash. These results validate
the same runtime binaries, but do not establish fresh resolution of the final
JAR.

The combined SBT command exited 1 at codegen. Its launcher and shutdown hooks
reported java.io.IOException: Input/output error while reading JARs; the
host drive had about 6 MB free. This does not invalidate the completed test
results, but the current-target codegen gate remains open.

Remote CI on head 4691a1e6e3 confirms the publication blocker:
Compile & Style Check
failed in lightgbm / update with
unresolved dependency: com.microsoft.ml.lightgbm#lightgbmlib;4.7.0: not found.
Both Maven Central endpoints were tried. This is a dependency-delivery failure
caused by the proposed pin, not a flaky test. Azure build
239250018
also failed in lightgbm / update for the same missing dependency during
Prewarm sbt bootstrap cache. The bootstrap command exhausted seven attempts
to resolve the unpublished package. The build used refs/pull/2753/merge, merge SHA
2bce4d7c289b5d49842ce563f06150efdbe1a2b7, with reason=pullRequest.
The Test CI helpers job passed all 282 tests and 63 subtests, including the
new package checks. The Spark test jobs did not run; Azure published no test
runs for this build. Copilot review was requested, but no current-head automated
review arrived within the bounded wait or by the final audit. Review remains
incomplete.

The exact checked-in distributed dense regression crashes with the published
3.3.510 binaries at iteration zero in SerialTreeLearner::SplitInner+0xf7b.
It passes with the 4.7.0 binaries. The test checks two actual active native
workers and a first split selecting 30 of 101 categories at
maxCatThreshold=32, rather than relying only on configured task counts.

Additional bounded local evidence: identical package bytes from Windows and
Linux, old native/Spark model prediction parity, reuse of old serialized
references, and Windows x86-64 JNI training/prediction. Native hashes were
checked. These local processes do not replace multi-executor Fabric evidence.
The earlier Fabric A/B in the issue used a same-era reconstruction, not this
official 4.7.0 package.

Still required before merge/release:

• Authorized Maven publication, including sources/Javadoc, signatures and
  repository metadata, followed by empty-cache dependency resolution.
• Current-target codegen completion and current-head full Azure/GitHub CI
  against the published artifact.
• macOS and multi-executor Fabric qualification, active-port compatibility,
  continued-training coverage and representative before/after performance.
• Current-head review completion and independent human approval.

Does this PR change any dependencies?

• No. You can skip this section.
• Yes. Make sure the dependencies are resolved correctly, and list changes here.

com.microsoft.ml.lightgbm:lightgbmlib changes from 3.3.510 to 4.7.0.
Resolution is verified against the isolated prepared Maven repository only,
not Maven Central. The official upstream source commit is
8f7036f03627054d5a54a6f965b13f4b9ff2cb63.

Does this PR add a new feature? If so, have you added samples on website?

• No. You can skip this section.
• Yes. Make sure you have added samples following below steps.

1. Find the corresponding markdown file for your new feature in website/docs/documentation folder.
   Make sure you choose the correct class estimators/transformers and namespace.
2. Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
3. Make sure the DocTable points to correct API link.
4. Navigate to website folder, and run npm start to make sure the website renders correctly.
5. Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
6. Make sure the WebsiteSamplesTests job pass in the pipeline.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.