Fix CUDA sanity check for EB >=5.4.0

[casparvl]

Since EB 5.4.0, the build dependencies are no longer available during the sanity check phase. This breaks our CUDA sanity check, which relies on cuobjdump being available.

This PR tries to promote CUDA to a regular dep temporarily, hoping that this will make the CUDA sanity check pass. A test on CUDA-Samples 12.9 shows that this is NOT the case. My AI friend thinks it's because the fake module that is generated is being generated based on data structures that are already available, so modifying the existing self.cfg instance has no effect.

Anyway, this is at least a description of the issue,and a starting point for the solution...

[casparvl]

It loads the module, but I guess a subshell with an isolated environment may be started? I'm not sure. It still doesn't find cuobjdump, that's for sure:

== Loading CUDA module 'CUDA/12.9.1', temporarily making
available for the (CUDA) sanity check
  >> file 'bin/deviceQuery' found: OK
  >> file 'bin/matrixMul' found: OK
  >> file 'bin/cudaOpenMP' found: OK
  >> loading modules:
CUDA-Samples/12.9-GCC-14.3.0-CUDA-12.9.1...
== ... (took 26 secs)
== FAILED: Installation ended unsuccessfully: Failed to
execute CUDA sanity check: cuobjdump not found
CUDA module must be loaded for sanity check (or cuobjdump
available in PATH) (took 4 mins 50 secs)

[ocaisa]

I knew this would bite us eventually easybuilders/easybuild-framework#4692 (comment)

[ocaisa]

Since it just needs to be in the PATH and you know the location, you could modify the default environment to include that location

[casparvl]

Ok, according to deepwiki indeed the sanity check does a module purge. load_fake_module(purge=True, ...).

Some hint on how to resolve this may be gotten from https://deepwiki.com/search/in-the-sanity-check-step-a-fak_317cc243-13e3-4627-a8ed-fbec38bfa715?mode=fast

[casparvl]

Explicitely adding the location of the bin dir from the CUDA module's prefix to PATH might work... it won't be affected by a module purge. Since we RPATH anyway, it should just work - as long as we can get to that binary. It's not the most robust solution maybe, but the fact that sanity_check itself so heavily modifies the environment means that changes in pre_sanitycheck_hook have a high change of not surviving until cuobjdump is called... (I'm not even certain for the PATH change, but I think that will survive...).

Maybe the most robust way is to do the module load I do now, then do a which cuobjdump, get the prefix for that in some temporary env var (A), then unload the module again, then add A to the PATH.

[casparvl]

Ok, testing this now, but not sure I can report the result - someone else may have to pick this up...

[ocaisa]

Probably want to store the original value in the attribute and restore it when removing that attribute

[casparvl]

Yeah, that was a 'TODO' (see the post-hook). But... it doesn't work. I don't have time to look into it any further right now... If you're up for it, you can give it a go. Otherwise, I expect Bob will look into it somewhere next week - he hit it when trying to compile ESPResSo with CUDA support.

[laraPPr]

In the case of LAMMPS we also need a CUDA during th running of the tests. So this hook would not fix the failing sanity check fo LAMMPS.

[laraPPr]

Should there than be another hook for LAMMPS which loads CUDA in pre_sanity_check_hook and unloads in post_sanity_check_hook?