Improve software-layer-script workflow - #147
Conversation
…-layer-scripts, instead of from the deployed stuff in software.eessi.io
bedroge
left a comment
There was a problem hiding this comment.
Just to summarize how this used to work and how it will work after merging this PR:
Previously, an updated hooks file was copied to the CVMFS repo (with the overlay) before the builds were started (by EESSI-install-software.sh), and this was only done if a PR diff file showed that the hooks file was modified by the PR.
In the new situation it will just use the version from the git clone of software-layer-scripts; the build script clones a specific commit of that repo. Additional CI will make sure that software-layer PR tarballs can only be deployed if that commit belongs to a merged PR.
One thing that maybe requires some attention is possible situations where a software layer PR modifies that software-layer-scripts commit, does the builds, and then undoes those changes regarding the used commit. A reviewer may not immediately spot that the builds were done with a different commit (the "Files changed" tab won't show it, you would have to look at individual commits or notice that there was a commit after the builds were done), but maybe the CI or bot itself could also catch that?
| source ${TOPDIR}/load_easybuild_module.sh ${eb_version} | ||
|
|
||
| # Set EASYBUILD_HOOKS to use the hooks from the software-layer-scripts, to make it easier to use updated hooks | ||
| export EASYBUILD_HOOKS=$TOPDIR/eb_hooks.py |
There was a problem hiding this comment.
We should make sure we can also pass the $TOPDIR/eb_hooks.py to install_cuda_and_libraries.sh as an optional argument. (since install_cuda_and_libraries.sh loads its own EESSI-extend, just setting an environment variable is not sufficient)
There was a problem hiding this comment.
Suggestion by Bob: instead, let's revise how EESSI-extend behaves: make it listen to an EESSI_EASYBUILD_HOOKS_OVERRIDE variable. That's also useful for sites in general if they want to extend our hooks.
This is also more robust, since any child process that would invoke EESSI-extend will have this environment variable set.
…hooks being used during builds to the eb_hooks.py from the software-layer-scripts clone
Changes needed to make builds use everything from the cloned software-layer-scripts, instead of from the deployed stuff in software.eessi.io.
This is a starting point for https://gitlab.com/eessi/support/-/issues/217.
Split off from 2422804 which was tested in e.g. EESSI/software-layer#1351 (comment)
We may not want to merge this until we have sufficient checks (CI, bot) in place to ensure that tarballs can only be deployed if they were build from a merge commit of
software-layer-scripts. That means we need:software-layer-scriptscommit was merged Clone specific software-layer-commit and implement CI to check merged status software-layer#1353bot/commit_sha(i.e. the same one as the CI check uses).