Skip to content

Improve software-layer-script workflow - #147

Draft
casparvl wants to merge 4 commits into
EESSI:mainfrom
casparvl:improve_software_layer_scripts_workflow
Draft

Improve software-layer-script workflow#147
casparvl wants to merge 4 commits into
EESSI:mainfrom
casparvl:improve_software_layer_scripts_workflow

Conversation

@casparvl

@casparvl casparvl commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

Changes needed to make builds use everything from the cloned software-layer-scripts, instead of from the deployed stuff in software.eessi.io.

This is a starting point for https://gitlab.com/eessi/support/-/issues/217.

Split off from 2422804 which was tested in e.g. EESSI/software-layer#1351 (comment)

We may not want to merge this until we have sufficient checks (CI, bot) in place to ensure that tarballs can only be deployed if they were build from a merge commit of software-layer-scripts. That means we need:

…-layer-scripts, instead of from the deployed stuff in software.eessi.io
casparvl pushed a commit to casparvl/software-layer-scripts that referenced this pull request Jan 13, 2026
@casparvl
casparvl requested a review from bedroge January 27, 2026 10:07

@bedroge bedroge left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to summarize how this used to work and how it will work after merging this PR:

Previously, an updated hooks file was copied to the CVMFS repo (with the overlay) before the builds were started (by EESSI-install-software.sh), and this was only done if a PR diff file showed that the hooks file was modified by the PR.

In the new situation it will just use the version from the git clone of software-layer-scripts; the build script clones a specific commit of that repo. Additional CI will make sure that software-layer PR tarballs can only be deployed if that commit belongs to a merged PR.

One thing that maybe requires some attention is possible situations where a software layer PR modifies that software-layer-scripts commit, does the builds, and then undoes those changes regarding the used commit. A reviewer may not immediately spot that the builds were done with a different commit (the "Files changed" tab won't show it, you would have to look at individual commits or notice that there was a commit after the builds were done), but maybe the CI or bot itself could also catch that?

Comment thread EESSI-install-software.sh
source ${TOPDIR}/load_easybuild_module.sh ${eb_version}

# Set EASYBUILD_HOOKS to use the hooks from the software-layer-scripts, to make it easier to use updated hooks
export EASYBUILD_HOOKS=$TOPDIR/eb_hooks.py

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should make sure we can also pass the $TOPDIR/eb_hooks.py to install_cuda_and_libraries.sh as an optional argument. (since install_cuda_and_libraries.sh loads its own EESSI-extend, just setting an environment variable is not sufficient)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion by Bob: instead, let's revise how EESSI-extend behaves: make it listen to an EESSI_EASYBUILD_HOOKS_OVERRIDE variable. That's also useful for sites in general if they want to extend our hooks.

This is also more robust, since any child process that would invoke EESSI-extend will have this environment variable set.

Caspar van Leeuwen added 2 commits August 10, 2026 15:42
…hooks being used during builds to the eb_hooks.py from the software-layer-scripts clone
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants