Updates to the way that the bundle script (and friends) find integtests

[bieryAtFnal]

Description

Sometime in the medium past, Pawel noticed that the script that we use to run integtests (the "bundle" script) wasn't handling the case in which a Python package had been installed with pip install -e. The "-e" qualifier confuses things because the Python package does not appear underneath the virtual environment directory (.venv) in the same way as when we use the pip install command without the "-e" qualifier.

As part of fixing that, I looked into ways to more cleanly locate integtests, and I made some changes to the bundle script itself, and its two helper scripts, list_repos_with_integtests.sh and list_available_integtests.sh.

I'll attach some slides to this PR with additional information.

There are lots of ways to set up a DUNE-DAQ software environment, so testing these change fully could become time consuming. I will post sample instructions here, and there are testing suggestions in the attached slides. The general pattern for the testing is to set up different types of software environments and confirm that the expected set of integtests is found in each of them.

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260922_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/bundle_script_repo_location

cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -r all -k 'min|basic|no_dup' --verb 2

Type of change

• Optimization (non-breaking change that improves code/performance)
• Bug fix (non-breaking change which fixes an issue)

Testing checklist

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas

[bieryAtFnal]

FindingIntegtests_22Sep2026_KAB.pdf

[bieryAtFnal]

Here is a newer version of the slides. The main change is that I fixed some typos in the suggested software-area-setup commands on Slide 8.

FindingIntegtests_28Sep2026_KAB.pdf

[PawelPlesniak]

Following the slides, I ran the following tests. The intended behaviour was achieved - when a python-only repository was installed with -e, the tests were not found and not listed under list_available_integtests.sh. If attenmpted to be used, the expected error message was shown:

*** No integtests were found in the "drunc" repo(s) [with "" repos excluded].

I chose to run only tests with local .venvs, as these are the ones modified on pip install. I also excluded fddaq-v5.6.0 as this one did not have a formal integtest structure for drunc. I will not merge this PR, as there may be some follow up discussion here.

1 - test with a candidate release.

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -b candidate fddaq-v5.7.0-rc3-a9
cd fddaq-v5.7.0-rc3-a9
source env.sh
dbt-build
dbt-workarea-env
dunedaq_integtest_bundle.sh -r all -k 'min|basic|no_dup'

list_repos_with_integtests.sh
list_available_integtests.sh

cd sourcecode
git clone git@github.com:DUNE-DAQ/daqsystemtest.git --branch v4.9.1
cd ..
dbt-build
dbt-workarea-env
dunedaq_integtest_bundle.sh -k 'min'

cd pythoncode
git clone git@github.com:DUNE-DAQ/drunc.git --branch v1.2.1
cd drunc

pip install .
dunedaq_integtest_bundle.sh -r drunc -k 'basic'
list_repos_with_integtests.sh
list_available_integtests.sh

pip install -e .
dunedaq_integtest_bundle.sh -r drunc -k 'basic'
list_repos_with_integtests.sh
list_available_integtests.sh

2 - test with a nightly

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260922_A9
cd NFD_DEV_260922_A9
source env.sh
dbt-build
dbt-workarea-env
dunedaq_integtest_bundle.sh -r all -k 'min|basic|no_dup'

list_repos_with_integtests.sh
list_available_integtests.sh

cd sourcecode
git clone git@github.com:DUNE-DAQ/daqsystemtest.git
cd ..
dbt-build
dbt-workarea-env
dunedaq_integtest_bundle.sh -k 'min'

cd pythoncode
git clone git@github.com:DUNE-DAQ/drunc.git
cd drunc

pip install .
dunedaq_integtest_bundle.sh -r drunc -k 'basic'
list_repos_with_integtests.sh
list_available_integtests.sh

pip install -e .
dunedaq_integtest_bundle.sh -r drunc -k 'basic'
list_repos_with_integtests.sh
list_available_integtests.sh

[jcfreeman2]

Approved. Rather than repeat what Pawel did, I instead took advantage of the root access available in ghcr.io/dune-daq/nightly-release-alma9:development_v5. I copied over the three files in daqsystemtest which differ on this feature branch relative to develop into /cvmfs (thus creating a de-facto Spack installation of daqsystemtest on this feature branch). Then I created an internal tester account which I logged into and rather than creating a local work area, I just did dbt-setup-release for the nightly in the container, and confirmed a few things:

• That list_repos_with_integtests.sh and list_available_integtests.sh worked
• That dunedaq_integtest_bundle.sh -r all -k 'min|basic|no_dup' --verb 2 worked (in the sense of, attempted to run the tests - they failed b/c they won't work out of the box in the container)