Improve the reliability of control process startup in integration tests

[bieryAtFnal]

Description

These changes are coupled with DUNE-DAQ/drunc#1068.

While working on some minor fixes for integration tests that use the Kubernetes process manager (e.g. here), I noticed that the startup of the drunc-unified-shell takes a little longer the first time that we start it in a fresh local software area on np04-srv-019 with the K8s process manager compared to the SSH process manager. And, the integrationtest infrastructure was not handling that well. Commands would be sent to the drunc-unified-shell before it was ready to receive them, and the system would not run successfully.

To improve this situation, I added 'process startup wait conditions' to the startup of control process in the integrationtest infrastructure. That translated to enhancing the wait conditions that already existed in the integrationtest code and adding their use to the startup of the control processes.

The updated InformationAboutSpecialVariables.md document includes information about the wait conditions that are currently available and how they can be used. In addition, the updated integtest basic_multiapp_test.py in the drunc repo has examples of specifying control applications to be run, commands to be sent to the control applications, and wait conditions that are useful in both cases.

There are also a few additional changes to help improve reliability and provide feedback to users in this PR.

To test these changes, I have run the full suite of regression/integration tests (39 of them) with the SSH process manager. There were a few failures, but those were unrelated to these changes. I also ran the daqsystemtest integtests with the K8s PM. Those mostly succeeded. There were a few problems which were unrelated to these changes (e.g. extra triggers when running with TPG), and those problems need to be investigated (separately from this PR).

Here are sample instructions for testing the improved behavior of an integration test with the K8s PM on np04-srv-019 the first time that a test is run in a fresh software area. The interesting thing to look at in the console output from the first integtest that is run is the time that it takes the drunc-unified-shell to start up. It typically takes 10-12 seconds, and the new code handles that just fine.

echo ""
echo "These steps need to be run on np04-srv-019!"
echo ""
sleep 3

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260923_A9 ${DATE_PREFIX}FDDevAsyncPMFixes_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevAsyncPMFixes_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/process_startup_improvements
git clone https://github.com/DUNE-DAQ/drunc.git -b kbiery/process_startup_improvements
cd ..

dbt-workarea-env
dbt-build -j 12
dbt-workarea-env

export KUBECONFIG=/nfs/home/np04daq/.kube/config.019

kubectl delete ns readout 2>/dev/null
kubectl delete ns fakedata 2>/dev/null
sleep 2

dunedaq_integtest_bundle.sh -r local --verb 5 -k 'min|basic' --pytest-options "--process-manager-type k8s"

echo ""
echo -e "\U1F535 \U2705 Note that the requested tests passed with the new code. \U2705 \U1F535"
echo ""
echo ""

Type of change

• Optimization (non-breaking change that improves code/performance)
• Bug fix (non-breaking change which fixes an issue)

Testing checklist

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas

[PawelPlesniak]

Thank you Kurt, the documentation is clear and the example provided ran successfully on np04-srv-019

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Tue Sep 29 03:15:20 PM CEST 2026
Log file is: /tmp/pytest-of-pplesnia/dunedaq_integtest_bundle_20260929151258.log

⮕ Running daqsystemtest/minimal_system_quick_test.py ⬅
========================= 4 passed ✅ in 64.88s (0:01:04) =========================
⮕ Running drunc/basic_multiapp_test.py ⬅
========================= 3 passed ✅ in 73.12s (0:01:13) =========================