Added support for multiple TRACE debug levels for a given TRACE_NAME in integtests...

[bieryAtFnal]

... and, added error checking for the value specified in the trace_debug_levels integtest config param.

Description

When I recently tried to set two TRACE levels (e.g. level 10 and level 13) for a given TRACE_NAME (e.g. DefaultRequestHandlerModel.hxx) in an integtest file, I realized that the initial support for setting TRACE debug levels in our integtest files did not handle more than one level. The changes in this PR remedy that situation.

I also added error checking for the contents of the trace_debug_levels integtest config parameter. With these changes, users should get notified when various types of invalid values have been specified.

It will be useful to test these changes in conjunction with DUNE-DAQ/dfmodules#516, and the following suggested instructions make use of those changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_261005_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop
git clone https://github.com/DUNE-DAQ/dfmodules.git -b kbiery/useful_debug_traces_in_integtests

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/trace_debug_multiple_levels
cd ..

sed -i 's/{"fast": {"DefaultRequestHandlerModel.hxx": \[10,13\]}}/{"fast": {"DefaultRequestHandlerModel.hxx": \[10,13\]}, "slow": "blahblah", "fast_and_slow": {"pdq": \["jkl", 19\]}, "slow_and_fast": {"xyz": {"hello", "world"}}, "none": {"abc": 17}}/' sourcecode/dfmodules/integtest/large_trigger_record_test.py

echo ""
echo -e "\U1F535 \U2705 Note that we have locally-modified the large_trigger_record_test to insert some invalid TRACE settings. \U2705 \U1F535"
echo ""
echo ""
sleep 3

. ./env.sh
dbt-build -j 12
dbt-workarea-env

export TRACE_FILE=$DBT_AREA_ROOT/log/${USER}_dunedaq.trace

dunedaq_integtest_bundle.sh -r dfmodules -k large --verb 2 --pytest-options "-k 125P"

echo ""
echo -e "\U1F535 \U2705 Note that several warnings were reported, as expected. \U2705 \U1F535"
echo ""
echo ""
sleep 3

dunedaq_integtest_bundle.sh -r dfmodules -k insufficient --verb 2

tshow | tdelta -ct 1 | grep DFOModule | grep -v 'stop delayed' | head -15

echo ""
echo -e "\U1F535 \U2705 Note that DFOModule TRACE messages for levels 21-24 were written to the TRACE memory buffer, \U2705 \U1F535"
echo -e "\U1F535 \U2705 as requested by the 'trace_debug_levels' integtest config param in that test. \U2705 \U1F535"
echo ""
echo ""

Type of change

• Bug fix (non-breaking change which fixes an issue)

Testing checklist

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas