SIMPLE-SAMPLE NETWORKS from LONGITUDINAL CLINICAL DATA

The proposed pipeline turns per-patient clinical time series (ICU vitals/labs, chemotherapy lab panels, single-cell/omics profiles, ...) into patient-specific networks, where nodes are variables (lab tests, genes, ...) and edges are subject-specific associations between them. Graph, node and edge features derived from these networks are then used to train classifiers (e.g. mortality, adverse event risk, tumour subtype).

The pipeline has two stages, both driven by a single CLI entry point:

1. Build — construct one patient network per admission/sample and export it as an edge-weight vector.
2. Train — turn those networks into tabular features (node/edge/graph metrics) and train/evaluate a classifier.

python -m tpnets.pipeline.run_pipeline --mode {build|train} ...

---

Contents

• Quick start
• Configuration
• Pipeline architecture
  • 1. Dataset loading & preprocessing
  • 2. Time strategies
  • 3. Edge construction
  • 4. Association measures
  • 5. Output artifacts & directory layout
  • 6. Feature engineering
  • 7. Model training
• Usage
• Extending the pipeline

---

Quick start

1. Create the conda environment: conda env create -f environment.yml

2. Create a .env file (loaded via python-dotenv) defining the dataset roots used by tpnets/config/constants.py:

   MIMIC_CHEMO_PATH=/path/to/mimic_chemo
   UKER_CHEMO_PATH=/path/to/uker_chemo
   ICU_PATH=/path/to/icu_datasets
   OMIC_PATH=/path/to/omic_datasets
   

3. Run the pipeline (see Usage below).

---

Configuration

File	Purpose
tpnets/config/constants.py	Resolves dataset roots from env vars and defines DATASET_CONFIG, a per-cohort dict of {base, data_file, cohort_file, CV_folds_path, top_features} paths. Adding a cohort means adding an entry here.
tpnets/config/*_grid.json	Grids of build parameters (chemo_grid.json, omic_grid.json, experiment_grid.json, chemo_test_grid.json) consumed via --build_all <name>. Each key maps to a list of values; the pipeline runs the cartesian product.
tpnets/config/train_grid.json	Grid of training parameters, consumed via --train_all train_grid.
tpnets/config/ml_params.yaml	Per-model default hyperparameters and hyperparameter-search grids (Random Forest, CatBoost, XGBoost, Logistic Regression, Gradient Boosting).
tpnets/config/ml_params_best.yaml	Optional dataset-specific best-known hyperparameters, keyed by model name → cohort. Overrides the grid's default when present.

Grid combinations with a significant key are de-duplicated by run_utils.trim_combos: when significant=False, threshold is forced to "NA", and when both significant=False and zscores=False, zmode is forced to "NA" too — this collapses combinations that would otherwise be identical runs.

---

Pipeline architecture

flowchart TD
    A[PNDataset: load cohort + time series csv.gz] --> B[PNDatasetFromSSN: apply time strategy]
    B --> C["ExperimentRunner (one per fold/config)"]
    C --> D["EdgeMethod subclass (SSN / LIONESS / SWEET / ...)"]
    D --> E["Aggregator (measure x na_aware x incremental)"]
    E --> F[per-sample edge vectors: train / val / test]
    F --> G["stored in memory and/or HDF5 / NPZ"]
    G --> H["GraphFilter (optional z-score + significance masking)"]
    H --> I["FeatureComputer (node / edge / graph features)"]
    I --> J["FeatureEngineer (merge with original data)"]
    J --> K["ModelFactory + MLTrainer (fit / tune / evaluate)"]
    K --> L["results.csv, feature importances, splits"]

Loading

1. Dataset loading & preprocessing

tpnets/datasets/ts_dataset.py::PNDataset is the base loader:

• Reads cohort_file (labels + demographics) and data_file (long-format time series: subject_id, hadm_id, bin, itemid, value), both .csv.gz, using the paths resolved by DATASET_CONFIG[args.cohort].
• Loads the CV split (train_ids, val_ids, test_ids) from CV_folds_path/fold_{fold}.pkl, and enforces no id overlap between splits. When --subset N is passed, splits are subsampled (train=N, val=N/5, test=N/2) for quick smoke runs.
• Optionally restricts to a preselected feature list (top_features, used by the MIMIC/UKER chemo cohorts).
• Drops variables not observed in the training split and constant-in-train variables (both computed on train only, to avoid test leakage).
• Builds normalized static/demographic features (age, gender) with missing indicators, using train-only mean/std.
• Computes the binary/multiclass target y from the cohort's label column.
• Detects dataset_type (static vs temporal) from the max number of observations per (hadm_id, itemid).

tpnets/datasets/pn_dataset.py::PNDatasetFromSSN extends it to prepare the data for edge construction:

• Reconciles --time_strategy with the detected dataset_type (forces TS0 for static data, TS1 for temporal data requested as TS0).
• --na_aware controls whether remaining cross-patient gaps are imputed (na_aware=False → KNN imputer, k=5, fit on train only) or left as NaN for the edge builder to mask (na_aware=True).
• Produces one or more (agg_df, mask_df) pairs — one per graph to build — according to the time strategy (below).

2. Time strategies

Strategy	Applies to	Behaviour
TS0	static data	Pivot hadm_id x itemid; one graph.
TS1	temporal	One graph per summary statistic in --stats (default mean median max min), computed per (hadm_id, itemid).
TS2	temporal	One graph per time bin, after forward/backward-fill imputation within each (hadm_id, itemid) series.
TS3	temporal	Time bins stacked into one long table (all (hadm_id, bin) treated as rows); a single aggregated graph.

3. Edge construction

tpnets/edges/edge_builder.py::EdgeMethod is the (formula-agnostic) engine: it loads the per-graph (agg_df, mask_df), builds the training aggregate A^(N) via the resolved measure/estimator, then for every sample computes a per-sample edge vector — leave-one-out (A^(N\q)) for training samples, add-one (A^(N+q)) for validation/test samples — and applies the sample's own missingness mask (edges touching an unobserved feature are set to NaN).

Every concrete method (tpnets/edges/edge_methods.py) is just a choice of three roles plugged into one shared formula:

e_ij^(q) = gamma_q * (A^(N)_ij - A^(N\q)_ij) + lambda * A^(N\q)_ij
gamma_q  = scale(N) * sample_weight(N, idx, single_sample)

Method	scale(N)	background (λ)	sample_weight
SSN	1	0	1 (pure differential network)
LIONESS	N	1	1 (linear reconstruction of the full aggregate)
SWEET	0.1·N	1	normalized mean Pearson-similarity of the sample to the training cohort
Example	0.1·N	1	normalized mean Jaccard-similarity of the sample's missingness mask to the training cohort
ExampleIndependent	0.1·N	1	fraction of features observed in the sample itself (no cohort-wide precomputation)

Methods whose weight depends on the whole training cohort (SWEET, Example) precompute it once in calculate_training_weights() and evaluate held-out samples incrementally (never rebuilding the full similarity matrix).

Selected via --edge_method {SSN|LIONESS|SWEET|Example|ExampleIndependent}.

4. Association measures

The aggregate matrix A is produced by a measure resolved from a triple: measure name (--measure, e.g. PCC), na_aware (--na_aware, keep/mask NaNs vs. reject them) and incremental (--incremental, whether leave-one-out/add-one updates are computed in O(p²) via cached moments rather than a full O(n·p²) recompute).

tpnets/edges/measures/core.py::resolve() looks for measures/<name>/<name>[_NA_aware][_incremental].py and degrades gracefully when the exact cell is missing:

• missing incremental variant → falls back to full recompute at the same NA level (slower, same numbers), logged as a note;
• missing NA-aware variant, but the measure is separable (entry (i,j) depends only on columns i,j) → falls back to a generic pairwise-deletion computation, logged as a note;
• missing NA-aware variant and not separable → raises.

Incremental measures use Welford's online algorithm (tpnets/edges/measures/welford.py) to maintain (n, mean, M2) (or the NA-aware pairwise variant (non_nan, mean, S, C)) and update/downdate it per sample without recomputing from scratch. Available measures: PCC, Cosine, Spearman, MIC, Mutual_Information, HSIC (folders under tpnets/edges/measures/); adding one is described in Extending the pipeline.

5. Output artifacts & directory layout

tpnets/datasets/data_utils.py::add_output_paths builds the graph-build output path (rooted at graph_results/, see constants.GRAPH_RESULTS):

graph_results/
  <cohort>/fold_<fold>/<sub_dir>/<bin or "static">/<pipeline>/<exp_name>/
    <imp|noimp>/<time_strategy>/<edge_method>/<measure>[_NA_aware][_incremental]/
      final/<stat-or-bin-name>/     # e.g. mean, median, tw_0, tw_agg, static
      dev/<stat-or-bin-name>/       # only produced when hyperparameter tuning is requested
  <cohort>/fold_<fold>/<sub_dir>/<bin or "static">/original_data/
    final/{train,test}.parquet, y.parquet
    dev/{train,val,test}.parquet

sub_dir reflects --subset/--subsample (full, sub_<n>, frac_<pct>, or both). Each .../<measure...>/ folder holds a log.txt (via Logger) and, per split, the edge vectors — kept in memory (stored_outputs, passed straight into training) and/or persisted as edge_vec.h5 (HDF5, one dataset per split + node/id metadata) or edge_vec.npz (compressed NumPy), controlled by save_edge_h5/save_edge_vec in ExperimentRunner.save_artifacts.

Training results add another layer under the same graph-build path (MLTrainer.set_output_path):

.../<measure...>/(sig_edges|all_edges)/(zscores|edgescores)/<zmode>/<threshold>/(tuned|default)/
  <model>_results.csv
  <model>_<comb>_feat_imp.csv
  splits/<comb>/{train,test}.parquet

6. Feature engineering

• GraphFilter (tpnets/features/graph_filter.py) loads the edge vectors (in-memory dict, .h5, or .npz) and optionally applies:
  • --zscores: standardize edges, --zmode graphwise (one mean/std over all train edges) or edgewise (per-edge mean/std over train samples only);
  • --significant: mask non-significant edges to NaN, --threshold p_val (two-sided normal threshold at --zmode's p-value, default 0.05) or min (lowest per-sample max-|z|, guaranteeing every sample keeps ≥1 edge).
• FeatureComputer (tpnets/features/feature_computer.py) turns the (optionally filtered) edge vectors into three feature families, requested via --features {edge,node,graph}:
  • node: weighted degree centrality per node;
  • edge: top-N edges by training-set variance (n_edges, cohort-dependent default set in GraphFilter.get_n_edges), same indices reused across splits, remaining NaNs zero-filled;
  • graph: per-sample topological/spectral summary computed by GraphMetricComputer — edge counts (total/pos/neg), connected components, weight-distribution moments, node-strength stats, spectral gap/radius, signed Katz centrality bounds, global/local efficiency, triangle count, degree assortativity, Louvain modularity, negative-subgraph clustering.
• FeatureEngineer (tpnets/ml_training/ml_features.py) merges the original (non-graph) time-series-derived features saved by ExperimentRunner.save_original_data with every requested graph-derived feature family, across every build "source" (each stat/bin/TS output), producing a (features, names) pair — names records which columns belong to which family/source, so training can try feature-family combinations.

7. Model training

• ModelFactory (tpnets/ml_training/ml_factory.py) is a small model registry (catboost_under, xgb_under, rf_balanced) that resolves default / best-known / grid-search hyperparameters from the YAML configs above and pairs each model with an imbalance sampler (RandomUnderSampler, or none).
• MLTrainer (tpnets/ml_training/ml_trainer.py):
  • iterates over feature-family combinations (all, original, edge, node, graph, and any per-source combination) discovered by FeatureEngineer;
  • if --hptune, runs grid_search on the dev stage (train/val, never test) to pick hyperparameters by validation ROC-AUC, then refits on the final stage (train+val merged, evaluated on test);
  • saves per-combination train/test splits, feature importances (native model importance if available), and evaluation metrics — ROC-AUC, PR-AUC, recall, precision, balanced accuracy, F1 — to <model>_results.csv.
• ExperimentTrainSingle trains one configuration; ExperimentTrainGrid (driven by --train_all <grid>) iterates the cartesian product of tpnets/config/train_grid.json.

---

Usage

1. Build graphs (no training)

Build with every parameter combination in a grid file:

python -m tpnets.pipeline.run_pipeline --mode build --build_all chemo_grid

Build a single, fully specified configuration:

python -m tpnets.pipeline.run_pipeline --mode build \
  --cohort mimic_cohort_aplasia_45_days --fold 0 --bin day \
  --time_strategy TS1 --edge_method SSN --measure PCC

2. Build + train

All build configurations x all training configurations:

python -m tpnets.pipeline.run_pipeline --mode train \
  --build_all chemo_grid --train_all train_grid

All build configurations, default training config:

python -m tpnets.pipeline.run_pipeline --mode train --build_all chemo_grid

All build configurations, one specified training config:

python -m tpnets.pipeline.run_pipeline --mode train --build_all chemo_grid \
  --significant --zscores --zmode graphwise --threshold p_val

One build configuration, all training configurations:

python -m tpnets.pipeline.run_pipeline --mode train \
  --cohort mimic_cohort_aplasia_45_days --fold 0 --bin day \
  --time_strategy TS1 --edge_method SSN --measure PCC --train_all train_grid

One build configuration, one training configuration:

python -m tpnets.pipeline.run_pipeline --mode train \
  --cohort mimic_cohort_aplasia_45_days --fold 0 --bin day \
  --time_strategy TS1 --edge_method SSN --measure PCC \
  --zscores --zmode edgewise

Key CLI arguments

Argument	Default	Meaning
--mode	build	build (graphs only) or train (graphs + ML).
--cohort	—	Key into DATASET_CONFIG (required unless --build_all).
--fold	—	CV fold id, selects fold_<fold>.pkl.
--bin	None	Time-bin column name for temporal cohorts (e.g. day, 6hour); None/omitted ⇒ static.
--time_strategy	—	TS0/TS1/TS2/TS3 (see above).
--subset / --subsample	None	Subsample by count / by percentage, for quick runs.
--edge_method	—	SSN, LIONESS, SWEET, Example, ExampleIndependent.
--measure	PCC	Association measure name (measures/ subfolder).
--na_aware / --no-na_aware	True	Keep NaNs (mask edges) vs. impute remaining gaps.
--incremental / --no-incremental	True	Use incremental leave-one-out updates when available.
--stats	mean median max min	Summary statistics for TS1.
--significant	False	Mask non-significant edges.
--zscores	False	Standardize edges before/instead of raw scores.
--zmode	graphwise	graphwise or edgewise standardization.
--threshold	p_val	p_val or min significance threshold.
--features	edge node graph	Feature families to compute for training.
--model	catboost_under	Key into ModelFactory's registry.
--hptune	False	Grid-search hyperparameters on the dev split.
--build_all	None	Name of a JSON grid file (in tpnets/config/) for the build stage.
--train_all	None	Name of a JSON grid file for the training stage.

---

Extending the pipeline

• New edge construction method: subclass GenericEdgeMethod (tpnets/edges/edge_methods.py) and override scale, background, and sample_weight (and calculate_training_weights if the weight needs cohort-wide precomputation); register it in METHOD_MAP in tpnets/pipeline/exp_build.py.
• New association measure: add tpnets/edges/measures/<Name>/<Name>.py with recompute(X) and a separable flag; optionally add <Name>_NA_aware.py, <Name>_incremental.py, <Name>_NA_aware_incremental.py cells (built from welford.make_moment_aggregate/make_pairwise_moment_aggregate if the measure is a moment-based statistic) for speed. Missing cells degrade gracefully (see Association measures).
• New dataset/cohort: add an entry to DATASET_CONFIG in tpnets/config/constants.py pointing at the cohort/data/fold files, and (if needed) a preprocessing script under tpnets/preprocessing/.
• New model: register it in ModelFactory.registry (tpnets/ml_training/ml_factory.py) and add its default/grid hyperparameters to tpnets/config/ml_params.yaml.