From b0ac5f3cc85bd8cba98ba14ce009188d444990f2 Mon Sep 17 00:00:00 2001 From: Tony Bagnall Date: Fri, 11 Sep 2026 16:51:30 +0100 Subject: [PATCH 1/3] Ingest EmoPain: the dataset is complete and now scored DisjointCNN, TimesURL and XCM were the last three without it, and all seven of the Hali run's estimators returned. EmoPain joins the scored set, taking it from 56 to 57 of 64, and DisjointCNN, TimesURL and XCM each reach 66. The dataset never needed rescaling. aeon's check_collection_variance flags pairs with std <= 1e-7 and a nonzero range, and all 1733 of EmoPain's sit in channels 13 to 25, which are sparse rather than mis-scaled: 84% of their values are below 1e-5 while their peaks reach 1.55, the same range as the motion channels. aeon relegated the check to a warning in #3598, and running on Hali with an aeon past that commit was the whole fix. XCM's result carries 'window_size': 0.8, so it is the fixed-parameter run the other 65 use rather than the cross-validated search the lookup name now resolves to. That was pinned in the Hali configuration for exactly this reason. Accuracies span 0.628 to 0.848, with PatchMTSC and TimesURL highest and DisjointCNN lowest, so nothing looks degenerate despite the flat channels. Co-Authored-By: Claude Opus 5 --- README.md | 54 +++++++++---------- .../DisjointCNN/DisjointCNN_accuracy.csv | 1 + .../DisjointCNN/DisjointCNN_auroc.csv | 1 + .../DisjointCNN/DisjointCNN_balacc.csv | 1 + .../multiverse/DisjointCNN/DisjointCNN_f1.csv | 1 + .../DisjointCNN/DisjointCNN_logloss.csv | 1 + .../DisjointCNN/DisjointCNN_sensitivity.csv | 1 + .../DisjointCNN/DisjointCNN_specificity.csv | 1 + .../multiverse/TimesURL/TimesURL_accuracy.csv | 1 + .../multiverse/TimesURL/TimesURL_auroc.csv | 1 + .../multiverse/TimesURL/TimesURL_balacc.csv | 1 + results/multiverse/TimesURL/TimesURL_f1.csv | 1 + .../multiverse/TimesURL/TimesURL_logloss.csv | 1 + .../TimesURL/TimesURL_sensitivity.csv | 1 + .../TimesURL/TimesURL_specificity.csv | 1 + results/multiverse/XCM/XCM_accuracy.csv | 1 + results/multiverse/XCM/XCM_auroc.csv | 1 + results/multiverse/XCM/XCM_balacc.csv | 1 + results/multiverse/XCM/XCM_f1.csv | 1 + results/multiverse/XCM/XCM_logloss.csv | 1 + results/multiverse/XCM/XCM_sensitivity.csv | 1 + results/multiverse/XCM/XCM_specificity.csv | 1 + results/multiverse/datasets.html | 2 +- results/multiverse/leaderboard.html | 6 +-- 24 files changed, 52 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index da32d64..d07121b 100644 --- a/README.md +++ b/README.md @@ -40,33 +40,33 @@ The current paper version describes: | # | Estimator | Accuracy rank | Accuracy | Balanced accuracy | AUROC | F1 | Log loss ↓ | Sensitivity | Specificity | |---|---|---|---|---|---|---|---|---|---| -| 1 | HC2 | **7.64** | **0.7917** | **0.7557** | **0.8935** | **0.7302** | **0.5350** | 0.7469 | **0.7998** | -| 2 | MRHydra | 9.08 | 0.7794 | 0.7520 | 0.8040 | 0.7266 | 7.9526 | **0.7577** | 0.7768 | -| 3 | RDST | 9.46 | 0.7729 | 0.7386 | 0.7928 | 0.7075 | 8.1867 | 0.7172 | 0.7902 | -| 4 | RIST | 9.85 | 0.7744 | 0.7451 | 0.8679 | 0.7174 | 0.6150 | 0.7416 | 0.7743 | -| 5 | CIF | 10.23 | 0.7770 | 0.7487 | 0.8842 | 0.7246 | 0.6442 | 0.7459 | 0.7780 | -| 6 | DrCIF | 10.25 | 0.7731 | 0.7433 | 0.8745 | 0.7189 | 0.6430 | 0.7400 | 0.7736 | -| 7 | QUANT | 10.70 | 0.7668 | 0.7404 | 0.8694 | 0.7166 | 0.7285 | 0.7491 | 0.7570 | -| 8 | LITETime-MV | 10.95 | 0.7511 | 0.7320 | 0.8503 | 0.6878 | 1.3004 | 0.7167 | 0.7660 | -| 9 | Arsenal | 11.04 | 0.7663 | 0.7335 | 0.8419 | 0.7061 | 3.6444 | 0.7266 | 0.7752 | -| 10 | ROCKET | 11.09 | 0.7690 | 0.7362 | 0.7925 | 0.7065 | 8.3274 | 0.7228 | 0.7798 | -| 11 | STSF | 11.29 | 0.7727 | 0.7508 | 0.8723 | 0.7164 | 0.6685 | 0.7412 | 0.7845 | -| 12 | H-InceptionTime | 11.61 | 0.7421 | 0.7208 | 0.8447 | 0.6853 | 1.3448 | 0.7227 | 0.7436 | -| 13 | ConvTran | 12.88 | 0.7490 | 0.7177 | 0.8529 | 0.6862 | 0.8826 | 0.7234 | 0.7419 | -| 14 | DisjointCNN | 13.08 | 0.7296 | 0.7057 | 0.8246 | 0.6641 | 2.1100 | 0.6815 | 0.7431 | -| 15 | PatchMTSC | 13.37 | 0.7454 | 0.6990 | 0.8250 | 0.6671 | 0.7818 | 0.6981 | 0.7397 | -| 16 | Catch22 | 13.47 | 0.7463 | 0.7177 | 0.8605 | 0.6929 | 0.7068 | 0.7229 | 0.7420 | -| 17 | TSF | 14.01 | 0.7426 | 0.7175 | 0.8571 | 0.6896 | 0.9987 | 0.7095 | 0.7521 | -| 18 | STC | 14.03 | 0.7507 | 0.7137 | 0.8624 | 0.6803 | 0.6468 | 0.7036 | 0.7611 | -| 19 | TDE | 14.99 | 0.7251 | 0.6834 | 0.8339 | 0.6446 | 0.8524 | 0.6759 | 0.7349 | -| 20 | TS2Vec | 15.21 | 0.7201 | 0.6809 | 0.7994 | 0.6527 | 0.7835 | 0.6879 | 0.7144 | -| 21 | XCM | 16.58 | 0.6706 | 0.6370 | 0.7893 | 0.5771 | 2.1798 | 0.6167 | 0.6875 | -| 22 | Summary | 16.92 | 0.6845 | 0.6570 | 0.8113 | 0.6299 | 0.9645 | 0.6621 | 0.6852 | -| 23 | TimesNet | 17.24 | 0.7020 | 0.6709 | 0.8218 | 0.6396 | 1.2889 | 0.6810 | 0.6934 | -| 24 | TimesURL | 17.36 | 0.6950 | 0.6562 | 0.7831 | 0.6052 | 0.9868 | 0.6312 | 0.7016 | -| 25 | Dummy | 22.69 | 0.3709 | 0.3067 | 0.5000 | 0.1626 | 1.3928 | 0.2987 | 0.3880 | - -Average over the 56 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. +| 1 | HC2 | **7.63** | **0.7927** | **0.7512** | **0.8944** | **0.7318** | **0.5321** | 0.7487 | **0.8007** | +| 2 | MRHydra | 8.98 | 0.7811 | 0.7502 | 0.8047 | 0.7292 | 7.8914 | **0.7598** | 0.7785 | +| 3 | RDST | 9.32 | 0.7747 | 0.7366 | 0.7937 | 0.7105 | 8.1197 | 0.7200 | 0.7918 | +| 4 | RIST | 9.74 | 0.7761 | 0.7434 | 0.8693 | 0.7201 | 0.6103 | 0.7439 | 0.7761 | +| 5 | DrCIF | 10.23 | 0.7744 | 0.7408 | 0.8755 | 0.7210 | 0.6386 | 0.7418 | 0.7749 | +| 6 | CIF | 10.32 | 0.7778 | 0.7449 | 0.8848 | 0.7262 | 0.6402 | 0.7472 | 0.7788 | +| 7 | QUANT | 10.73 | 0.7680 | 0.7377 | 0.8707 | 0.7188 | 0.7224 | 0.7506 | 0.7584 | +| 8 | LITETime-MV | 10.94 | 0.7526 | 0.7292 | 0.8518 | 0.6908 | 1.2944 | 0.7188 | 0.7672 | +| 9 | Arsenal | 11.15 | 0.7671 | 0.7290 | 0.8413 | 0.7075 | 3.6366 | 0.7281 | 0.7759 | +| 10 | ROCKET | 11.21 | 0.7696 | 0.7319 | 0.7900 | 0.7076 | 8.3060 | 0.7242 | 0.7802 | +| 11 | STSF | 11.22 | 0.7740 | 0.7487 | 0.8728 | 0.7187 | 0.6667 | 0.7431 | 0.7856 | +| 12 | H-InceptionTime | 11.81 | 0.7411 | 0.7169 | 0.8455 | 0.6864 | 1.3447 | 0.7220 | 0.7425 | +| 13 | ConvTran | 13.07 | 0.7478 | 0.7140 | 0.8542 | 0.6872 | 0.8894 | 0.7226 | 0.7408 | +| 14 | Catch22 | 13.25 | 0.7488 | 0.7177 | 0.8620 | 0.6962 | 0.7009 | 0.7258 | 0.7445 | +| 15 | PatchMTSC | 13.25 | 0.7472 | 0.6982 | 0.8270 | 0.6704 | 0.7772 | 0.7007 | 0.7416 | +| 16 | DisjointCNN | 13.29 | 0.7278 | 0.7027 | 0.8258 | 0.6648 | 2.1048 | 0.6806 | 0.7411 | +| 17 | STC | 14.00 | 0.7521 | 0.7095 | 0.8632 | 0.6825 | 0.6427 | 0.7059 | 0.7623 | +| 18 | TSF | 14.04 | 0.7439 | 0.7142 | 0.8581 | 0.6917 | 0.9887 | 0.7114 | 0.7533 | +| 19 | TDE | 14.97 | 0.7269 | 0.6807 | 0.8355 | 0.6475 | 0.8456 | 0.6786 | 0.7365 | +| 20 | TS2Vec | 15.31 | 0.7209 | 0.6796 | 0.8015 | 0.6552 | 0.7822 | 0.6892 | 0.7153 | +| 21 | XCM | 16.68 | 0.6710 | 0.6353 | 0.7908 | 0.5801 | 2.1677 | 0.6180 | 0.6876 | +| 22 | Summary | 16.71 | 0.6878 | 0.6572 | 0.8137 | 0.6342 | 0.9545 | 0.6657 | 0.6885 | +| 23 | TimesURL | 17.24 | 0.6974 | 0.6532 | 0.7841 | 0.6089 | 0.9807 | 0.6348 | 0.7040 | +| 24 | TimesNet | 17.29 | 0.7033 | 0.6673 | 0.8237 | 0.6424 | 1.2793 | 0.6827 | 0.6949 | +| 25 | Dummy | 22.62 | 0.3781 | 0.3072 | 0.5000 | 0.1719 | 1.3799 | 0.3072 | 0.3950 | + +Average over the 57 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. Rebuilt with `python -m multiverse.experiments.tables`, which also writes a sortable diff --git a/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv b/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv index 7f5f584..60c54d6 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7430285915990117 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.6106870229007634 +EmoPain,0.6281690140845071 Epilepsy,0.9782608695652174 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.40476190476190477 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv b/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv index aea739e..080c66b 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.8060149855068997 DuckDuckGeese,0.8734999999999999 ERing,0.9996049382716049 EigenWorms,0.939828804098903 +EmoPain,0.8969922963526262 Epilepsy,0.9996474772895348 EthanolConcentration,0.5151667219374849 EyesOpenShut,0.4331065759637188 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv b/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv index 4414cbd..0530412 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.743105311212915 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.5226262626262625 +EmoPain,0.535280162966401 Epilepsy,0.9779411764705882 EthanolConcentration,0.2776223776223776 EyesOpenShut,0.40476190476190477 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_f1.csv b/results/multiverse/DisjointCNN/DisjointCNN_f1.csv index f31ecf7..1d8a28c 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_f1.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7888631090487239 DuckDuckGeese,0.5616361416361415 ERing,0.9550962651682063 EigenWorms,0.5339473211773573 +EmoPain,0.7048428320655101 Epilepsy,0.9780092879822783 EthanolConcentration,0.268875415691347 EyesOpenShut,0.4186046511627907 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv b/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv index 6c07e9f..286e320 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,5.777233621887509 DuckDuckGeese,2.043216171575822 ERing,0.13835083108857268 EigenWorms,1.1175401751396294 +EmoPain,1.811924632950105 Epilepsy,0.09597854665684544 EthanolConcentration,2.8107859500435426 EyesOpenShut,3.9011231082747133 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv b/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv index 1025959..95b8bc4 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.96045197740113 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.6106870229007634 +EmoPain,0.6281690140845071 Epilepsy,0.9782608695652174 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.42857142857142855 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv b/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv index 2d69a80..78be908 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.5257586450247 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.6106870229007634 +EmoPain,0.6281690140845071 Epilepsy,0.9782608695652174 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.38095238095238093 diff --git a/results/multiverse/TimesURL/TimesURL_accuracy.csv b/results/multiverse/TimesURL/TimesURL_accuracy.csv index 28e7b38..17f210b 100644 --- a/results/multiverse/TimesURL/TimesURL_accuracy.csv +++ b/results/multiverse/TimesURL/TimesURL_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.6840804800564773 DuckDuckGeese,0.5 ERing,0.8851851851851852 EigenWorms,0.8549618320610687 +EmoPain,0.8366197183098592 Epilepsy,0.9420289855072463 EthanolConcentration,0.23954372623574144 EyesOpenShut,0.5 diff --git a/results/multiverse/TimesURL/TimesURL_auroc.csv b/results/multiverse/TimesURL/TimesURL_auroc.csv index 5d44e67..d6ed61b 100644 --- a/results/multiverse/TimesURL/TimesURL_auroc.csv +++ b/results/multiverse/TimesURL/TimesURL_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7748470948012232 DuckDuckGeese,0.7725 ERing,0.9819753086419752 EigenWorms,0.9485469975641881 +EmoPain,0.8388880647503131 Epilepsy,0.9827082206548413 EthanolConcentration,0.5270873799468113 EyesOpenShut,0.4331065759637188 diff --git a/results/multiverse/TimesURL/TimesURL_balacc.csv b/results/multiverse/TimesURL/TimesURL_balacc.csv index 05a2ad2..2891a50 100644 --- a/results/multiverse/TimesURL/TimesURL_balacc.csv +++ b/results/multiverse/TimesURL/TimesURL_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.6841642943435045 DuckDuckGeese,0.5 ERing,0.8851851851851853 EigenWorms,0.8226472078645992 +EmoPain,0.48561151079136694 Epilepsy,0.9441573926868044 EthanolConcentration,0.23933566433566433 EyesOpenShut,0.5 diff --git a/results/multiverse/TimesURL/TimesURL_f1.csv b/results/multiverse/TimesURL/TimesURL_f1.csv index 6988ae1..519c325 100644 --- a/results/multiverse/TimesURL/TimesURL_f1.csv +++ b/results/multiverse/TimesURL/TimesURL_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7446504992867332 DuckDuckGeese,0.4913478260869566 ERing,0.8852171572332718 EigenWorms,0.8520288596624473 +EmoPain,0.8150529324542688 Epilepsy,0.9418486609229948 EthanolConcentration,0.2415233857922727 EyesOpenShut,0.6666666666666666 diff --git a/results/multiverse/TimesURL/TimesURL_logloss.csv b/results/multiverse/TimesURL/TimesURL_logloss.csv index 769b055..230cb78 100644 --- a/results/multiverse/TimesURL/TimesURL_logloss.csv +++ b/results/multiverse/TimesURL/TimesURL_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,0.6811406478510141 DuckDuckGeese,1.5354294896775442 ERing,0.5886295866172955 EigenWorms,0.6568276741658519 +EmoPain,0.63623861409052 Epilepsy,0.4325868494205753 EthanolConcentration,1.9707683257768949 EyesOpenShut,1.1269007394531552 diff --git a/results/multiverse/TimesURL/TimesURL_sensitivity.csv b/results/multiverse/TimesURL/TimesURL_sensitivity.csv index da22960..05bb815 100644 --- a/results/multiverse/TimesURL/TimesURL_sensitivity.csv +++ b/results/multiverse/TimesURL/TimesURL_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.9216101694915254 DuckDuckGeese,0.5 ERing,0.8851851851851852 EigenWorms,0.8549618320610687 +EmoPain,0.8366197183098592 Epilepsy,0.9420289855072463 EthanolConcentration,0.23954372623574144 EyesOpenShut,1.0 diff --git a/results/multiverse/TimesURL/TimesURL_specificity.csv b/results/multiverse/TimesURL/TimesURL_specificity.csv index 6ae92ac..2487839 100644 --- a/results/multiverse/TimesURL/TimesURL_specificity.csv +++ b/results/multiverse/TimesURL/TimesURL_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.44671841919548344 DuckDuckGeese,0.5 ERing,0.8851851851851852 EigenWorms,0.8549618320610687 +EmoPain,0.8366197183098592 Epilepsy,0.9420289855072463 EthanolConcentration,0.23954372623574144 EyesOpenShut,0.0 diff --git a/results/multiverse/XCM/XCM_accuracy.csv b/results/multiverse/XCM/XCM_accuracy.csv index fcddf01..12f5b88 100644 --- a/results/multiverse/XCM/XCM_accuracy.csv +++ b/results/multiverse/XCM/XCM_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7518531591951995 DuckDuckGeese,0.54 ERing,0.3037037037037037 EigenWorms,0.5038167938931297 +EmoPain,0.6957746478873239 Epilepsy,0.9130434782608695 EthanolConcentration,0.2889733840304182 EyesOpenShut,0.5 diff --git a/results/multiverse/XCM/XCM_auroc.csv b/results/multiverse/XCM/XCM_auroc.csv index e1ed21e..604df0a 100644 --- a/results/multiverse/XCM/XCM_auroc.csv +++ b/results/multiverse/XCM/XCM_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7769353372486634 DuckDuckGeese,0.8835 ERing,0.9387489711934157 EigenWorms,0.8730167577171467 +EmoPain,0.8743557734499935 Epilepsy,0.9757065171400083 EthanolConcentration,0.5558911771202598 EyesOpenShut,0.2380952380952381 diff --git a/results/multiverse/XCM/XCM_balacc.csv b/results/multiverse/XCM/XCM_balacc.csv index 9211a6b..6569b6e 100644 --- a/results/multiverse/XCM/XCM_balacc.csv +++ b/results/multiverse/XCM/XCM_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7519220303099171 DuckDuckGeese,0.54 ERing,0.30370370370370375 EigenWorms,0.36 +EmoPain,0.5431706247904902 Epilepsy,0.9141494435612083 EthanolConcentration,0.28805361305361304 EyesOpenShut,0.5 diff --git a/results/multiverse/XCM/XCM_f1.csv b/results/multiverse/XCM/XCM_f1.csv index 2576c12..6a5c16f 100644 --- a/results/multiverse/XCM/XCM_f1.csv +++ b/results/multiverse/XCM/XCM_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7923190546528803 DuckDuckGeese,0.5323809523809524 ERing,0.1517735755778025 EigenWorms,0.40159309658148024 +EmoPain,0.7492379034906649 Epilepsy,0.9112318840579711 EthanolConcentration,0.24591210249853798 EyesOpenShut,0.6666666666666666 diff --git a/results/multiverse/XCM/XCM_logloss.csv b/results/multiverse/XCM/XCM_logloss.csv index e9de0b4..c1ab07c 100644 --- a/results/multiverse/XCM/XCM_logloss.csv +++ b/results/multiverse/XCM/XCM_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,5.681748011476673 DuckDuckGeese,1.4476585961906254 ERing,1.5511330924068825 EigenWorms,2.7894721126602673 +EmoPain,1.4895746842659396 Epilepsy,0.3715004381698749 EthanolConcentration,1.529273939296933 EyesOpenShut,4.248528533555482 diff --git a/results/multiverse/XCM/XCM_sensitivity.csv b/results/multiverse/XCM/XCM_sensitivity.csv index e9242f0..b18a5d3 100644 --- a/results/multiverse/XCM/XCM_sensitivity.csv +++ b/results/multiverse/XCM/XCM_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.9470338983050848 DuckDuckGeese,0.54 ERing,0.3037037037037037 EigenWorms,0.5038167938931297 +EmoPain,0.6957746478873239 Epilepsy,0.9130434782608695 EthanolConcentration,0.2889733840304182 EyesOpenShut,1.0 diff --git a/results/multiverse/XCM/XCM_specificity.csv b/results/multiverse/XCM/XCM_specificity.csv index bc472f0..dcf3a82 100644 --- a/results/multiverse/XCM/XCM_specificity.csv +++ b/results/multiverse/XCM/XCM_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.5568101623147494 DuckDuckGeese,0.54 ERing,0.3037037037037037 EigenWorms,0.5038167938931297 +EmoPain,0.6957746478873239 Epilepsy,0.9130434782608695 EthanolConcentration,0.2889733840304182 EyesOpenShut,0.0 diff --git a/results/multiverse/datasets.html b/results/multiverse/datasets.html index 9ede74a..6cf926b 100644 --- a/results/multiverse/datasets.html +++ b/results/multiverse/datasets.html @@ -60,7 +60,7 @@ details { margin-top: .6rem; } summary { cursor: pointer; color: var(--accent); } code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: .9em; } -tr.nosignal td { background: rgba(214, 158, 46, .16); }tr.saturated td { background: rgba(56, 161, 105, .14); }

Multiverse-core datasets: accuracy

64 datasets · accuracy · best of up to 24 estimators against the Dummy baseline · built 2026-09-11

DatasetDummyMedianBestBest estimatorGain over dummySpreadEstimators
KINECAL-QSEO0.94120.94120.9412Arsenal0.00000.647124
BIDMC32SpO2_disc0.71530.65740.7203ROCKET0.00500.190124
Locust20220.91120.90820.9206MRHydra0.00940.052423
Heartbeat0.72200.74880.7854CIF0.06340.126824
HouseholdPowerConsumption2_disc0.72160.76530.7872DisjointCNN0.06560.097724
MotorImagery0.50000.51000.5900HC20.09000.130024
AutomotiveRoadTrials0.75320.78570.8442CIF0.09090.233824
EyesOpenShut0.50000.50000.5952STSF0.09520.190524
EmoPain0.78310.83380.8845Catch220.10140.202821
AppliancesEnergy_disc0.80950.82140.9286DrCIF0.11900.428624
BeijingPM10Quality_disc0.71120.82420.8417STSF0.13050.126024
Alzheimers0.41860.37210.5581MRHydra0.13950.302323
PhotoStimulation0.41670.38890.5833ROCKET0.16670.388923
FaceDetection0.50000.62510.6850H-InceptionTime0.18500.170524
BeijingPM25Quality_disc0.69770.87510.8879ConvTran0.19020.128824
AtrialFibrillation0.33330.26670.5333TS2Vec0.20000.466724
HouseholdPowerConsumption1_disc0.77840.91110.9825STSF0.20410.864424
LowCost0.50000.63170.7300TSF0.23000.248324
BoneProbAgeGroup0.47640.64380.7124H-InceptionTime0.23600.184324
StandWalkJump0.33330.40000.6000MRHydra0.26670.400024
CrowdSourced0.50020.71900.7734LITETime-MV0.27320.176824
BenzeneConcentration_disc0.68970.81070.9768STSF0.28700.577024
FordChallenge0.62320.88430.9360QUANT0.31280.312824
BIDMC32HR_disc0.65070.80120.9637RIST0.31300.635324
STEW0.50000.74020.8385Arsenal0.33850.210022
BoneIntensitiesAgeGroup0.47640.79890.8202HC20.34380.256224
PhonemeSpectra0.02560.27100.3746H-InceptionTime0.34890.291124
KERAAL-RTK0.57140.78570.9286HC20.35710.571424
LSST0.31510.62900.6752HC20.36010.452124
HandMovementDirection0.20270.41220.6081TSF0.40540.351424
DuckDuckGeese0.20000.46000.6400H-InceptionTime0.44000.480024
SelfRegulationSCP10.50170.85320.9454MRHydra0.44370.392524
IEEEPPG_disc0.26050.45480.7078ConvTran0.44730.438324
AsphaltRegularityCoordinates0.50730.97870.9947H-InceptionTime0.48740.061324
MindReading0.23120.52830.7243LITETime-MV0.49310.385924
EthanolConcentration0.25100.43350.7490STC0.49810.513324
UIPRMD-DS-C0.50000.83331.0000Catch220.50000.388924
WISDM0.36640.86580.8965MRHydra0.53000.131024
Blink0.44440.98671.0000Arsenal0.55560.428924
EigenWorms0.41980.86260.9771MRHydra0.55730.557323
KIMORE-PR-C0.14290.42860.7143LITETime-MV0.57140.571424
AsphaltObstaclesCoordinates0.28390.81970.8670MRHydra0.58310.199524
CounterMovementJump0.33520.75700.9274Arsenal0.59220.458124
Handwriting0.03760.37290.6529H-InceptionTime0.61530.478824
RacketSports0.28290.87830.9079RDST0.62500.125024
UCDHE-Rowing-MC0.20450.73410.8295PatchMTSC0.62500.259124
USCActivity0.11380.69240.7473HC20.63340.166422
IRDS-SFL0.20690.79310.8966XCM0.68970.482824
Skoda0.23560.94550.9646H-InceptionTime0.72900.119924
Epilepsy0.26810.98191.0000HC20.73190.101424
Tiselac0.06280.81360.8373STSF0.77450.204420
MotionSenseHAR0.20380.98871.0000DrCIF0.79620.101924
NATOPS0.16670.89170.9667LITETime-MV0.80000.155624
UCIActivity0.19160.97690.9983LITETime-MV0.80670.174924
UWaveGestureLibrary0.12500.90940.9406Arsenal0.81560.553124
ERing0.16670.94070.9963MRHydra0.82960.692624
PEMS-SF0.11560.86991.0000CIF0.88440.317924
SpokenArabicDigits0.10000.98160.9945DisjointCNN0.89450.129124
Libras0.06670.88890.9722RIST0.90560.338924
JapaneseVowels0.08380.97030.9946LITETime-MV0.91080.208124
Cricket0.08330.97921.0000Arsenal0.91670.069424
CharacterTrajectories0.06480.98960.9958H-InceptionTime0.93110.044624
TactileTextureRecognition0.05140.99851.0000H-InceptionTime0.94860.234924
ArticularyWordRecognition0.04000.98000.9933Arsenal0.95330.050024

One row per dataset. Dummy is the no-skill floor. Median, best and spread are over the other estimators, so the baseline cannot flatter them. Gain over dummy is best minus dummy, how much skill was found at all; spread is best minus worst, how much the choice of estimator mattered. The two answer different questions, and a single range would conflate them.

3 of 64 datasets gained 0.05 or less over the baseline (shaded amber) and 14 have a best of 0.99 or more (shaded green). Both separate estimators poorly, for opposite reasons. Best is a maximum over many estimators, so it is optimistic by construction: read it as what the archive can currently do on a problem, not as what any one method delivers.