diff --git a/README.md b/README.md index da32d64..51081f1 100644 --- a/README.md +++ b/README.md @@ -40,33 +40,33 @@ The current paper version describes: | # | Estimator | Accuracy rank | Accuracy | Balanced accuracy | AUROC | F1 | Log loss ↓ | Sensitivity | Specificity | |---|---|---|---|---|---|---|---|---|---| -| 1 | HC2 | **7.64** | **0.7917** | **0.7557** | **0.8935** | **0.7302** | **0.5350** | 0.7469 | **0.7998** | -| 2 | MRHydra | 9.08 | 0.7794 | 0.7520 | 0.8040 | 0.7266 | 7.9526 | **0.7577** | 0.7768 | -| 3 | RDST | 9.46 | 0.7729 | 0.7386 | 0.7928 | 0.7075 | 8.1867 | 0.7172 | 0.7902 | -| 4 | RIST | 9.85 | 0.7744 | 0.7451 | 0.8679 | 0.7174 | 0.6150 | 0.7416 | 0.7743 | -| 5 | CIF | 10.23 | 0.7770 | 0.7487 | 0.8842 | 0.7246 | 0.6442 | 0.7459 | 0.7780 | -| 6 | DrCIF | 10.25 | 0.7731 | 0.7433 | 0.8745 | 0.7189 | 0.6430 | 0.7400 | 0.7736 | -| 7 | QUANT | 10.70 | 0.7668 | 0.7404 | 0.8694 | 0.7166 | 0.7285 | 0.7491 | 0.7570 | -| 8 | LITETime-MV | 10.95 | 0.7511 | 0.7320 | 0.8503 | 0.6878 | 1.3004 | 0.7167 | 0.7660 | -| 9 | Arsenal | 11.04 | 0.7663 | 0.7335 | 0.8419 | 0.7061 | 3.6444 | 0.7266 | 0.7752 | -| 10 | ROCKET | 11.09 | 0.7690 | 0.7362 | 0.7925 | 0.7065 | 8.3274 | 0.7228 | 0.7798 | -| 11 | STSF | 11.29 | 0.7727 | 0.7508 | 0.8723 | 0.7164 | 0.6685 | 0.7412 | 0.7845 | -| 12 | H-InceptionTime | 11.61 | 0.7421 | 0.7208 | 0.8447 | 0.6853 | 1.3448 | 0.7227 | 0.7436 | -| 13 | ConvTran | 12.88 | 0.7490 | 0.7177 | 0.8529 | 0.6862 | 0.8826 | 0.7234 | 0.7419 | -| 14 | DisjointCNN | 13.08 | 0.7296 | 0.7057 | 0.8246 | 0.6641 | 2.1100 | 0.6815 | 0.7431 | -| 15 | PatchMTSC | 13.37 | 0.7454 | 0.6990 | 0.8250 | 0.6671 | 0.7818 | 0.6981 | 0.7397 | -| 16 | Catch22 | 13.47 | 0.7463 | 0.7177 | 0.8605 | 0.6929 | 0.7068 | 0.7229 | 0.7420 | -| 17 | TSF | 14.01 | 0.7426 | 0.7175 | 0.8571 | 0.6896 | 0.9987 | 0.7095 | 0.7521 | -| 18 | STC | 14.03 | 0.7507 | 0.7137 | 0.8624 | 0.6803 | 0.6468 | 0.7036 | 0.7611 | -| 19 | TDE | 14.99 | 0.7251 | 0.6834 | 0.8339 | 0.6446 | 0.8524 | 0.6759 | 0.7349 | -| 20 | TS2Vec | 15.21 | 0.7201 | 0.6809 | 0.7994 | 0.6527 | 0.7835 | 0.6879 | 0.7144 | -| 21 | XCM | 16.58 | 0.6706 | 0.6370 | 0.7893 | 0.5771 | 2.1798 | 0.6167 | 0.6875 | -| 22 | Summary | 16.92 | 0.6845 | 0.6570 | 0.8113 | 0.6299 | 0.9645 | 0.6621 | 0.6852 | -| 23 | TimesNet | 17.24 | 0.7020 | 0.6709 | 0.8218 | 0.6396 | 1.2889 | 0.6810 | 0.6934 | -| 24 | TimesURL | 17.36 | 0.6950 | 0.6562 | 0.7831 | 0.6052 | 0.9868 | 0.6312 | 0.7016 | -| 25 | Dummy | 22.69 | 0.3709 | 0.3067 | 0.5000 | 0.1626 | 1.3928 | 0.2987 | 0.3880 | - -Average over the 56 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. +| 1 | HC2 | **7.52** | **0.7936** | **0.7528** | **0.8949** | **0.7337** | **0.5307** | 0.7499 | **0.8018** | +| 2 | MRHydra | 9.02 | 0.7806 | 0.7503 | 0.8039 | 0.7293 | 7.9077 | **0.7584** | 0.7794 | +| 3 | RDST | 9.34 | 0.7746 | 0.7371 | 0.7933 | 0.7113 | 8.1235 | 0.7199 | 0.7924 | +| 4 | RIST | 9.84 | 0.7750 | 0.7428 | 0.8685 | 0.7196 | 0.6087 | 0.7424 | 0.7758 | +| 5 | CIF | 10.19 | 0.7782 | 0.7459 | 0.8847 | 0.7273 | 0.6376 | 0.7473 | 0.7799 | +| 6 | DrCIF | 10.36 | 0.7731 | 0.7401 | 0.8743 | 0.7202 | 0.6370 | 0.7396 | 0.7752 | +| 7 | QUANT | 10.87 | 0.7667 | 0.7370 | 0.8693 | 0.7177 | 0.7198 | 0.7476 | 0.7594 | +| 8 | Arsenal | 10.99 | 0.7683 | 0.7309 | 0.8425 | 0.7096 | 3.5868 | 0.7293 | 0.7777 | +| 9 | ROCKET | 11.14 | 0.7698 | 0.7328 | 0.7899 | 0.7088 | 8.2968 | 0.7245 | 0.7811 | +| 10 | LITETime-MV | 11.16 | 0.7505 | 0.7274 | 0.8511 | 0.6864 | 1.3223 | 0.7113 | 0.7707 | +| 11 | STSF | 11.23 | 0.7735 | 0.7486 | 0.8722 | 0.7188 | 0.6639 | 0.7417 | 0.7862 | +| 12 | H-InceptionTime | 11.76 | 0.7418 | 0.7179 | 0.8460 | 0.6881 | 1.3451 | 0.7233 | 0.7428 | +| 13 | ConvTran | 12.95 | 0.7485 | 0.7152 | 0.8545 | 0.6892 | 0.8930 | 0.7251 | 0.7401 | +| 14 | Catch22 | 13.09 | 0.7495 | 0.7189 | 0.8622 | 0.6977 | 0.6971 | 0.7263 | 0.7459 | +| 15 | PatchMTSC | 13.16 | 0.7478 | 0.6996 | 0.8276 | 0.6723 | 0.7796 | 0.7018 | 0.7425 | +| 16 | DisjointCNN | 13.44 | 0.7267 | 0.7021 | 0.8267 | 0.6662 | 2.1025 | 0.6857 | 0.7344 | +| 17 | STC | 13.84 | 0.7527 | 0.7109 | 0.8633 | 0.6843 | 0.6397 | 0.7072 | 0.7629 | +| 18 | TSF | 14.06 | 0.7434 | 0.7142 | 0.8570 | 0.6918 | 0.9812 | 0.7105 | 0.7536 | +| 19 | TDE | 15.01 | 0.7266 | 0.6811 | 0.8344 | 0.6488 | 0.8411 | 0.6800 | 0.7351 | +| 20 | TS2Vec | 15.41 | 0.7203 | 0.6798 | 0.8005 | 0.6556 | 0.7806 | 0.6886 | 0.7154 | +| 21 | Summary | 16.66 | 0.6886 | 0.6585 | 0.8135 | 0.6357 | 0.9475 | 0.6661 | 0.6901 | +| 22 | XCM | 16.78 | 0.6707 | 0.6357 | 0.7914 | 0.5827 | 2.1902 | 0.6233 | 0.6824 | +| 23 | TimesNet | 17.22 | 0.7039 | 0.6685 | 0.8236 | 0.6437 | 1.2796 | 0.6824 | 0.6968 | +| 24 | TimesURL | 17.28 | 0.6973 | 0.6538 | 0.7828 | 0.6099 | 0.9919 | 0.6345 | 0.7050 | +| 25 | Dummy | 22.66 | 0.3802 | 0.3105 | 0.5000 | 0.1804 | 1.3681 | 0.3192 | 0.3882 | + +Average over the 58 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. Rebuilt with `python -m multiverse.experiments.tables`, which also writes a sortable diff --git a/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv b/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv index 7f5f584..60c54d6 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7430285915990117 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.6106870229007634 +EmoPain,0.6281690140845071 Epilepsy,0.9782608695652174 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.40476190476190477 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv b/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv index aea739e..080c66b 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.8060149855068997 DuckDuckGeese,0.8734999999999999 ERing,0.9996049382716049 EigenWorms,0.939828804098903 +EmoPain,0.8969922963526262 Epilepsy,0.9996474772895348 EthanolConcentration,0.5151667219374849 EyesOpenShut,0.4331065759637188 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv b/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv index 4414cbd..0530412 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.743105311212915 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.5226262626262625 +EmoPain,0.535280162966401 Epilepsy,0.9779411764705882 EthanolConcentration,0.2776223776223776 EyesOpenShut,0.40476190476190477 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_f1.csv b/results/multiverse/DisjointCNN/DisjointCNN_f1.csv index f31ecf7..1d8a28c 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_f1.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7888631090487239 DuckDuckGeese,0.5616361416361415 ERing,0.9550962651682063 EigenWorms,0.5339473211773573 +EmoPain,0.7048428320655101 Epilepsy,0.9780092879822783 EthanolConcentration,0.268875415691347 EyesOpenShut,0.4186046511627907 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv b/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv index 6c07e9f..286e320 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,5.777233621887509 DuckDuckGeese,2.043216171575822 ERing,0.13835083108857268 EigenWorms,1.1175401751396294 +EmoPain,1.811924632950105 Epilepsy,0.09597854665684544 EthanolConcentration,2.8107859500435426 EyesOpenShut,3.9011231082747133 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv b/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv index 1025959..95b8bc4 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.96045197740113 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.6106870229007634 +EmoPain,0.6281690140845071 Epilepsy,0.9782608695652174 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.42857142857142855 diff --git a/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv b/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv index 2d69a80..78be908 100644 --- a/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv +++ b/results/multiverse/DisjointCNN/DisjointCNN_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.5257586450247 DuckDuckGeese,0.6 ERing,0.9555555555555556 EigenWorms,0.6106870229007634 +EmoPain,0.6281690140845071 Epilepsy,0.9782608695652174 EthanolConcentration,0.27756653992395436 EyesOpenShut,0.38095238095238093 diff --git a/results/multiverse/HC2/HC2_accuracy.csv b/results/multiverse/HC2/HC2_accuracy.csv index dc2bbce..513d3fb 100644 --- a/results/multiverse/HC2/HC2_accuracy.csv +++ b/results/multiverse/HC2/HC2_accuracy.csv @@ -52,11 +52,13 @@ PenDigits,0.9797026872498571 PhonemeSpectra,0.31404712198031615 PhotoStimulation,0.4166666666666667 RacketSports,0.9013157894736842 +STEW,0.8448372615039281 SelfRegulationSCP1,0.8907849829351536 Skoda,0.960382030420941 SpokenArabicDigits,0.9881764438381082 StandWalkJump,0.4 TactileTextureRecognition,1.0 +Tiselac,0.7703741152679474 UCDHE-Rowing-MC,0.7886363636363637 UCIActivity,0.9862557267805081 UIPRMD-DS-C,1.0 diff --git a/results/multiverse/HC2/HC2_auroc.csv b/results/multiverse/HC2/HC2_auroc.csv index 0abee34..f207251 100644 --- a/results/multiverse/HC2/HC2_auroc.csv +++ b/results/multiverse/HC2/HC2_auroc.csv @@ -52,11 +52,13 @@ PenDigits,0.9994088617415299 PhonemeSpectra,0.9118050226287818 PhotoStimulation,0.5684676434676434 RacketSports,0.9745211293715941 +STEW,0.9225323915045203 SelfRegulationSCP1,0.9619327182927966 Skoda,0.9969445738605183 SpokenArabicDigits,0.9995223846604337 StandWalkJump,0.5133333333333333 TactileTextureRecognition,1.0 +Tiselac,0.9562308908810502 UCDHE-Rowing-MC,0.9547150195459019 UCIActivity,0.9998589849536378 UIPRMD-DS-C,1.0 diff --git a/results/multiverse/HC2/HC2_balacc.csv b/results/multiverse/HC2/HC2_balacc.csv index c7a3756..7d4ae29 100644 --- a/results/multiverse/HC2/HC2_balacc.csv +++ b/results/multiverse/HC2/HC2_balacc.csv @@ -52,11 +52,13 @@ PenDigits,0.9801359297644916 PhonemeSpectra,0.31409028727770183 PhotoStimulation,0.3333333333333333 RacketSports,0.9093023255813953 +STEW,0.8448372615039281 SelfRegulationSCP1,0.8908070077346006 Skoda,0.9539202462208749 SpokenArabicDigits,0.9881797426317975 StandWalkJump,0.4000000000000001 TactileTextureRecognition,1.0 +Tiselac,0.5605762982066498 UCDHE-Rowing-MC,0.7924444444444444 UCIActivity,0.9866506989269137 UIPRMD-DS-C,1.0 diff --git a/results/multiverse/HC2/HC2_f1.csv b/results/multiverse/HC2/HC2_f1.csv index 25b3f60..a9bdc1e 100644 --- a/results/multiverse/HC2/HC2_f1.csv +++ b/results/multiverse/HC2/HC2_f1.csv @@ -52,11 +52,13 @@ PenDigits,0.9796541213707934 PhonemeSpectra,0.3059648052811076 PhotoStimulation,0.25510204081632654 RacketSports,0.9012583564979668 +STEW,0.8410462776659959 SelfRegulationSCP1,0.891156462585034 Skoda,0.9606548944216594 SpokenArabicDigits,0.9881454126634625 StandWalkJump,0.38888888888888895 TactileTextureRecognition,1.0 +Tiselac,0.7538778677818783 UCDHE-Rowing-MC,0.7867549047609365 UCIActivity,0.9861188502098307 UIPRMD-DS-C,1.0 diff --git a/results/multiverse/HC2/HC2_logloss.csv b/results/multiverse/HC2/HC2_logloss.csv index cee6115..961d4ad 100644 --- a/results/multiverse/HC2/HC2_logloss.csv +++ b/results/multiverse/HC2/HC2_logloss.csv @@ -52,11 +52,13 @@ PenDigits,0.13733158187254543 PhonemeSpectra,2.35291964631169 PhotoStimulation,1.1126576647106405 RacketSports,0.37416977911349175 +STEW,0.4523506267538209 SelfRegulationSCP1,0.2967984397676627 Skoda,0.2823476491551809 SpokenArabicDigits,0.18906090756363803 StandWalkJump,1.110307611747587 TactileTextureRecognition,0.08844470538136215 +Tiselac,0.7204977347835975 UCDHE-Rowing-MC,0.7059100588662615 UCIActivity,0.20648496301861372 UIPRMD-DS-C,0.22566949629880087 diff --git a/results/multiverse/HC2/HC2_sensitivity.csv b/results/multiverse/HC2/HC2_sensitivity.csv index f46f301..8508e37 100644 --- a/results/multiverse/HC2/HC2_sensitivity.csv +++ b/results/multiverse/HC2/HC2_sensitivity.csv @@ -52,11 +52,13 @@ PenDigits,0.9797026872498571 PhonemeSpectra,0.31404712198031615 PhotoStimulation,0.4166666666666667 RacketSports,0.9013157894736842 +STEW,0.8209876543209876 SelfRegulationSCP1,0.8972602739726028 Skoda,0.960382030420941 SpokenArabicDigits,0.9881764438381082 StandWalkJump,0.4 TactileTextureRecognition,1.0 +Tiselac,0.7703741152679474 UCDHE-Rowing-MC,0.7886363636363637 UCIActivity,0.9862557267805081 UIPRMD-DS-C,1.0 diff --git a/results/multiverse/HC2/HC2_specificity.csv b/results/multiverse/HC2/HC2_specificity.csv index 1f1f8a5..fa69522 100644 --- a/results/multiverse/HC2/HC2_specificity.csv +++ b/results/multiverse/HC2/HC2_specificity.csv @@ -52,11 +52,13 @@ PenDigits,0.9797026872498571 PhonemeSpectra,0.31404712198031615 PhotoStimulation,0.4166666666666667 RacketSports,0.9013157894736842 +STEW,0.8686868686868687 SelfRegulationSCP1,0.8843537414965986 Skoda,0.960382030420941 SpokenArabicDigits,0.9881764438381082 StandWalkJump,0.4 TactileTextureRecognition,1.0 +Tiselac,0.7703741152679474 UCDHE-Rowing-MC,0.7886363636363637 UCIActivity,0.9862557267805081 UIPRMD-DS-C,1.0 diff --git a/results/multiverse/TDE/TDE_accuracy.csv b/results/multiverse/TDE/TDE_accuracy.csv index c734d40..de46f31 100644 --- a/results/multiverse/TDE/TDE_accuracy.csv +++ b/results/multiverse/TDE/TDE_accuracy.csv @@ -52,6 +52,7 @@ PenDigits,0.8847913093196113 PhonemeSpectra,0.2496271995228154 PhotoStimulation,0.3888888888888889 RacketSports,0.8355263157894737 +STEW,0.7063692480359147 SelfRegulationSCP1,0.8191126279863481 Skoda,0.9204103289706402 SpokenArabicDigits,0.9195088676671214 diff --git a/results/multiverse/TDE/TDE_auroc.csv b/results/multiverse/TDE/TDE_auroc.csv index 5ce469b..c419ea9 100644 --- a/results/multiverse/TDE/TDE_auroc.csv +++ b/results/multiverse/TDE/TDE_auroc.csv @@ -52,6 +52,7 @@ PenDigits,0.9875063182374243 PhonemeSpectra,0.8266142054522128 PhotoStimulation,0.5783821733821735 RacketSports,0.957631859177761 +STEW,0.7749237606139964 SelfRegulationSCP1,0.9258456807380486 Skoda,0.9952000827469243 SpokenArabicDigits,0.9933580427386813 diff --git a/results/multiverse/TDE/TDE_balacc.csv b/results/multiverse/TDE/TDE_balacc.csv index d3a9758..b6e682d 100644 --- a/results/multiverse/TDE/TDE_balacc.csv +++ b/results/multiverse/TDE/TDE_balacc.csv @@ -52,6 +52,7 @@ PenDigits,0.8848021308180737 PhonemeSpectra,0.24968606404994917 PhotoStimulation,0.3111111111111111 RacketSports,0.8502906976744187 +STEW,0.7063692480359147 SelfRegulationSCP1,0.8195415152362315 Skoda,0.9120942331520815 SpokenArabicDigits,0.9195205479452054 diff --git a/results/multiverse/TDE/TDE_f1.csv b/results/multiverse/TDE/TDE_f1.csv index bb6086f..7b5f385 100644 --- a/results/multiverse/TDE/TDE_f1.csv +++ b/results/multiverse/TDE/TDE_f1.csv @@ -52,6 +52,7 @@ PenDigits,0.884181517464376 PhonemeSpectra,0.23874020317394176 PhotoStimulation,0.24305555555555555 RacketSports,0.8335474101921471 +STEW,0.7216385157600744 SelfRegulationSCP1,0.8389057750759878 Skoda,0.921771233546782 SpokenArabicDigits,0.9181939983167722 diff --git a/results/multiverse/TDE/TDE_logloss.csv b/results/multiverse/TDE/TDE_logloss.csv index eff47aa..7bb82d4 100644 --- a/results/multiverse/TDE/TDE_logloss.csv +++ b/results/multiverse/TDE/TDE_logloss.csv @@ -52,6 +52,7 @@ PenDigits,0.6939331589120361 PhonemeSpectra,6.517339029931114 PhotoStimulation,1.1283870794760338 RacketSports,0.41070629899733835 +STEW,0.5864372029017902 SelfRegulationSCP1,0.5171146591886021 Skoda,0.1875074463467951 SpokenArabicDigits,0.4570068697595056 diff --git a/results/multiverse/TDE/TDE_sensitivity.csv b/results/multiverse/TDE/TDE_sensitivity.csv index d2eade9..8121f01 100644 --- a/results/multiverse/TDE/TDE_sensitivity.csv +++ b/results/multiverse/TDE/TDE_sensitivity.csv @@ -52,6 +52,7 @@ PenDigits,0.8847913093196113 PhonemeSpectra,0.2496271995228154 PhotoStimulation,0.3888888888888889 RacketSports,0.8355263157894737 +STEW,0.7612233445566778 SelfRegulationSCP1,0.9452054794520548 Skoda,0.9204103289706402 SpokenArabicDigits,0.9195088676671214 diff --git a/results/multiverse/TDE/TDE_specificity.csv b/results/multiverse/TDE/TDE_specificity.csv index 5b5e9ec..ceb0314 100644 --- a/results/multiverse/TDE/TDE_specificity.csv +++ b/results/multiverse/TDE/TDE_specificity.csv @@ -52,6 +52,7 @@ PenDigits,0.8847913093196113 PhonemeSpectra,0.2496271995228154 PhotoStimulation,0.3888888888888889 RacketSports,0.8355263157894737 +STEW,0.6515151515151515 SelfRegulationSCP1,0.6938775510204082 Skoda,0.9204103289706402 SpokenArabicDigits,0.9195088676671214 diff --git a/results/multiverse/TimesURL/TimesURL_accuracy.csv b/results/multiverse/TimesURL/TimesURL_accuracy.csv index 28e7b38..17f210b 100644 --- a/results/multiverse/TimesURL/TimesURL_accuracy.csv +++ b/results/multiverse/TimesURL/TimesURL_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.6840804800564773 DuckDuckGeese,0.5 ERing,0.8851851851851852 EigenWorms,0.8549618320610687 +EmoPain,0.8366197183098592 Epilepsy,0.9420289855072463 EthanolConcentration,0.23954372623574144 EyesOpenShut,0.5 diff --git a/results/multiverse/TimesURL/TimesURL_auroc.csv b/results/multiverse/TimesURL/TimesURL_auroc.csv index 5d44e67..d6ed61b 100644 --- a/results/multiverse/TimesURL/TimesURL_auroc.csv +++ b/results/multiverse/TimesURL/TimesURL_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7748470948012232 DuckDuckGeese,0.7725 ERing,0.9819753086419752 EigenWorms,0.9485469975641881 +EmoPain,0.8388880647503131 Epilepsy,0.9827082206548413 EthanolConcentration,0.5270873799468113 EyesOpenShut,0.4331065759637188 diff --git a/results/multiverse/TimesURL/TimesURL_balacc.csv b/results/multiverse/TimesURL/TimesURL_balacc.csv index 05a2ad2..2891a50 100644 --- a/results/multiverse/TimesURL/TimesURL_balacc.csv +++ b/results/multiverse/TimesURL/TimesURL_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.6841642943435045 DuckDuckGeese,0.5 ERing,0.8851851851851853 EigenWorms,0.8226472078645992 +EmoPain,0.48561151079136694 Epilepsy,0.9441573926868044 EthanolConcentration,0.23933566433566433 EyesOpenShut,0.5 diff --git a/results/multiverse/TimesURL/TimesURL_f1.csv b/results/multiverse/TimesURL/TimesURL_f1.csv index 6988ae1..519c325 100644 --- a/results/multiverse/TimesURL/TimesURL_f1.csv +++ b/results/multiverse/TimesURL/TimesURL_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7446504992867332 DuckDuckGeese,0.4913478260869566 ERing,0.8852171572332718 EigenWorms,0.8520288596624473 +EmoPain,0.8150529324542688 Epilepsy,0.9418486609229948 EthanolConcentration,0.2415233857922727 EyesOpenShut,0.6666666666666666 diff --git a/results/multiverse/TimesURL/TimesURL_logloss.csv b/results/multiverse/TimesURL/TimesURL_logloss.csv index 769b055..230cb78 100644 --- a/results/multiverse/TimesURL/TimesURL_logloss.csv +++ b/results/multiverse/TimesURL/TimesURL_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,0.6811406478510141 DuckDuckGeese,1.5354294896775442 ERing,0.5886295866172955 EigenWorms,0.6568276741658519 +EmoPain,0.63623861409052 Epilepsy,0.4325868494205753 EthanolConcentration,1.9707683257768949 EyesOpenShut,1.1269007394531552 diff --git a/results/multiverse/TimesURL/TimesURL_sensitivity.csv b/results/multiverse/TimesURL/TimesURL_sensitivity.csv index da22960..05bb815 100644 --- a/results/multiverse/TimesURL/TimesURL_sensitivity.csv +++ b/results/multiverse/TimesURL/TimesURL_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.9216101694915254 DuckDuckGeese,0.5 ERing,0.8851851851851852 EigenWorms,0.8549618320610687 +EmoPain,0.8366197183098592 Epilepsy,0.9420289855072463 EthanolConcentration,0.23954372623574144 EyesOpenShut,1.0 diff --git a/results/multiverse/TimesURL/TimesURL_specificity.csv b/results/multiverse/TimesURL/TimesURL_specificity.csv index 6ae92ac..2487839 100644 --- a/results/multiverse/TimesURL/TimesURL_specificity.csv +++ b/results/multiverse/TimesURL/TimesURL_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.44671841919548344 DuckDuckGeese,0.5 ERing,0.8851851851851852 EigenWorms,0.8549618320610687 +EmoPain,0.8366197183098592 Epilepsy,0.9420289855072463 EthanolConcentration,0.23954372623574144 EyesOpenShut,0.0 diff --git a/results/multiverse/XCM/XCM_accuracy.csv b/results/multiverse/XCM/XCM_accuracy.csv index fcddf01..12f5b88 100644 --- a/results/multiverse/XCM/XCM_accuracy.csv +++ b/results/multiverse/XCM/XCM_accuracy.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7518531591951995 DuckDuckGeese,0.54 ERing,0.3037037037037037 EigenWorms,0.5038167938931297 +EmoPain,0.6957746478873239 Epilepsy,0.9130434782608695 EthanolConcentration,0.2889733840304182 EyesOpenShut,0.5 diff --git a/results/multiverse/XCM/XCM_auroc.csv b/results/multiverse/XCM/XCM_auroc.csv index e1ed21e..604df0a 100644 --- a/results/multiverse/XCM/XCM_auroc.csv +++ b/results/multiverse/XCM/XCM_auroc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7769353372486634 DuckDuckGeese,0.8835 ERing,0.9387489711934157 EigenWorms,0.8730167577171467 +EmoPain,0.8743557734499935 Epilepsy,0.9757065171400083 EthanolConcentration,0.5558911771202598 EyesOpenShut,0.2380952380952381 diff --git a/results/multiverse/XCM/XCM_balacc.csv b/results/multiverse/XCM/XCM_balacc.csv index 9211a6b..6569b6e 100644 --- a/results/multiverse/XCM/XCM_balacc.csv +++ b/results/multiverse/XCM/XCM_balacc.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7519220303099171 DuckDuckGeese,0.54 ERing,0.30370370370370375 EigenWorms,0.36 +EmoPain,0.5431706247904902 Epilepsy,0.9141494435612083 EthanolConcentration,0.28805361305361304 EyesOpenShut,0.5 diff --git a/results/multiverse/XCM/XCM_f1.csv b/results/multiverse/XCM/XCM_f1.csv index 2576c12..6a5c16f 100644 --- a/results/multiverse/XCM/XCM_f1.csv +++ b/results/multiverse/XCM/XCM_f1.csv @@ -22,6 +22,7 @@ CrowdSourced,0.7923190546528803 DuckDuckGeese,0.5323809523809524 ERing,0.1517735755778025 EigenWorms,0.40159309658148024 +EmoPain,0.7492379034906649 Epilepsy,0.9112318840579711 EthanolConcentration,0.24591210249853798 EyesOpenShut,0.6666666666666666 diff --git a/results/multiverse/XCM/XCM_logloss.csv b/results/multiverse/XCM/XCM_logloss.csv index e9de0b4..c1ab07c 100644 --- a/results/multiverse/XCM/XCM_logloss.csv +++ b/results/multiverse/XCM/XCM_logloss.csv @@ -22,6 +22,7 @@ CrowdSourced,5.681748011476673 DuckDuckGeese,1.4476585961906254 ERing,1.5511330924068825 EigenWorms,2.7894721126602673 +EmoPain,1.4895746842659396 Epilepsy,0.3715004381698749 EthanolConcentration,1.529273939296933 EyesOpenShut,4.248528533555482 diff --git a/results/multiverse/XCM/XCM_sensitivity.csv b/results/multiverse/XCM/XCM_sensitivity.csv index e9242f0..b18a5d3 100644 --- a/results/multiverse/XCM/XCM_sensitivity.csv +++ b/results/multiverse/XCM/XCM_sensitivity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.9470338983050848 DuckDuckGeese,0.54 ERing,0.3037037037037037 EigenWorms,0.5038167938931297 +EmoPain,0.6957746478873239 Epilepsy,0.9130434782608695 EthanolConcentration,0.2889733840304182 EyesOpenShut,1.0 diff --git a/results/multiverse/XCM/XCM_specificity.csv b/results/multiverse/XCM/XCM_specificity.csv index bc472f0..dcf3a82 100644 --- a/results/multiverse/XCM/XCM_specificity.csv +++ b/results/multiverse/XCM/XCM_specificity.csv @@ -22,6 +22,7 @@ CrowdSourced,0.5568101623147494 DuckDuckGeese,0.54 ERing,0.3037037037037037 EigenWorms,0.5038167938931297 +EmoPain,0.6957746478873239 Epilepsy,0.9130434782608695 EthanolConcentration,0.2889733840304182 EyesOpenShut,0.0 diff --git a/results/multiverse/datasets.html b/results/multiverse/datasets.html index 9ede74a..ea26c9c 100644 --- a/results/multiverse/datasets.html +++ b/results/multiverse/datasets.html @@ -60,7 +60,7 @@ details { margin-top: .6rem; } summary { cursor: pointer; color: var(--accent); } code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: .9em; } -tr.nosignal td { background: rgba(214, 158, 46, .16); }tr.saturated td { background: rgba(56, 161, 105, .14); }

Multiverse-core datasets: accuracy

64 datasets · accuracy · best of up to 24 estimators against the Dummy baseline · built 2026-09-11

DatasetDummyMedianBestBest estimatorGain over dummySpreadEstimators
KINECAL-QSEO0.94120.94120.9412Arsenal0.00000.647124
BIDMC32SpO2_disc0.71530.65740.7203ROCKET0.00500.190124
Locust20220.91120.90820.9206MRHydra0.00940.052423
Heartbeat0.72200.74880.7854CIF0.06340.126824
HouseholdPowerConsumption2_disc0.72160.76530.7872DisjointCNN0.06560.097724
MotorImagery0.50000.51000.5900HC20.09000.130024
AutomotiveRoadTrials0.75320.78570.8442CIF0.09090.233824
EyesOpenShut0.50000.50000.5952STSF0.09520.190524
EmoPain0.78310.83380.8845Catch220.10140.202821
AppliancesEnergy_disc0.80950.82140.9286DrCIF0.11900.428624
BeijingPM10Quality_disc0.71120.82420.8417STSF0.13050.126024
Alzheimers0.41860.37210.5581MRHydra0.13950.302323
PhotoStimulation0.41670.38890.5833ROCKET0.16670.388923
FaceDetection0.50000.62510.6850H-InceptionTime0.18500.170524
BeijingPM25Quality_disc0.69770.87510.8879ConvTran0.19020.128824
AtrialFibrillation0.33330.26670.5333TS2Vec0.20000.466724
HouseholdPowerConsumption1_disc0.77840.91110.9825STSF0.20410.864424
LowCost0.50000.63170.7300TSF0.23000.248324
BoneProbAgeGroup0.47640.64380.7124H-InceptionTime0.23600.184324
StandWalkJump0.33330.40000.6000MRHydra0.26670.400024
CrowdSourced0.50020.71900.7734LITETime-MV0.27320.176824
BenzeneConcentration_disc0.68970.81070.9768STSF0.28700.577024
FordChallenge0.62320.88430.9360QUANT0.31280.312824
BIDMC32HR_disc0.65070.80120.9637RIST0.31300.635324
STEW0.50000.74020.8385Arsenal0.33850.210022
BoneIntensitiesAgeGroup0.47640.79890.8202HC20.34380.256224
PhonemeSpectra0.02560.27100.3746H-InceptionTime0.34890.291124
KERAAL-RTK0.57140.78570.9286HC20.35710.571424
LSST0.31510.62900.6752HC20.36010.452124
HandMovementDirection0.20270.41220.6081TSF0.40540.351424
DuckDuckGeese0.20000.46000.6400H-InceptionTime0.44000.480024
SelfRegulationSCP10.50170.85320.9454MRHydra0.44370.392524
IEEEPPG_disc0.26050.45480.7078ConvTran0.44730.438324
AsphaltRegularityCoordinates0.50730.97870.9947H-InceptionTime0.48740.061324
MindReading0.23120.52830.7243LITETime-MV0.49310.385924
EthanolConcentration0.25100.43350.7490STC0.49810.513324
UIPRMD-DS-C0.50000.83331.0000Catch220.50000.388924
WISDM0.36640.86580.8965MRHydra0.53000.131024
Blink0.44440.98671.0000Arsenal0.55560.428924
EigenWorms0.41980.86260.9771MRHydra0.55730.557323
KIMORE-PR-C0.14290.42860.7143LITETime-MV0.57140.571424
AsphaltObstaclesCoordinates0.28390.81970.8670MRHydra0.58310.199524
CounterMovementJump0.33520.75700.9274Arsenal0.59220.458124
Handwriting0.03760.37290.6529H-InceptionTime0.61530.478824
RacketSports0.28290.87830.9079RDST0.62500.125024
UCDHE-Rowing-MC0.20450.73410.8295PatchMTSC0.62500.259124
USCActivity0.11380.69240.7473HC20.63340.166422
IRDS-SFL0.20690.79310.8966XCM0.68970.482824
Skoda0.23560.94550.9646H-InceptionTime0.72900.119924
Epilepsy0.26810.98191.0000HC20.73190.101424
Tiselac0.06280.81360.8373STSF0.77450.204420
MotionSenseHAR0.20380.98871.0000DrCIF0.79620.101924
NATOPS0.16670.89170.9667LITETime-MV0.80000.155624
UCIActivity0.19160.97690.9983LITETime-MV0.80670.174924
UWaveGestureLibrary0.12500.90940.9406Arsenal0.81560.553124
ERing0.16670.94070.9963MRHydra0.82960.692624
PEMS-SF0.11560.86991.0000CIF0.88440.317924
SpokenArabicDigits0.10000.98160.9945DisjointCNN0.89450.129124
Libras0.06670.88890.9722RIST0.90560.338924
JapaneseVowels0.08380.97030.9946LITETime-MV0.91080.208124
Cricket0.08330.97921.0000Arsenal0.91670.069424
CharacterTrajectories0.06480.98960.9958H-InceptionTime0.93110.044624
TactileTextureRecognition0.05140.99851.0000H-InceptionTime0.94860.234924
ArticularyWordRecognition0.04000.98000.9933Arsenal0.95330.050024

One row per dataset. Dummy is the no-skill floor. Median, best and spread are over the other estimators, so the baseline cannot flatter them. Gain over dummy is best minus dummy, how much skill was found at all; spread is best minus worst, how much the choice of estimator mattered. The two answer different questions, and a single range would conflate them.

3 of 64 datasets gained 0.05 or less over the baseline (shaded amber) and 14 have a best of 0.99 or more (shaded green). Both separate estimators poorly, for opposite reasons. Best is a maximum over many estimators, so it is optimistic by construction: read it as what the archive can currently do on a problem, not as what any one method delivers.