# Unified_CFM vs Baselines — Performance Comparison Table

Live tracking. Last updated: 2026-04-30.

Two reproduction modes are tracked separately:

- **Method reproduction** (≡ "ours-method-only"): we use the baseline's
  model/training algorithm but feed it our 20-d packet-derived canonical flow
  features (or 9-d packet sequences for AT/Kitsune). Tests whether the
  baseline's architecture beats Unified_CFM on the **same data substrate**.

- **True reproduction** (≡ "ours-true-repro"): we use the baseline's full
  pipeline — including their feature engineering (CICFlowMeter CSV /
  AfterImage / image encoding). Tests whether their published numbers
  reproduce on our datasets.

"paper" = the number quoted in the source paper; not run by us.

## Headline AUROC table

Two columns per baseline:
- "ours-method-only" = baseline model on our 20-d / 9-d features, 3-seed mean±std.
- "ours-true-repro"  = baseline's full pipeline (CICFlowMeter CSV + their feature subsets), single seed for now.

Raw AUROC reported. abs() = sign-agnostic max(AUROC, 1−AUROC) for inverted-signal baselines.

| Protocol | **Unified_CFM** | Shafir NF paper | Shafir NF method-only (ours) | **Shafir NF true-repro (ours)** | Kitsune paper | Kitsune Path B method-only (ours) | AT method-only (ours) |
|---|---:|---:|---:|---:|---:|---:|---:|
| ISCXTor2016 within | **0.9945 ± 0.0011** | 0.8731 | 0.9422 ± 0.0075 | **0.7562** [^4] | 0.7800 | 0.5653 ± 0.0226 | 0.4122 ± 0.0503 (abs 0.59) |
| CICIDS2017 within (σ=0.6) | **0.9858 ± 0.0021** | 0.9303 | 0.9256 ± 0.0188 | **0.8678** [^4] | 0.8500 | 0.7023 ± 0.0310 | 0.5009 ± 0.2107 (abs 0.66) |
| CICDDoS2019 within | **0.9960 ± 0.0010** | 0.9300 | 0.8903 ± 0.0386 | 0.5926 [^5] | — | 0.4710 ± 0.0039 | 0.4777 ± 0.3325 (abs 0.75) |
| IDS2017→DDoS2019 forward | 0.9109 ± 0.0032 | 0.8900 | **0.9210 ± 0.0111** | 0.7831 | — | 0.4905 ± 0.0751 | 0.5404 ± 0.1495 (abs 0.63) |
| DDoS2019→IDS2017 reverse | 0.5999 (single) [^1] | **0.9300** | 0.7247 ± 0.0035 | 0.7473 | — | 0.7483 ± 0.0137 | 0.4767 ± 0.2597 (abs 0.70) |
| CICIoT2023 within (single seed) | **0.9618** [^2] | F1=0.9951 [^3] | 0.8996 [^2] | 0.7398 [^4] | — | — | — |

**Bold** = best per row.

[^1]: Reverse `terminal_norm` 0.5999 single-seed. Our **PNA** score on this
    protocol is 0.9089 (3-seed mean) — beats both Shafir NF reproduced (0.7247)
    and Shafir paper (0.93).

[^2]: CICIoT2023 single seed. Our 20-d canonical features. Shafir NF (ours)
    used 5-d Shafir-selected features (HTTPS, Protocol_Type, Magnitude,
    Variance, fin_count) computed from our packets.

[^3]: Shafir paper Table VIII reports F1=0.9951 (not AUROC) on CICIoT2023.
    They used the official CICFlowMeter pipeline on full pcap; not directly
    AUROC-comparable. Our F1@P95 on this protocol = **0.9463** (with our 20-d
    canonical features).

[^4]: Shafir NF true-reproduction uses their **paper-specified SHAP-selected
    feature subsets**: ISCXTor 4 features (Flow IAT Std, Flow Bytes/s,
    Flow Packets/s, Bwd IAT Max — paper §V-A), CICIDS2017 5 features (Bwd
    Packet Length Mean, Fwd Packets/s, ACK Flag Count, Total Length of Bwd
    Packets, Flow Duration — paper §V-B), CICIoT2023 5 features (HTTPS,
    Protocol Type, Magnitude, Variance, fin_count — paper §V-D). Driver
    uses **single NF**; paper headline numbers (0.8731 / 0.9303 / 0.93) come
    from a **2-NF ensemble** which we don't reproduce. Expected single-NF
    underperformance vs ensemble: ~0.05-0.15 AUROC.

[^5]: CICDDoS2019 true-repro uses CICIDS2017 best-5 feature subset (paper
    §V-C: "comparable feature subset shared with CICIDS2017"). Result is
    weak (0.59) — those 5 features are tuned for CICIDS attacks, not DrDoS.
    Paper's 0.93 number is presumably with CICDDoS-specific feature
    selection (not specified in paper).

## CICIoT2023 thresholded F1 table

| Method | flow features | F1@P95 | F1@P99 | TPR@1%FPR |
|---|---|---:|---:|---:|
| **Unified_CFM 20-d canonical** | 20-d packet-derived | **0.9463** | **0.9004** | **0.8208** |
| Unified_CFM Shafir-5 | 5-d Shafir SHAP-selected | 0.9266 | 0.8704 | 0.7726 |
| Shafir NF (ours), Shafir-5 | 5-d Shafir SHAP-selected | 0.9053 | 0.0652 | — |
| Shafir NF (paper) | 5-d via official CICFlowMeter | **0.9951** | — | — |

## Summary by within / cross direction

**Within-dataset (4 protocols)**: Unified_CFM beats Shafir NF reproduction on **4/4** (ISCXTor +0.052, CICIDS +0.060, CICDDoS +0.106, CICIoT2023 +0.062).
**Forward cross (1 protocol)**: Shafir NF (ours) 0.921 narrowly edges Unified_CFM 0.911 (+0.010 — within noise).
**Reverse cross (1 protocol)**: Shafir NF (ours) 0.725 beats Unified_CFM `terminal_norm` 0.600, but Unified_CFM `PNA` 0.909 beats both. Reverse score-of-record for Unified_CFM is PNA, not terminal_norm.

## Sources

- Unified_CFM main: `RESULTS.md`, `RESULTS_THRESHOLDED.md`
- Unified_CFM CICIoT2023: `artifacts/runs/unified_cfm_ciciot2023_2026_04_29/{phase1/phase1_summary.json, phase1/thresholded.json}`
- Unified_CFM PNA on reverse: `artifacts/phase_new_scores_2026_04_29/pna/reverse_cross_seed{42,43,44}.json`
- Shafir NF (ours): `artifacts/baselines/shafir_nf_2026_04_29/summary.md`
- Kitsune (ours): `artifacts/baselines/kitsune_2026_04_29/summary.md`
- Shafir paper baselines: `artifacts/locked_baselines.md`

## Reproduction-mode coverage table

For each baseline, what's the achievable reproduction mode and current status:

| Baseline | Method-only (ours-features) | True-repro (their-features) | Comment |
|---|---|---|---|
| **Shafir NF** | ✅ 15 cells | ✅ 6 cells (single seed, CSV+SHAP-5) | True-repro single-NF; paper headline uses 2-NF ensemble |
| **Kitsune** | ✅ 15 cells (Path B, KitNET on 9-d) | ❌ blocked (Path A) | Path A pcap streaming starts at 1471 pkt/s, slows to 581 pkt/s as AfterImage state grows. CICIDS2017 alone needs 7-9 hours; CICDDoS2019 / CICIoT2023 prohibitive. Single-pcap test on ISCXTor showed only 22 unique 5-tuples per pcap → flow coverage gaps. Documented as infeasible at our data scale. |
| **Anomaly-Transformer** | ✅ 15 cells | n/a | AT has no domain-specific feature pipeline (raw time-series only); method-only IS the faithful reproduction here |
| **ConMD** | not started | not started | Repo is just a 878-line model class — would require writing entire distillation training loop + image-like preprocessing from scratch. ~2 days work, no guaranteed paper-number reproduction. |
| **9 image-AD methods** (FastFlow, PatchCore, PADIM, STFPM, DFKDE, DFM, DRAEM, CFlow, RD4AD via `anomalib`) | not started | not started | Requires flow→image encoder which Shafir didn't open-source. Any reproduction would be "our flow→image + their image-AD model" → method-only at best. ~2 days work. |
| **TSLANet** | n/a | n/a | Vendored repo at the noted commit has no anomaly detection module. Skipped. |
| **ganomaly / RD4AD / STFPM standalone** | not started | not started | Would be subsumed by anomalib path; standalone repos have no native traffic-data support. |
