Merge RESULTS_THRESHOLDED.md into RESULTS.md (section D); rewrite README.md for JANUS
This commit is contained in:
57
RESULTS.md
57
RESULTS.md
@@ -159,10 +159,10 @@ This is **+0.31 over our own legacy memory baseline of 0.62**. The "main
|
||||
attack direction" recorded in `reverse_cross_score_redirection_2026_04_25`
|
||||
is now substantially solved.
|
||||
|
||||
Thresholded F1 / Precision / Recall / TPR@FPR (unsupervised protocol, τ from
|
||||
benign-val percentile) are reported separately in `RESULTS_THRESHOLDED.md`.
|
||||
Headline thresholded numbers: CICDDoS2019 within `terminal_norm` F1=0.993 ± 0.001
|
||||
at τ=P95; cross `terminal_norm` F1=0.632 ± 0.051 at τ=P95 (precision ≈ 0.95, recall ≈ 0.47).
|
||||
Thresholded F1 / Precision / Recall / TPR@FPR under the unsupervised threshold
|
||||
protocol are reported in **section D** below. Headline thresholded numbers:
|
||||
CICDDoS2019 within `terminal_norm` F1=0.993 ± 0.001 at τ=P95; cross `terminal_norm`
|
||||
F1=0.632 ± 0.051 at τ=P95 (precision ≈ 0.95, recall ≈ 0.47).
|
||||
|
||||
> **Note on cross-dataset baseline**: Shafir's Table IX is asymmetric.
|
||||
> The IDS2017→DDoS2019 direction (which we evaluate) reads **0.89**, not
|
||||
@@ -175,6 +175,52 @@ at τ=P95; cross `terminal_norm` F1=0.632 ± 0.051 at τ=P95 (precision ≈ 0.95
|
||||
> Single-policy σ=0.6 also beats Shafir on 4/4. Full 4×2 sensitivity table
|
||||
> in `artifacts/sigma_validation.md`.
|
||||
|
||||
### D. Thresholded operating-point metrics
|
||||
|
||||
⚠️ Numbers in this section are from the **Unified_CFM legacy** recipe (σ=0.1
|
||||
within, σ=0.6 cross, λ=0.3, single fixed score). Equivalent thresholded
|
||||
numbers for current JANUS + Mahalanobis-OAS have not been recomputed yet;
|
||||
the AUROC tables (A/B/C above) are the authoritative JANUS comparison.
|
||||
|
||||
**Protocol**: τ is set from a benign-val half (A); F1 / Precision / Recall /
|
||||
FPR are measured on benign-val half B + attack. AUROC / AUPRC use full
|
||||
benign val + attack. TPR@FPR is measured on the test half. Both percentiles
|
||||
τ ∈ {P95, P99} are reported because they correspond to different operating
|
||||
points and F1 is sensitive to that choice.
|
||||
|
||||
**CICDDoS2019 within** (σ=0.1, λ=0.3):
|
||||
|
||||
| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| `terminal_norm` | 0.9960 ± 0.0011 | 0.9975 ± 0.0008 | 0.9932 ± 0.0012 | 0.9881 ± 0.0015 | 0.9983 ± 0.0008 | 0.0481 ± 0.0061 | 0.9112 ± 0.0402 | 0.9013 ± 0.0540 | 0.9980 ± 0.0014 |
|
||||
| `terminal_flow` | 0.9885 ± 0.0028 | 0.9918 ± 0.0017 | 0.9788 ± 0.0086 | 0.9868 ± 0.0009 | 0.9710 ± 0.0163 | 0.0517 ± 0.0030 | 0.7752 ± 0.0128 | 0.6052 ± 0.0347 | 0.9697 ± 0.0169 |
|
||||
|
||||
**CICIDS2017 → CICDDoS2019 cross** (σ=0.6, λ=0.3):
|
||||
|
||||
| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| `terminal_norm` | 0.9109 ± 0.0032 | 0.8974 ± 0.0047 | 0.6321 ± 0.0513 | 0.9545 ± 0.0045 | 0.4745 ± 0.0550 | 0.0441 ± 0.0011 | 0.4202 ± 0.0171 | 0.2685 ± 0.0139 | 0.4940 ± 0.0399 |
|
||||
| `terminal_flow` | 0.9197 ± 0.0036 | 0.8957 ± 0.0086 | 0.6324 ± 0.0585 | 0.9517 ± 0.0055 | 0.4762 ± 0.0639 | 0.0469 ± 0.0019 | 0.4028 ± 0.0049 | 0.2534 ± 0.0039 | 0.4776 ± 0.0636 |
|
||||
|
||||
**Reading**:
|
||||
|
||||
- *Within-dataset CICDDoS2019* saturates: at τ=P95 F1 ≈ 0.99 with balanced
|
||||
precision and recall ≈ 0.99; at τ=P99 (≈1% FPR) F1 ≈ 0.91 with TPR@1%FPR
|
||||
≈ 0.90. The model is a working detector at fixed thresholds, not just an
|
||||
AUROC artifact.
|
||||
- *Cross-dataset CICIDS2017→CICDDoS2019* keeps AUROC ≈ 0.91 but at fixed τ
|
||||
shows precision ≈ 0.95 / recall ≈ 0.50 at P95 and ≈0.27 at 1% FPR — the
|
||||
cross-dataset domain shift compresses the score gap, so source-calibrated
|
||||
thresholds are conservative on target. **AUROC alone overstates
|
||||
deployability cross-dataset; thresholded numbers are the honest figure.**
|
||||
|
||||
**TIPSO-GAN comparability**: TIPSO-GAN's CICDDoS2019 F1 ≈ 0.99 is reported
|
||||
under a **supervised** protocol (model has seen attack examples). Our F1
|
||||
≈ 0.99 on CICDDoS2019 within is achieved under the **unsupervised** protocol
|
||||
(benign-only training, threshold from benign val), which is the strictly
|
||||
harder setting. Direct F1 numerical equivalence; protocol asymmetry is in
|
||||
our favor.
|
||||
|
||||
## Methodological contribution: `flow_consistency` diagnostic score
|
||||
|
||||
Phase 2 masked-prediction consistency loss unlocks a new score that is
|
||||
@@ -319,9 +365,6 @@ artifacts.
|
||||
|
||||
## Source artifacts
|
||||
|
||||
- `RESULTS_THRESHOLDED.md` — F1 / Precision / Recall / TPR@FPR under unsupervised
|
||||
threshold protocol (τ = benign-val P95/P99) for CICDDoS2019 within and
|
||||
CICIDS2017→CICDDoS2019 cross.
|
||||
- `artifacts/locked_baselines.md` — verified Shafir baselines (PDF inspection trail).
|
||||
- `artifacts/sigma_validation.md` — full 4×2 σ-sensitivity table (σ ∈ {0.1, 0.6} ×
|
||||
4 tasks, 3 seeds each) and per-task σ-selection protocol.
|
||||
|
||||
Reference in New Issue
Block a user