Update README headline + 3x3 cross table; add cross_3x3 tooling to scripts/aggregate
This commit is contained in:
50
README.md
50
README.md
@@ -15,15 +15,28 @@ JANUS is the first NIDS method to use Flow Matching as the training paradigm in
|
||||
|
||||
3-seed mean ± std AUROC. Selection-bias-free Mahalanobis-OAS aggregator on the 10-d JANUS score vector, fit on benign val only.
|
||||
|
||||
### Within-dataset
|
||||
|
||||
| Task | Shafir 2026 SOTA | **JANUS** | Δ |
|
||||
|---|---|---|---|
|
||||
| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9908 ± 0.0012** | **+0.118** |
|
||||
| CICIDS2017 within | 0.9303 | **0.9845 ± 0.0030** | **+0.054** |
|
||||
| CICDDoS2019 within | 0.93 | **0.9913 ± 0.0009** | **+0.061** |
|
||||
| CICIDS2017 → CICDDoS2019 cross | 0.89 | **0.9594 ± 0.0046** | **+0.07** |
|
||||
| CICDDoS2019 → CICIDS2017 reverse cross | 0.93 | **0.9301 ± 0.0122** | matches |
|
||||
| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9909 ± 0.0013** | **+0.118** |
|
||||
| CICIDS2017 within | 0.9303 | **0.9826 ± 0.0035** | **+0.052** |
|
||||
| CICDDoS2019 within | 0.93 | **0.9918 ± 0.0005** | **+0.062** |
|
||||
| CICIoT2023 within | F1=0.9951 (different metric) | 0.9590 ± 0.0022 (AUROC) | N/A — metric mismatch |
|
||||
|
||||
3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIDS2017→CICDDoS2019 cross also beats; reverse direction matches. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table.
|
||||
3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table.
|
||||
|
||||
### 3×3 cross-dataset transfer matrix
|
||||
|
||||
Source (rows) trained on 10K benign of source dataset; target (columns) tested on full target benign + **all** target attacks. Aggregator fit on target benign val only — no attack labels at any stage. Diagonal italic = within-dataset.
|
||||
|
||||
| Source ↓ / Target → | CICIDS17 | CICDDoS19 | CICIoT23 |
|
||||
|---|---|---|---|
|
||||
| **CICIDS17** | _0.9826 ± 0.0035_ | **0.9690 ± 0.0047** | 0.8698 ± 0.0031 |
|
||||
| **CICDDoS19** | 0.9413 ± 0.0212 | _0.9918 ± 0.0005_ | 0.8767 ± 0.0068 |
|
||||
| **CICIoT23** | 0.9394 ± 0.0063 | 0.9030 ± 0.0075 | _0.9590 ± 0.0022_ |
|
||||
|
||||
Forward CICIDS17→CICDDoS19 (0.969) beats Shafir 0.89 by **+0.08**; reverse CICDDoS19→CICIDS17 (0.941) approximately matches Shafir 0.93. CICIoT23 is hardest both as source and target — its IoT-protocol diversity makes the "benign of source ≈ benign of target" assumption brittle. Full table at `artifacts/route_comparison/CROSS_MATRIX_3x3.md`.
|
||||
|
||||
## Layout
|
||||
|
||||
@@ -44,15 +57,19 @@ scripts/ Workspace-level pcap → artifact pipeline,
|
||||
download/ UNB/CIC dataset downloaders.
|
||||
baselines/ Third-party baseline runners (Kitsune, Shafir-NF,
|
||||
Anomaly-Transformer).
|
||||
aggregate/ Mahalanobis-OAS score-router + cross-matrix
|
||||
orchestration. aggregate_score_router.py is the
|
||||
deployable score path; run_cross_3x3.sh +
|
||||
cross_3x3_table.py produce the cross matrix.
|
||||
tests/ Data-contract unit tests.
|
||||
```
|
||||
|
||||
The following directories are **gitignored** (live on the dev box, not in the repo):
|
||||
|
||||
```
|
||||
artifacts/ All run outputs (checkpoints, eval JSONs, score npzs,
|
||||
figures). Score-router aggregator at
|
||||
artifacts/route_comparison/aggregate_score_router.py.
|
||||
artifacts/ All run outputs (checkpoints, eval JSONs, score
|
||||
npzs, figures). Per-(dataset × seed) model dirs at
|
||||
artifacts/route_comparison/janus_<ds>_seed<N>/.
|
||||
datasets/ Raw + processed datasets (~1 TB).
|
||||
baselines/ Third-party baseline forks (Kitsune-py,
|
||||
Anomaly-Transformer, ConMD, ganomaly, TIPSO-GAN, ...).
|
||||
@@ -85,11 +102,18 @@ uv run --no-sync python train.py --config configs/cicids2017_seed42.yaml
|
||||
uv run --no-sync python eval_phase1.py \
|
||||
--model-dir <model_dir> --out-dir <eval_dir>
|
||||
|
||||
# Cross-dataset evaluation
|
||||
# Single cross-dataset eval
|
||||
uv run --no-sync python eval_cross.py \
|
||||
--src-model <src_model_dir> \
|
||||
--tgt-data datasets/<tgt>/processed/ \
|
||||
--model-dir <src_model_dir> \
|
||||
--target-store datasets/<tgt>/processed/full_store \
|
||||
--target-flows datasets/<tgt>/processed/flows.parquet \
|
||||
--target-flow-features datasets/<tgt>/processed/flow_features.parquet \
|
||||
--benign-label normal --n-benign 10000 --n-attack 1000000 \
|
||||
--out <result.json>
|
||||
|
||||
# 3×3 cross matrix (6 off-diagonal directions × 3 seeds, 2-GPU parallel)
|
||||
bash ../scripts/aggregate/run_cross_3x3.sh
|
||||
uv run --no-sync python ../scripts/aggregate/cross_3x3_table.py
|
||||
```
|
||||
|
||||
JANUS hyper-parameters (locked in `Mixed_CFM/configs/<dataset>_seed*.yaml`):
|
||||
@@ -122,7 +146,7 @@ d²(s) = (s − μ)ᵀ Σ⁻¹ (s − μ), where (μ, Σ) come from sklearn.c
|
||||
fit on benign val ONLY (no attack labels).
|
||||
```
|
||||
|
||||
Reference implementation: `artifacts/route_comparison/aggregate_score_router.py` (artifacts/ is gitignored; the script lives on the dev box).
|
||||
Reference implementation: `scripts/aggregate/aggregate_score_router.py`. It reads `artifacts/route_comparison/janus_<ds>_seed*/phase1_scores.npz` and `artifacts/route_comparison/cross/janus_seed*_<src>_to_<tgt>.npz`, then writes `artifacts/route_comparison/SCORE_ROUTER.md` (within-dataset rows) and `artifacts/route_comparison/CROSS_MATRIX_3x3.md` (cross matrix, via `cross_3x3_table.py`).
|
||||
|
||||
## Tests
|
||||
|
||||
|
||||
Reference in New Issue
Block a user