diff --git a/README.md b/README.md index 728bfe7..20e3515 100644 --- a/README.md +++ b/README.md @@ -15,15 +15,28 @@ JANUS is the first NIDS method to use Flow Matching as the training paradigm in 3-seed mean ± std AUROC. Selection-bias-free Mahalanobis-OAS aggregator on the 10-d JANUS score vector, fit on benign val only. +### Within-dataset + | Task | Shafir 2026 SOTA | **JANUS** | Δ | |---|---|---|---| -| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9908 ± 0.0012** | **+0.118** | -| CICIDS2017 within | 0.9303 | **0.9845 ± 0.0030** | **+0.054** | -| CICDDoS2019 within | 0.93 | **0.9913 ± 0.0009** | **+0.061** | -| CICIDS2017 → CICDDoS2019 cross | 0.89 | **0.9594 ± 0.0046** | **+0.07** | -| CICDDoS2019 → CICIDS2017 reverse cross | 0.93 | **0.9301 ± 0.0122** | matches | +| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9909 ± 0.0013** | **+0.118** | +| CICIDS2017 within | 0.9303 | **0.9826 ± 0.0035** | **+0.052** | +| CICDDoS2019 within | 0.93 | **0.9918 ± 0.0005** | **+0.062** | +| CICIoT2023 within | F1=0.9951 (different metric) | 0.9590 ± 0.0022 (AUROC) | N/A — metric mismatch | -3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIDS2017→CICDDoS2019 cross also beats; reverse direction matches. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table. +3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table. + +### 3×3 cross-dataset transfer matrix + +Source (rows) trained on 10K benign of source dataset; target (columns) tested on full target benign + **all** target attacks. Aggregator fit on target benign val only — no attack labels at any stage. Diagonal italic = within-dataset. + +| Source ↓ / Target → | CICIDS17 | CICDDoS19 | CICIoT23 | +|---|---|---|---| +| **CICIDS17** | _0.9826 ± 0.0035_ | **0.9690 ± 0.0047** | 0.8698 ± 0.0031 | +| **CICDDoS19** | 0.9413 ± 0.0212 | _0.9918 ± 0.0005_ | 0.8767 ± 0.0068 | +| **CICIoT23** | 0.9394 ± 0.0063 | 0.9030 ± 0.0075 | _0.9590 ± 0.0022_ | + +Forward CICIDS17→CICDDoS19 (0.969) beats Shafir 0.89 by **+0.08**; reverse CICDDoS19→CICIDS17 (0.941) approximately matches Shafir 0.93. CICIoT23 is hardest both as source and target — its IoT-protocol diversity makes the "benign of source ≈ benign of target" assumption brittle. Full table at `artifacts/route_comparison/CROSS_MATRIX_3x3.md`. ## Layout @@ -44,15 +57,19 @@ scripts/ Workspace-level pcap → artifact pipeline, download/ UNB/CIC dataset downloaders. baselines/ Third-party baseline runners (Kitsune, Shafir-NF, Anomaly-Transformer). + aggregate/ Mahalanobis-OAS score-router + cross-matrix + orchestration. aggregate_score_router.py is the + deployable score path; run_cross_3x3.sh + + cross_3x3_table.py produce the cross matrix. tests/ Data-contract unit tests. ``` The following directories are **gitignored** (live on the dev box, not in the repo): ``` -artifacts/ All run outputs (checkpoints, eval JSONs, score npzs, - figures). Score-router aggregator at - artifacts/route_comparison/aggregate_score_router.py. +artifacts/ All run outputs (checkpoints, eval JSONs, score + npzs, figures). Per-(dataset × seed) model dirs at + artifacts/route_comparison/janus__seed/. datasets/ Raw + processed datasets (~1 TB). baselines/ Third-party baseline forks (Kitsune-py, Anomaly-Transformer, ConMD, ganomaly, TIPSO-GAN, ...). @@ -85,11 +102,18 @@ uv run --no-sync python train.py --config configs/cicids2017_seed42.yaml uv run --no-sync python eval_phase1.py \ --model-dir --out-dir -# Cross-dataset evaluation +# Single cross-dataset eval uv run --no-sync python eval_cross.py \ - --src-model \ - --tgt-data datasets//processed/ \ + --model-dir \ + --target-store datasets//processed/full_store \ + --target-flows datasets//processed/flows.parquet \ + --target-flow-features datasets//processed/flow_features.parquet \ + --benign-label normal --n-benign 10000 --n-attack 1000000 \ --out + +# 3×3 cross matrix (6 off-diagonal directions × 3 seeds, 2-GPU parallel) +bash ../scripts/aggregate/run_cross_3x3.sh +uv run --no-sync python ../scripts/aggregate/cross_3x3_table.py ``` JANUS hyper-parameters (locked in `Mixed_CFM/configs/_seed*.yaml`): @@ -122,7 +146,7 @@ d²(s) = (s − μ)ᵀ Σ⁻¹ (s − μ), where (μ, Σ) come from sklearn.c fit on benign val ONLY (no attack labels). ``` -Reference implementation: `artifacts/route_comparison/aggregate_score_router.py` (artifacts/ is gitignored; the script lives on the dev box). +Reference implementation: `scripts/aggregate/aggregate_score_router.py`. It reads `artifacts/route_comparison/janus__seed*/phase1_scores.npz` and `artifacts/route_comparison/cross/janus_seed*__to_.npz`, then writes `artifacts/route_comparison/SCORE_ROUTER.md` (within-dataset rows) and `artifacts/route_comparison/CROSS_MATRIX_3x3.md` (cross matrix, via `cross_3x3_table.py`). ## Tests diff --git a/scripts/aggregate/cross_3x3_table.py b/scripts/aggregate/cross_3x3_table.py new file mode 100644 index 0000000..fa839c0 --- /dev/null +++ b/scripts/aggregate/cross_3x3_table.py @@ -0,0 +1,122 @@ +import json +from pathlib import Path +import numpy as np +from sklearn.covariance import OAS +from sklearn.metrics import roc_auc_score + +ROOT = Path("/home/chy/JANUS/artifacts/route_comparison") +CROSS = ROOT / "cross" +DATASETS = ["cicids2017", "cicddos2019", "ciciot2023"] +SEEDS = [42, 43, 44] + + +def load_cell(src, tgt, seed): + if src == tgt: + path = ROOT / f"janus_{src}_seed{seed}/phase1_scores.npz" + prefix_b, prefix_a = "val_", "atk_" + else: + path = CROSS / f"janus_seed{seed}_{src}_to_{tgt}.npz" + prefix_b, prefix_a = "b_", "a_" + z = np.load(path, allow_pickle=True) + keys = sorted( + k.replace(prefix_b, "") + for k in z.files + if k.startswith(prefix_b) and not k.endswith("labels") + ) + val_S = np.stack([z[f"{prefix_b}{k}"] for k in keys], axis=1) + atk_S = np.stack([z[f"{prefix_a}{k}"] for k in keys], axis=1) + val_S = np.nan_to_num(val_S, nan=0.0, posinf=1e6, neginf=-1e6) + atk_S = np.nan_to_num(atk_S, nan=0.0, posinf=1e6, neginf=-1e6) + return val_S, atk_S + + +def mahal_oas_auroc(val_S, atk_S): + K = val_S.shape[1] + mu = val_S.mean(axis=0) + oas = OAS().fit(val_S) + inv_cov = np.linalg.inv(oas.covariance_ + 1e-9 * np.eye(K)) + + def d2(S): + d = S - mu + return np.einsum("ni,ij,nj->n", d, inv_cov, d) + + s_val = d2(val_S) + s_atk = d2(atk_S) + s = np.r_[s_val, s_atk] + s = np.nan_to_num(s, nan=0.0, posinf=1e12, neginf=-1e12) + y = np.r_[np.zeros(val_S.shape[0]), np.ones(atk_S.shape[0])] + return float(roc_auc_score(y, s)) + + +cells = {} +sample_counts = {} +for src in DATASETS: + for tgt in DATASETS: + aucs = [] + n_val_seen = n_atk_seen = None + for s in SEEDS: + val_S, atk_S = load_cell(src, tgt, s) + auc = mahal_oas_auroc(val_S, atk_S) + aucs.append(auc) + n_val_seen, n_atk_seen = val_S.shape[0], atk_S.shape[0] + a = np.array(aucs) + cells[(src, tgt)] = (a.mean(), a.std()) + sample_counts[(src, tgt)] = (n_val_seen, n_atk_seen) + + +def short(name): + return {"cicids2017": "CICIDS17", "cicddos2019": "CICDDoS19", "ciciot2023": "CICIoT23"}[name] + + +print("# 3×3 cross-dataset AUROC matrix (Mahalanobis-OAS, 3-seed mean ± std)\n") +print("Rows = source (training), columns = target (test). Diagonal = within-dataset.") +print("Aggregator fit on target benign val only; tested on target benign + ALL target attacks.\n") + +header = "| Source ↓ / Target → | " + " | ".join(short(t) for t in DATASETS) + " |" +sep = "|" + "|".join(["---"] * (len(DATASETS) + 1)) + "|" +print(header) +print(sep) +for src in DATASETS: + row = [short(src)] + for tgt in DATASETS: + m, sd = cells[(src, tgt)] + cell = f"{m:.4f} ± {sd:.4f}" + if src == tgt: + cell = f"_{cell}_" + row.append(cell) + print("| " + " | ".join(row) + " |") + +print("\n## Sample counts (target benign / all target attacks)\n") +print(header) +print(sep) +for src in DATASETS: + row = [short(src)] + for tgt in DATASETS: + n_b, n_a = sample_counts[(src, tgt)] + row.append(f"{n_b}b / {n_a}a") + print("| " + " | ".join(row) + " |") + +out_md = ROOT / "CROSS_MATRIX_3x3.md" +with out_md.open("w") as f: + f.write("# 3×3 cross-dataset AUROC matrix (Mahalanobis-OAS, 3-seed mean ± std)\n\n") + f.write("Rows = source (training), columns = target (test). Diagonal italic = within-dataset.\n") + f.write("Aggregator fit on target benign val only; tested on target benign + ALL target attacks.\n\n") + f.write(header + "\n" + sep + "\n") + for src in DATASETS: + row = [short(src)] + for tgt in DATASETS: + m, sd = cells[(src, tgt)] + cell = f"{m:.4f} ± {sd:.4f}" + if src == tgt: + cell = f"_{cell}_" + row.append(cell) + f.write("| " + " | ".join(row) + " |\n") + f.write("\n## Sample counts (target benign / all target attacks)\n\n") + f.write(header + "\n" + sep + "\n") + for src in DATASETS: + row = [short(src)] + for tgt in DATASETS: + n_b, n_a = sample_counts[(src, tgt)] + row.append(f"{n_b}b / {n_a}a") + f.write("| " + " | ".join(row) + " |\n") +print(f"\n[wrote] {out_md}") diff --git a/scripts/aggregate/run_cross_3x3.sh b/scripts/aggregate/run_cross_3x3.sh new file mode 100755 index 0000000..db658b7 --- /dev/null +++ b/scripts/aggregate/run_cross_3x3.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +set -e +ROOT=/home/chy/JANUS +EVAL=${ROOT}/Mixed_CFM/eval_cross.py +CROSS_DIR=${ROOT}/artifacts/route_comparison/cross +mkdir -p ${CROSS_DIR} + +declare -A STORE FLOWS FEATS +STORE[cicids2017]=${ROOT}/datasets/cicids2017/processed/full_store +FLOWS[cicids2017]=${ROOT}/datasets/cicids2017/processed/flows.parquet +FEATS[cicids2017]=${ROOT}/datasets/cicids2017/processed/flow_features.parquet +STORE[cicddos2019]=${ROOT}/datasets/cicddos2019/processed/full_store +FLOWS[cicddos2019]=${ROOT}/datasets/cicddos2019/processed/flows.parquet +FEATS[cicddos2019]=${ROOT}/datasets/cicddos2019/processed/flow_features.parquet +STORE[ciciot2023]=${ROOT}/datasets/ciciot2023/processed/full_store +FLOWS[ciciot2023]=${ROOT}/datasets/ciciot2023/processed/full_store/flows.parquet +FEATS[ciciot2023]=${ROOT}/datasets/ciciot2023/processed/flow_features.parquet + +run_one() { + local gpu=$1 src=$2 tgt=$3 seed=$4 + local md=${ROOT}/artifacts/route_comparison/janus_${src}_seed${seed} + local out=${CROSS_DIR}/janus_seed${seed}_${src}_to_${tgt}.json + if [ -f "${out}" ]; then echo "[skip] ${src}→${tgt} seed${seed}"; return; fi + if [ ! -f "${md}/model.pt" ]; then echo "[missing model] ${md}/model.pt"; return; fi + echo "[gpu${gpu}] ${src} → ${tgt} seed${seed}" + cd ${ROOT}/Mixed_CFM + CUDA_VISIBLE_DEVICES=${gpu} stdbuf -oL uv run --no-sync python -u ${EVAL} \ + --model-dir ${md} \ + --target-store ${STORE[$tgt]} --target-flows ${FLOWS[$tgt]} --target-flow-features ${FEATS[$tgt]} \ + --benign-label normal --n-benign 10000 --n-attack 1000000 \ + --out ${out} --seed ${seed} --T 64 --batch-size 512 --n-steps 16 \ + > ${CROSS_DIR}/janus_seed${seed}_${src}_to_${tgt}.log 2>&1 +} + +GPU0_DIRS=("cicids2017:cicddos2019" "cicids2017:ciciot2023" "cicddos2019:cicids2017") +GPU1_DIRS=("cicddos2019:ciciot2023" "ciciot2023:cicids2017" "ciciot2023:cicddos2019") + +{ +for dir in "${GPU0_DIRS[@]}"; do + src=${dir%:*}; tgt=${dir#*:} + for seed in 42 43 44; do + run_one 0 ${src} ${tgt} ${seed} + done +done +echo "[gpu0 done]" +} & +G0=$! + +{ +for dir in "${GPU1_DIRS[@]}"; do + src=${dir%:*}; tgt=${dir#*:} + for seed in 42 43 44; do + run_one 1 ${src} ${tgt} ${seed} + done +done +echo "[gpu1 done]" +} & +G1=$! + +wait $G0 $G1 +echo "[all done]" +ls ${CROSS_DIR}/*.json | wc -l