BattleTag/JANUS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README headline + 3x3 cross table; add cross_3x3 tooling to scripts/aggregate

Browse Source
This commit is contained in:
BattleTag
2026-05-07 23:56:10 +08:00
parent 1fc0260309
commit dc22e20616
3 changed files with 221 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
README.md
View File

@@ -15,15 +15,28 @@ JANUS is the first NIDS method to use Flow Matching as the training paradigm in
3-seed mean ± std AUROC. Selection-bias-free Mahalanobis-OAS aggregator on the 10-d JANUS score vector, fit on benign val only.
### Within-dataset
| Task | Shafir 2026 SOTA | **JANUS** | Δ |
|---|---|---|---|
| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9908 ± 0.0012** | **+0.118** |
| CICIDS2017 within | 0.9303 | **0.9845 ± 0.0030** | **+0.054** |
| CICDDoS2019 within | 0.93 | **0.9913 ± 0.0009** | **+0.061** |
| CICIDS2017 → CICDDoS2019 cross | 0.89 | **0.9594 ± 0.0046** | **+0.07** |
| CICDDoS2019 → CICIDS2017 reverse cross | 0.93 | **0.9301 ± 0.0122** | matches |
| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9909 ± 0.0013** | **+0.118** |
| CICIDS2017 within | 0.9303 | **0.9826 ± 0.0035** | **+0.052** |
| CICDDoS2019 within | 0.93 | **0.9918 ± 0.0005** | **+0.062** |
| CICIoT2023 within | F1=0.9951 (different metric) | 0.9590 ± 0.0022 (AUROC) | N/A — metric mismatch |
3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIDS2017→CICDDoS2019 cross also beats; reverse direction matches. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table.
3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table.
### 3×3 cross-dataset transfer matrix
Source (rows) trained on 10K benign of source dataset; target (columns) tested on full target benign + **all** target attacks. Aggregator fit on target benign val only — no attack labels at any stage. Diagonal italic = within-dataset.
| Source ↓ / Target → | CICIDS17 | CICDDoS19 | CICIoT23 |
|---|---|---|---|
| **CICIDS17** | _0.9826 ± 0.0035_ | **0.9690 ± 0.0047** | 0.8698 ± 0.0031 |
| **CICDDoS19** | 0.9413 ± 0.0212 | _0.9918 ± 0.0005_ | 0.8767 ± 0.0068 |
| **CICIoT23** | 0.9394 ± 0.0063 | 0.9030 ± 0.0075 | _0.9590 ± 0.0022_ |
Forward CICIDS17→CICDDoS19 (0.969) beats Shafir 0.89 by **+0.08**; reverse CICDDoS19→CICIDS17 (0.941) approximately matches Shafir 0.93. CICIoT23 is hardest both as source and target — its IoT-protocol diversity makes the "benign of source ≈ benign of target" assumption brittle. Full table at `artifacts/route_comparison/CROSS_MATRIX_3x3.md`.
## Layout
@@ -44,15 +57,19 @@ scripts/ Workspace-level pcap → artifact pipeline,
download/ UNB/CIC dataset downloaders.
baselines/ Third-party baseline runners (Kitsune, Shafir-NF,
Anomaly-Transformer).
aggregate/ Mahalanobis-OAS score-router + cross-matrix
orchestration. aggregate_score_router.py is the
deployable score path; run_cross_3x3.sh +
cross_3x3_table.py produce the cross matrix.
tests/ Data-contract unit tests.
```
The following directories are **gitignored** (live on the dev box, not in the repo):
```
artifacts/ All run outputs (checkpoints, eval JSONs, score npzs,
figures). Score-router aggregator at
artifacts/route_comparison/aggregate_score_router.py.
artifacts/ All run outputs (checkpoints, eval JSONs, score
npzs, figures). Per-(dataset × seed) model dirs at
artifacts/route_comparison/janus_<ds>_seed<N>/.
datasets/ Raw + processed datasets (~1 TB).
baselines/ Third-party baseline forks (Kitsune-py,
Anomaly-Transformer, ConMD, ganomaly, TIPSO-GAN, ...).
@@ -85,11 +102,18 @@ uv run --no-sync python train.py --config configs/cicids2017_seed42.yaml
uv run --no-sync python eval_phase1.py \
--model-dir <model_dir> --out-dir <eval_dir>
# Cross-dataset evaluation
# Single cross-dataset eval
uv run --no-sync python eval_cross.py \
--src-model <src_model_dir> \
--tgt-data datasets/<tgt>/processed/ \
--model-dir <src_model_dir> \
--target-store datasets/<tgt>/processed/full_store \
--target-flows datasets/<tgt>/processed/flows.parquet \
--target-flow-features datasets/<tgt>/processed/flow_features.parquet \
--benign-label normal --n-benign 10000 --n-attack 1000000 \
--out <result.json>
# 3×3 cross matrix (6 off-diagonal directions × 3 seeds, 2-GPU parallel)
bash ../scripts/aggregate/run_cross_3x3.sh
uv run --no-sync python ../scripts/aggregate/cross_3x3_table.py
```
JANUS hyper-parameters (locked in `Mixed_CFM/configs/<dataset>_seed*.yaml`):
@@ -122,7 +146,7 @@ d²(s) = (s − μ)ᵀ Σ⁻¹ (s − μ), where (μ, Σ) come from sklearn.c
fit on benign val ONLY (no attack labels).
```
Reference implementation: `artifacts/route_comparison/aggregate_score_router.py` (artifacts/ is gitignored; the script lives on the dev box).
Reference implementation: `scripts/aggregate/aggregate_score_router.py`. It reads `artifacts/route_comparison/janus_<ds>_seed*/phase1_scores.npz` and `artifacts/route_comparison/cross/janus_seed*_<src>_to_<tgt>.npz`, then writes `artifacts/route_comparison/SCORE_ROUTER.md` (within-dataset rows) and `artifacts/route_comparison/CROSS_MATRIX_3x3.md` (cross matrix, via `cross_3x3_table.py`).
## Tests

122
scripts/aggregate/cross_3x3_table.py Normal file
View File

@@ -0,0 +1,122 @@
import json
from pathlib import Path
import numpy as np
from sklearn.covariance import OAS
from sklearn.metrics import roc_auc_score
ROOT = Path("/home/chy/JANUS/artifacts/route_comparison")
CROSS = ROOT / "cross"
DATASETS = ["cicids2017", "cicddos2019", "ciciot2023"]
SEEDS = [42, 43, 44]
def load_cell(src, tgt, seed):
if src == tgt:
path = ROOT / f"janus_{src}_seed{seed}/phase1_scores.npz"
prefix_b, prefix_a = "val_", "atk_"
else:
path = CROSS / f"janus_seed{seed}_{src}_to_{tgt}.npz"
prefix_b, prefix_a = "b_", "a_"
z = np.load(path, allow_pickle=True)
keys = sorted(
k.replace(prefix_b, "")
for k in z.files
if k.startswith(prefix_b) and not k.endswith("labels")
)
val_S = np.stack([z[f"{prefix_b}{k}"] for k in keys], axis=1)
atk_S = np.stack([z[f"{prefix_a}{k}"] for k in keys], axis=1)
val_S = np.nan_to_num(val_S, nan=0.0, posinf=1e6, neginf=-1e6)
atk_S = np.nan_to_num(atk_S, nan=0.0, posinf=1e6, neginf=-1e6)
return val_S, atk_S
def mahal_oas_auroc(val_S, atk_S):
K = val_S.shape[1]
mu = val_S.mean(axis=0)
oas = OAS().fit(val_S)
inv_cov = np.linalg.inv(oas.covariance_ + 1e-9 * np.eye(K))
def d2(S):
d = S - mu
return np.einsum("ni,ij,nj->n", d, inv_cov, d)
s_val = d2(val_S)
s_atk = d2(atk_S)
s = np.r_[s_val, s_atk]
s = np.nan_to_num(s, nan=0.0, posinf=1e12, neginf=-1e12)
y = np.r_[np.zeros(val_S.shape[0]), np.ones(atk_S.shape[0])]
return float(roc_auc_score(y, s))
cells = {}
sample_counts = {}
for src in DATASETS:
for tgt in DATASETS:
aucs = []
n_val_seen = n_atk_seen = None
for s in SEEDS:
val_S, atk_S = load_cell(src, tgt, s)
auc = mahal_oas_auroc(val_S, atk_S)
aucs.append(auc)
n_val_seen, n_atk_seen = val_S.shape[0], atk_S.shape[0]
a = np.array(aucs)
cells[(src, tgt)] = (a.mean(), a.std())
sample_counts[(src, tgt)] = (n_val_seen, n_atk_seen)
def short(name):
return {"cicids2017": "CICIDS17", "cicddos2019": "CICDDoS19", "ciciot2023": "CICIoT23"}[name]
print("# 3×3 cross-dataset AUROC matrix (Mahalanobis-OAS, 3-seed mean ± std)\n")
print("Rows = source (training), columns = target (test). Diagonal = within-dataset.")
print("Aggregator fit on target benign val only; tested on target benign + ALL target attacks.\n")
header = "| Source ↓ / Target → | " + " | ".join(short(t) for t in DATASETS) + " |"
sep = "|" + "|".join(["---"] * (len(DATASETS) + 1)) + "|"
print(header)
print(sep)
for src in DATASETS:
row = [short(src)]
for tgt in DATASETS:
m, sd = cells[(src, tgt)]
cell = f"{m:.4f} ± {sd:.4f}"
if src == tgt:
cell = f"_{cell}_"
row.append(cell)
print("| " + " | ".join(row) + " |")
print("\n## Sample counts (target benign / all target attacks)\n")
print(header)
print(sep)
for src in DATASETS:
row = [short(src)]
for tgt in DATASETS:
n_b, n_a = sample_counts[(src, tgt)]
row.append(f"{n_b}b / {n_a}a")
print("| " + " | ".join(row) + " |")
out_md = ROOT / "CROSS_MATRIX_3x3.md"
with out_md.open("w") as f:
f.write("# 3×3 cross-dataset AUROC matrix (Mahalanobis-OAS, 3-seed mean ± std)\n\n")
f.write("Rows = source (training), columns = target (test). Diagonal italic = within-dataset.\n")
f.write("Aggregator fit on target benign val only; tested on target benign + ALL target attacks.\n\n")
f.write(header + "\n" + sep + "\n")
for src in DATASETS:
row = [short(src)]
for tgt in DATASETS:
m, sd = cells[(src, tgt)]
cell = f"{m:.4f} ± {sd:.4f}"
if src == tgt:
cell = f"_{cell}_"
row.append(cell)
f.write("| " + " | ".join(row) + " |\n")
f.write("\n## Sample counts (target benign / all target attacks)\n\n")
f.write(header + "\n" + sep + "\n")
for src in DATASETS:
row = [short(src)]
for tgt in DATASETS:
n_b, n_a = sample_counts[(src, tgt)]
row.append(f"{n_b}b / {n_a}a")
f.write("| " + " | ".join(row) + " |\n")
print(f"\n[wrote] {out_md}")

62
scripts/aggregate/run_cross_3x3.sh Executable file
View File

@@ -0,0 +1,62 @@
#!/usr/bin/env bash
set -e
ROOT=/home/chy/JANUS
EVAL=${ROOT}/Mixed_CFM/eval_cross.py
CROSS_DIR=${ROOT}/artifacts/route_comparison/cross
mkdir -p ${CROSS_DIR}
declare -A STORE FLOWS FEATS
STORE[cicids2017]=${ROOT}/datasets/cicids2017/processed/full_store
FLOWS[cicids2017]=${ROOT}/datasets/cicids2017/processed/flows.parquet
FEATS[cicids2017]=${ROOT}/datasets/cicids2017/processed/flow_features.parquet
STORE[cicddos2019]=${ROOT}/datasets/cicddos2019/processed/full_store
FLOWS[cicddos2019]=${ROOT}/datasets/cicddos2019/processed/flows.parquet
FEATS[cicddos2019]=${ROOT}/datasets/cicddos2019/processed/flow_features.parquet
STORE[ciciot2023]=${ROOT}/datasets/ciciot2023/processed/full_store
FLOWS[ciciot2023]=${ROOT}/datasets/ciciot2023/processed/full_store/flows.parquet
FEATS[ciciot2023]=${ROOT}/datasets/ciciot2023/processed/flow_features.parquet
run_one() {
local gpu=$1 src=$2 tgt=$3 seed=$4
local md=${ROOT}/artifacts/route_comparison/janus_${src}_seed${seed}
local out=${CROSS_DIR}/janus_seed${seed}_${src}_to_${tgt}.json
if [ -f "${out}" ]; then echo "[skip] ${src}→${tgt} seed${seed}"; return; fi
if [ ! -f "${md}/model.pt" ]; then echo "[missing model] ${md}/model.pt"; return; fi
echo "[gpu${gpu}] ${src} → ${tgt} seed${seed}"
cd ${ROOT}/Mixed_CFM
CUDA_VISIBLE_DEVICES=${gpu} stdbuf -oL uv run --no-sync python -u ${EVAL} \
--model-dir ${md} \
--target-store ${STORE[$tgt]} --target-flows ${FLOWS[$tgt]} --target-flow-features ${FEATS[$tgt]} \
--benign-label normal --n-benign 10000 --n-attack 1000000 \
--out ${out} --seed ${seed} --T 64 --batch-size 512 --n-steps 16 \
> ${CROSS_DIR}/janus_seed${seed}_${src}_to_${tgt}.log 2>&1
}
GPU0_DIRS=("cicids2017:cicddos2019" "cicids2017:ciciot2023" "cicddos2019:cicids2017")
GPU1_DIRS=("cicddos2019:ciciot2023" "ciciot2023:cicids2017" "ciciot2023:cicddos2019")
{
for dir in "${GPU0_DIRS[@]}"; do
src=${dir%:*}; tgt=${dir#*:}
for seed in 42 43 44; do
run_one 0 ${src} ${tgt} ${seed}
done
done
echo "[gpu0 done]"
} &
G0=$!
{
for dir in "${GPU1_DIRS[@]}"; do
src=${dir%:*}; tgt=${dir#*:}
for seed in 42 43 44; do
run_one 1 ${src} ${tgt} ${seed}
done
done
echo "[gpu1 done]"
} &
G1=$!
wait $G0 $G1
echo "[all done]"
ls ${CROSS_DIR}/*.json | wc -l