BattleTag/JANUS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: code, paper, small artifacts

Browse Source
This commit is contained in:
BattleTag
2026-05-07 20:47:30 +08:00
commit fae2db8cff
322 changed files with 33159 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
.gitignore vendored Normal file
View File

@@ -0,0 +1,25 @@
.venv/
venv/
env/
__pycache__/
*.pyc
*.pyo
*.pyd
*.egg-info/
.pytest_cache/
.mypy_cache/
.ruff_cache/
.DS_Store
Thumbs.db
.idea/
.vscode/
.claude/
*.swp
*.swo
/datasets/
/baselines/
*.tmp

172
CLAUDE.md Normal file
View File

@@ -0,0 +1,172 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Repo shape
This is a **workspace-style repo with three sibling model packages** plus a
shared data contract. The root intentionally keeps only workspace-level
files; all model/training/eval code lives under one of the three packages.
- `common/data_contract.py` — **single source of truth** for the canonical
9-d packet schema (`PACKET_FEATURE_NAMES`) and 20-d packet-derived flow
schema (`CANONICAL_FLOW_FEATURE_NAMES`), label normalization, canonical
5-tuple, packet preprocessing helpers, and `compute_flow_features_from_packets`.
All three packages import from here.
- `Packet_CFM/` — packet-sequence OT-CFM with explicit σ-band benign
distribution learning. Has its own `CLAUDE.md` for internal details.
- `Flow_CFM/` — flow-level CFM on the workspace-canonical 20-d packet-derived
`flow_features.parquet`. Legacy 61-d CICFlowMeter CSV caches are still
available only for reproduction via the `--legacy-csv-features` flag.
- `Unified_CFM/` — **current SOTA model**. Unified token CFM over
`[FLOW_TOKEN, PACKET_1, ..., PACKET_T]` with masked-prediction consistency
loss (Phase 2). All within-dataset SOTAs (ISCXTor2016 / CICIDS2017 /
CICDDoS2019) come from here.
- `scripts/` — **workspace-level** scripts shared across all packages:
- `download/` — UNB/CIC dataset downloaders (Token-cookie + `cic_download.py`
recursive crawler). See `scripts/download/README.md` before touching.
- `extract_<dataset>.py` + `extract_lib.py` — pcap→artifact drivers that write
`datasets/<name>/processed/{packets.npz, flows.parquet, flow_features.parquet}`,
all row-aligned by `flow_id = arange(N)`.
- `generate_flow_features.py` — one-shot tool to upgrade an existing
`packets.npz` + `flows.parquet` pair to a canonical `flow_features.parquet`
without re-extracting pcap. Supports `--source-store` for sharded stores.
- `csv_adapter.py`, `convert_npz_splits_to_store.py`, `eval_cross_dataset_protocol.py`,
`merge_*.py`, `auto_transfer_*.sh` — cross-package tooling.
- `datasets/<name>/raw/` and `datasets/<name>/processed/` — shared dataset store.
- `artifacts/{runs,phase0_*,phase1_*,phase25_*,verify_*}/` — **all outputs go
here**, not `runs/` at root. Phase summary reports live in `artifacts/phase*/`.
- `paper/` — paper PDFs we compare against (Shafir 2026 NF, ConMD 2026,
TIPSO-GAN 2026, Lipman 2210.02747).
There is no `archive_v1/` at root; old flow-stat v1 code has been removed.
`Flow_CFM/checkpoints_archive/` retains historical checkpoints for reproduction.
## Data contract (read this before touching data code)
Every processed dataset under `datasets/<name>/processed/` ships an aligned
triple, all with the same row order (`flow_id = arange(N)`):
```
packets.npz # packet_tokens [N, T_full, 9], packet_lengths [N], flow_id [N]
# OR full_store/ (PacketShardStore directory) for large datasets
flows.parquet # flow_id + label + 5-tuple metadata (src_ip, dst_ip, ports, protocol)
flow_features.parquet # flow_id + label + 20 canonical packet-derived features
```
Optional / legacy:
- `flow_features_csv.parquet` — Flow_CFM's 61-d CICFlowMeter cache (paper
reproduction only; not row-aligned with packets in general)
The 20 canonical flow features are computed by
`common.data_contract.compute_flow_features_from_packets(packet_tokens, lens)`
and cover Shafir 2026's top-SHAP categories (size/IAT/active-idle/rate/flags)
in a packet-derivable way.
## Python env
- `requires-python = ">=3.14"`; PyTorch pinned to the `pytorch-cu128` index
(`torch>=2.9.1`), plus `mamba-ssm`, `causal-conv1d`, `scapy`, `dpkt`, `pyarrow`.
- Two `pyproject.toml` files: root (`/pyproject.toml`) and `Packet_CFM/pyproject.toml`.
They are **not declared as a uv workspace** — each resolves independently.
Run `uv run ...` from whichever directory owns the entry point you are invoking.
- `Flow_CFM/` and `Unified_CFM/` have no `pyproject.toml`; they use the root
venv (`uv run --no-sync python <script.py>`).
- Scripts under `scripts/download/` are pure stdlib — invoke with `python3`.
## Running things
**Unified_CFM** (SOTA model, run from `Unified_CFM/`):
```bash
cd Unified_CFM
uv run --no-sync python train.py --config configs/cicids2017_baseline.yaml
# Phase 2 with consistency loss:
uv run --no-sync python train.py --config configs/cicids2017_consistency.yaml
```
Best hyperparameters from the σ × λ sweeps:
- `lambda_flow = lambda_packet = 0.3`
- `sigma = 0.6` for cross-dataset transfer
- `sigma = 0.1` is fine for within-dataset (and marginally better on ISCXTor2016)
**Phase 1 / 2 evaluation**:
```bash
# Per-attack-class AUROC over 34 scores (terminal_norm primary, plus curvature,
# Jacobian-Hutchinson, time-profile velocity, flow_consistency diagnostics).
uv run --no-sync python artifacts/verify_2026_04_24/eval_phase1_unified.py \
--model-dir <model_dir> --out-dir <eval_dir> \
--batch-size 256 --jacobian-n-eps 4 \
--n-val-cap 10000 --n-atk-cap 30000
# Cross-dataset CICIDS2017 → CICDDoS2019:
uv run --no-sync python artifacts/verify_2026_04_24/eval_phase2_cross_cicddos2019.py \
--model-dir <model_dir> --out <result.json> \
--n-benign 10000 --n-attack 10000 --seed 42
```
**Packet_CFM entry points** (run from `Packet_CFM/`):
```bash
cd Packet_CFM
uv run python -m train --config configs/n10k.yaml
uv run python -m detect --save-dir ../artifacts/runs/<run>
uv run python -m eval.per_class --save-dir ../artifacts/runs/<run>
uv run python -m run_phase1 --sigmas 0.0 0.1 0.2 0.3
```
**Flow_CFM entry points** (run from `Flow_CFM/`): see `Flow_CFM/README_migration.md`.
**Tests**:
```bash
uv run --no-sync python -m pytest Packet_CFM/tests/ tests/common/ Unified_CFM/tests/
```
(43 passing — common data contract + Unified_CFM Phase 1/2 score functions
+ Packet_CFM existing tests.)
## Adding a new dataset
Write one driver at `scripts/extract_<name>.py` that calls
`extract_lib.extract_dataset(...)` (see `scripts/extract_cicids2017.py` as
the reference template). The driver hardcodes CSV column names, timestamp
formats, benign aliases, and drop patterns as module constants, then feeds
`extract_lib` a per-day `(canonical_key → [(row_idx, ts_epoch)])` mapping
and a per-day pcap file map. No YAML is needed.
The extract pipeline writes all three artifacts (packets.npz, flows.parquet,
flow_features.parquet) row-aligned. To upgrade an existing artifact pair
that lacks `flow_features.parquet`, run
`scripts/generate_flow_features.py --packets-npz ... --flows-parquet ... --out ...`
(or `--source-store` for sharded stores).
Common gotcha: if CSV timestamps and pcap epochs are in different time zones,
`extract_lib` prints a diagnostic with the recommended `--time-offset`; rerun
with that value.
## Conventions worth preserving
- Do not create a new `runs/` at repo root — outputs belong under `artifacts/`.
- `scripts/download/` stays at the root (shared by all packages).
- When adding new cross-package tooling, put it in root `scripts/`. Only move
it into `Packet_CFM/scripts/` if it depends on that package's imports.
- Phase reports live in `artifacts/phase*/` — keep the timestamp suffix
(`_2026_04_25`) so future runs don't overwrite history.
- The 9-d packet schema and 20-d canonical flow schema are FIXED in
`common/data_contract.py`. Do not extend them ad-hoc; if you need new
features, propose them with evidence (Shafir-style SHAP analysis or
Phase 1-style per-attack ablation).
## Current state of the work (2026-04-25)
- Phase 0 baselines + Shafir-protocol verification: ✓
- Phase 1 (34-score expansion + per-attack-class table): ✓
- Phase 2 (masked-prediction consistency loss): ✓ — multi-seed at λ=0.3
- Phase 2.5 (σ × λ sweep + multi-seed at σ=0.6): ✓
- Cross-dataset multi-seed: ✓ — also SOTA after baseline lock
- Shafir baselines locked from PDF: ✓ — `artifacts/locked_baselines.md`
- 4 of 4 reported tasks beat Shafir SOTA (final table: `RESULTS.md`)
- Architecture is finalized; remaining work is paper writing
(P1 skeleton, P2 thresholded F1/Precision/Recall metrics).

1
Mixed_CFM/__init__.py Normal file
View File

@@ -0,0 +1 @@
pass

42
Mixed_CFM/configs/cicddos2019_ac_combo_seed42.yaml Normal file
View File

@@ -0,0 +1,42 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_cicddos2019_seed42
source_store: /home/chy/JANUS/datasets/cicddos2019/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/cicddos2019/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicddos2019/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 20000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 10000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

42
Mixed_CFM/configs/cicddos2019_ac_combo_seed43.yaml Normal file
View File

@@ -0,0 +1,42 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_cicddos2019_seed43
source_store: /home/chy/JANUS/datasets/cicddos2019/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/cicddos2019/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicddos2019/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: normal
val_cap: 20000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 10000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

42
Mixed_CFM/configs/cicddos2019_ac_combo_seed44.yaml Normal file
View File

@@ -0,0 +1,42 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_cicddos2019_seed44
source_store: /home/chy/JANUS/datasets/cicddos2019/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/cicddos2019/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicddos2019/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: normal
val_cap: 20000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 10000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

40
Mixed_CFM/configs/cicids2017_ac_combo_seed42.yaml Normal file
View File

@@ -0,0 +1,40 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_cicids2017_seed42
packets_npz: /home/chy/JANUS/datasets/cicids2017/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/cicids2017/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicids2017/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

40
Mixed_CFM/configs/cicids2017_ac_combo_seed43.yaml Normal file
View File

@@ -0,0 +1,40 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_cicids2017_seed43
packets_npz: /home/chy/JANUS/datasets/cicids2017/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/cicids2017/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicids2017/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: normal
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

40
Mixed_CFM/configs/cicids2017_ac_combo_seed44.yaml Normal file
View File

@@ -0,0 +1,40 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_cicids2017_seed44
packets_npz: /home/chy/JANUS/datasets/cicids2017/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/cicids2017/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicids2017/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: normal
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

44
Mixed_CFM/configs/ciciot2023_ac_combo_seed42.yaml Normal file
View File

@@ -0,0 +1,44 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_ciciot2023_seed42
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
device: auto
reference_mode: causal_packets

44
Mixed_CFM/configs/ciciot2023_ac_combo_seed43.yaml Normal file
View File

@@ -0,0 +1,44 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_ciciot2023_seed43
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
device: auto
reference_mode: causal_packets

44
Mixed_CFM/configs/ciciot2023_ac_combo_seed44.yaml Normal file
View File

@@ -0,0 +1,44 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_ciciot2023_seed44
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
device: auto
reference_mode: causal_packets

42
Mixed_CFM/configs/ciciot2023_seed42.yaml Normal file
View File

@@ -0,0 +1,42 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_c_mixed_ciciot2023_seed42
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
device: auto

42
Mixed_CFM/configs/ciciot2023_seed43.yaml Normal file
View File

@@ -0,0 +1,42 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_c_mixed_ciciot2023_seed43
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
device: auto

42
Mixed_CFM/configs/ciciot2023_seed44.yaml Normal file
View File

@@ -0,0 +1,42 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_c_mixed_ciciot2023_seed44
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
device: auto

40
Mixed_CFM/configs/iscxtor2016_ac_combo_seed42.yaml Normal file
View File

@@ -0,0 +1,40 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_iscxtor2016_seed42
packets_npz: /home/chy/JANUS/datasets/iscxtor2016/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/iscxtor2016/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/iscxtor2016/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: nontor
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

40
Mixed_CFM/configs/iscxtor2016_ac_combo_seed43.yaml Normal file
View File

@@ -0,0 +1,40 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_iscxtor2016_seed43
packets_npz: /home/chy/JANUS/datasets/iscxtor2016/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/iscxtor2016/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/iscxtor2016/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: nontor
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

40
Mixed_CFM/configs/iscxtor2016_ac_combo_seed44.yaml Normal file
View File

@@ -0,0 +1,40 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_ac_combo_iscxtor2016_seed44
packets_npz: /home/chy/JANUS/datasets/iscxtor2016/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/iscxtor2016/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/iscxtor2016/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: nontor
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_disc: 1.0
reference_mode: causal_packets
device: auto

155
Mixed_CFM/data.py Normal file
View File

@@ -0,0 +1,155 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
import numpy as np
import pandas as pd
import sys as _sys
from pathlib import Path as _Path
_sys.path.insert(0, str(_Path(__file__).resolve().parents[1]))
from common.data_contract import PACKET_FEATURE_NAMES, PACKET_CONTINUOUS_CHANNEL_IDX, PACKET_BINARY_CHANNEL_IDX, fit_packet_stats as _fit_packet_stats, zscore as _zscore
import importlib.util as _ilu
_UDATA_NAME = 'unified_cfm_data'
if _UDATA_NAME not in _sys.modules:
_udata_spec = _ilu.spec_from_file_location(_UDATA_NAME, _Path(__file__).resolve().parents[1] / 'Unified_CFM' / 'data.py')
_udata = _ilu.module_from_spec(_udata_spec)
_sys.modules[_UDATA_NAME] = _udata
_udata_spec.loader.exec_module(_udata)
else:
_udata = _sys.modules[_UDATA_NAME]
DEFAULT_FLOW_META_COLUMNS = _udata.DEFAULT_FLOW_META_COLUMNS
_read_aligned_flow_features = _udata._read_aligned_flow_features
_preprocess_flow = _udata._preprocess_flow
@dataclass
class MixedData:
train_cont: np.ndarray
val_cont: np.ndarray
attack_cont: np.ndarray
train_disc: np.ndarray
val_disc: np.ndarray
attack_disc: np.ndarray
train_flow: np.ndarray
val_flow: np.ndarray
attack_flow: np.ndarray
train_len: np.ndarray
val_len: np.ndarray
attack_len: np.ndarray
attack_labels: np.ndarray
cont_mean: np.ndarray
cont_std: np.ndarray
flow_mean: np.ndarray
flow_std: np.ndarray
flow_feature_names: tuple[str, ...]
packet_feature_names: tuple[str, ...] = PACKET_FEATURE_NAMES
@property
def T(self) -> int:
return int(self.train_cont.shape[1])
@property
def n_cont(self) -> int:
return int(self.train_cont.shape[2])
@property
def n_disc(self) -> int:
return int(self.train_disc.shape[2])
@property
def flow_dim(self) -> int:
return int(self.train_flow.shape[1])
def _zscore_cont(train_x: np.ndarray, val_x: np.ndarray, attack_x: np.ndarray, train_l: np.ndarray, val_l: np.ndarray, attack_l: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
(mean, std) = _fit_packet_stats(train_x, train_l)
def prep(x: np.ndarray, l: np.ndarray) -> np.ndarray:
z = _zscore(x, mean, std)
T = x.shape[1]
m = np.arange(T)[None, :] < l[:, None]
return (z * m[:, :, None]).astype(np.float32)
return (prep(train_x, train_l), prep(val_x, val_l), prep(attack_x, attack_l), mean, std)
def load_mixed_data(*, packets_npz: Path | None=None, source_store: Path | None=None, flows_parquet: Path, flow_features_path: Path, flow_feature_columns: Optional[list[str]]=None, flow_features_align: str='auto', T: int=64, split_seed: int=42, train_ratio: float=0.8, benign_label: str='normal', min_len: int=2, attack_cap: int | None=None, val_cap: int | None=None) -> MixedData:
if (packets_npz is None) == (source_store is None):
raise ValueError('pass exactly one of packets_npz or source_store')
flows_parquet = Path(flows_parquet)
print(f'[data] flows={flows_parquet} packets={(packets_npz if packets_npz else source_store)}')
flow_cols = ['flow_id', 'label', 'src_ip', 'src_port', 'dst_ip', 'dst_port', 'protocol']
flows = pd.read_parquet(flows_parquet, columns=flow_cols)
labels_full = flows['label'].to_numpy().astype(str)
flow_id = flows['flow_id'].to_numpy()
tokens_full: np.ndarray | None = None
store = None
if packets_npz is not None:
pz = np.load(Path(packets_npz))
tokens_full = pz['packet_tokens'].astype(np.float32)
lens_full = pz['packet_lengths'].astype(np.int32)
if T > tokens_full.shape[1]:
raise ValueError(f'requested T={T} > stored {tokens_full.shape[1]}')
tokens_full = tokens_full[:, :T].copy()
lens_full = np.minimum(lens_full, T).astype(np.int32)
if 'flow_id' in pz.files and (not np.array_equal(pz['flow_id'], flow_id)):
raise ValueError('packets_npz / flows_parquet not row-aligned')
else:
from common.packet_store import PacketShardStore
store = PacketShardStore.open(Path(source_store))
store_id = store.read_flows(columns=['flow_id'])['flow_id'].to_numpy()
if not np.array_equal(store_id, flow_id):
raise ValueError('source_store / flows_parquet not row-aligned')
lens_full = np.minimum(store.manifest['packet_length'].to_numpy(dtype=np.int32), T)
(flow_features, flow_names) = _read_aligned_flow_features(Path(flow_features_path), flows, feature_columns=flow_feature_columns, align=flow_features_align)
keep = lens_full >= min_len
labels = labels_full[keep]
flow_features = flow_features[keep]
lens = lens_full[keep]
global_idx = np.flatnonzero(keep).astype(np.int64)
materialized = tokens_full[keep] if tokens_full is not None else None
print(f'[data] kept {keep.sum():,} of {len(keep):,} (min_len={min_len})')
benign = np.where(labels == benign_label)[0]
attack = np.where(labels != benign_label)[0]
rng = np.random.default_rng(split_seed)
rng.shuffle(benign)
n_train = int(len(benign) * train_ratio)
train_local = benign[:n_train]
val_local = benign[n_train:]
if val_cap is not None and len(val_local) > val_cap:
val_local = np.sort(rng.choice(val_local, size=val_cap, replace=False))
if attack_cap is not None and len(attack) > attack_cap:
attack = np.sort(rng.choice(attack, size=attack_cap, replace=False))
print(f'[data] train={len(train_local):,} val={len(val_local):,} attack={len(attack):,}')
def _materialize(idx_local: np.ndarray) -> np.ndarray:
if materialized is not None:
return materialized[idx_local].astype(np.float32, copy=False)
assert store is not None
g = global_idx[idx_local]
(tok, _) = store.read_packets(g.astype(np.int64), T=T)
return tok.astype(np.float32, copy=False)
tr_p = _materialize(train_local)
va_p = _materialize(val_local)
at_p = _materialize(attack)
tr_l = lens[train_local]
va_l = lens[val_local]
at_l = lens[attack]
tr_f = flow_features[train_local]
va_f = flow_features[val_local]
at_f = flow_features[attack]
cont_idx = list(PACKET_CONTINUOUS_CHANNEL_IDX)
disc_idx = list(PACKET_BINARY_CHANNEL_IDX)
tr_cont = tr_p[..., cont_idx]
va_cont = va_p[..., cont_idx]
at_cont = at_p[..., cont_idx]
tr_disc = tr_p[..., disc_idx].astype(np.int8)
va_disc = va_p[..., disc_idx].astype(np.int8)
at_disc = at_p[..., disc_idx].astype(np.int8)
(tr_cont, va_cont, at_cont, c_mean, c_std) = _zscore_cont(tr_cont, va_cont, at_cont, tr_l, va_l, at_l)
(tr_flow, va_flow, at_flow, f_mean, f_std) = _preprocess_flow(tr_f, va_f, at_f)
return MixedData(train_cont=tr_cont, val_cont=va_cont, attack_cont=at_cont, train_disc=tr_disc, val_disc=va_disc, attack_disc=at_disc, train_flow=tr_flow, val_flow=va_flow, attack_flow=at_flow, train_len=tr_l, val_len=va_l, attack_len=at_l, attack_labels=labels[attack], cont_mean=c_mean, cont_std=c_std, flow_mean=f_mean, flow_std=f_std, flow_feature_names=tuple(flow_names))
def subsample_train(data: MixedData, n: int, seed: int) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
if n <= 0 or n >= len(data.train_cont):
return (data.train_flow, data.train_cont, data.train_disc, data.train_len)
rng = np.random.default_rng(seed)
idx = rng.choice(len(data.train_cont), n, replace=False)
idx.sort()
return (data.train_flow[idx], data.train_cont[idx], data.train_disc[idx], data.train_len[idx])

180
Mixed_CFM/eval_cross.py Normal file
View File

@@ -0,0 +1,180 @@
from __future__ import annotations
import argparse
import json
import sys as _sys
import time
from pathlib import Path
import numpy as np
import pandas as pd
import torch
from sklearn.metrics import average_precision_score, roc_auc_score
REPO = Path(__file__).resolve().parents[1]
_sys.path.insert(0, str(REPO))
from common.data_contract import PACKET_CONTINUOUS_CHANNEL_IDX, PACKET_BINARY_CHANNEL_IDX, zscore as _zscore
from common.packet_store import PacketShardStore
_sys.path.insert(0, str(Path(__file__).resolve().parent))
from model import MixedCFMConfig, MixedTokenCFM
def _device(arg: str) -> torch.device:
if arg == 'auto':
return torch.device('cuda' if torch.cuda.is_available() else 'cpu')
return torch.device(arg)
def _score_batch(model, flow_z, cont_z, disc_int, lens, device, batch_size=256, n_steps=16):
out: dict[str, list[np.ndarray]] = {}
for start in range(0, len(flow_z), batch_size):
sl = slice(start, start + batch_size)
f = torch.from_numpy(flow_z[sl]).float().to(device)
c = torch.from_numpy(cont_z[sl]).float().to(device)
d = torch.from_numpy(disc_int[sl]).long().to(device)
l = torch.from_numpy(lens[sl]).long().to(device)
with torch.no_grad():
traj = model.trajectory_metrics(f, c, d, l, n_steps=n_steps)
nll = model.disc_nll_score(f, c, d, l)
for src in (traj, nll):
for (k, v) in src.items():
out.setdefault(k, []).append(v.detach().cpu().numpy())
if start // batch_size % 20 == 0:
print(f'[score] {min(start + batch_size, len(flow_z)):,}/{len(flow_z):,}', flush=True)
return {k: np.concatenate(v) for (k, v) in out.items()}
def main() -> None:
p = argparse.ArgumentParser()
p.add_argument('--model-dir', type=Path, required=True)
p.add_argument('--target-store', type=Path, default=None, help='Sharded packet store (mutually exclusive with --target-packets-npz).')
p.add_argument('--target-packets-npz', type=Path, default=None, help='Monolithic packets.npz (for datasets without full_store).')
p.add_argument('--target-flows', type=Path, required=True)
p.add_argument('--target-flow-features', type=Path, required=True)
p.add_argument('--out', type=Path, required=True)
p.add_argument('--n-benign', type=int, default=10000)
p.add_argument('--n-attack', type=int, default=10000)
p.add_argument('--benign-label', type=str, default='normal', help="Label string of benign class in target dataset (e.g. 'nontor' for ISCXTor2016).")
p.add_argument('--seed', type=int, default=42)
p.add_argument('--T', type=int, default=64)
p.add_argument('--batch-size', type=int, default=256)
p.add_argument('--n-steps', type=int, default=16)
p.add_argument('--device', type=str, default='auto')
args = p.parse_args()
if (args.target_store is None) == (args.target_packets_npz is None):
p.error('pass exactly one of --target-store or --target-packets-npz')
device = _device(args.device)
ckpt = torch.load(args.model_dir / 'model.pt', map_location='cpu', weights_only=False)
model_cfg = MixedCFMConfig(**ckpt['model_cfg'])
model = MixedTokenCFM(model_cfg).to(device)
model.load_state_dict(ckpt['model_state_dict'])
model.eval()
cont_mean = np.asarray(ckpt['cont_mean'], dtype=np.float32)
cont_std = np.asarray(ckpt['cont_std'], dtype=np.float32)
flow_mean = np.asarray(ckpt['flow_mean'], dtype=np.float32)
flow_std = np.asarray(ckpt['flow_std'], dtype=np.float32)
flow_names = [str(n) for n in ckpt['flow_feature_names']]
print(f'[model] T={args.T} flow_dim={model_cfg.flow_dim}')
flows = pd.read_parquet(args.target_flows, columns=['flow_id', 'label'])
ff = pd.read_parquet(args.target_flow_features)
if not np.array_equal(flows['flow_id'].to_numpy(dtype=np.uint64), ff['flow_id'].to_numpy(dtype=np.uint64)):
raise ValueError('flows and flow_features not row-aligned')
labels = flows['label'].astype(str).to_numpy()
print(f'[data] {len(flows):,} target rows')
rng = np.random.default_rng(args.seed)
benign_idx = np.flatnonzero(labels == args.benign_label)
attack_idx = np.flatnonzero(labels != args.benign_label)
n_benign = min(args.n_benign, len(benign_idx))
if n_benign < args.n_benign:
print(f'[warn] only {len(benign_idx)} benign rows available (asked {args.n_benign})')
b_sel = np.sort(rng.choice(benign_idx, size=n_benign, replace=False))
atk_classes = sorted(set(labels[attack_idx]))
per_class = max(1, args.n_attack // len(atk_classes))
a_sel_chunks = []
for cls in atk_classes:
pool = attack_idx[labels[attack_idx] == cls]
k = min(per_class, len(pool))
if k:
a_sel_chunks.append(rng.choice(pool, size=k, replace=False))
a_sel = np.sort(np.concatenate(a_sel_chunks))
if len(a_sel) > args.n_attack:
a_sel = np.sort(rng.choice(a_sel, size=args.n_attack, replace=False))
print(f'[sample] benign={len(b_sel):,} attack={len(a_sel):,} ({len(atk_classes)} classes)')
cont_idx = list(PACKET_CONTINUOUS_CHANNEL_IDX)
disc_idx = list(PACKET_BINARY_CHANNEL_IDX)
if args.target_store is not None:
store = PacketShardStore.open(args.target_store)
npz_tokens = None
npz_lens = None
else:
store = None
pz = np.load(args.target_packets_npz)
npz_tokens = pz['packet_tokens'][:, :args.T].astype(np.float32)
npz_lens = np.minimum(pz['packet_lengths'], args.T).astype(np.int32)
def _materialize(idx: np.ndarray):
if store is not None:
(tok, l) = store.read_packets(idx, T=args.T)
else:
tok = npz_tokens[idx]
l = npz_lens[idx]
l = np.minimum(l, args.T).astype(np.int32)
f = ff.iloc[idx][flow_names].to_numpy(dtype=np.float64)
f = np.nan_to_num(f, nan=0.0, posinf=0.0, neginf=0.0).astype(np.float32)
return (tok.astype(np.float32), l, f)
print('[read] benign...')
(b_tok, b_len, b_flow) = _materialize(b_sel)
print('[read] attack...')
(a_tok, a_len, a_flow) = _materialize(a_sel)
if cont_mean.shape == (9,):
cm = cont_mean[cont_idx]
cs = cont_std[cont_idx]
else:
cm = cont_mean
cs = cont_std
def _prep(tok, lens):
cont = tok[..., cont_idx]
disc = tok[..., disc_idx].astype(np.int8)
z = _zscore(cont, cm, cs)
m = np.arange(args.T)[None, :] < lens[:, None]
cont_z = (z * m[:, :, None]).astype(np.float32)
return (cont_z, disc)
(b_cont, b_disc) = _prep(b_tok, b_len)
(a_cont, a_disc) = _prep(a_tok, a_len)
b_flow_z = ((b_flow - flow_mean) / np.maximum(flow_std, 1e-06)).astype(np.float32)
a_flow_z = ((a_flow - flow_mean) / np.maximum(flow_std, 1e-06)).astype(np.float32)
t0 = time.time()
print('[eval] benign...')
b_scores = _score_batch(model, b_flow_z, b_cont, b_disc, b_len, device, batch_size=args.batch_size, n_steps=args.n_steps)
print(f'[eval] benign done {time.time() - t0:.1f}s')
t0 = time.time()
print('[eval] attack...')
a_scores = _score_batch(model, a_flow_z, a_cont, a_disc, a_len, device, batch_size=args.batch_size, n_steps=args.n_steps)
print(f'[eval] attack done {time.time() - t0:.1f}s')
keys = sorted(set(b_scores) & set(a_scores))
overall = {}
for k in keys:
y = np.r_[np.zeros(len(b_scores[k])), np.ones(len(a_scores[k]))]
s = np.r_[b_scores[k], a_scores[k]]
s = np.nan_to_num(s, nan=0.0, posinf=1000000000000.0, neginf=-1000000000000.0)
overall[k] = {'auroc': float(roc_auc_score(y, s)), 'auprc': float(average_precision_score(y, s))}
a_labels = labels[a_sel]
per_cls = {}
for cls in sorted(set(a_labels)):
m = a_labels == cls
per_cls[cls] = {'_n': float(m.sum())}
for k in keys:
y = np.r_[np.zeros(len(b_scores[k])), np.ones(int(m.sum()))]
s = np.r_[b_scores[k], a_scores[k][m]]
s = np.nan_to_num(s, nan=0.0, posinf=1000000000000.0, neginf=-1000000000000.0)
try:
per_cls[cls][k] = float(roc_auc_score(y, s))
except ValueError:
per_cls[cls][k] = float('nan')
out = {'model_dir': str(args.model_dir), 'target_store': str(args.target_store), 'n_benign': len(b_sel), 'n_attack': len(a_sel), 'n_score_keys': len(keys), 'overall': overall, 'per_class': per_cls}
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(json.dumps(out, indent=2))
npz = args.out.with_suffix('.npz')
save = {'a_labels': a_labels.astype(str)}
for k in keys:
save[f'b_{k}'] = b_scores[k].astype(np.float32)
save[f'a_{k}'] = a_scores[k].astype(np.float32)
np.savez(npz, **save)
print(f'[saved] {args.out}')
if __name__ == '__main__':
main()

109
Mixed_CFM/eval_phase1.py Normal file
View File

@@ -0,0 +1,109 @@
from __future__ import annotations
import argparse
import json
import sys as _sys
import time
from pathlib import Path
from pathlib import Path as _Path
import numpy as np
import torch
import yaml
from sklearn.metrics import average_precision_score, roc_auc_score
_sys.path.insert(0, str(_Path(__file__).resolve().parent))
from data import load_mixed_data
from model import MixedCFMConfig, MixedTokenCFM
def _device(arg: str) -> torch.device:
if arg == 'auto':
return torch.device('cuda' if torch.cuda.is_available() else 'cpu')
return torch.device(arg)
def _score_batch(model: MixedTokenCFM, flow_np: np.ndarray, cont_np: np.ndarray, disc_np: np.ndarray, len_np: np.ndarray, device: torch.device, *, batch_size: int, n_steps: int) -> dict[str, np.ndarray]:
out: dict[str, list[np.ndarray]] = {}
for start in range(0, len(flow_np), batch_size):
sl = slice(start, start + batch_size)
flow = torch.from_numpy(flow_np[sl]).float().to(device)
cont = torch.from_numpy(cont_np[sl]).float().to(device)
disc = torch.from_numpy(disc_np[sl]).long().to(device)
lens = torch.from_numpy(len_np[sl]).long().to(device)
with torch.no_grad():
traj = model.trajectory_metrics(flow, cont, disc, lens, n_steps=n_steps)
nll = model.disc_nll_score(flow, cont, disc, lens)
for d in (traj, nll):
for (k, v) in d.items():
out.setdefault(k, []).append(v.detach().cpu().numpy())
print(f'[score] {min(start + batch_size, len(flow_np)):,}/{len(flow_np):,}', flush=True)
return {k: np.concatenate(v, axis=0) for (k, v) in out.items()}
def _auroc_safe(y, s) -> float:
try:
return float(roc_auc_score(y, s))
except ValueError:
return float('nan')
def _auprc_safe(y, s) -> float:
try:
return float(average_precision_score(y, s))
except ValueError:
return float('nan')
def main() -> None:
p = argparse.ArgumentParser()
p.add_argument('--model-dir', type=Path, required=True)
p.add_argument('--out-dir', type=Path, required=True)
p.add_argument('--n-val-cap', type=int, default=None)
p.add_argument('--n-atk-cap', type=int, default=None)
p.add_argument('--batch-size', type=int, default=256)
p.add_argument('--n-steps', type=int, default=16)
p.add_argument('--device', type=str, default='auto')
args = p.parse_args()
device = _device(args.device)
args.out_dir.mkdir(parents=True, exist_ok=True)
cfg = yaml.safe_load((args.model_dir / 'config.yaml').read_text())
ckpt = torch.load(args.model_dir / 'model.pt', map_location='cpu', weights_only=False)
model_cfg = MixedCFMConfig(**ckpt['model_cfg'])
model = MixedTokenCFM(model_cfg).to(device)
model.load_state_dict(ckpt['model_state_dict'])
model.eval()
print(f'[model] T={model_cfg.T} flow_dim={model_cfg.flow_dim}')
data = load_mixed_data(packets_npz=Path(cfg['packets_npz']) if cfg.get('packets_npz') else None, source_store=Path(cfg['source_store']) if cfg.get('source_store') else None, flows_parquet=Path(cfg['flows_parquet']), flow_features_path=Path(cfg['flow_features_path']), flow_features_align=str(cfg.get('flow_features_align', 'auto')), T=int(cfg['T']), split_seed=int(cfg.get('data_seed', cfg.get('seed', 42))), train_ratio=float(cfg.get('train_ratio', 0.8)), benign_label=str(cfg.get('benign_label', 'normal')), min_len=int(cfg.get('min_len', 2)), attack_cap=int(cfg['attack_cap']) if cfg.get('attack_cap') else None, val_cap=int(cfg['val_cap']) if cfg.get('val_cap') else None)
print(f'[data] val={len(data.val_flow):,} attack={len(data.attack_flow):,}')
rng = np.random.default_rng(0)
(val_flow, val_cont, val_disc, val_len) = (data.val_flow, data.val_cont, data.val_disc, data.val_len)
(atk_flow, atk_cont, atk_disc, atk_len) = (data.attack_flow, data.attack_cont, data.attack_disc, data.attack_len)
atk_labels = data.attack_labels
if args.n_val_cap is not None and len(val_flow) > args.n_val_cap:
idx = np.sort(rng.choice(len(val_flow), size=args.n_val_cap, replace=False))
(val_flow, val_cont, val_disc, val_len) = (val_flow[idx], val_cont[idx], val_disc[idx], val_len[idx])
if args.n_atk_cap is not None and len(atk_flow) > args.n_atk_cap:
idx = np.sort(rng.choice(len(atk_flow), size=args.n_atk_cap, replace=False))
(atk_flow, atk_cont, atk_disc, atk_len) = (atk_flow[idx], atk_cont[idx], atk_disc[idx], atk_len[idx])
atk_labels = atk_labels[idx]
print(f'[eval] scoring val={len(val_flow):,} atk={len(atk_flow):,}')
t0 = time.time()
val = _score_batch(model, val_flow, val_cont, val_disc, val_len, device, batch_size=args.batch_size, n_steps=args.n_steps)
print(f'[eval] val done {time.time() - t0:.1f}s')
t0 = time.time()
atk = _score_batch(model, atk_flow, atk_cont, atk_disc, atk_len, device, batch_size=args.batch_size, n_steps=args.n_steps)
print(f'[eval] atk done {time.time() - t0:.1f}s')
keys = sorted(set(val) & set(atk))
overall: dict[str, dict[str, float]] = {}
for k in keys:
y = np.r_[np.zeros(len(val[k])), np.ones(len(atk[k]))]
s = np.r_[val[k], atk[k]]
s = np.nan_to_num(s, nan=0.0, posinf=1000000000000.0, neginf=-1000000000000.0)
overall[k] = {'auroc': _auroc_safe(y, s), 'auprc': _auprc_safe(y, s)}
per_class: dict[str, dict[str, float]] = {}
for c in sorted(set(atk_labels.tolist())):
m = atk_labels == c
per_class[c] = {'_n': float(m.sum())}
for k in keys:
y = np.r_[np.zeros(len(val[k])), np.ones(int(m.sum()))]
s = np.r_[val[k], atk[k][m]]
s = np.nan_to_num(s, nan=0.0, posinf=1000000000000.0, neginf=-1000000000000.0)
per_class[c][k] = _auroc_safe(y, s)
np.savez(args.out_dir / 'phase1_scores.npz', val_labels=np.array(['normal'] * len(val_flow)), atk_labels=atk_labels.astype(str), **{f'val_{k}': val[k] for k in keys}, **{f'atk_{k}': atk[k] for k in keys})
json.dump({'overall': overall, 'per_class': per_class}, open(args.out_dir / 'phase1_summary.json', 'w'), indent=2)
print(f'[wrote] {args.out_dir}/phase1_summary.json keys={len(keys)}')
if __name__ == '__main__':
main()

244
Mixed_CFM/model.py Normal file
View File

@@ -0,0 +1,244 @@
from __future__ import annotations
import math
from dataclasses import dataclass, field
import torch
import torch.nn as nn
import torch.nn.functional as F
import importlib.util as _ilu
import sys as _sys
from pathlib import Path as _Path
_UNIFIED_NAME = 'unified_cfm_model'
if _UNIFIED_NAME not in _sys.modules:
_unified_spec = _ilu.spec_from_file_location(_UNIFIED_NAME, _Path(__file__).resolve().parents[1] / 'Unified_CFM' / 'model.py')
_unified = _ilu.module_from_spec(_unified_spec)
_sys.modules[_UNIFIED_NAME] = _unified
_unified_spec.loader.exec_module(_unified)
else:
_unified = _sys.modules[_UNIFIED_NAME]
AdaLNBlock = _unified.AdaLNBlock
SinusoidalTimeEmb = _unified.SinusoidalTimeEmb
_sinkhorn_coupling = _unified._sinkhorn_coupling
@dataclass
class MixedCFMConfig:
T: int = 64
flow_dim: int = 20
n_cont_pkt: int = 3
n_disc_pkt: int = 6
cont_pkt_idx: tuple[int, ...] = (0, 1, 8)
disc_pkt_idx: tuple[int, ...] = (2, 3, 4, 5, 6, 7)
n_disc_classes: int = 2
token_dim: int | None = None
d_model: int = 128
n_layers: int = 4
n_heads: int = 4
mlp_ratio: float = 4.0
time_dim: int = 64
sigma: float = 0.1
use_ot: bool = False
reference_mode: str | None = None
lambda_disc: float = 1.0
disc_path: str = 'uniform'
disc_embed_scale: float = 1.0
def __post_init__(self) -> None:
if len(self.cont_pkt_idx) != self.n_cont_pkt:
raise ValueError('cont_pkt_idx length mismatch n_cont_pkt')
if len(self.disc_pkt_idx) != self.n_disc_pkt:
raise ValueError('disc_pkt_idx length mismatch n_disc_pkt')
if self.disc_path != 'uniform':
raise NotImplementedError(f'disc_path={self.disc_path}')
class MixedVelocity(nn.Module):
def __init__(self, token_dim: int, seq_len: int, n_disc: int, n_classes: int, d_model: int=128, n_layers: int=4, n_heads: int=4, mlp_ratio: float=4.0, time_dim: int=64, reference_mode: str | None=None) -> None:
super().__init__()
if reference_mode not in (None, 'causal_packets', 'causal_all'):
raise ValueError(f'reference_mode={reference_mode!r}')
self.token_dim = token_dim
self.seq_len = seq_len
self.n_disc = n_disc
self.n_classes = n_classes
self.reference_mode = reference_mode
self.input_proj = nn.Linear(token_dim, d_model)
self.pos_emb = nn.Parameter(torch.zeros(1, seq_len, d_model))
self.type_emb = nn.Embedding(2, d_model)
nn.init.trunc_normal_(self.pos_emb, std=0.02)
nn.init.normal_(self.type_emb.weight, std=0.02)
self.time_emb = SinusoidalTimeEmb(time_dim)
self.cond_mlp = nn.Sequential(nn.Linear(time_dim, d_model), nn.SiLU(), nn.Linear(d_model, d_model))
self.blocks = nn.ModuleList([AdaLNBlock(d_model, n_heads, mlp_ratio, cond_dim=d_model) for _ in range(n_layers)])
self.out_norm = nn.LayerNorm(d_model, elementwise_affine=False)
self.head_v = nn.Linear(d_model, token_dim)
self.head_disc = nn.Linear(d_model, n_disc * n_classes)
for layer in (self.head_v, self.head_disc):
nn.init.zeros_(layer.weight)
nn.init.zeros_(layer.bias)
type_ids = torch.ones(seq_len, dtype=torch.long)
type_ids[0] = 0
self.register_buffer('type_ids', type_ids, persistent=False)
def _attn_mask(self, L: int, device: torch.device) -> torch.Tensor | None:
if self.reference_mode is None:
return None
if self.reference_mode == 'causal_packets':
mask = torch.zeros((L, L), dtype=torch.bool, device=device)
if L > 1:
mask[1:, 1:] = torch.triu(torch.ones(L - 1, L - 1, dtype=torch.bool, device=device), diagonal=1)
return mask
return torch.triu(torch.ones(L, L, dtype=torch.bool, device=device), diagonal=1)
def forward(self, x: torch.Tensor, t: torch.Tensor, key_padding_mask: torch.Tensor | None=None) -> tuple[torch.Tensor, torch.Tensor]:
(B, L, _) = x.shape
if t.dim() == 0:
t = t.expand(B)
h = self.input_proj(x)
h = h + self.pos_emb[:, :L, :] + self.type_emb(self.type_ids[:L])[None, :, :]
cond = self.cond_mlp(self.time_emb(t))
attn_mask = self._attn_mask(L, x.device)
for block in self.blocks:
h = block(h, cond, key_padding_mask, attn_mask=attn_mask)
h = self.out_norm(h)
v = self.head_v(h)
d = self.head_disc(h).view(B, L, self.n_disc, self.n_classes)
return (v, d)
class MixedTokenCFM(nn.Module):
def __init__(self, cfg: MixedCFMConfig) -> None:
super().__init__()
self.cfg = cfg
cont_size = cfg.n_cont_pkt + cfg.n_disc_pkt
self.token_dim = cfg.token_dim or 1 + max(cfg.flow_dim, cont_size)
if self.token_dim < 1 + max(cfg.flow_dim, cont_size):
raise ValueError('token_dim too small')
self.seq_len = cfg.T + 1
self.velocity = MixedVelocity(token_dim=self.token_dim, seq_len=self.seq_len, n_disc=cfg.n_disc_pkt, n_classes=cfg.n_disc_classes, d_model=cfg.d_model, n_layers=cfg.n_layers, n_heads=cfg.n_heads, mlp_ratio=cfg.mlp_ratio, time_dim=cfg.time_dim, reference_mode=cfg.reference_mode)
def _embed_disc(self, x_disc_int: torch.Tensor) -> torch.Tensor:
s = self.cfg.disc_embed_scale
return (x_disc_int.float() - 0.5) * s
def build_tokens(self, flow: torch.Tensor, packets_cont: torch.Tensor, x_disc_t_int: torch.Tensor) -> torch.Tensor:
(B, T, Cp) = packets_cont.shape
assert T == self.cfg.T and Cp == self.cfg.n_cont_pkt
z = packets_cont.new_zeros((B, T + 1, self.token_dim))
z[:, 0, 0] = -1.0
z[:, 0, 1:1 + self.cfg.flow_dim] = flow
z[:, 1:, 0] = 1.0
z[:, 1:, 1:1 + self.cfg.n_cont_pkt] = packets_cont
z[:, 1:, 1 + self.cfg.n_cont_pkt:1 + self.cfg.n_cont_pkt + self.cfg.n_disc_pkt] = self._embed_disc(x_disc_t_int)
return z
def key_padding_mask(self, lens: torch.Tensor) -> torch.Tensor:
B = lens.shape[0]
idx = torch.arange(self.cfg.T, device=lens.device)[None, :]
packet_real = idx < lens[:, None]
real = torch.cat([torch.ones(B, 1, dtype=torch.bool, device=lens.device), packet_real], dim=1)
return ~real
def _loss_mask(self, lens: torch.Tensor) -> torch.Tensor:
return (~self.key_padding_mask(lens)).float()
def compute_loss(self, flow: torch.Tensor, packets_cont: torch.Tensor, packets_disc: torch.Tensor, lens: torch.Tensor, *, return_components: bool=False) -> torch.Tensor | dict[str, torch.Tensor]:
(B, T, _) = packets_cont.shape
device = packets_cont.device
mask = self._loss_mask(lens)
kpm = mask == 0
x_1_cont = self.build_tokens(flow, packets_cont, torch.zeros_like(packets_disc))
x_0_cont = torch.randn_like(x_1_cont)
if self.cfg.use_ot:
flat0 = (x_0_cont * mask[:, :, None]).reshape(B, -1)
flat1 = (x_1_cont * mask[:, :, None]).reshape(B, -1)
col = _sinkhorn_coupling(torch.cdist(flat0.float(), flat1.float()))
x_1_cont = x_1_cont[col]
packets_cont = packets_cont[col]
packets_disc = packets_disc[col]
flow = flow[col]
lens = lens[col]
mask = self._loss_mask(lens)
kpm = mask == 0
t = torch.rand(B, device=device)
x_t_cont = (1.0 - t[:, None, None]) * x_0_cont + t[:, None, None] * x_1_cont
if self.cfg.sigma > 0:
std = self.cfg.sigma * torch.sqrt(t * (1.0 - t))[:, None, None]
x_t_cont = x_t_cont + std * torch.randn_like(x_t_cont)
target_cont = x_1_cont - x_0_cont
u = torch.rand(B, T, self.cfg.n_disc_pkt, device=device)
keep = u < t[:, None, None]
rand_disc = torch.randint(0, self.cfg.n_disc_classes, packets_disc.shape, device=device)
x_disc_t = torch.where(keep, packets_disc, rand_disc)
disc_start = 1 + self.cfg.n_cont_pkt
x_t_full = x_t_cont.clone()
x_t_full[:, 1:, disc_start:disc_start + self.cfg.n_disc_pkt] = self._embed_disc(x_disc_t)
(v_pred, d_logits) = self.velocity(x_t_full, t, key_padding_mask=kpm)
v_err = (v_pred - target_cont).square()
v_err[:, :, disc_start:disc_start + self.cfg.n_disc_pkt] = 0.0
v_per_token = v_err.mean(dim=-1)
per_sample = (v_per_token * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
L_cont = per_sample.mean()
pkt_logits = d_logits[:, 1:]
pkt_real = mask[:, 1:].bool()
corrupt = ~keep & pkt_real[:, :, None]
flat_logits = pkt_logits.reshape(-1, self.cfg.n_disc_classes)
flat_targets = packets_disc.reshape(-1).long()
flat_ce = F.cross_entropy(flat_logits, flat_targets, reduction='none')
flat_ce = flat_ce.view(B, T, self.cfg.n_disc_pkt)
flat_ce = flat_ce * corrupt.float()
denom = corrupt.float().sum().clamp_min(1.0)
L_disc = flat_ce.sum() / denom
total = L_cont + self.cfg.lambda_disc * L_disc
if return_components:
return {'total': total, 'main': L_cont.detach(), 'aux_disc': L_disc.detach(), 'aux_flow': L_cont.new_zeros(()), 'aux_packet': L_cont.new_zeros(())}
return total
@torch.no_grad()
def trajectory_metrics(self, flow: torch.Tensor, packets_cont: torch.Tensor, packets_disc: torch.Tensor, lens: torch.Tensor, n_steps: int=16) -> dict[str, torch.Tensor]:
z = self.build_tokens(flow, packets_cont, packets_disc)
mask = self._loss_mask(lens)
kpm = mask == 0
B = z.shape[0]
dt = 1.0 / n_steps
disc_start = 1 + self.cfg.n_cont_pkt
disc_end = disc_start + self.cfg.n_disc_pkt
disc_embed = z[:, 1:, disc_start:disc_end].clone()
for k in range(n_steps):
t_val = 1.0 - k * dt
t = torch.full((B,), t_val, device=z.device)
(v, _) = self.velocity(z, t, key_padding_mask=kpm)
v[:, :, disc_start:disc_end] = 0.0
z = z - v * dt
z[:, 1:, disc_start:disc_end] = disc_embed
z_real = z * mask[:, :, None]
z_cont = z_real.clone()
z_cont[:, 1:, disc_start:disc_end] = 0.0
packet_count = mask[:, 1:].sum(dim=-1).clamp_min(1.0)
terminal = z_cont.reshape(B, -1).norm(dim=-1) / (mask.sum(dim=-1) * self.token_dim).clamp_min(1.0).sqrt()
terminal_flow = z_cont[:, 0].norm(dim=-1) / math.sqrt(self.token_dim)
terminal_packet = (z_cont[:, 1:] * mask[:, 1:, None]).reshape(B, -1).norm(dim=-1) / (packet_count * self.token_dim).sqrt()
return {'terminal_norm': terminal, 'terminal_flow': terminal_flow, 'terminal_packet': terminal_packet}
@torch.no_grad()
def disc_nll_score(self, flow: torch.Tensor, packets_cont: torch.Tensor, packets_disc: torch.Tensor, lens: torch.Tensor, t_eval: float=0.5) -> dict[str, torch.Tensor]:
(B, T, _) = packets_cont.shape
device = packets_cont.device
mask = self._loss_mask(lens)
kpm = mask == 0
z = self.build_tokens(flow, packets_cont, packets_disc)
t = torch.full((B,), float(t_eval), device=device)
(_, d_logits) = self.velocity(z, t, key_padding_mask=kpm)
pkt_logits = d_logits[:, 1:]
flat_logits = pkt_logits.reshape(-1, self.cfg.n_disc_classes)
flat_targets = packets_disc.reshape(-1).long()
ce = F.cross_entropy(flat_logits, flat_targets, reduction='none')
ce = ce.view(B, T, self.cfg.n_disc_pkt)
pkt_real = mask[:, 1:].bool().float()
per_sample = (ce.sum(dim=-1) * pkt_real).sum(dim=-1) / pkt_real.sum(dim=-1).clamp_min(1.0)
per_ch = (ce * pkt_real[:, :, None]).sum(dim=1) / pkt_real.sum(dim=1).clamp_min(1.0)[:, None]
out = {'disc_nll_total': per_sample}
for (c, idx) in enumerate(self.cfg.disc_pkt_idx):
out[f'disc_nll_ch{idx}'] = per_ch[:, c]
return out
def param_count(self) -> int:
return sum((p.numel() for p in self.parameters()))

141
Mixed_CFM/train.py Normal file
View File

@@ -0,0 +1,141 @@
from __future__ import annotations
import argparse
import json
import sys as _sys
import time
from dataclasses import asdict
from pathlib import Path
from pathlib import Path as _Path
from typing import Any
import numpy as np
import torch
import yaml
from sklearn.metrics import roc_auc_score
from torch.utils.data import DataLoader, TensorDataset
_sys.path.insert(0, str(_Path(__file__).resolve().parent))
from data import MixedData, load_mixed_data, subsample_train
from model import MixedCFMConfig, MixedTokenCFM
def _device(arg: str) -> torch.device:
if arg == 'auto':
return torch.device('cuda' if torch.cuda.is_available() else 'cpu')
return torch.device(arg)
def _batch_score(model: MixedTokenCFM, flow_np: np.ndarray, cont_np: np.ndarray, disc_np: np.ndarray, len_np: np.ndarray, device: torch.device, *, batch_size: int, n_steps: int) -> dict[str, np.ndarray]:
out: dict[str, list[np.ndarray]] = {}
model.eval()
for start in range(0, len(flow_np), batch_size):
sl = slice(start, start + batch_size)
flow = torch.from_numpy(flow_np[sl]).float().to(device)
cont = torch.from_numpy(cont_np[sl]).float().to(device)
disc = torch.from_numpy(disc_np[sl]).long().to(device)
lens = torch.from_numpy(len_np[sl]).long().to(device)
m = model.trajectory_metrics(flow, cont, disc, lens, n_steps=n_steps)
d = model.disc_nll_score(flow, cont, disc, lens)
for src in (m, d):
for (k, v) in src.items():
out.setdefault(k, []).append(v.detach().cpu().numpy())
return {k: np.concatenate(v, axis=0) for (k, v) in out.items()}
def _quick_eval(model: MixedTokenCFM, data: MixedData, device: torch.device, cfg: dict[str, Any]) -> dict[str, float]:
n_eval = int(cfg.get('eval_n', 2000))
rng = np.random.default_rng(0)
def pick(n: int) -> np.ndarray:
m = min(n_eval, n)
return rng.choice(n, m, replace=False)
vi = pick(len(data.val_flow))
ai = pick(len(data.attack_flow))
v = _batch_score(model, data.val_flow[vi], data.val_cont[vi], data.val_disc[vi], data.val_len[vi], device, batch_size=int(cfg.get('eval_batch_size', 512)), n_steps=int(cfg.get('eval_n_steps', 8)))
a = _batch_score(model, data.attack_flow[ai], data.attack_cont[ai], data.attack_disc[ai], data.attack_len[ai], device, batch_size=int(cfg.get('eval_batch_size', 512)), n_steps=int(cfg.get('eval_n_steps', 8)))
y = np.concatenate([np.zeros(len(vi)), np.ones(len(ai))])
out: dict[str, float] = {}
for k in sorted(v.keys()):
s = np.concatenate([v[k], a[k]])
s = np.nan_to_num(s, nan=0.0, posinf=1000000000000.0, neginf=-1000000000000.0)
out[f'auroc_{k}'] = float(roc_auc_score(y, s))
return out
def train(cfg: dict[str, Any]) -> Path:
device = _device(str(cfg.get('device', 'auto')))
save_dir = Path(cfg['save_dir'])
save_dir.mkdir(parents=True, exist_ok=True)
with open(save_dir / 'config.yaml', 'w') as f:
yaml.safe_dump(cfg, f)
seed = int(cfg.get('seed', 42))
data_seed = int(cfg.get('data_seed', seed))
torch.manual_seed(seed)
np.random.seed(seed)
print(f'Device: {device} seed=model:{seed}/data:{data_seed}')
data = load_mixed_data(packets_npz=Path(cfg['packets_npz']) if cfg.get('packets_npz') else None, source_store=Path(cfg['source_store']) if cfg.get('source_store') else None, flows_parquet=Path(cfg['flows_parquet']), flow_features_path=Path(cfg['flow_features_path']), flow_feature_columns=cfg.get('flow_feature_columns'), flow_features_align=str(cfg.get('flow_features_align', 'auto')), T=int(cfg['T']), split_seed=data_seed, train_ratio=float(cfg.get('train_ratio', 0.8)), benign_label=str(cfg.get('benign_label', 'normal')), min_len=int(cfg.get('min_len', 2)), attack_cap=int(cfg['attack_cap']) if cfg.get('attack_cap') else None, val_cap=int(cfg['val_cap']) if cfg.get('val_cap') else None)
print(f'[data] T={data.T} cont={data.n_cont} disc={data.n_disc} flow={data.flow_dim} train={len(data.train_flow):,} val={len(data.val_flow):,} attack={len(data.attack_flow):,}')
(tr_f, tr_c, tr_d, tr_l) = subsample_train(data, int(cfg.get('n_train', 0)), data_seed)
ds = TensorDataset(torch.from_numpy(tr_f).float(), torch.from_numpy(tr_c).float(), torch.from_numpy(tr_d).long(), torch.from_numpy(tr_l).long())
loader = DataLoader(ds, batch_size=int(cfg['batch_size']), shuffle=True, drop_last=True, num_workers=int(cfg.get('num_workers', 0)), pin_memory=device.type == 'cuda')
print(f'[data] training on {len(ds):,} flows')
model_cfg = MixedCFMConfig(T=data.T, flow_dim=data.flow_dim, token_dim=cfg.get('token_dim'), d_model=int(cfg['d_model']), n_layers=int(cfg['n_layers']), n_heads=int(cfg['n_heads']), mlp_ratio=float(cfg.get('mlp_ratio', 4.0)), time_dim=int(cfg.get('time_dim', 64)), sigma=float(cfg.get('sigma', 0.1)), use_ot=bool(cfg.get('use_ot', False)), reference_mode=cfg.get('reference_mode'), lambda_disc=float(cfg.get('lambda_disc', 1.0)))
model = MixedTokenCFM(model_cfg).to(device)
print(f'[model] params={model.param_count():,} token_dim={model.token_dim} sigma={model_cfg.sigma} use_ot={model_cfg.use_ot} lambda_disc={model_cfg.lambda_disc}')
opt = torch.optim.AdamW(model.parameters(), lr=float(cfg['lr']), weight_decay=float(cfg.get('weight_decay', 0.01)))
total_steps = max(1, int(cfg['epochs']) * len(loader))
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=total_steps)
history: dict[str, list[Any]] = {'epoch': [], 'loss': [], 'eval': []}
for epoch in range(1, int(cfg['epochs']) + 1):
model.train()
losses: list[float] = []
ldisc_sum = 0.0
n_batches = 0
t0 = time.time()
for (flow, cont, disc, lens) in loader:
flow = flow.to(device, non_blocking=True)
cont = cont.to(device, non_blocking=True)
disc = disc.to(device, non_blocking=True)
lens = lens.to(device, non_blocking=True)
comp = model.compute_loss(flow, cont, disc, lens, return_components=True)
loss = comp['total']
ldisc_sum += float(comp['aux_disc'].item())
opt.zero_grad(set_to_none=True)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), float(cfg.get('grad_clip', 1.0)))
opt.step()
sched.step()
losses.append(float(loss.item()))
n_batches += 1
mean_loss = float(np.mean(losses)) if losses else float('nan')
eval_metrics: dict[str, float] | None = None
if epoch % int(cfg.get('eval_every', 5)) == 0 or epoch == int(cfg['epochs']):
eval_metrics = _quick_eval(model, data, device, cfg)
history['epoch'].append(epoch)
history['loss'].append(mean_loss)
history['eval'].append(eval_metrics)
elapsed = time.time() - t0
tail = ''
if eval_metrics:
t = eval_metrics.get('auroc_terminal_norm', float('nan'))
n = eval_metrics.get('auroc_disc_nll_total', float('nan'))
tail = f' auroc_term={t:.3f} auroc_disc={n:.3f}'
if n_batches:
tail += f' L_disc={ldisc_sum / n_batches:.4f}'
print(f"[epoch {epoch:>3d}/{cfg['epochs']:<3d}] ({elapsed:.1f}s) loss={mean_loss:.4f}{tail}")
if not np.isfinite(mean_loss):
raise RuntimeError(f'non-finite loss at epoch {epoch}')
payload = {'model_state_dict': model.state_dict(), 'model_cfg': asdict(model_cfg), 'cont_mean': data.cont_mean, 'cont_std': data.cont_std, 'flow_mean': data.flow_mean, 'flow_std': data.flow_std, 'flow_feature_names': np.asarray(data.flow_feature_names), 'packet_feature_names': np.asarray(data.packet_feature_names)}
torch.save(payload, save_dir / 'model.pt')
with open(save_dir / 'history.json', 'w') as f:
json.dump(history, f, indent=2, default=str)
print(f"[saved] {save_dir / 'model.pt'}")
return save_dir
def main() -> None:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('--config', type=Path, required=True)
p.add_argument('--override', type=str, nargs='*', default=[])
args = p.parse_args()
with open(args.config) as f:
cfg = yaml.safe_load(f)
for ov in args.override:
(k, v) = ov.split('=', 1)
cfg[k] = yaml.safe_load(v)
train(cfg)
if __name__ == '__main__':
main()

57
README.md Normal file
View File

@@ -0,0 +1,57 @@
# mambafortrafficmodeling
Network traffic anomaly detection with continuous flow matching (CFM). Three
sibling model packages over a shared canonical data contract.
## Layout
- `common/data_contract.py` — single source of truth for the canonical
packet schema (9-d) and flow schema (20-d, packet-derived). All three
packages import constants and helpers from here.
- `Packet_CFM/` — packet-sequence OT-CFM with explicit σ-band benign
distribution learning.
- `Flow_CFM/` — flow-level CFM on the workspace-canonical 20-d packet-derived
`flow_features.parquet`. Legacy 61-d CICFlowMeter CSV caches are kept only
for paper reproduction (`--legacy-csv-features` flag).
- `Unified_CFM/` — unified packet+flow token CFM. **Current SOTA model** —
used for all main results (within-dataset SOTA on ISCXTor2016 / CICIDS2017
/ CICDDoS2019, near-SOTA cross-dataset).
- `datasets/<name>/processed/` — canonical artifact bundle:
- `packets.npz` (small/medium) or `full_store/` (large, sharded)
- `flows.parquet` (label + 5-tuple metadata)
- `flow_features.parquet` (20-d packet-derived, row-aligned)
- `scripts/` — workspace-level pcap → artifact extraction, CSV adapters,
cross-package eval tooling. `scripts/download/` is also here.
- `artifacts/` — run outputs (training checkpoints, eval JSONs, reports).
Phase 0 / 1 / 2 / 2.5 experiment summaries live under
`artifacts/phase{0,1,2}*` directories.
- `paper/` — paper PDFs we compare against (Shafir 2026 NF, ConMD 2026,
TIPSO-GAN 2026, Lipman 2210.02747 flow matching).
The root keeps only workspace-level files. All model/training/eval code
lives under one of the three packages.
## Current best results (Unified_CFM, λ=0.3, 3 seeds)
Shafir baselines verified from paper PDF tables — see `artifacts/locked_baselines.md`.
| Task | Shafir 2026 SOTA | Our best | Δ |
|---|---|---|---|
| ISCXTor2016 (NonTor → Tor) | 0.8731 (Table VI) | 0.9945 ± 0.0011 (σ=0.1) | **+0.121** |
| CICIDS2017 within (10k/10k Shafir protocol) | 0.9303 (Table VII) | **0.9858 ± 0.0021** (σ=0.6) | **+0.055** |
| CICDDoS2019 within | 0.93 (Table IX) | **0.9958 ± 0.0010** (σ=0.1) | **+0.066** |
| CICIDS2017 → CICDDoS2019 cross (`terminal_norm`) | 0.89 (Table IX, IDS→DDoS row) | **0.9109 ± 0.0032** (σ=0.6) | **+0.021** |
| CICIDS2017 → CICDDoS2019 cross (`terminal_flow`) | 0.89 | **0.9197 ± 0.0036** | **+0.030** |
**4 of 4 reported tasks achieve SOTA**. Cross-dataset baseline was previously misread as 0.93; the IDS→DDoS direction in Shafir Table IX is 0.89.
Plus an architectural contribution: a `flow_consistency` diagnostic score
that lifts from random (~0.6) to discriminative (~0.9) only when the model
is trained with the masked-prediction consistency loss. On SSH-Patator (the
hardest CICIDS2017 class for `terminal_norm` at 0.64) it reaches 0.94.
Authoritative result tables live in `RESULTS.md` (root) and
`artifacts/locked_baselines.md` (Shafir baseline verification trail).
Thresholded F1 / Precision / Recall / TPR@FPR under unsupervised threshold
protocol: `RESULTS_THRESHOLDED.md`.
Per-attack-family multi-seed analysis: `artifacts/phase25_multiseed_2026_04_25/PER_ATTACK_TABLE.md`.

341
RESULTS.md Normal file
View File

@@ -0,0 +1,341 @@
# Final Results
## Main-line model: JANUS
**JANUS** (Joint Anomaly via Normalizing-flows of Unified States) is the
current main-line model. Codebase identifier is `Mixed_CFM/`; JANUS is the
external/published name.
JANUS = a packet-causal Transformer backbone with two output heads:
- **Continuous Flow Matching head** over (size, IAT, win) packet channels
- **Discrete Flow Matching (DFM) head** over the 6 binary protocol-flag /
direction channels
trained jointly (σ=0.1, lambda_disc=1.0, use_ot=true, no Phase-2
consistency loss). Downstream uses a **single deployable scalar score**:
the Mahalanobis-OAS distance over the 10-d score vector emitted by JANUS,
fit on benign val only (no attack labels).
JANUS is the first NIDS method to use Flow Matching as the training paradigm
in mixed continuous-discrete state spaces over packet sequences.
All numbers reported are 3-seed mean ± std. Two model families are tracked:
- **Unified_CFM** (legacy / our previous internal recipe): single Transformer
over [FLOW + packets] with Phase-2 consistency loss; λ=0.3. Strongest
single-fixed-score (`terminal_norm`) within-dataset baseline.
- **JANUS = A+C combo** (current main line, 2026-05-01): see above.
**New SOTA on cross-dataset transfer** under Mahalanobis auto-routing;
matches legacy within-dataset under the same protocol. See
`artifacts/route_comparison/SCORE_ROUTER.md`.
## Caveats that travel with all external claims
1. **CICIoT2023 vs Shafir is a metric mismatch, not a +SOTA result.** Shafir
reports F1=0.9951 with threshold tuned by Youden's J (TPR−FPR) on a
1K+1K balanced val set (uses attack labels for threshold selection only)
and tested on 10K+10K balanced. We report AUROC=0.9594 (Mahalanobis-OAS).
Different metric. CICIoT2023 should be presented as "additional
benchmark, no Shafir AUROC published" rather than "+SOTA". To make it
directly comparable, either reproduce Shafir's threshold protocol on
JANUS's d² to compute F1, or run Shafir's GitHub
`lshafir/NF-anomaly-detection` to extract NF AUROC.
2. **Reverse cross (CICDDoS2019→CICIDS2017) matches Shafir, does not beat.**
JANUS gets 0.9301 ± 0.0122. Shafir Table IX row 3 reports 0.93. The
"+0.31" gain is vs our own legacy `terminal_norm` (0.62), not vs Shafir.
3. **Cross-dataset is calibrated cross-domain transfer, not zero-shot.** The
Mahalanobis-OAS aggregator is fit on the **target** dataset's benign val
(unsupervised — no attack labels). Comparison vs Shafir is fair (his NF
threshold also calibrated on target benign), but the language must be
"calibrated cross-domain transfer" not "zero-shot transfer".
4. **Aggregator selection (OAS over LedoitWolf / plain Mahal / max-z) was
post-hoc.** OAS picked because consistently top across all cells in
`SCORE_ROUTER.md`; differences vs LedoitWolf ≤ 0.005. Strict pre-
registration would say "we evaluated 5 benign-only aggregators and OAS
performed best".
## Headline performance
External SOTA baselines (Shafir 2026 NF + Shapley) verified directly from
the paper (`artifacts/locked_baselines.md`). Unified_CFM "legacy" rows are
*our* previous internal recipe (Phase-2 consistency loss + per-task σ); they
are reported as internal ablation, NOT as the SOTA-comparison baseline.
### A. vs External SOTA — within-dataset, JANUS + Mahalanobis-OAS (no selection bias)
| Task | **Shafir 2026 SOTA** | **JANUS + Mahalanobis-OAS** | **Δ vs Shafir** |
|---|---|---|---|
| ISCXTor2016 (NonTor → Tor) | 0.8731 (AUROC) | **0.9908 ± 0.0012** | **+0.118** ⭐⭐ |
| CICIDS2017 within | 0.9303 (AUROC) | **0.9845 ± 0.0030** | **+0.054** ⭐ |
| CICDDoS2019 within | 0.93 (AUROC) | **0.9913 ± 0.0009** | **+0.061** ⭐ |
| CICIoT2023 within | F1=0.9951 (no AUROC) | 0.9594 ± 0.0028 (AUROC) | **N/A — metric mismatch, see Caveat 1** |
**3/3 directly comparable within-dataset benchmarks: JANUS sets new SOTA vs
external Shafir baselines, with margins +0.054 to +0.118 — all far outside
seed std.** This holds under fully selection-bias-free eval (single
Mahalanobis-OAS aggregator on the 10-d score vector, fit on benign val
only, no attack labels). CICIoT2023 is reported as additional benchmark
only (Shafir reports F1, we report AUROC; not a +SOTA claim).
### A'. Reference only — best per-channel fixed score (per-dataset selection-biased; do NOT use as headline SOTA)
⚠️ **Selection-biased**: the channel chosen per row (`terminal_norm` vs
`terminal_packet`) requires looking at attack-label AUROC to pick. Use this
table as ablation upper bound only, not as the SOTA claim. The honest
external SOTA claim is in table A above.
| Task | Shafir 2026 | JANUS (best fixed channel) | Δ vs Shafir |
|---|---|---|---|
| ISCXTor2016 | 0.8731 | 0.9954 ± 0.0007 (`terminal_norm`) | +0.122 |
| CICIDS2017 | 0.9303 | 0.9932 ± 0.0013 (`terminal_packet`) | +0.063 |
| CICDDoS2019 | 0.93 | 0.9970 ± 0.0005 (`terminal_norm`) | +0.067 |
| CICIoT2023 | F1=0.9951 (different metric) | 0.9671 ± 0.0002 (`terminal_packet`) | N/A |
### B. Internal ablation — JANUS vs our previous Unified_CFM legacy
This is for tracking how JANUS does relative to our own previous internal
best (not for the SOTA claim — Unified_CFM legacy is also our work).
Within-dataset AUROC has saturated above 0.99; differences ≤ 0.005 are seed
noise and the regime has no resolving power. The discriminating axis is
cross-dataset (next section).
| Task | Legacy Unified_CFM | JANUS + Mahalanobis-OAS | JANUS (best fixed) |
|---|---|---|---|
| ISCXTor2016 | 0.9945 ± 0.0011 | 0.9908 ± 0.0012 | 0.9954 ± 0.0007 |
| CICIDS2017 | 0.9858 ± 0.0021 | 0.9845 ± 0.0030 | 0.9932 ± 0.0013 |
| CICDDoS2019 | 0.9960 ± 0.0010 | 0.9913 ± 0.0009 | 0.9970 ± 0.0005 |
| CICIoT2023 | 0.9612 ± 0.0017 | 0.9594 ± 0.0028 | 0.9671 ± 0.0002 |
JANUS + Mahalanobis-OAS ties the legacy recipe within seed std on every
within-dataset task (all gaps ≤ 0.005, all overlapping). Best-fixed (per-
dataset selection-biased) strictly beats legacy on 4/4 but cannot be cited
as a clean SOTA claim. The decisive value-add is on cross-dataset transfer.
### C. Cross-dataset transfer — JANUS + Mahalanobis-OAS
⚠️ **Δ columns are vs our own legacy** (not vs Shafir). vs Shafir: forward
beats (+0.07 over 0.89), reverse matches (0.93 = 0.93). See Caveats above.
| Task | Legacy `terminal_norm` | **JANUS + Mahalanobis-OAS** | Δ vs legacy | Shafir | vs Shafir |
|---|---|---|---|---|---|
| **CICIoT2023 → CICIDS2017** | 0.7700 ± 0.0133 | **0.8983 ± 0.0098** | **+0.128** | (n/a) | (n/a) |
| **CICIoT2023 → CICDDoS2019** | 0.7473 ± 0.0223 | **0.8944 ± 0.0068** | **+0.147** | (n/a) | (n/a) |
| **CICIDS2017 → CICDDoS2019** (forward) | 0.911 (legacy SOTA) | **0.9594 ± 0.0046** | +0.048 | 0.89 | **+0.07** |
| **CICDDoS2019 → CICIDS2017** (reverse) | 0.62 (legacy) | **0.9301 ± 0.0122** | **+0.31** | 0.93 | **0 (matches)** |
Full 4×4 cross matrix at `artifacts/route_comparison/CROSS_MATRIX.md`. All
12 off-diagonal directions tested (3 seeds each = 36 cross evaluations).
**Average off-diagonal improvement: +0.175 over `terminal_norm`**
(0.660 → 0.835). The four "source-likeness collapse" cells where
`terminal_norm` ≤ 0.57 (essentially random) are all recovered to ≥ 0.75.
See `artifacts/route_comparison/SCORE_ROUTER.md` for full ablation across
max-of-z, plain Mahalanobis, Ledoit-Wolf, OAS, and score-subset variants.
### Reverse cross (CICDDoS2019 → CICIDS2017) — 2026-05-01 update
The reverse direction was the project's "stuck" failure mode (memory note
`reverse_cross_score_redirection_2026_04_25`). Three model variants compared:
| Model | `terminal_norm` | best single score (post-hoc) | **Mahalanobis-OAS** |
|---|---|---|---|
| Legacy Unified + consistency | 0.626 | `pna_packet_median` 0.882 | 0.824 |
| Legacy Unified no consistency | 0.554 | `pna_packet_median` 0.852 | 0.893 |
| **JANUS (new)** | 0.519 | `disc_nll_total` **0.903 ± 0.012** | **0.930 ± 0.015** |
`terminal_norm` collapses (≈ random) across **all** model variants — this is
the source-likeness-classifier failure mode confirmed at the architecture
level, not just a single-recipe artifact. The recovery path is:
1. **DFM head** gives a `disc_nll` score that captures protocol-flag
distribution, which is genuinely transfer-stable.
2. **Mahalanobis-OAS** on the 10-d score vector aggregates `disc_nll` with
the (broken-but-not-useless) terminal scores into a 0.93 ± 0.015 AUROC.
3. Compared to Shafir's reverse 0.93 on this direction, JANUS +
Mahalanobis-OAS **matches** that benchmark (0.93 = 0.93). Does NOT beat.
This is **+0.31 over our own legacy memory baseline of 0.62**. The "main
attack direction" recorded in `reverse_cross_score_redirection_2026_04_25`
is now substantially solved.
Thresholded F1 / Precision / Recall / TPR@FPR (unsupervised protocol, τ from
benign-val percentile) are reported separately in `RESULTS_THRESHOLDED.md`.
Headline thresholded numbers: CICDDoS2019 within `terminal_norm` F1=0.993 ± 0.001
at τ=P95; cross `terminal_norm` F1=0.632 ± 0.051 at τ=P95 (precision ≈ 0.95, recall ≈ 0.47).
> **Note on cross-dataset baseline**: Shafir's Table IX is asymmetric.
> The IDS2017→DDoS2019 direction (which we evaluate) reads **0.89**, not
> 0.93. The 0.93 number is the reverse direction (DDoS2019→IDS2017),
> which we have not evaluated. See `artifacts/locked_baselines.md`.
> **Note on σ choice**: headline numbers use per-task best σ (σ=0.1 for ISCXTor2016
> and CICDDoS2019; σ=0.6 for CICIDS2017 within and cross). Within-dataset
> tasks are σ-insensitive within seed noise; cross-dataset requires σ=0.6.
> Single-policy σ=0.6 also beats Shafir on 4/4. Full 4×2 sensitivity table
> in `artifacts/sigma_validation.md`.
## Methodological contribution: `flow_consistency` diagnostic score
Phase 2 masked-prediction consistency loss unlocks a new score that is
discriminative **only when the model is trained with the consistency loss**:
| Dataset | baseline (no aux) | Phase 2 (λ=0.3, σ=0.1) |
|---|---|---|
| ISCXTor2016 | 0.6543 | 0.9011 ± 0.0125 (+0.247) |
| CICIDS2017 | 0.5745 | 0.8770 ± 0.0039 (+0.302) |
| CICDDoS2019 | 0.9084 | 0.9459 ± 0.0188 (+0.038) |
On **SSH-Patator** — the worst class in CICIDS2017 for `terminal_norm`
(0.6407 ± 0.0675) — `flow_consistency` reaches 0.94, providing a reliable
detector where standard density scores fail.
## Per-attack-family pattern
`terminal_norm` dominates on volumetric attacks (DDoS, DoS, Portscan, all
DrDoS_*) — saturated 0.97-0.99. Decomposed scores compete only on
brute-force / app-layer attacks where flow-level signal is strong but
packet-level signal is weak:
| Class | n | terminal_norm | best decomposed score | best AUROC |
|---|---|---|---|---|
| SSH-Patator | 168 | 0.6407 ± 0.0675 | `kinetic_flow` | 0.9458 ± 0.0080 |
| FTP-Patator | 256 | 0.8963 ± 0.0015 | `terminal_flow` | 0.9773 ± 0.0049 |
| DoS GoldenEye | 448 | 0.9760 ± 0.0008 | `terminal_flow` | 0.9868 ± 0.0015 |
Outside these classes, `terminal_norm` is the right primary; decomposed
scores are diagnostic only.
## What the experiments proved
1. **JANUS sets new SOTA vs external Shafir 2026 NF on 3/3 directly
comparable within-dataset benchmarks** under unbiased Mahalanobis-OAS
eval (+0.054 to +0.118, all margins outside 3-seed std). CICIoT2023 is
metric-mismatched (F1 vs AUROC) and reported as additional benchmark.
2. **Within-dataset is saturated**: JANUS + Mahalanobis-OAS ties our own
internal Unified_CFM legacy within ±0.005 (all in seed std). At AUROC
> 0.99 the regime has no resolving power; benchmarks here cannot
distinguish models. The right axis is cross-dataset.
3. **JANUS recovers the previously catastrophic reverse cross direction**:
CICDDoS2019→CICIDS2017 from legacy `terminal_norm` 0.62 → JANUS
Mahalanobis-OAS 0.93. Matches Shafir's 0.93 on the same direction
(does not exceed). The "source-likeness collapse" failure mode of
`terminal_norm` is confirmed at the architecture level (≤ 0.63 across
3 distinct backbones) and is broken by the DFM head + Mahalanobis route.
4. **Discrete Flow Matching on flag/direction channels unlocks a new score
family** (`disc_nll_total`) that is independent of `terminal_norm`. It is
the single best cross→CICIDS2017 fixed score across all 5 routes
(0.9191). Without it the Mahalanobis aggregator has nothing to recover
reverse cross with.
5. **Causal-packet attention reduces multi-seed std** by ~2-8× on every
dataset, indicating the protocol-causal prior is a stabilizer for CFM
training.
6. **Phase-2 consistency loss is no longer the lead mechanism**: useful for
the `flow_consistency` diagnostic family, but JANUS's `terminal_packet`
and `disc_nll_total` heads cover its function without the masked-
prediction aux loss.
7. **σ-band noise is a transfer-friendly regularizer** — σ=0.6 cross-dataset
AUROC is +0.02 over σ=0.1, matching the σ=0.6 sweet spot from Packet_CFM.
8. **Per-attack-family analysis is the right reporting frame** — averaged
AUROC hides the SSH-Patator-style cases where decomposed scores save
the day.
## What the experiments disproved
1. **Curvature as primary score**: 0.32-0.91 across datasets, much weaker
than `terminal_norm`. Has diagnostic value on SSH-Patator (+0.30) but
should not lead reporting.
2. **Jacobian-Hutchinson as primary score**: 0.32-0.59 on ISCXTor2016 —
below random for some sub-scores. Failed.
3. **Time-profile velocity scores**: at best +0.005 over `terminal_norm`
on average. Some per-class wins on brute-force but not enough to lead.
## Configuration
```yaml
# CURRENT SOTA: JANUS (Mixed_CFM + causal-packet attention).
# Configs at: Mixed_CFM/configs/<dataset>_ac_combo_seed{42,43,44}.yaml
model:
T: 64
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
use_ot: true
reference_mode: causal_packets # ← Route A: packet-causal attention
training:
n_train: 10000
epochs: 50
batch_size: 256
lr: 3.0e-4
# Mixed CFM packet preprocessing: cont channels z-scored,
# disc channels (direction + 5 TCP flags) kept as int {0,1}
sigma: 0.1
lambda_disc: 1.0 # ← Route C: DFM cross-entropy weight
scoring (per dataset best):
ISCXTor2016 / CICDDoS2019: terminal_norm
CICIDS2017 / CICIoT2023: terminal_packet
cross→CICIDS2017: disc_nll_total
```
### Legacy config (Unified_CFM with Phase-2 consistency)
Kept for reference; superseded by JANUS on cross-dataset (within-dataset is
saturated and JANUS ties legacy in noise):
```yaml
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
sigma: 0.1 within / 0.6 cross
```
## Stability
JANUS std vs legacy Unified_CFM std (3 seeds):
| Dataset | Legacy std | **JANUS std** | std reduction |
|---|---|---|---|
| ISCXTor2016 | 0.0011 | **0.0007** | 1.6× |
| CICIDS2017 | 0.0021 | **0.0013** | 1.6× |
| CICDDoS2019 | 0.0010 | **0.0005** | 2× |
| CICIoT2023 | 0.0017 | **0.0002** | **8×** |
Causal-packet attention is the dominant contributor to std reduction —
isolated Route A also halved std on terminal_norm in CICIoT2023 (Route A
alone: 0.0006 vs baseline 0.0017).
Legacy reference (kept for completeness):
- `terminal_norm` ISCXTor2016: ±0.0011 (σ=0.1) / ±0.0019 (σ=0.6)
- `terminal_norm` CICIDS2017: ±0.0021 (σ=0.6)
- `terminal_norm` CICDDoS2019: ±0.0010 (σ=0.1)
- cross `terminal_norm` σ=0.6: ±0.0032
- cross `terminal_flow` σ=0.6: ±0.0036
The +0.121 on ISCXTor2016 and +0.055 on CICIDS2017 are not single-seed
artifacts.
## Source artifacts
- `RESULTS_THRESHOLDED.md` — F1 / Precision / Recall / TPR@FPR under unsupervised
threshold protocol (τ = benign-val P95/P99) for CICDDoS2019 within and
CICIDS2017→CICDDoS2019 cross.
- `artifacts/locked_baselines.md` — verified Shafir baselines (PDF inspection trail).
- `artifacts/sigma_validation.md` — full 4×2 σ-sensitivity table (σ ∈ {0.1, 0.6} ×
4 tasks, 3 seeds each) and per-task σ-selection protocol.
- `artifacts/reverse_cross.md` — reverse direction CICDDoS2019 → CICIDS2017
evaluation (3 seeds × 2 σ × 16 scores). Asymmetry finding.
- `artifacts/phase25_multiseed_2026_04_25/PER_ATTACK_TABLE.md` — per-attack
multi-seed table (granular `terminal_norm` vs decomposed scores per class).
- `artifacts/phase{0,1,25}*/<config_name>_seed*/phase1_summary.json` — raw
per-seed eval results across all experiments.
- `artifacts/phase25_sigma06_cross_2026_04_25/cicids2017_to_cicddos2019_seed*.json` —
3-seed cross-dataset eval JSONs.
- Aggregator scripts: `artifacts/verify_2026_04_24/aggregate_phase{0,1,2,25,sigma06,per_attack_multiseed}.py`.
- Orchestrator scripts: `artifacts/verify_2026_04_24/run_phase*.sh`.
Phase summary markdown reports were superseded by this `RESULTS.md` and
removed during the 2026-04-25 baseline-lock cleanup. The aggregator
scripts can regenerate any historical view from the raw JSON results.

34
RESULTS_THRESHOLDED.md Normal file
View File

@@ -0,0 +1,34 @@
# Thresholded metrics — unsupervised AD protocol
3-seed mean ± std. Threshold τ is set on benign-val half A; F1 / Precision / Recall / FPR are measured on benign-val half B + attack. AUROC/AUPRC use full benign val + attack. TPR@FPR is measured on the test half.
Both percentiles are reported because P95 and P99 give different operating points; F1 numbers are sensitive to that choice.
Primary score: `terminal_norm`. `terminal_flow` is reported on cross because RESULTS.md headlines both.
## CICDDoS2019 within (σ=0.1, λ=0.3)
| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR |
|---|---|---|---|---|---|---|---|---|---|
| `terminal_norm` | 0.9960 ± 0.0011 | 0.9975 ± 0.0008 | 0.9932 ± 0.0012 | 0.9881 ± 0.0015 | 0.9983 ± 0.0008 | 0.0481 ± 0.0061 | 0.9112 ± 0.0402 | 0.9013 ± 0.0540 | 0.9980 ± 0.0014 |
| `terminal_flow` | 0.9885 ± 0.0028 | 0.9918 ± 0.0017 | 0.9788 ± 0.0086 | 0.9868 ± 0.0009 | 0.9710 ± 0.0163 | 0.0517 ± 0.0030 | 0.7752 ± 0.0128 | 0.6052 ± 0.0347 | 0.9697 ± 0.0169 |
## CICIDS2017 → CICDDoS2019 cross (σ=0.6, λ=0.3)
| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR |
|---|---|---|---|---|---|---|---|---|---|
| `terminal_norm` | 0.9109 ± 0.0032 | 0.8974 ± 0.0047 | 0.6321 ± 0.0513 | 0.9545 ± 0.0045 | 0.4745 ± 0.0550 | 0.0441 ± 0.0011 | 0.4202 ± 0.0171 | 0.2685 ± 0.0139 | 0.4940 ± 0.0399 |
| `terminal_flow` | 0.9197 ± 0.0036 | 0.8957 ± 0.0086 | 0.6324 ± 0.0585 | 0.9517 ± 0.0055 | 0.4762 ± 0.0639 | 0.0469 ± 0.0019 | 0.4028 ± 0.0049 | 0.2534 ± 0.0039 | 0.4776 ± 0.0636 |
## Reading
- **Within-dataset (CICDDoS2019)**: at τ=P95, `terminal_norm` reaches F1 ≈ 0.99 with precision ≈ 0.99 and recall ≈ 0.99 — saturation. At τ=P99 (≈1% FPR), F1 ≈ 0.91 / TPR@1%FPR ≈ 0.90. The model is a working detector at fixed thresholds, not just an AUROC artifact.
- **Cross-dataset (CICIDS2017 → CICDDoS2019)**: AUROC stays high (≈ 0.91) but at fixed thresholds Precision is high (≈0.95) and Recall drops to ≈0.50 at P95 / ≈0.27 at 1% FPR. The cross-dataset domain shift compresses the score gap, so a source-calibrated threshold is conservative on target — false positives stay low, but a substantial fraction of target-domain attacks score below the source benign P95. **AUROC alone overstates deployability cross-dataset; thresholded numbers are the honest figure.**
- TIPSO-GAN comparability: TIPSO-GAN's CIC-DDoS2019 F1 ≈ 0.99 is reported under a **supervised** protocol (model has seen attack examples). Our F1 ≈ 0.99 on CICDDoS2019 within is achieved under the **unsupervised** protocol (benign-only training, threshold from benign-val), which is the strictly harder setting. Direct F1 numerical equivalence; protocol asymmetry is in our favor.
## Source artifacts
- `artifacts/verify_2026_04_24/thresholded_metrics.py` — per-file metric tool.
- `artifacts/verify_2026_04_24/aggregate_thresholded.py` — this aggregator.
- Within: `artifacts/phase1_2026_04_25/cicddos2019_lambda0p3_seed*/thresholded_metrics.json` (computed from existing `phase1_scores.npz`).
- Cross: `artifacts/phase25_sigma06_cross_2026_04_25/with_scores/thresholded_seed*.json` (raw scores re-saved by patched `eval_phase2_cross_cicddos2019.py`).

133
Unified_CFM/README.md Normal file
View File

@@ -0,0 +1,133 @@
# Unified_CFM
A single multi-scale OT-CFM over one token sequence per flow:
```text
[FLOW_TOKEN, PACKET_1, ..., PACKET_T]
```
This is **not** a Flow-CFM + Packet-CFM ensemble. Flow-level and packet-level
signals interact inside one Transformer velocity field, and a Phase 2
masked-prediction consistency loss explicitly trains the cross-modal
dependency.
This is the **current SOTA model** in the repo (within-dataset SOTA on
ISCXTor2016 / CICIDS2017 / CICDDoS2019; near-SOTA cross-dataset).
## Model
`UnifiedTokenCFM` uses fixed tokenization to avoid latent-collapse shortcuts:
```text
flow token: [type=-1, normalized 20-d canonical flow features, zero pad]
packet token: [type=+1, normalized 9-d packet features, zero pad]
```
Velocity field: 4-layer AdaLN-Zero Transformer (`d_model=128, n_heads=4`),
sinusoidal time embedding (`time_dim=64`). Total ≈ 1.23M parameters.
Loss with Phase 2 consistency:
```
L = L_main + λ_flow · L_mask_flow + λ_packet · L_mask_packet
L_main: standard OT-CFM velocity regression with σ-band noise +
Sinkhorn OT coupling.
L_mask_flow: zero out the flow token's input at x_t; predict v[flow]
from packet context only.
L_mask_packet: zero out a random 50% of real packet tokens at x_t;
predict their velocities from flow + remaining packets.
```
Best hyperparameters from the σ × λ sweeps:
```
lambda_flow = lambda_packet = 0.3
packet_mask_ratio = 0.5
sigma = 0.6 # cross-dataset best; σ=0.1 marginally better for some within
use_ot = True
```
## Scores
The model exposes three classes of scores at inference:
```text
# primary
terminal_norm
# decomposed (analysis only)
terminal_flow terminal_packet
arc_length kinetic_energy kinetic_flow kinetic_packet
velocity_total velocity_flow velocity_packet
# Phase 1 diagnostics
curvature_total curvature_flow curvature_packet # ∫ ||dv/dt||² dt
kappa2_speed2norm_packet_{mean,median,trimmed10_mean} # packet curvature / speed²
jacobian_total jacobian_flow jacobian_packet # Hutchinson VJP estimate of ||∂v/∂x||_F²
velocity_*_t{01..10} # 18 time-profile scores
# Phase 2 cross-modal consistency
flow_consistency packet_consistency consistency_total
```
`terminal_norm` is the paper's primary score. The decomposed and diagnostic
scores serve **per-attack-family analysis** — they are NOT competing
SOTA claims. Multi-seed std on `terminal_norm` is ≤ 0.005 across all our
runs.
The Phase 2 consistency scores have a notable property: they are
**discriminative only when the model is trained with the consistency loss**.
On a baseline model `flow_consistency` is roughly random (0.57 on
CICIDS2017); after Phase 2 training it lifts to 0.88. On SSH-Patator,
where standard density scores struggle (`terminal_norm` 0.64), Phase 2
`flow_consistency` reaches 0.94.
## Train
```bash
# baseline (no consistency loss)
uv run python Unified_CFM/train.py --config Unified_CFM/configs/cicids2017_baseline.yaml
# Phase 2 with consistency loss (λ=0.1, σ=0.1)
uv run python Unified_CFM/train.py --config Unified_CFM/configs/cicids2017_consistency.yaml
# σ × λ sweeps and multi-seed orchestrators live in
# artifacts/verify_2026_04_24/run_*.sh
```
The intended setup is to use the workspace-canonical 20-d packet-derived
flow feature file:
```yaml
flow_features_path: datasets/cicids2017/processed/flow_features.parquet
flow_features_align: auto
```
`flow_features.parquet` is row-aligned with the Packet_CFM artifacts via
`flow_id`. With `flow_features_align: auto`, the loader uses direct
row/`flow_id` alignment when possible; scan alignment remains only for
legacy full CSV-derived caches.
For large datasets where a monolithic `packets.npz` would exceed memory,
the loader supports the sharded backend:
```yaml
source_store: datasets/cicddos2019/processed/full_store
val_cap: 20000
attack_cap: 20000
```
If `flow_features_path` is empty, the loader derives compact 16-d flow-level
statistics from the packet sequence. That fallback is for debugging only;
new runs should use the canonical 20-d file generated by
`scripts/generate_flow_features.py`.
## Evaluation
`artifacts/verify_2026_04_24/eval_phase1_unified.py` runs Phase 1 + Phase 2
score battery on a trained checkpoint, with per-attack-class AUROC.
`artifacts/verify_2026_04_24/eval_phase2_cross_cicddos2019.py` runs
cross-dataset CICIDS2017→CICDDoS2019 evaluation under the standard
10k benign + 10k stratified attack protocol.

1
Unified_CFM/__init__.py Normal file
View File

@@ -0,0 +1 @@
pass

45
Unified_CFM/configs/cicddos2019_reference_blockdiag.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/phaseC_reference_2026_04_25/cicddos2019_ref_blockdiag_seed42
source_store: /home/chy/JANUS/datasets/cicddos2019/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/cicddos2019/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicddos2019/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 20000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode: block_diagonal
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.0
lambda_packet: 0.0
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/cicddos2019_reference_independent.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/phaseC_reference_2026_04_25/cicddos2019_ref_independent_seed42
source_store: /home/chy/JANUS/datasets/cicddos2019/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/cicddos2019/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicddos2019/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 20000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode: independent_token
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 10000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.0
lambda_packet: 0.0
packet_mask_ratio: 0.5
device: auto

41
Unified_CFM/configs/cicddos2019_within.yaml Normal file
View File

@@ -0,0 +1,41 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_cicddos2019_within_2026_04_25
source_store: /home/chy/JANUS/datasets/cicddos2019/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/cicddos2019/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicddos2019/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 20000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 10000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
device: auto

43
Unified_CFM/configs/cicddos2019_within_consistency.yaml Normal file
View File

@@ -0,0 +1,43 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_cicddos2019_within_consistency_2026_04_25
source_store: /home/chy/JANUS/datasets/cicddos2019/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/cicddos2019/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicddos2019/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 20000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 10000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.1
lambda_packet: 0.1
packet_mask_ratio: 0.5
device: auto

38
Unified_CFM/configs/cicids2017_baseline.yaml Normal file
View File

@@ -0,0 +1,38 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_cicids2017_canonical_2026_04_24
packets_npz: /home/chy/JANUS/datasets/cicids2017/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/cicids2017/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicids2017/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 2
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
device: auto

43
Unified_CFM/configs/cicids2017_consistency.yaml Normal file
View File

@@ -0,0 +1,43 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_cicids2017_consistency_2026_04_25
packets_npz: /home/chy/JANUS/datasets/cicids2017/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/cicids2017/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/cicids2017/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 2
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.1
lambda_packet: 0.1
packet_mask_ratio: 0.5
device: auto

43
Unified_CFM/configs/ciciot2023.yaml Normal file
View File

@@ -0,0 +1,43 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_ciciot2023_2026_04_29
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 10000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/ciciot2023_baseline_seed42.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/baseline_ciciot2023_seed42
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/ciciot2023_baseline_seed43.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/baseline_ciciot2023_seed43
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/ciciot2023_baseline_seed44.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/baseline_ciciot2023_seed44
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/ciciot2023_route_a_causal.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_a_causal_ciciot2023_seed42
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode: causal_packets
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/ciciot2023_route_a_causal_seed43.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_a_causal_ciciot2023_seed43
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode: causal_packets
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/ciciot2023_route_a_causal_seed44.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_a_causal_ciciot2023_seed44
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
reference_mode: causal_packets
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

44
Unified_CFM/configs/ciciot2023_route_b_spectral_seed42.yaml Normal file
View File

@@ -0,0 +1,44 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_b_spectral_ciciot2023_seed42
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features_spectral.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

44
Unified_CFM/configs/ciciot2023_route_b_spectral_seed43.yaml Normal file
View File

@@ -0,0 +1,44 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_b_spectral_ciciot2023_seed43
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features_spectral.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 43
data_seed: 43
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

44
Unified_CFM/configs/ciciot2023_route_b_spectral_seed44.yaml Normal file
View File

@@ -0,0 +1,44 @@
save_dir: /home/chy/JANUS/artifacts/route_comparison/route_b_spectral_ciciot2023_seed44
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features_spectral.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 44
data_seed: 44
train_ratio: 0.8
benign_label: normal
val_cap: 10000
attack_cap: 20000
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

45
Unified_CFM/configs/ciciot2023_shafir5.yaml Normal file
View File

@@ -0,0 +1,45 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_ciciot2023_shafir5_2026_04_29
source_store: /home/chy/JANUS/datasets/ciciot2023/processed/full_store
flows_parquet: /home/chy/JANUS/datasets/ciciot2023/processed/full_store/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/ciciot2023/processed/flow_features_shafir5.parquet
flow_feature_columns: ["HTTPS", "Protocol_Type", "Magnitude", "Variance", "fin_count"]
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: normal
val_cap: 10000
flow_dim: 5
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 0
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.3
lambda_packet: 0.3
packet_mask_ratio: 0.5
device: auto

39
Unified_CFM/configs/iscxtor2016.yaml Normal file
View File

@@ -0,0 +1,39 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_iscxtor2016_2026_04_25
packets_npz: /home/chy/JANUS/datasets/iscxtor2016/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/iscxtor2016/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/iscxtor2016/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: nontor
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 2
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
device: auto

41
Unified_CFM/configs/iscxtor2016_consistency.yaml Normal file
View File

@@ -0,0 +1,41 @@
save_dir: /home/chy/JANUS/artifacts/runs/unified_cfm_iscxtor2016_consistency_2026_04_25
packets_npz: /home/chy/JANUS/datasets/iscxtor2016/processed/packets.npz
flows_parquet: /home/chy/JANUS/datasets/iscxtor2016/processed/flows.parquet
flow_features_path: /home/chy/JANUS/datasets/iscxtor2016/processed/flow_features.parquet
flow_features_align: auto
T: 64
n_train: 10000
min_len: 2
packet_preprocess: mixed_dequant
seed: 42
data_seed: 42
train_ratio: 0.8
benign_label: nontor
d_model: 128
n_layers: 4
n_heads: 4
mlp_ratio: 4.0
time_dim: 64
token_dim:
batch_size: 256
num_workers: 2
epochs: 50
lr: 3.0e-4
weight_decay: 0.01
grad_clip: 1.0
eval_every: 10
eval_n: 20000
eval_batch_size: 512
eval_n_steps: 8
sigma: 0.1
use_ot: true
lambda_flow: 0.1
lambda_packet: 0.1
packet_mask_ratio: 0.5
device: auto

275
Unified_CFM/data.py Normal file
View File

@@ -0,0 +1,275 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
import numpy as np
import pandas as pd
import sys as _sys
from pathlib import Path as _Path
_sys.path.insert(0, str(_Path(__file__).resolve().parents[1]))
from common.data_contract import PACKET_FEATURE_NAMES, PACKET_CONTINUOUS_CHANNEL_IDX as CONTINUOUS_CHANNEL_IDX, PACKET_BINARY_CHANNEL_IDX as BINARY_CHANNEL_IDX, canonical_5tuple as _canonical_key, fit_packet_stats as _fit_packet_stats, zscore as _zscore, apply_mixed_dequant as _apply_mixed_dequant
DEFAULT_FLOW_META_COLUMNS = {'flow_id', 'label', 'day', 'service', 'src_ip', 'dst_ip', 'src_port', 'dst_port', 'protocol', 'timestamp', 'start_ts', 'n_pkts'}
DERIVED_FLOW_FEATURE_NAMES = ('log_len', 'fwd_frac', 'bwd_frac', 'log_size_mean', 'log_size_std', 'log_size_min', 'log_size_max', 'log_dt_mean', 'log_dt_std', 'log_dt_max', 'syn_frac', 'fin_frac', 'rst_frac', 'psh_frac', 'ack_frac', 'log_win_mean')
@dataclass
class UnifiedData:
train_flow: np.ndarray
val_flow: np.ndarray
attack_flow: np.ndarray
train_packets: np.ndarray
val_packets: np.ndarray
attack_packets: np.ndarray
train_len: np.ndarray
val_len: np.ndarray
attack_len: np.ndarray
attack_labels: np.ndarray
packet_mean: np.ndarray
packet_std: np.ndarray
flow_mean: np.ndarray
flow_std: np.ndarray
packet_preprocess: str
flow_feature_names: tuple[str, ...]
packet_feature_names: tuple[str, ...] = PACKET_FEATURE_NAMES
@property
def T(self) -> int:
return int(self.train_packets.shape[1])
@property
def packet_dim(self) -> int:
return int(self.train_packets.shape[2])
@property
def flow_dim(self) -> int:
return int(self.train_flow.shape[1])
def _preprocess_packets(train_x: np.ndarray, val_x: np.ndarray, attack_x: np.ndarray, train_l: np.ndarray, val_l: np.ndarray, attack_l: np.ndarray, preprocess: str, seed: int) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
if preprocess not in ('zscore', 'mixed_dequant'):
raise ValueError("packet_preprocess must be 'zscore' or 'mixed_dequant'")
(mean, std) = _fit_packet_stats(train_x, train_l)
def prep(x: np.ndarray, l: np.ndarray, tag: str) -> np.ndarray:
if preprocess == 'zscore':
z = _zscore(x, mean, std)
mask = np.arange(x.shape[1])[None, :] < l[:, None]
return (z * mask[:, :, None]).astype(np.float32)
return _apply_mixed_dequant(x, l, mean, std, split_tag=tag, seed=seed)
return (prep(train_x, train_l, 'train'), prep(val_x, val_l, 'val'), prep(attack_x, attack_l, 'attack'), mean, std)
def _derive_flow_features(tokens: np.ndarray, lens: np.ndarray) -> np.ndarray:
(N, T, _) = tokens.shape
out = np.zeros((N, len(DERIVED_FLOW_FEATURE_NAMES)), dtype=np.float32)
for i in range(N):
n = int(max(lens[i], 1))
x = tokens[i, :n]
direction = x[:, 2]
size = x[:, 0]
dt = x[:, 1]
win = x[:, 8]
out[i, 0] = np.log1p(n)
out[i, 1] = np.mean(direction < 0.5)
out[i, 2] = np.mean(direction >= 0.5)
out[i, 3] = size.mean()
out[i, 4] = size.std()
out[i, 5] = size.min()
out[i, 6] = size.max()
out[i, 7] = dt.mean()
out[i, 8] = dt.std()
out[i, 9] = dt.max()
out[i, 10] = x[:, 3].mean()
out[i, 11] = x[:, 4].mean()
out[i, 12] = x[:, 5].mean()
out[i, 13] = x[:, 6].mean()
out[i, 14] = x[:, 7].mean()
out[i, 15] = win.mean()
return out
def _read_flow_features(path: Path, *, expected_rows: int, feature_columns: Optional[list[str]]=None) -> tuple[np.ndarray, tuple[str, ...], np.ndarray | None]:
path = Path(path)
if path.suffix == '.npz':
data = np.load(path, allow_pickle=True)
x = data['features'].astype(np.float32)
raw_names = data['feature_names'] if 'feature_names' in data.files else np.arange(x.shape[1])
names = tuple((str(v) for v in raw_names))
flow_id = data['flow_id'] if 'flow_id' in data.files else None
elif path.suffix in ('.parquet', '.pq'):
df = pd.read_parquet(path)
flow_id = df['flow_id'].to_numpy() if 'flow_id' in df.columns else None
if feature_columns:
cols = feature_columns
else:
cols = [c for c in df.columns if c not in DEFAULT_FLOW_META_COLUMNS and pd.api.types.is_numeric_dtype(df[c])]
if not cols:
raise ValueError(f'no numeric flow feature columns found in {path}')
x = df[cols].to_numpy(dtype=np.float32)
names = tuple(cols)
else:
raise ValueError(f'unsupported flow feature file: {path}')
if len(x) != expected_rows:
raise ValueError(f'flow feature row count {len(x):,} != packet row count {expected_rows:,}')
x = np.nan_to_num(x, nan=0.0, posinf=0.0, neginf=0.0).astype(np.float32)
return (x, names, flow_id)
def _feature_columns_from_df(df: pd.DataFrame, requested: Optional[list[str]]) -> list[str]:
if requested:
return requested
return [c for c in df.columns if c not in DEFAULT_FLOW_META_COLUMNS and pd.api.types.is_numeric_dtype(df[c])]
def _align_flow_features_by_scan(feature_df: pd.DataFrame, packet_flows: pd.DataFrame, *, feature_columns: list[str]) -> tuple[np.ndarray, tuple[str, ...]]:
required = ['label', 'src_ip', 'src_port', 'dst_ip', 'dst_port', 'protocol']
missing_feature = [c for c in required if c not in feature_df.columns]
missing_packet = [c for c in required if c not in packet_flows.columns]
if missing_feature or missing_packet:
raise ValueError(f'scan alignment requires label + 5-tuple metadata. missing in feature_df={missing_feature}, packet_flows={missing_packet}')
packet_keys = [(str(lbl), _canonical_key(src, sp, dst, dp, proto)) for (lbl, src, sp, dst, dp, proto) in zip(packet_flows['label'].to_numpy(), packet_flows['src_ip'].to_numpy(), packet_flows['src_port'].to_numpy(), packet_flows['dst_ip'].to_numpy(), packet_flows['dst_port'].to_numpy(), packet_flows['protocol'].to_numpy())]
labels = feature_df['label'].to_numpy()
src_ip = feature_df['src_ip'].to_numpy()
src_port = feature_df['src_port'].to_numpy()
dst_ip = feature_df['dst_ip'].to_numpy()
dst_port = feature_df['dst_port'].to_numpy()
protocol = feature_df['protocol'].to_numpy()
matched: list[int] = []
j = 0
n_csv = len(feature_df)
for (i, target) in enumerate(packet_keys):
while j < n_csv:
cand = (str(labels[j]), _canonical_key(src_ip[j], src_port[j], dst_ip[j], dst_port[j], protocol[j]))
j += 1
if cand == target:
matched.append(j - 1)
break
else:
raise ValueError(f'failed to align packet flow row {i:,}/{len(packet_keys):,}; the CSV cache may not be the same one used for packet extraction')
print(f'[data] scan-aligned CSV flow features: matched={len(matched):,} from csv_rows={n_csv:,} skipped={matched[-1] + 1 - len(matched):,}')
x = feature_df.iloc[matched][feature_columns].to_numpy(dtype=np.float32)
x = np.nan_to_num(x, nan=0.0, posinf=0.0, neginf=0.0).astype(np.float32)
return (x, tuple(feature_columns))
def _read_aligned_flow_features(path: Path, packet_flows: pd.DataFrame, *, feature_columns: Optional[list[str]]=None, align: str='auto') -> tuple[np.ndarray, tuple[str, ...]]:
path = Path(path)
if align not in ('auto', 'row', 'scan'):
raise ValueError("flow_features_align must be 'auto', 'row', or 'scan'")
if path.suffix == '.npz':
(x, names, flow_id) = _read_flow_features(path, expected_rows=len(packet_flows), feature_columns=feature_columns)
packet_id = packet_flows['flow_id'].to_numpy() if 'flow_id' in packet_flows else None
if flow_id is not None and packet_id is not None and (not np.array_equal(flow_id, packet_id)):
raise ValueError('NPZ flow_id does not align with Packet_CFM flows')
return (x, names)
if path.suffix not in ('.parquet', '.pq'):
raise ValueError(f'unsupported flow feature file: {path}')
feature_df = pd.read_parquet(path)
cols = _feature_columns_from_df(feature_df, feature_columns)
if not cols:
raise ValueError(f'no numeric flow feature columns found in {path}')
packet_id = packet_flows['flow_id'].to_numpy() if 'flow_id' in packet_flows else None
if len(feature_df) == len(packet_flows):
feature_id = feature_df['flow_id'].to_numpy() if 'flow_id' in feature_df.columns else None
if feature_id is None or packet_id is None or np.array_equal(feature_id, packet_id):
x = feature_df[cols].to_numpy(dtype=np.float32)
x = np.nan_to_num(x, nan=0.0, posinf=0.0, neginf=0.0).astype(np.float32)
return (x, tuple(cols))
if align == 'row':
raise ValueError("flow_id mismatch with flow_features_align='row'")
if align == 'row':
raise ValueError(f'row alignment requested but feature rows={len(feature_df):,} packet rows={len(packet_flows):,}')
return _align_flow_features_by_scan(feature_df, packet_flows, feature_columns=cols)
def _preprocess_flow(train: np.ndarray, val: np.ndarray, attack: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
mean = train.mean(axis=0).astype(np.float32)
std = train.std(axis=0).astype(np.float32)
return (_zscore(train, mean, std), _zscore(val, mean, std), _zscore(attack, mean, std), mean, std)
def load_unified_data(*, packets_npz: Path | None=None, source_store: Path | None=None, flows_parquet: Path, flow_features_path: Path | None=None, flow_feature_columns: Optional[list[str]]=None, flow_features_align: str='auto', T: int=128, split_seed: int=42, train_ratio: float=0.8, benign_label: str='normal', min_len: int=2, packet_preprocess: str='mixed_dequant', attack_cap: int | None=None, val_cap: int | None=None) -> UnifiedData:
if (packets_npz is None) == (source_store is None):
raise ValueError('pass exactly one of packets_npz or source_store')
flows_parquet = Path(flows_parquet)
print(f'[data] flows={flows_parquet} packets_source={(packets_npz if packets_npz else source_store)}')
flow_cols = ['flow_id', 'label']
if flow_features_path is not None:
flow_cols += ['src_ip', 'src_port', 'dst_ip', 'dst_port', 'protocol']
flows = pd.read_parquet(flows_parquet, columns=flow_cols)
labels_full = flows['label'].to_numpy().astype(str)
flow_id = flows['flow_id'].to_numpy()
tokens_full: np.ndarray | None = None
store = None
if packets_npz is not None:
pz = np.load(Path(packets_npz))
tokens_full = pz['packet_tokens'].astype(np.float32)
lens_full = pz['packet_lengths'].astype(np.int32)
packet_flow_id = pz['flow_id'] if 'flow_id' in pz.files else None
if T > tokens_full.shape[1]:
raise ValueError(f'requested T={T} > stored T_full={tokens_full.shape[1]}')
tokens_full = tokens_full[:, :T].copy()
lens_full = np.minimum(lens_full, T).astype(np.int32)
if packet_flow_id is not None and (not np.array_equal(packet_flow_id, flow_id)):
raise ValueError('packets_npz and flows_parquet are not row-aligned by flow_id')
else:
if flow_features_path is None:
raise ValueError('source_store path requires flow_features_path (derived features need tokens in memory)')
from common.packet_store import PacketShardStore
store = PacketShardStore.open(Path(source_store))
store_flow_id = store.read_flows(columns=['flow_id'])['flow_id'].to_numpy()
if not np.array_equal(store_flow_id, flow_id):
raise ValueError('source_store and flows_parquet are not row-aligned by flow_id')
lens_full = np.minimum(store.manifest['packet_length'].to_numpy(dtype=np.int32), T)
if flow_features_path is None:
assert tokens_full is not None
flow_features = _derive_flow_features(tokens_full, lens_full)
flow_names = DERIVED_FLOW_FEATURE_NAMES
print(f'[data] using derived flow features D={flow_features.shape[1]}')
else:
(flow_features, flow_names) = _read_aligned_flow_features(Path(flow_features_path), flows, feature_columns=flow_feature_columns, align=flow_features_align)
print(f'[data] using external flow features D={flow_features.shape[1]}')
keep = lens_full >= min_len
labels = labels_full[keep]
flow_features = flow_features[keep]
lens = lens_full[keep]
global_idx = np.flatnonzero(keep).astype(np.int64)
if tokens_full is not None:
materialized_tokens = tokens_full[keep]
else:
materialized_tokens = None
print(f'[data] rows total={len(keep):,} keep len>={min_len}: {keep.sum():,}')
benign_local = np.where(labels == benign_label)[0]
attack_local = np.where(labels != benign_label)[0]
rng = np.random.default_rng(split_seed)
rng.shuffle(benign_local)
n_train = int(len(benign_local) * train_ratio)
train_local = benign_local[:n_train]
val_local = benign_local[n_train:]
if val_cap is not None and len(val_local) > val_cap:
val_local = np.sort(rng.choice(val_local, size=val_cap, replace=False))
if attack_cap is not None and len(attack_local) > attack_cap:
attack_local = np.sort(rng.choice(attack_local, size=attack_cap, replace=False))
print(f'[data] benign={len(benign_local):,} attack={len(attack_local):,} -> train={len(train_local):,} val={len(val_local):,}')
def _materialize(local_indices: np.ndarray) -> np.ndarray:
if materialized_tokens is not None:
return materialized_tokens[local_indices].astype(np.float32, copy=False)
assert store is not None
g = global_idx[local_indices]
(tok, _) = store.read_packets(g.astype(np.int64), T=T)
return tok.astype(np.float32, copy=False)
tr_p_raw = _materialize(train_local)
va_p_raw = _materialize(val_local)
at_p_raw = _materialize(attack_local)
tr_l = lens[train_local]
va_l = lens[val_local]
at_l = lens[attack_local]
tr_f_raw = flow_features[train_local]
va_f_raw = flow_features[val_local]
at_f_raw = flow_features[attack_local]
train_idx = train_local
val_idx = val_local
attack_idx = attack_local
(tr_p, va_p, at_p, p_mean, p_std) = _preprocess_packets(tr_p_raw, va_p_raw, at_p_raw, tr_l, va_l, at_l, preprocess=packet_preprocess, seed=split_seed)
(tr_f, va_f, at_f, f_mean, f_std) = _preprocess_flow(tr_f_raw, va_f_raw, at_f_raw)
return UnifiedData(train_flow=tr_f, val_flow=va_f, attack_flow=at_f, train_packets=tr_p, val_packets=va_p, attack_packets=at_p, train_len=tr_l, val_len=va_l, attack_len=at_l, attack_labels=labels[attack_idx], packet_mean=p_mean, packet_std=p_std, flow_mean=f_mean, flow_std=f_std, packet_preprocess=packet_preprocess, flow_feature_names=tuple(flow_names))
def subsample_train(data: UnifiedData, n_train: int, seed: int) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
if n_train <= 0 or n_train >= len(data.train_flow):
return (data.train_flow, data.train_packets, data.train_len)
rng = np.random.default_rng(seed)
idx = rng.choice(len(data.train_flow), n_train, replace=False)
idx.sort()
return (data.train_flow[idx], data.train_packets[idx], data.train_len[idx])

588
Unified_CFM/model.py Normal file
View File

@@ -0,0 +1,588 @@
from __future__ import annotations
import math
from dataclasses import dataclass
import torch
import torch.nn as nn
from torchdiffeq import odeint
@torch.no_grad()
def _sinkhorn_coupling(C: torch.Tensor, reg: float=0.05, n_iter: int=20) -> torch.Tensor:
C = C.float()
log_k = -C / reg
B = C.shape[0]
log_u = torch.zeros(B, device=C.device)
log_v = torch.zeros(B, device=C.device)
for _ in range(n_iter):
log_v = -torch.logsumexp(log_k + log_u.unsqueeze(1), dim=0)
log_u = -torch.logsumexp(log_k + log_v.unsqueeze(0), dim=1)
log_p = log_u.unsqueeze(1) + log_k + log_v.unsqueeze(0)
return log_p.argmax(dim=1)
class SinusoidalTimeEmb(nn.Module):
def __init__(self, dim: int) -> None:
super().__init__()
if dim % 2 != 0:
raise ValueError('time embedding dimension must be even')
self.dim = dim
def forward(self, t: torch.Tensor) -> torch.Tensor:
half = self.dim // 2
freqs = torch.exp(-math.log(10000) * torch.arange(half, device=t.device, dtype=t.dtype) / max(half - 1, 1))
args = t[:, None] * freqs[None, :]
return torch.cat([args.sin(), args.cos()], dim=-1)
class AdaLNBlock(nn.Module):
def __init__(self, d_model: int, n_heads: int, mlp_ratio: float, cond_dim: int) -> None:
super().__init__()
self.norm1 = nn.LayerNorm(d_model, elementwise_affine=False)
self.attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
self.norm2 = nn.LayerNorm(d_model, elementwise_affine=False)
hidden = int(d_model * mlp_ratio)
self.mlp = nn.Sequential(nn.Linear(d_model, hidden), nn.GELU(), nn.Linear(hidden, d_model))
self.cond_proj = nn.Linear(cond_dim, 6 * d_model)
nn.init.zeros_(self.cond_proj.weight)
nn.init.zeros_(self.cond_proj.bias)
@staticmethod
def _modulate(x: torch.Tensor, gamma: torch.Tensor, beta: torch.Tensor) -> torch.Tensor:
return x * (1.0 + gamma[:, None, :]) + beta[:, None, :]
def forward(self, x: torch.Tensor, cond: torch.Tensor, key_padding_mask: torch.Tensor | None, attn_mask: torch.Tensor | None=None) -> torch.Tensor:
(g1, b1, a1, g2, b2, a2) = self.cond_proj(cond).chunk(6, dim=-1)
h = self._modulate(self.norm1(x), g1, b1)
(attn_out, _) = self.attn(h, h, h, key_padding_mask=key_padding_mask, attn_mask=attn_mask, need_weights=False)
x = x + a1[:, None, :] * attn_out
h = self._modulate(self.norm2(x), g2, b2)
return x + a2[:, None, :] * self.mlp(h)
class UnifiedVelocity(nn.Module):
def __init__(self, token_dim: int, seq_len: int, d_model: int=128, n_layers: int=4, n_heads: int=4, mlp_ratio: float=4.0, time_dim: int=64, reference_mode: str | None=None) -> None:
super().__init__()
if reference_mode not in (None, 'independent_token', 'block_diagonal', 'causal_packets', 'causal_all'):
raise ValueError(f'unknown reference_mode={reference_mode!r}')
self.token_dim = token_dim
self.seq_len = seq_len
self.reference_mode = reference_mode
self.input_proj = nn.Linear(token_dim, d_model)
self.pos_emb = nn.Parameter(torch.zeros(1, seq_len, d_model))
self.type_emb = nn.Embedding(2, d_model)
nn.init.trunc_normal_(self.pos_emb, std=0.02)
nn.init.normal_(self.type_emb.weight, std=0.02)
self.time_emb = SinusoidalTimeEmb(time_dim)
self.cond_mlp = nn.Sequential(nn.Linear(time_dim, d_model), nn.SiLU(), nn.Linear(d_model, d_model))
self.blocks = nn.ModuleList([AdaLNBlock(d_model, n_heads, mlp_ratio, cond_dim=d_model) for _ in range(n_layers)])
self.out_norm = nn.LayerNorm(d_model, elementwise_affine=False)
self.out = nn.Linear(d_model, token_dim)
nn.init.zeros_(self.out.weight)
nn.init.zeros_(self.out.bias)
type_ids = torch.ones(seq_len, dtype=torch.long)
type_ids[0] = 0
self.register_buffer('type_ids', type_ids, persistent=False)
def forward(self, x: torch.Tensor, t: torch.Tensor, key_padding_mask: torch.Tensor | None=None, attn_mask_override: torch.Tensor | None=None) -> torch.Tensor:
(B, L, _) = x.shape
if L > self.seq_len:
raise ValueError(f'sequence length {L} exceeds configured {self.seq_len}')
if t.dim() == 0:
t = t.expand(B)
h = self.input_proj(x)
h = h + self.pos_emb[:, :L, :]
h = h + self.type_emb(self.type_ids[:L])[None, :, :]
cond = self.cond_mlp(self.time_emb(t))
if attn_mask_override is not None:
attn_mask = attn_mask_override
else:
attn_mask = self._reference_attn_mask(L, x.device)
for block in self.blocks:
h = block(h, cond, key_padding_mask, attn_mask=attn_mask)
return self.out(self.out_norm(h))
def _reference_attn_mask(self, L: int, device: torch.device) -> torch.Tensor | None:
if self.reference_mode is None:
return None
if self.reference_mode == 'independent_token':
return ~torch.eye(L, dtype=torch.bool, device=device)
if self.reference_mode == 'block_diagonal':
mask = torch.ones((L, L), dtype=torch.bool, device=device)
mask[0, 0] = False
if L > 1:
mask[1:, 1:] = False
return mask
if self.reference_mode == 'causal_packets':
mask = torch.zeros((L, L), dtype=torch.bool, device=device)
if L > 1:
packet_causal = torch.triu(torch.ones(L - 1, L - 1, dtype=torch.bool, device=device), diagonal=1)
mask[1:, 1:] = packet_causal
return mask
if self.reference_mode == 'causal_all':
return torch.triu(torch.ones(L, L, dtype=torch.bool, device=device), diagonal=1)
raise AssertionError(self.reference_mode)
@dataclass
class UnifiedCFMConfig:
T: int = 128
packet_dim: int = 9
flow_dim: int = 16
token_dim: int | None = None
d_model: int = 128
n_layers: int = 4
n_heads: int = 4
mlp_ratio: float = 4.0
time_dim: int = 64
sigma: float = 0.1
use_ot: bool = False
reference_mode: str | None = None
class UnifiedTokenCFM(nn.Module):
def __init__(self, cfg: UnifiedCFMConfig) -> None:
super().__init__()
self.cfg = cfg
self.token_dim = cfg.token_dim or 1 + max(cfg.flow_dim, cfg.packet_dim)
if self.token_dim < 1 + max(cfg.flow_dim, cfg.packet_dim):
raise ValueError('token_dim is too small for flow_dim/packet_dim')
self.seq_len = cfg.T + 1
self.velocity = UnifiedVelocity(token_dim=self.token_dim, seq_len=self.seq_len, d_model=cfg.d_model, n_layers=cfg.n_layers, n_heads=cfg.n_heads, mlp_ratio=cfg.mlp_ratio, time_dim=cfg.time_dim, reference_mode=cfg.reference_mode)
def build_tokens(self, flow: torch.Tensor, packets: torch.Tensor) -> torch.Tensor:
(B, T, Dp) = packets.shape
if T != self.cfg.T:
raise ValueError(f'packet T={T} but config T={self.cfg.T}')
if Dp != self.cfg.packet_dim:
raise ValueError(f'packet_dim={Dp} but config packet_dim={self.cfg.packet_dim}')
if flow.shape[-1] != self.cfg.flow_dim:
raise ValueError(f'flow_dim={flow.shape[-1]} but config flow_dim={self.cfg.flow_dim}')
z = packets.new_zeros((B, T + 1, self.token_dim))
z[:, 0, 0] = -1.0
z[:, 0, 1:1 + self.cfg.flow_dim] = flow
z[:, 1:, 0] = 1.0
z[:, 1:, 1:1 + self.cfg.packet_dim] = packets
return z
def key_padding_mask(self, lens: torch.Tensor) -> torch.Tensor:
B = lens.shape[0]
idx = torch.arange(self.cfg.T, device=lens.device)[None, :]
packet_real = idx < lens[:, None]
real = torch.cat([torch.ones(B, 1, dtype=torch.bool, device=lens.device), packet_real], dim=1)
return ~real
def _loss_mask(self, lens: torch.Tensor) -> torch.Tensor:
return (~self.key_padding_mask(lens)).float()
@staticmethod
def _masked_trimmed_mean(values: torch.Tensor, mask: torch.Tensor, trim_frac: float=0.1) -> torch.Tensor:
out = values.new_zeros(values.shape[0])
for i in range(values.shape[0]):
v = values[i][mask[i] > 0]
if v.numel() == 0:
continue
if v.numel() < 5:
out[i] = v.mean()
continue
v_sorted = torch.sort(v).values
lo = int(trim_frac * v_sorted.numel())
hi = int((1.0 - trim_frac) * v_sorted.numel())
if hi <= lo:
out[i] = v_sorted.mean()
else:
out[i] = v_sorted[lo:hi].mean()
return out
@staticmethod
def _masked_median(values: torch.Tensor, mask: torch.Tensor) -> torch.Tensor:
out = values.new_zeros(values.shape[0])
for i in range(values.shape[0]):
v = values[i][mask[i] > 0]
if v.numel() == 0:
continue
v_sorted = torch.sort(v).values
mid = v_sorted.numel() // 2
if v_sorted.numel() % 2:
out[i] = v_sorted[mid]
else:
out[i] = 0.5 * (v_sorted[mid - 1] + v_sorted[mid])
return out
def compute_loss(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, *, lambda_flow: float=0.0, lambda_packet: float=0.0, packet_mask_ratio: float=0.5, return_components: bool=False) -> torch.Tensor | dict[str, torch.Tensor]:
x1 = self.build_tokens(flow, packets)
B = x1.shape[0]
x0 = torch.randn_like(x1)
mask = self._loss_mask(lens)
kpm = mask == 0
if self.cfg.use_ot:
flat0 = (x0 * mask[:, :, None]).reshape(B, -1)
flat1 = (x1 * mask[:, :, None]).reshape(B, -1)
col = _sinkhorn_coupling(torch.cdist(flat0.float(), flat1.float()))
x1 = x1[col]
flow = flow[col]
packets = packets[col]
lens = lens[col]
mask = self._loss_mask(lens)
kpm = mask == 0
t = torch.rand(B, device=x1.device)
x_t = (1.0 - t[:, None, None]) * x0 + t[:, None, None] * x1
if self.cfg.sigma > 0:
std = self.cfg.sigma * torch.sqrt(t * (1.0 - t))[:, None, None]
x_t = x_t + std * torch.randn_like(x_t)
target = x1 - x0
pred = self.velocity(x_t, t, key_padding_mask=kpm)
sq = (pred - target).square().mean(dim=-1)
per_sample = (sq * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
main_loss = per_sample.mean()
aux_flow_loss = x1.new_zeros(())
aux_packet_loss = x1.new_zeros(())
if lambda_flow > 0.0:
x_t_mf = x_t.clone()
x_t_mf[:, 0, :] = 0.0
pred_mf = self.velocity(x_t_mf, t, key_padding_mask=kpm)
err = (pred_mf[:, 0] - target[:, 0]).square().mean(dim=-1)
aux_flow_loss = err.mean()
if lambda_packet > 0.0:
packet_real = mask[:, 1:] > 0
rand_draw = torch.rand(packet_real.shape, device=x1.device)
mask_pkt = (rand_draw < packet_mask_ratio) & packet_real
pkt_mask_full = torch.cat([torch.zeros(B, 1, dtype=torch.bool, device=x1.device), mask_pkt], dim=1)
x_t_mp = x_t.clone()
x_t_mp[pkt_mask_full] = 0.0
pred_mp = self.velocity(x_t_mp, t, key_padding_mask=kpm)
sq_mp = (pred_mp - target).square().mean(dim=-1)
mask_f = pkt_mask_full.float()
denom = mask_f.sum(dim=-1).clamp_min(1.0)
aux_packet_loss = ((sq_mp * mask_f).sum(dim=-1) / denom).mean()
total = main_loss + lambda_flow * aux_flow_loss + lambda_packet * aux_packet_loss
if return_components:
return {'total': total, 'main': main_loss.detach(), 'aux_flow': aux_flow_loss.detach(), 'aux_packet': aux_packet_loss.detach()}
return total
@torch.no_grad()
def velocity_score(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, t_eval: tuple[float, ...]=(0.5, 0.75, 1.0)) -> dict[str, torch.Tensor]:
x = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
total = torch.zeros(x.shape[0], device=x.device)
flow_s = torch.zeros_like(total)
packet_s = torch.zeros_like(total)
packet_count = mask[:, 1:].sum(dim=-1).clamp_min(1.0)
for t_val in t_eval:
t = torch.full((x.shape[0],), float(t_val), device=x.device)
v = self.velocity(x, t, key_padding_mask=kpm)
e = v.square().mean(dim=-1)
total = total + (e * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
flow_s = flow_s + e[:, 0]
packet_s = packet_s + (e[:, 1:] * mask[:, 1:]).sum(dim=-1) / packet_count
denom = float(len(t_eval))
return {'velocity_total': total / denom, 'velocity_flow': flow_s / denom, 'velocity_packet': packet_s / denom}
@torch.no_grad()
def trajectory_metrics(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, n_steps: int=16) -> dict[str, torch.Tensor]:
z = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
B = z.shape[0]
dt = 1.0 / n_steps
total_arc = torch.zeros(B, device=z.device)
total_ke = torch.zeros(B, device=z.device)
flow_ke = torch.zeros(B, device=z.device)
packet_ke = torch.zeros(B, device=z.device)
total_curv = torch.zeros(B, device=z.device)
flow_curv = torch.zeros(B, device=z.device)
packet_curv = torch.zeros(B, device=z.device)
packet_kappa2_speed2 = torch.zeros(B, max(z.shape[1] - 1, 0), device=z.device)
packet_count = mask[:, 1:].sum(dim=-1).clamp_min(1.0)
v_prev = None
v_prev_norm = None
for k in range(n_steps):
t_val = 1.0 - k * dt
t = torch.full((B,), t_val, device=z.device)
v = self.velocity(z, t, key_padding_mask=kpm)
e = v.square().mean(dim=-1)
v_norm = v.square().sum(dim=-1).clamp_min(1e-12).sqrt()
total_ke = total_ke + (e * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0) * dt
flow_ke = flow_ke + e[:, 0] * dt
packet_ke = packet_ke + (e[:, 1:] * mask[:, 1:]).sum(dim=-1) / packet_count * dt
if v_prev is not None:
dv = v - v_prev
dve = dv.square().mean(dim=-1)
total_curv = total_curv + (dve * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
flow_curv = flow_curv + dve[:, 0]
packet_curv = packet_curv + (dve[:, 1:] * mask[:, 1:]).sum(dim=-1) / packet_count
dv2_sum = dv[:, 1:].square().sum(dim=-1)
assert v_prev_norm is not None
v_avg = 0.5 * (v_norm[:, 1:] + v_prev_norm[:, 1:])
packet_kappa2_speed2 = packet_kappa2_speed2 + dv2_sum / v_avg.square().clamp_min(1e-06)
v_prev = v
v_prev_norm = v_norm
z_new = z - v * dt
dz = (z_new - z) * mask[:, :, None]
total_arc = total_arc + dz.reshape(B, -1).norm(dim=-1) / mask.sum(dim=-1).sqrt()
z = z_new
z_masked = z * mask[:, :, None]
terminal = z_masked.reshape(B, -1).norm(dim=-1) / (mask.sum(dim=-1) * self.token_dim).clamp_min(1.0).sqrt()
terminal_flow = z[:, 0].norm(dim=-1) / math.sqrt(self.token_dim)
terminal_packet = (z[:, 1:] * mask[:, 1:, None]).reshape(B, -1).norm(dim=-1) / (packet_count * self.token_dim).sqrt()
packet_mask = mask[:, 1:]
kappa2_speed2_mean = (packet_kappa2_speed2 * packet_mask).sum(dim=-1) / packet_count
kappa2_speed2_median = self._masked_median(packet_kappa2_speed2, packet_mask)
kappa2_speed2_trimmed = self._masked_trimmed_mean(packet_kappa2_speed2, packet_mask)
return {'terminal_norm': terminal, 'terminal_flow': terminal_flow, 'terminal_packet': terminal_packet, 'arc_length': total_arc, 'kinetic_energy': total_ke, 'kinetic_flow': flow_ke, 'kinetic_packet': packet_ke, 'curvature_total': total_curv, 'curvature_flow': flow_curv, 'curvature_packet': packet_curv, 'kappa2_speed2norm_packet_mean': kappa2_speed2_mean, 'kappa2_speed2norm_packet_median': kappa2_speed2_median, 'kappa2_speed2norm_packet_trimmed10_mean': kappa2_speed2_trimmed}
@torch.no_grad()
def score_profile_vt(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, t_eval: tuple[float, ...]=(0.1, 0.3, 0.5, 0.7, 0.9, 1.0)) -> dict[str, torch.Tensor]:
x = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
packet_count = mask[:, 1:].sum(dim=-1).clamp_min(1.0)
out: dict[str, torch.Tensor] = {}
for t_val in t_eval:
t = torch.full((x.shape[0],), float(t_val), device=x.device)
v = self.velocity(x, t, key_padding_mask=kpm)
e = v.square().mean(dim=-1)
tag = f't{int(round(t_val * 10)):02d}'
out[f'velocity_total_{tag}'] = (e * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
out[f'velocity_flow_{tag}'] = e[:, 0]
out[f'velocity_packet_{tag}'] = (e[:, 1:] * mask[:, 1:]).sum(dim=-1) / packet_count
return out
@torch.no_grad()
def consistency_score(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, t_eval: float=0.5) -> dict[str, torch.Tensor]:
x = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
B = x.shape[0]
packet_count = mask[:, 1:].sum(dim=-1).clamp_min(1.0)
t = torch.full((B,), float(t_eval), device=x.device)
v_full = self.velocity(x, t, key_padding_mask=kpm)
x_mf = x.clone()
x_mf[:, 0, :] = 0.0
v_mf = self.velocity(x_mf, t, key_padding_mask=kpm)
flow_cons = (v_full[:, 0] - v_mf[:, 0]).square().mean(dim=-1)
x_mp = x.clone()
pkt_mask_full = mask[:, 1:] > 0
idx_pkt_mask = torch.cat([torch.zeros(B, 1, dtype=torch.bool, device=x.device), pkt_mask_full], dim=1)
x_mp[idx_pkt_mask] = 0.0
v_mp = self.velocity(x_mp, t, key_padding_mask=kpm)
diff = (v_full - v_mp).square().mean(dim=-1)
packet_cons = (diff[:, 1:] * mask[:, 1:]).sum(dim=-1) / packet_count
return {'flow_consistency': flow_cons, 'packet_consistency': packet_cons, 'consistency_total': flow_cons + packet_cons}
def jacobian_hutchinson(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, t_eval: tuple[float, ...]=(0.5,), n_eps: int=4, generator: torch.Generator | None=None) -> dict[str, torch.Tensor]:
x = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
B = x.shape[0]
packet_count = mask[:, 1:].sum(dim=-1).clamp_min(1.0)
total = torch.zeros(B, device=x.device)
flow_j = torch.zeros(B, device=x.device)
packet_j = torch.zeros(B, device=x.device)
n_draws = n_eps * len(t_eval)
for t_val in t_eval:
t_current = torch.full((B,), float(t_val), device=x.device)
for _ in range(n_eps):
x_req = x.detach().clone().requires_grad_(True)
v = self.velocity(x_req, t_current, key_padding_mask=kpm)
eps = torch.randn(v.shape, device=v.device, generator=generator)
(g,) = torch.autograd.grad(outputs=v, inputs=x_req, grad_outputs=eps, retain_graph=False, create_graph=False)
e = g.square().mean(dim=-1)
total = total + (e * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
flow_j = flow_j + e[:, 0]
packet_j = packet_j + (e[:, 1:] * mask[:, 1:]).sum(dim=-1) / packet_count
return {'jacobian_total': (total / n_draws).detach(), 'jacobian_flow': (flow_j / n_draws).detach(), 'jacobian_packet': (packet_j / n_draws).detach()}
@torch.no_grad()
def pna_score(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, n_steps: int=16, flow_masked: bool=False) -> dict[str, torch.Tensor]:
eps_v2 = 1e-06
dt = 1.0 / n_steps
z = self.build_tokens(flow, packets)
if flow_masked:
z = z.clone()
z[:, 0, :] = 0.0
mask = self._loss_mask(lens)
kpm = mask == 0
(B, L, _) = z.shape
pna = torch.zeros(B, L, device=z.device)
v_prev: torch.Tensor | None = None
v_norm_prev: torch.Tensor | None = None
for k in range(n_steps):
t_val = 1.0 - k * dt
t = torch.full((B,), t_val, device=z.device)
v = self.velocity(z, t, key_padding_mask=kpm)
v_norm = (v.square().sum(dim=-1) + 1e-12).sqrt()
if v_prev is not None:
dv2 = (v - v_prev).square().sum(dim=-1)
v_avg2 = (0.5 * (v_norm + v_norm_prev)).square().clamp_min(eps_v2)
pna = pna + dv2 / v_avg2
v_prev = v
v_norm_prev = v_norm
z = z - v * dt
if flow_masked:
z[:, 0, :] = 0.0
flow_pna = pna[:, 0]
packet_pna = pna[:, 1:]
packet_mask = mask[:, 1:]
packet_count = packet_mask.sum(dim=-1).clamp_min(1.0)
pna_median = self._masked_median(packet_pna, packet_mask)
pna_mean = (packet_pna * packet_mask).sum(dim=-1) / packet_count
masked_for_max = packet_pna.masked_fill(packet_mask == 0, float('-inf'))
pna_max = masked_for_max.max(dim=-1).values
pna_trimmed = self._masked_trimmed_mean(packet_pna, packet_mask)
return {'pna_packet_median': pna_median, 'pna_packet_mean': pna_mean, 'pna_packet_max': pna_max, 'pna_packet_trimmed10_mean': pna_trimmed, 'pna_flow': flow_pna}
@torch.no_grad()
def causal_consistency_score(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, t_eval: float=0.5) -> dict[str, torch.Tensor]:
x = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
(B, L, _) = x.shape
t = torch.full((B,), float(t_eval), device=x.device)
v_full = self.velocity(x, t, key_padding_mask=kpm)
causal = torch.triu(torch.ones(L, L, dtype=torch.bool, device=x.device), diagonal=1)
v_causal = self.velocity(x, t, key_padding_mask=kpm, attn_mask_override=causal)
diff = (v_full - v_causal).square().mean(dim=-1)
flow_surprisal = diff[:, 0]
packet_diff = diff[:, 1:]
packet_mask = mask[:, 1:]
packet_count = packet_mask.sum(dim=-1).clamp_min(1.0)
packet_mean = (packet_diff * packet_mask).sum(dim=-1) / packet_count
packet_median = self._masked_median(packet_diff, packet_mask)
masked_for_max = packet_diff.masked_fill(packet_mask == 0, float('-inf'))
packet_max = masked_for_max.max(dim=-1).values
packet_trimmed = self._masked_trimmed_mean(packet_diff, packet_mask)
total = (diff * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
return {'causal_surprisal_total': total, 'causal_surprisal_flow': flow_surprisal, 'causal_surprisal_packet_mean': packet_mean, 'causal_surprisal_packet_median': packet_median, 'causal_surprisal_packet_max': packet_max, 'causal_surprisal_packet_trimmed10_mean': packet_trimmed}
@torch.no_grad()
def direction_consistency_score(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, t_eval: tuple[float, ...]=(0.2, 0.4, 0.6, 0.8, 1.0)) -> dict[str, torch.Tensor]:
x = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
(B, L, _) = x.shape
t_eval = tuple(t_eval)
if len(t_eval) < 2:
raise ValueError('direction_consistency_score needs >=2 t values')
prev_v: torch.Tensor | None = None
drift = x.new_zeros(B, L)
n_pairs = len(t_eval) - 1
for t_val in t_eval:
t = torch.full((B,), float(t_val), device=x.device)
v = self.velocity(x, t, key_padding_mask=kpm)
if prev_v is not None:
num = (prev_v * v).sum(dim=-1)
denom = prev_v.norm(dim=-1).clamp_min(1e-08) * v.norm(dim=-1).clamp_min(1e-08)
cos = num / denom
drift = drift + (1.0 - cos)
prev_v = v
drift = drift / max(n_pairs, 1)
flow_drift = drift[:, 0]
packet_drift = drift[:, 1:]
packet_mask = mask[:, 1:]
packet_count = packet_mask.sum(dim=-1).clamp_min(1.0)
packet_mean = (packet_drift * packet_mask).sum(dim=-1) / packet_count
packet_median = self._masked_median(packet_drift, packet_mask)
masked_for_max = packet_drift.masked_fill(packet_mask == 0, float('-inf'))
packet_max = masked_for_max.max(dim=-1).values
packet_trimmed = self._masked_trimmed_mean(packet_drift, packet_mask)
total = (drift * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
return {'direction_drift_total': total, 'direction_drift_flow': flow_drift, 'direction_drift_packet_mean': packet_mean, 'direction_drift_packet_median': packet_median, 'direction_drift_packet_max': packet_max, 'direction_drift_packet_trimmed10_mean': packet_trimmed}
def inverse_flow_nll_score(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, n_steps: int=16, n_eps: int=4, compute_divergence: bool=True, generator: torch.Generator | None=None) -> dict[str, torch.Tensor]:
z = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
(B, L, D) = z.shape
dt = 1.0 / n_steps
accum_div = torch.zeros(B, device=z.device)
if compute_divergence:
for k in range(n_steps):
t_val = 1.0 - k * dt
t = torch.full((B,), t_val, device=z.device)
z_req = z.detach().clone().requires_grad_(True)
v = self.velocity(z_req, t, key_padding_mask=kpm)
div_step = torch.zeros(B, device=z.device)
for j in range(n_eps):
eps = torch.randn_like(v)
eps_masked = eps * mask[:, :, None]
retain = j < n_eps - 1
(g,) = torch.autograd.grad(outputs=v, inputs=z_req, grad_outputs=eps_masked, retain_graph=retain, create_graph=False)
div_step = div_step + (eps_masked * g).sum(dim=(1, 2))
div_step = div_step / float(n_eps)
accum_div = accum_div + div_step * dt
with torch.no_grad():
z = (z_req - v * dt).detach()
else:
with torch.no_grad():
for k in range(n_steps):
t_val = 1.0 - k * dt
t = torch.full((B,), t_val, device=z.device)
v = self.velocity(z, t, key_padding_mask=kpm)
z = z - v * dt
with torch.no_grad():
z_masked = z * mask[:, :, None]
n_real = mask.sum(dim=-1).clamp_min(1.0)
x0_quadratic = z_masked.reshape(B, -1).square().sum(dim=-1) / (n_real * float(D))
nll_x0_only = x0_quadratic
nll_div_only = accum_div / (n_real * float(D))
nll_full = nll_x0_only + nll_div_only
return {'nll_x0_only': nll_x0_only.detach(), 'nll_div_only': nll_div_only.detach(), 'nll_full': nll_full.detach()}
def jacobian_spectral_score(self, flow: torch.Tensor, packets: torch.Tensor, lens: torch.Tensor, t_eval: float=0.5, n_eps: int=4, generator: torch.Generator | None=None) -> dict[str, torch.Tensor]:
x = self.build_tokens(flow, packets)
mask = self._loss_mask(lens)
kpm = mask == 0
(B, L, D) = x.shape
t = torch.full((B,), float(t_eval), device=x.device)
packet_mask = mask[:, 1:]
packet_count = packet_mask.sum(dim=-1).clamp_min(1.0)
norms_total: list[torch.Tensor] = []
norms_flow: list[torch.Tensor] = []
norms_packet: list[torch.Tensor] = []
for _ in range(n_eps):
x_req = x.detach().clone().requires_grad_(True)
v = self.velocity(x_req, t, key_padding_mask=kpm)
eps = torch.randn(v.shape, device=v.device, generator=generator)
(g,) = torch.autograd.grad(outputs=v, inputs=x_req, grad_outputs=eps, retain_graph=False, create_graph=False)
e = g.square().mean(dim=-1)
n_total = (e * mask).sum(dim=-1) / mask.sum(dim=-1).clamp_min(1.0)
n_flow = e[:, 0]
n_packet = (e[:, 1:] * packet_mask).sum(dim=-1) / packet_count
norms_total.append(n_total.detach())
norms_flow.append(n_flow.detach())
norms_packet.append(n_packet.detach())
def _spectral_summary(samples: list[torch.Tensor]) -> dict[str, torch.Tensor]:
stack = torch.stack(samples, dim=1)
mean = stack.mean(dim=1).clamp_min(1e-12)
mx = stack.max(dim=1).values
mn = stack.min(dim=1).values
logfro = torch.log(mean)
aniso = mx / mean
min_over_max = mn / mx.clamp_min(1e-12)
p = stack / stack.sum(dim=1, keepdim=True).clamp_min(1e-12)
entropy = -(p * p.clamp_min(1e-12).log()).sum(dim=1)
eff_rank = torch.exp(entropy)
return {'logfro': logfro, 'anisotropy': aniso, 'min_over_max': min_over_max, 'eff_rank': eff_rank}
out: dict[str, torch.Tensor] = {}
for (tag, samples) in (('total', norms_total), ('flow', norms_flow), ('packet', norms_packet)):
summ = _spectral_summary(samples)
for (stat_name, val) in summ.items():
out[f'jac_{stat_name}_{tag}'] = val
return out
@torch.no_grad()
def sample(self, n: int, lens: torch.Tensor, device: torch.device, n_steps: int=50, method: str='euler') -> torch.Tensor:
z = torch.randn(n, self.seq_len, self.token_dim, device=device)
ts = torch.linspace(0.0, 1.0, n_steps + 1, device=device)
kpm = self.key_padding_mask(lens.to(device))
def f(t: torch.Tensor, x: torch.Tensor) -> torch.Tensor:
return self.velocity(x, t.expand(x.shape[0]), key_padding_mask=kpm)
if method == 'euler':
for i in range(n_steps):
z = z + f(ts[i], z) * (ts[i + 1] - ts[i])
return z
return odeint(f, z, ts, method=method)[-1]
def param_count(self) -> int:
return sum((p.numel() for p in self.parameters()))

157
Unified_CFM/tests/test_model_shapes.py Normal file
View File

@@ -0,0 +1,157 @@
import sys
from pathlib import Path
import torch
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from model import UnifiedCFMConfig, UnifiedTokenCFM
def _build_model():
return UnifiedTokenCFM(UnifiedCFMConfig(T=4, packet_dim=3, flow_dim=5, d_model=16, n_layers=1, n_heads=4, time_dim=8))
def _build_reference_model(reference_mode: str):
return UnifiedTokenCFM(UnifiedCFMConfig(T=4, packet_dim=3, flow_dim=5, d_model=16, n_layers=1, n_heads=4, time_dim=8, reference_mode=reference_mode))
def _sample_batch(seed: int=0):
torch.manual_seed(seed)
flow = torch.randn(2, 5)
packets = torch.randn(2, 4, 3)
lens = torch.tensor([4, 2])
return (flow, packets, lens)
def test_unified_cfm_shapes_and_scores():
model = _build_model()
(flow, packets, lens) = _sample_batch()
tokens = model.build_tokens(flow, packets)
assert tokens.shape == (2, 5, 6)
loss = model.compute_loss(flow, packets, lens)
assert loss.ndim == 0
assert torch.isfinite(loss)
traj = model.trajectory_metrics(flow, packets, lens, n_steps=2)
assert 'terminal_norm' in traj
assert traj['terminal_norm'].shape == (2,)
vel = model.velocity_score(flow, packets, lens)
assert set(vel) == {'velocity_total', 'velocity_flow', 'velocity_packet'}
def test_reference_mode_independent_token_shapes_and_scores():
model = _build_reference_model('independent_token')
(flow, packets, lens) = _sample_batch(seed=9)
loss = model.compute_loss(flow, packets, lens)
assert loss.ndim == 0
assert torch.isfinite(loss)
traj = model.trajectory_metrics(flow, packets, lens, n_steps=2)
assert traj['terminal_norm'].shape == (2,)
assert torch.all(torch.isfinite(traj['curvature_packet']))
def test_reference_mode_block_diagonal_shapes_and_scores():
model = _build_reference_model('block_diagonal')
(flow, packets, lens) = _sample_batch(seed=10)
loss = model.compute_loss(flow, packets, lens)
assert loss.ndim == 0
assert torch.isfinite(loss)
vel = model.velocity_score(flow, packets, lens)
assert set(vel) == {'velocity_total', 'velocity_flow', 'velocity_packet'}
def test_trajectory_curvature_keys_and_shapes():
model = _build_model()
(flow, packets, lens) = _sample_batch(seed=1)
traj = model.trajectory_metrics(flow, packets, lens, n_steps=4)
for key in ('curvature_total', 'curvature_flow', 'curvature_packet'):
assert key in traj, f'missing {key}'
assert traj[key].shape == (2,)
assert torch.all(torch.isfinite(traj[key]))
assert torch.all(traj[key] >= 0)
def test_trajectory_curvature_zero_with_one_step():
model = _build_model()
(flow, packets, lens) = _sample_batch(seed=2)
traj = model.trajectory_metrics(flow, packets, lens, n_steps=1)
for key in ('curvature_total', 'curvature_flow', 'curvature_packet'):
assert traj[key].abs().sum().item() == 0.0
def test_speed_normalized_packet_curvature_scores():
model = _build_model()
(flow, packets, lens) = _sample_batch(seed=11)
traj = model.trajectory_metrics(flow, packets, lens, n_steps=4)
keys = ('kappa2_speed2norm_packet_mean', 'kappa2_speed2norm_packet_median', 'kappa2_speed2norm_packet_trimmed10_mean')
for key in keys:
assert key in traj, f'missing {key}'
assert traj[key].shape == (2,)
assert torch.all(torch.isfinite(traj[key]))
assert torch.all(traj[key] >= 0)
one_step = model.trajectory_metrics(flow, packets, lens, n_steps=1)
for key in keys:
assert one_step[key].abs().sum().item() == 0.0
def test_score_profile_vt_shapes():
model = _build_model()
(flow, packets, lens) = _sample_batch(seed=3)
t_eval = (0.1, 0.3, 0.5, 0.7, 0.9, 1.0)
prof = model.score_profile_vt(flow, packets, lens, t_eval=t_eval)
assert len(prof) == 3 * len(t_eval)
for (k, v) in prof.items():
assert v.shape == (2,), k
assert torch.all(torch.isfinite(v))
assert torch.all(v >= 0)
assert 'velocity_total_t05' in prof
assert 'velocity_flow_t10' in prof
assert 'velocity_packet_t01' in prof
def test_compute_loss_backward_compat():
model = _build_model()
(flow, packets, lens) = _sample_batch(seed=5)
torch.manual_seed(0)
a = model.compute_loss(flow, packets, lens)
torch.manual_seed(0)
b = model.compute_loss(flow, packets, lens, lambda_flow=0.0, lambda_packet=0.0)
assert torch.allclose(a, b), f'λ=0 must match old loss; got {a.item()} vs {b.item()}'
def test_compute_loss_aux_components_finite():
model = _build_model()
(flow, packets, lens) = _sample_batch(seed=6)
torch.manual_seed(7)
comp = model.compute_loss(flow, packets, lens, lambda_flow=0.1, lambda_packet=0.1, return_components=True)
assert set(comp) == {'total', 'main', 'aux_flow', 'aux_packet'}
for (k, v) in comp.items():
assert torch.isfinite(v), k
assert v >= 0, f'{k} negative: {v.item()}'
def test_compute_loss_aux_affects_gradient():
model = _build_model()
with torch.no_grad():
model.velocity.out.weight.normal_(std=0.01)
for block in model.velocity.blocks:
block.cond_proj.weight.normal_(std=0.01)
(flow, packets, lens) = _sample_batch(seed=8)
torch.manual_seed(10)
total = model.compute_loss(flow, packets, lens, lambda_flow=1.0, lambda_packet=1.0)
total.backward()
some_grad = False
for p in model.parameters():
if p.grad is not None and p.grad.abs().sum().item() > 0:
some_grad = True
break
assert some_grad, 'no gradient flowed through aux losses'
def test_consistency_score_shapes():
model = _build_model()
(flow, packets, lens) = _sample_batch(seed=9)
cs = model.consistency_score(flow, packets, lens)
assert set(cs) == {'flow_consistency', 'packet_consistency', 'consistency_total'}
for (k, v) in cs.items():
assert v.shape == (2,), k
assert torch.all(torch.isfinite(v))
assert torch.all(v >= 0), k
def test_jacobian_hutchinson_shapes_and_nonneg():
model = _build_model()
with torch.no_grad():
model.velocity.out.weight.normal_(std=0.01)
for block in model.velocity.blocks:
block.cond_proj.weight.normal_(std=0.01)
(flow, packets, lens) = _sample_batch(seed=4)
gen = torch.Generator().manual_seed(42)
jac = model.jacobian_hutchinson(flow, packets, lens, t_eval=(0.5,), n_eps=2, generator=gen)
assert set(jac) == {'jacobian_total', 'jacobian_flow', 'jacobian_packet'}
for (k, v) in jac.items():
assert v.shape == (2,), k
assert torch.all(torch.isfinite(v))
assert torch.all(v >= 0), f'{k} has negative value'

147
Unified_CFM/train.py Normal file
View File

@@ -0,0 +1,147 @@
from __future__ import annotations
import argparse
import json
import time
from dataclasses import asdict
from pathlib import Path
from typing import Any
import numpy as np
import torch
import yaml
from sklearn.metrics import roc_auc_score
from torch.utils.data import DataLoader, TensorDataset
from data import UnifiedData, load_unified_data, subsample_train
from model import UnifiedCFMConfig, UnifiedTokenCFM
def _device(dev_arg: str) -> torch.device:
if dev_arg == 'auto':
return torch.device('cuda' if torch.cuda.is_available() else 'cpu')
return torch.device(dev_arg)
def _batch_score(model: UnifiedTokenCFM, flow_np: np.ndarray, packet_np: np.ndarray, len_np: np.ndarray, device: torch.device, *, batch_size: int, n_steps: int) -> dict[str, np.ndarray]:
out: dict[str, list[np.ndarray]] = {}
model.eval()
for start in range(0, len(flow_np), batch_size):
sl = slice(start, start + batch_size)
flow = torch.from_numpy(flow_np[sl]).float().to(device)
packets = torch.from_numpy(packet_np[sl]).float().to(device)
lens = torch.from_numpy(len_np[sl]).long().to(device)
metrics = model.trajectory_metrics(flow, packets, lens, n_steps=n_steps)
vel = model.velocity_score(flow, packets, lens)
metrics.update(vel)
for (k, v) in metrics.items():
out.setdefault(k, []).append(v.detach().cpu().numpy())
return {k: np.concatenate(v, axis=0) for (k, v) in out.items()}
def _quick_eval(model: UnifiedTokenCFM, data: UnifiedData, device: torch.device, cfg: dict[str, Any]) -> dict[str, float]:
n_eval = int(cfg.get('eval_n', 2000))
rng = np.random.default_rng(0)
def pick(n: int) -> np.ndarray:
m = min(n_eval, n)
return rng.choice(n, m, replace=False)
vi = pick(len(data.val_flow))
ai = pick(len(data.attack_flow))
v = _batch_score(model, data.val_flow[vi], data.val_packets[vi], data.val_len[vi], device, batch_size=int(cfg.get('eval_batch_size', 512)), n_steps=int(cfg.get('eval_n_steps', 8)))
a = _batch_score(model, data.attack_flow[ai], data.attack_packets[ai], data.attack_len[ai], device, batch_size=int(cfg.get('eval_batch_size', 512)), n_steps=int(cfg.get('eval_n_steps', 8)))
y = np.concatenate([np.zeros(len(vi)), np.ones(len(ai))])
result: dict[str, float] = {}
for key in sorted(v.keys()):
s = np.concatenate([v[key], a[key]])
s = np.nan_to_num(s, nan=0.0, posinf=1000000000000.0, neginf=-1000000000000.0)
result[f'auroc_{key}'] = float(roc_auc_score(y, s))
return result
def train(cfg: dict[str, Any]) -> Path:
device = _device(str(cfg.get('device', 'auto')))
save_dir = Path(cfg['save_dir'])
save_dir.mkdir(parents=True, exist_ok=True)
with open(save_dir / 'config.yaml', 'w') as f:
yaml.safe_dump(cfg, f)
seed = int(cfg.get('seed', 42))
data_seed = int(cfg.get('data_seed', seed))
torch.manual_seed(seed)
np.random.seed(seed)
print(f'Device: {device}')
print(f'[seed] model={seed} data={data_seed}')
feature_columns = cfg.get('flow_feature_columns')
data = load_unified_data(packets_npz=Path(cfg['packets_npz']) if cfg.get('packets_npz') else None, source_store=Path(cfg['source_store']) if cfg.get('source_store') else None, flows_parquet=Path(cfg['flows_parquet']), flow_features_path=Path(cfg['flow_features_path']) if cfg.get('flow_features_path') else None, flow_feature_columns=feature_columns, flow_features_align=str(cfg.get('flow_features_align', 'auto')), T=int(cfg['T']), split_seed=data_seed, train_ratio=float(cfg.get('train_ratio', 0.8)), benign_label=str(cfg.get('benign_label', 'normal')), min_len=int(cfg.get('min_len', 2)), packet_preprocess=str(cfg.get('packet_preprocess', 'mixed_dequant')), attack_cap=int(cfg['attack_cap']) if cfg.get('attack_cap') else None, val_cap=int(cfg['val_cap']) if cfg.get('val_cap') else None)
print(f'[data] T={data.T} packet_D={data.packet_dim} flow_D={data.flow_dim} train={len(data.train_flow):,} val={len(data.val_flow):,} attack={len(data.attack_flow):,}')
(tr_f, tr_p, tr_l) = subsample_train(data, int(cfg.get('n_train', 0)), data_seed)
ds = TensorDataset(torch.from_numpy(tr_f).float(), torch.from_numpy(tr_p).float(), torch.from_numpy(tr_l).long())
loader = DataLoader(ds, batch_size=int(cfg['batch_size']), shuffle=True, drop_last=True, num_workers=int(cfg.get('num_workers', 0)), pin_memory=device.type == 'cuda')
print(f'[data] using {len(ds):,} benign training flows')
model_cfg = UnifiedCFMConfig(T=data.T, packet_dim=data.packet_dim, flow_dim=data.flow_dim, token_dim=cfg.get('token_dim'), d_model=int(cfg['d_model']), n_layers=int(cfg['n_layers']), n_heads=int(cfg['n_heads']), mlp_ratio=float(cfg.get('mlp_ratio', 4.0)), time_dim=int(cfg.get('time_dim', 64)), sigma=float(cfg.get('sigma', 0.1)), use_ot=bool(cfg.get('use_ot', False)), reference_mode=cfg.get('reference_mode'))
model = UnifiedTokenCFM(model_cfg).to(device)
print(f'[model] params={model.param_count():,} token_dim={model.token_dim} seq_len={model.seq_len} sigma={model_cfg.sigma} use_ot={model_cfg.use_ot} reference_mode={model_cfg.reference_mode}')
opt = torch.optim.AdamW(model.parameters(), lr=float(cfg['lr']), weight_decay=float(cfg.get('weight_decay', 0.01)))
total_steps = max(1, int(cfg['epochs']) * len(loader))
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=total_steps)
history: dict[str, list[Any]] = {'epoch': [], 'loss': [], 'eval': []}
lambda_flow = float(cfg.get('lambda_flow', 0.0))
lambda_packet = float(cfg.get('lambda_packet', 0.0))
packet_mask_ratio = float(cfg.get('packet_mask_ratio', 0.5))
aux_enabled = lambda_flow > 0.0 or lambda_packet > 0.0
if aux_enabled:
print(f'[loss] λ_flow={lambda_flow} λ_packet={lambda_packet} packet_mask_ratio={packet_mask_ratio}')
for epoch in range(1, int(cfg['epochs']) + 1):
model.train()
losses: list[float] = []
aux_flow_sum = 0.0
aux_packet_sum = 0.0
n_steps_this_epoch = 0
t0 = time.time()
for (flow, packets, lens) in loader:
flow = flow.to(device, non_blocking=True)
packets = packets.to(device, non_blocking=True)
lens = lens.to(device, non_blocking=True)
if aux_enabled:
comp = model.compute_loss(flow, packets, lens, lambda_flow=lambda_flow, lambda_packet=lambda_packet, packet_mask_ratio=packet_mask_ratio, return_components=True)
loss = comp['total']
aux_flow_sum += float(comp['aux_flow'].item())
aux_packet_sum += float(comp['aux_packet'].item())
else:
loss = model.compute_loss(flow, packets, lens)
opt.zero_grad(set_to_none=True)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), float(cfg.get('grad_clip', 1.0)))
opt.step()
sched.step()
losses.append(float(loss.item()))
n_steps_this_epoch += 1
mean_loss = float(np.mean(losses)) if losses else float('nan')
eval_metrics: dict[str, float] | None = None
if epoch % int(cfg.get('eval_every', 5)) == 0 or epoch == int(cfg['epochs']):
eval_metrics = _quick_eval(model, data, device, cfg)
history['epoch'].append(epoch)
history['loss'].append(mean_loss)
history['eval'].append(eval_metrics)
elapsed = time.time() - t0
terminal = ''
if eval_metrics:
terminal = f" auroc_terminal={eval_metrics['auroc_terminal_norm']:.3f}"
if aux_enabled and n_steps_this_epoch:
terminal += f' aux_flow={aux_flow_sum / n_steps_this_epoch:.4f} aux_pkt={aux_packet_sum / n_steps_this_epoch:.4f}'
print(f"[epoch {epoch:>3d}/{cfg['epochs']:<3d}] ({elapsed:.1f}s) loss={mean_loss:.4f}{terminal}")
if not np.isfinite(mean_loss):
raise RuntimeError(f'non-finite loss at epoch {epoch}')
payload = {'model_state_dict': model.state_dict(), 'model_cfg': asdict(model_cfg), 'packet_mean': data.packet_mean, 'packet_std': data.packet_std, 'flow_mean': data.flow_mean, 'flow_std': data.flow_std, 'packet_preprocess': data.packet_preprocess, 'flow_feature_names': np.asarray(data.flow_feature_names), 'packet_feature_names': np.asarray(data.packet_feature_names)}
torch.save(payload, save_dir / 'model.pt')
with open(save_dir / 'history.json', 'w') as f:
json.dump(history, f, indent=2, default=str)
print(f"[saved] {save_dir / 'model.pt'}")
return save_dir
def main() -> None:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('--config', type=Path, required=True)
p.add_argument('--override', type=str, nargs='*', default=[])
args = p.parse_args()
with open(args.config) as f:
cfg = yaml.safe_load(f)
for override in args.override:
(key, value) = override.split('=', 1)
cfg[key] = yaml.safe_load(value)
train(cfg)
if __name__ == '__main__':
main()

103
artifacts/baselines/COMPARISON_TABLE.md Normal file
View File

@@ -0,0 +1,103 @@
# Unified_CFM vs Baselines — Performance Comparison Table
Live tracking. Last updated: 2026-04-30.
Two reproduction modes are tracked separately:
- **Method reproduction** (≡ "ours-method-only"): we use the baseline's
model/training algorithm but feed it our 20-d packet-derived canonical flow
features (or 9-d packet sequences for AT/Kitsune). Tests whether the
baseline's architecture beats Unified_CFM on the **same data substrate**.
- **True reproduction** (≡ "ours-true-repro"): we use the baseline's full
pipeline — including their feature engineering (CICFlowMeter CSV /
AfterImage / image encoding). Tests whether their published numbers
reproduce on our datasets.
"paper" = the number quoted in the source paper; not run by us.
## Headline AUROC table
Two columns per baseline:
- "ours-method-only" = baseline model on our 20-d / 9-d features, 3-seed mean±std.
- "ours-true-repro" = baseline's full pipeline (CICFlowMeter CSV + their feature subsets), single seed for now.
Raw AUROC reported. abs() = sign-agnostic max(AUROC, 1−AUROC) for inverted-signal baselines.
| Protocol | **Unified_CFM** | Shafir NF paper | Shafir NF method-only (ours) | **Shafir NF true-repro (ours)** | Kitsune paper | Kitsune Path B method-only (ours) | AT method-only (ours) |
|---|---:|---:|---:|---:|---:|---:|---:|
| ISCXTor2016 within | **0.9945 ± 0.0011** | 0.8731 | 0.9422 ± 0.0075 | **0.7562** [^4] | 0.7800 | 0.5653 ± 0.0226 | 0.4122 ± 0.0503 (abs 0.59) |
| CICIDS2017 within (σ=0.6) | **0.9858 ± 0.0021** | 0.9303 | 0.9256 ± 0.0188 | **0.8678** [^4] | 0.8500 | 0.7023 ± 0.0310 | 0.5009 ± 0.2107 (abs 0.66) |
| CICDDoS2019 within | **0.9960 ± 0.0010** | 0.9300 | 0.8903 ± 0.0386 | 0.5926 [^5] | — | 0.4710 ± 0.0039 | 0.4777 ± 0.3325 (abs 0.75) |
| IDS2017→DDoS2019 forward | 0.9109 ± 0.0032 | 0.8900 | **0.9210 ± 0.0111** | 0.7831 | — | 0.4905 ± 0.0751 | 0.5404 ± 0.1495 (abs 0.63) |
| DDoS2019→IDS2017 reverse | 0.5999 (single) [^1] | **0.9300** | 0.7247 ± 0.0035 | 0.7473 | — | 0.7483 ± 0.0137 | 0.4767 ± 0.2597 (abs 0.70) |
| CICIoT2023 within (single seed) | **0.9618** [^2] | F1=0.9951 [^3] | 0.8996 [^2] | 0.7398 [^4] | — | — | — |
**Bold** = best per row.
[^1]: Reverse `terminal_norm` 0.5999 single-seed. Our **PNA** score on this
protocol is 0.9089 (3-seed mean) — beats both Shafir NF reproduced (0.7247)
and Shafir paper (0.93).
[^2]: CICIoT2023 single seed. Our 20-d canonical features. Shafir NF (ours)
used 5-d Shafir-selected features (HTTPS, Protocol_Type, Magnitude,
Variance, fin_count) computed from our packets.
[^3]: Shafir paper Table VIII reports F1=0.9951 (not AUROC) on CICIoT2023.
They used the official CICFlowMeter pipeline on full pcap; not directly
AUROC-comparable. Our F1@P95 on this protocol = **0.9463** (with our 20-d
canonical features).
[^4]: Shafir NF true-reproduction uses their **paper-specified SHAP-selected
feature subsets**: ISCXTor 4 features (Flow IAT Std, Flow Bytes/s,
Flow Packets/s, Bwd IAT Max — paper §V-A), CICIDS2017 5 features (Bwd
Packet Length Mean, Fwd Packets/s, ACK Flag Count, Total Length of Bwd
Packets, Flow Duration — paper §V-B), CICIoT2023 5 features (HTTPS,
Protocol Type, Magnitude, Variance, fin_count — paper §V-D). Driver
uses **single NF**; paper headline numbers (0.8731 / 0.9303 / 0.93) come
from a **2-NF ensemble** which we don't reproduce. Expected single-NF
underperformance vs ensemble: ~0.05-0.15 AUROC.
[^5]: CICDDoS2019 true-repro uses CICIDS2017 best-5 feature subset (paper
§V-C: "comparable feature subset shared with CICIDS2017"). Result is
weak (0.59) — those 5 features are tuned for CICIDS attacks, not DrDoS.
Paper's 0.93 number is presumably with CICDDoS-specific feature
selection (not specified in paper).
## CICIoT2023 thresholded F1 table
| Method | flow features | F1@P95 | F1@P99 | TPR@1%FPR |
|---|---|---:|---:|---:|
| **Unified_CFM 20-d canonical** | 20-d packet-derived | **0.9463** | **0.9004** | **0.8208** |
| Unified_CFM Shafir-5 | 5-d Shafir SHAP-selected | 0.9266 | 0.8704 | 0.7726 |
| Shafir NF (ours), Shafir-5 | 5-d Shafir SHAP-selected | 0.9053 | 0.0652 | — |
| Shafir NF (paper) | 5-d via official CICFlowMeter | **0.9951** | — | — |
## Summary by within / cross direction
**Within-dataset (4 protocols)**: Unified_CFM beats Shafir NF reproduction on **4/4** (ISCXTor +0.052, CICIDS +0.060, CICDDoS +0.106, CICIoT2023 +0.062).
**Forward cross (1 protocol)**: Shafir NF (ours) 0.921 narrowly edges Unified_CFM 0.911 (+0.010 — within noise).
**Reverse cross (1 protocol)**: Shafir NF (ours) 0.725 beats Unified_CFM `terminal_norm` 0.600, but Unified_CFM `PNA` 0.909 beats both. Reverse score-of-record for Unified_CFM is PNA, not terminal_norm.
## Sources
- Unified_CFM main: `RESULTS.md`, `RESULTS_THRESHOLDED.md`
- Unified_CFM CICIoT2023: `artifacts/runs/unified_cfm_ciciot2023_2026_04_29/{phase1/phase1_summary.json, phase1/thresholded.json}`
- Unified_CFM PNA on reverse: `artifacts/phase_new_scores_2026_04_29/pna/reverse_cross_seed{42,43,44}.json`
- Shafir NF (ours): `artifacts/baselines/shafir_nf_2026_04_29/summary.md`
- Kitsune (ours): `artifacts/baselines/kitsune_2026_04_29/summary.md`
- Shafir paper baselines: `artifacts/locked_baselines.md`
## Reproduction-mode coverage table
For each baseline, what's the achievable reproduction mode and current status:
| Baseline | Method-only (ours-features) | True-repro (their-features) | Comment |
|---|---|---|---|
| **Shafir NF** | ✅ 15 cells | ✅ 6 cells (single seed, CSV+SHAP-5) | True-repro single-NF; paper headline uses 2-NF ensemble |
| **Kitsune** | ✅ 15 cells (Path B, KitNET on 9-d) | ❌ blocked (Path A) | Path A pcap streaming starts at 1471 pkt/s, slows to 581 pkt/s as AfterImage state grows. CICIDS2017 alone needs 7-9 hours; CICDDoS2019 / CICIoT2023 prohibitive. Single-pcap test on ISCXTor showed only 22 unique 5-tuples per pcap → flow coverage gaps. Documented as infeasible at our data scale. |
| **Anomaly-Transformer** | ✅ 15 cells | n/a | AT has no domain-specific feature pipeline (raw time-series only); method-only IS the faithful reproduction here |
| **ConMD** | not started | not started | Repo is just a 878-line model class — would require writing entire distillation training loop + image-like preprocessing from scratch. ~2 days work, no guaranteed paper-number reproduction. |
| **9 image-AD methods** (FastFlow, PatchCore, PADIM, STFPM, DFKDE, DFM, DRAEM, CFlow, RD4AD via `anomalib`) | not started | not started | Requires flow→image encoder which Shafir didn't open-source. Any reproduction would be "our flow→image + their image-AD model" → method-only at best. ~2 days work. |
| **TSLANet** | n/a | n/a | Vendored repo at the noted commit has no anomaly detection module. Skipped. |
| **ganomaly / RD4AD / STFPM standalone** | not started | not started | Would be subsumed by anomalib path; standalone repos have no native traffic-data support. |

320
artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed42.json Normal file
View File

@@ -0,0 +1,320 @@
{
"method": "anomaly_transformer",
"protocol": "cicddos_within",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed42",
"n_train": 10000,
"n_val": 10000,
"n_atk": 20000,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 23.69,
"loss_first_last": [
0.14222308861303934,
0.005586131493549181
],
"overall_by_agg": {
"mean": {
"auroc": 0.36776777750000006,
"auprc": 0.7022768182855437
},
"max": {
"auroc": 0.34954356750000004,
"auprc": 0.6604947928545277
},
"median": {
"auroc": 0.4061339125,
"auprc": 0.7025293226875595
},
"p90": {
"auroc": 0.36881044999999996,
"auprc": 0.6932174063535016
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.19116588908450705
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.18985000000000002
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.18985000000000002
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.505333219470538
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.19054429674099488
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.18985000000000002
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.2037271978021978
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.20567407574391344
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.18985000000000002
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.18985000000000002
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.18985000000000002
},
"Portmap": {
"_n": 417.0,
"auroc": 0.19371738609112713
},
"Syn": {
"_n": 3361.0,
"auroc": 0.9442166468313002
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.23980361663652802
},
"UDP": {
"_n": 1383.0,
"auroc": 0.20412762111352134
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.8218587514585765
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.44220000000000004
}
},
"max": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.19179731514084505
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.18974999999999997
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.18974999999999997
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.6647462852263023
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.19044361063464837
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.18974999999999997
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.2068688644688644
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.2098838142470694
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.18974999999999997
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.18974999999999997
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.18974999999999997
},
"Portmap": {
"_n": 417.0,
"auroc": 0.19352434052757791
},
"Syn": {
"_n": 3361.0,
"auroc": 0.8067483189526927
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.23951234177215186
},
"UDP": {
"_n": 1383.0,
"auroc": 0.20833101952277655
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.7022372812135356
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.44220000000000004
}
},
"median": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.28154999999999997
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.28154999999999997
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.28154999999999997
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.28356947053800163
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.28154999999999997
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.28154999999999997
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.2824230769230769
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.2818045536519387
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.28154999999999997
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.28154999999999997
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.28154999999999997
},
"Portmap": {
"_n": 417.0,
"auroc": 0.2837037170263788
},
"Syn": {
"_n": 3361.0,
"auroc": 0.8967866557572151
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.288824773960217
},
"UDP": {
"_n": 1383.0,
"auroc": 0.28154999999999997
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.7609575845974329
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.7525999999999999
}
},
"p90": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.2140569982394366
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.21365
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.21365
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.30573680614859094
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.21432414236706693
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.21365
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.22095155677655676
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.22327037871956718
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.21365
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.21365
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.21365
},
"Portmap": {
"_n": 417.0,
"auroc": 0.21740203836930455
},
"Syn": {
"_n": 3361.0,
"auroc": 0.9253638351681047
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.2627084086799277
},
"UDP": {
"_n": 1383.0,
"auroc": 0.22233470715835144
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.8148401400233373
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.5740000000000001
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed42.npz Normal file
View File

Binary file not shown.

320
artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed43.json Normal file
View File

@@ -0,0 +1,320 @@
{
"method": "anomaly_transformer",
"protocol": "cicddos_within",
"seed": 43,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed43",
"n_train": 10000,
"n_val": 10000,
"n_atk": 20000,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 24.46,
"loss_first_last": [
0.1440289024310776,
0.004007022972277637
],
"overall_by_agg": {
"mean": {
"auroc": 0.16187093,
"auprc": 0.6077831395684652
},
"max": {
"auroc": 0.16731178000000002,
"auprc": 0.6028034515372264
},
"median": {
"auroc": 0.18686188499999998,
"auprc": 0.556872781554051
},
"p90": {
"auroc": 0.1994999625,
"auprc": 0.616710814385669
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.017933482542524647
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.005402806563039751
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.007452684859154955
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.27283804855275445
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.07414517657192077
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.009164126712328793
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.3974829535095715
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.42633196573489635
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.005454378648874089
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.007531764705882378
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.06480254614894973
},
"Portmap": {
"_n": 407.0,
"auroc": 0.06535786240786241
},
"Syn": {
"_n": 3303.0,
"auroc": 0.12263094156827128
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.5118133650519031
},
"UDP": {
"_n": 1334.0,
"auroc": 0.43651510494752627
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.2210940389294404
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.16269999999999996
}
},
"max": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.017658281110116386
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.005101899827288433
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.007152772887323949
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.5034922035480859
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.06803212747631353
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.008623758561643841
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.3702902461257976
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.3996642470694319
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.005153628023352798
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.007226386554621853
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.060337873965626995
},
"Portmap": {
"_n": 407.0,
"auroc": 0.0621894348894349
},
"Syn": {
"_n": 3303.0,
"auroc": 0.12265498032092038
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.4962176903114187
},
"UDP": {
"_n": 1334.0,
"auroc": 0.4099449025487256
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.21177378345498785
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.16159999999999997
}
},
"median": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.12521938227394808
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.11268424006908465
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.12095347711267607
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.12790480859010273
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.23087037037037036
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.11873750000000002
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.24098144940747496
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.24872605951307486
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.11225166805671392
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.12170260504201681
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.2215689688096754
},
"Portmap": {
"_n": 407.0,
"auroc": 0.22320724815724818
},
"Syn": {
"_n": 3303.0,
"auroc": 0.22364801695428396
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.2383819204152249
},
"UDP": {
"_n": 1334.0,
"auroc": 0.24979988755622184
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.21808619221411196
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.4455
}
},
"p90": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.05750152193375112
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.04537931778929188
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.05087596830985915
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.114456162464986
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.14196339362618432
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.05058343321917808
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.4329668641750228
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.46167132551848505
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.04534462051709758
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.051506092436974786
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.13326954169318905
},
"Portmap": {
"_n": 407.0,
"auroc": 0.13529152334152333
},
"Syn": {
"_n": 3303.0,
"auroc": 0.1905550408719346
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.5306939013840829
},
"UDP": {
"_n": 1334.0,
"auroc": 0.4702869190404797
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.28141763990267643
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.31999999999999995
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed43.npz Normal file
View File

Binary file not shown.

320
artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed44.json Normal file
View File

@@ -0,0 +1,320 @@
{
"method": "anomaly_transformer",
"protocol": "cicddos_within",
"seed": 44,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed44",
"n_train": 10000,
"n_val": 10000,
"n_atk": 20000,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 23.12,
"loss_first_last": [
0.13797885122932965,
0.004073698047100555
],
"overall_by_agg": {
"mean": {
"auroc": 0.6323407125,
"auprc": 0.8247269083802768
},
"max": {
"auroc": 0.5918001275,
"auprc": 0.7644950552709932
},
"median": {
"auroc": 0.8401825849999998,
"auprc": 0.9044669345849929
},
"p90": {
"auroc": 0.6628086200000001,
"auprc": 0.8249129023839272
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.38358046387154326
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.3384721048182587
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.8784359464627152
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.42309863195057373
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.9832278733031674
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.37689410714285715
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.9443937277580071
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.9635801412180053
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.34371287813310286
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.8914378446115288
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.9836372795969773
},
"Portmap": {
"_n": 417.0,
"auroc": 0.9855167865707434
},
"Syn": {
"_n": 3418.0,
"auroc": 0.20813423054417787
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.9565173021925643
},
"UDP": {
"_n": 1353.0,
"auroc": 0.9599579822616406
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.35599000000000003
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.6828000000000001
}
},
"max": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.3705818911685994
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.3302398562975486
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.7660391013384321
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.616021359223301
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.8523205429864253
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.360296875
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.8770422153024912
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.899051279788173
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.3357694468452896
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.7734077694235588
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.8464110831234257
},
"Portmap": {
"_n": 417.0,
"auroc": 0.8579430455635492
},
"Syn": {
"_n": 3418.0,
"auroc": 0.21209106202457578
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.8926503336510964
},
"UDP": {
"_n": 1353.0,
"auroc": 0.8919618994826312
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.34583795321637434
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.9594
}
},
"median": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.951335816235504
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.9408696956889264
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.9935487571701721
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.49186902030008817
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.9981957466063348
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.9505358928571428
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.8902613879003559
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.8952235657546336
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.9460346585998273
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.9933953216374268
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.9987342569269522
},
"Portmap": {
"_n": 417.0,
"auroc": 0.9972292565947243
},
"Syn": {
"_n": 3418.0,
"auroc": 0.5606071679344644
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.8713183031458532
},
"UDP": {
"_n": 1353.0,
"auroc": 0.8944427937915743
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.6068018713450292
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.4114
}
},
"p90": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.5112844335414808
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.47495126796280646
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.8811490439770555
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.3321917034421889
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.9584939366515838
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.5054002232142857
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.9330716192170819
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.948352824360106
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.48074563526361275
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.8884563909774438
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.9560999370277078
},
"Portmap": {
"_n": 417.0,
"auroc": 0.9615719424460432
},
"Syn": {
"_n": 3418.0,
"auroc": 0.26969002340550025
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.9451080076263109
},
"UDP": {
"_n": 1353.0,
"auroc": 0.9463139320029565
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.3996357309941521
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.4225
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed44.npz Normal file
View File

Binary file not shown.

256
artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed42.json Normal file
View File

@@ -0,0 +1,256 @@
{
"method": "anomaly_transformer",
"protocol": "cicids_within",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed42",
"n_train": 10000,
"n_val": 10000,
"n_atk": 30000,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 22.45,
"loss_first_last": [
0.15159071482057812,
0.004075585566814753
],
"overall_by_agg": {
"mean": {
"auroc": 0.6001296883333334,
"auprc": 0.8383184815597481
},
"max": {
"auroc": 0.547795575,
"auprc": 0.7839841390353699
},
"median": {
"auroc": 0.36235965666666664,
"auprc": 0.7142202977874932
},
"p90": {
"auroc": 0.47742446,
"auprc": 0.7638875860346089
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 46.0,
"auroc": 0.8872391304347825
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.8693338404033378
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.8588349137931033
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.8672453141696944
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.8991410256410255
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.762110810810811
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.7410864406779661
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.46535
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.603313003492433
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.14459035543766577
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.6733217105263157
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.72576
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.7853
}
},
"max": {
"Botnet": {
"_n": 46.0,
"auroc": 0.7783739130434782
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.7791971662030597
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.798263146551724
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.7894506197905534
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.8176192307692307
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.7589518918918919
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.754200847457627
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.56245
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.5414751222351571
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.14076252519893898
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.7641335526315789
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.87258
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.9582999999999999
}
},
"median": {
"Botnet": {
"_n": 46.0,
"auroc": 0.20422608695652175
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.19559450625869262
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.17157586206896552
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.18297270784355632
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.190975
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.21840324324324326
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.15560000000000002
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.24850000000000003
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.7628175669383004
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.4828546206896552
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.15560000000000002
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.15560000000000002
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.15560000000000002
}
},
"p90": {
"Botnet": {
"_n": 46.0,
"auroc": 0.8840760869565217
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.5441651773296244
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.3994240301724138
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.4272720987390468
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.6002397435897436
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.38377540540540545
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.21240254237288134
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.33949999999999997
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.7373752735739231
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.3844263448275862
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.05923914473684209
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.05864999999999998
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.05864999999999998
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed42.npz Normal file
View File

Binary file not shown.

272
artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed43.json Normal file
View File

@@ -0,0 +1,272 @@
{
"method": "anomaly_transformer",
"protocol": "cicids_within",
"seed": 43,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed43",
"n_train": 10000,
"n_val": 10000,
"n_atk": 30000,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 23.73,
"loss_first_last": [
0.1467826651244224,
0.0038189603681852923
],
"overall_by_agg": {
"mean": {
"auroc": 0.25881962166666667,
"auprc": 0.6678241607690786
},
"max": {
"auroc": 0.25728142333333337,
"auprc": 0.6590998108230994
},
"median": {
"auroc": 0.29273710333333336,
"auprc": 0.6397626176311149
},
"p90": {
"auroc": 0.2794696483333333,
"auprc": 0.6818023724164499
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 39.0,
"auroc": 0.3123346153846153
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.6342495588494794
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.7240725672877847
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.12875773550916603
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.31539722222222216
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.15719341317365268
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.03223434579439251
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.9448
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.4443001539554714
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.05020893327711605
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.9247592896174862
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.9488333333333333
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.8862
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.9608599999999999
}
},
"max": {
"Botnet": {
"_n": 39.0,
"auroc": 0.3178192307692307
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.6361651402858655
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.7227036231884058
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.1345415810109145
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.3220927777777778
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.15693502994011976
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.03367242990654206
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.9434
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.4329837280909521
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.04342239907241488
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.9255346994535518
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.9508
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.8877499999999999
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.9626
}
},
"median": {
"Botnet": {
"_n": 39.0,
"auroc": 0.30856794871794874
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.2570666401976355
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.3667913043478261
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.16186963017908235
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.22024166666666672
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.21525359281437126
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.14395000000000002
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.14395000000000002
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.5923504263382284
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.31523249710129647
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.1541614754098361
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.14395000000000002
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.48135
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.14395000000000002
}
},
"p90": {
"Botnet": {
"_n": 39.0,
"auroc": 0.34030512820512826
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.6269999823539791
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.7092333333333333
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.13366804598919146
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.2878455555555556
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.13084820359281435
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.033217757009345775
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.43779999999999997
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.4858389152060635
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.10328855802677346
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.6608251366120219
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.5921666666666667
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.93145
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.52772
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed43.npz Normal file
View File

Binary file not shown.

240
artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed44.json Normal file
View File

@@ -0,0 +1,240 @@
{
"method": "anomaly_transformer",
"protocol": "cicids_within",
"seed": 44,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed44",
"n_train": 10000,
"n_val": 10000,
"n_atk": 30000,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 23.79,
"loss_first_last": [
0.14415377727414988,
0.0037091414420570754
],
"overall_by_agg": {
"mean": {
"auroc": 0.6436450233333333,
"auprc": 0.8672315471451842
},
"max": {
"auroc": 0.6878354799999999,
"auprc": 0.8657705943772598
},
"median": {
"auroc": 0.4987740683333334,
"auprc": 0.8127357475982513
},
"p90": {
"auroc": 0.5558856366666667,
"auprc": 0.8318653186696193
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 38.0,
"auroc": 0.46004210526315786
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.5145650968544518
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.7046408296943232
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.4318485673352435
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.6630583333333333
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.5808367088607596
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.3321348214285714
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.5555688679245283
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.9853483215454448
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.19467950310559007
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.3236285714285714
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.19369999999999998
}
},
"max": {
"Botnet": {
"_n": 38.0,
"auroc": 0.4877605263157895
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.63700774835614
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.8225126637554585
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.582189382362305
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.793025
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.7865386075949368
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.7114633928571429
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.5279190635066727
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.8887979731869522
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.6114167701863353
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.9434428571428571
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.18889999999999996
}
},
"median": {
"Botnet": {
"_n": 38.0,
"auroc": 0.31109868421052633
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.23562989159409986
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.21711735807860263
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.22242115568290352
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.2532523809523809
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.31956424050632914
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.14125
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.44260834100322133
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.990073931172807
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.14250807453416148
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.14125
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.3135
}
},
"p90": {
"Botnet": {
"_n": 38.0,
"auroc": 0.5111026315789474
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.35984901368402344
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.342010807860262
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.27933460150695105
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.4002023809523809
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.39653987341772157
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.1268642857142857
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.5756708582604694
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.9719925208487281
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.054568944099378874
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.02589999999999998
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.26039999999999996
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed44.npz Normal file
View File

Binary file not shown.

320
artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed42.json Normal file
View File

@@ -0,0 +1,320 @@
{
"method": "anomaly_transformer",
"protocol": "forward_cross",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed42",
"n_train": 10000,
"n_val": 10000,
"n_atk": 9846,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 22.48,
"loss_first_last": [
0.1520787082329581,
0.003957434124136462
],
"overall_by_agg": {
"mean": {
"auroc": 0.4892238015437741,
"auprc": 0.5018063179317858
},
"max": {
"auroc": 0.4701616494007718,
"auprc": 0.4553141881058038
},
"median": {
"auroc": 0.6214469022953484,
"auprc": 0.6244131030607472
},
"p90": {
"auroc": 0.5738190788137315,
"auprc": 0.5600867877876071
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.2418359693877551
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.2234609693877551
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.37049260204081635
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.6877126700680272
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.48235034013605443
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.22540357142857143
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.5496690476190476
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.5629209183673469
},
"LDAP": {
"_n": 588.0,
"auroc": 0.22110892857142855
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.38065323129251705
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.4970219387755102
},
"Portmap": {
"_n": 588.0,
"auroc": 0.48847789115646256
},
"Syn": {
"_n": 588.0,
"auroc": 0.9474401360544217
},
"TFTP": {
"_n": 588.0,
"auroc": 0.5612562925170068
},
"UDP": {
"_n": 588.0,
"auroc": 0.5692301870748299
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.9008552721088435
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.378726598173516
}
},
"max": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.23988681972789117
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.22176845238095239
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.36733971088435374
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.739343112244898
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.477965731292517
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.22383358843537415
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.5421444727891156
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.5541784013605442
},
"LDAP": {
"_n": 588.0,
"auroc": 0.219068962585034
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.3767042517006803
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.4928329081632653
},
"Portmap": {
"_n": 588.0,
"auroc": 0.4839550170068027
},
"Syn": {
"_n": 588.0,
"auroc": 0.785248044217687
},
"TFTP": {
"_n": 588.0,
"auroc": 0.5538645408163266
},
"UDP": {
"_n": 588.0,
"auroc": 0.5612477040816326
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.7573638605442178
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.3706054794520548
}
},
"median": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.4279896258503401
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.4071503401360545
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.5898164965986394
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.36098630952380956
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.7220323979591837
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.4041826530612244
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.7640134353741496
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.7767905612244899
},
"LDAP": {
"_n": 588.0,
"auroc": 0.40308545918367344
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.6017380952380953
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.737468962585034
},
"Portmap": {
"_n": 588.0,
"auroc": 0.727680612244898
},
"Syn": {
"_n": 588.0,
"auroc": 0.9082727040816327
},
"TFTP": {
"_n": 588.0,
"auroc": 0.7536788265306124
},
"UDP": {
"_n": 588.0,
"auroc": 0.781905612244898
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.8426772108843539
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.26392294520547943
}
},
"p90": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.3539172619047619
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.3341599489795919
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.4993686224489796
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.44331615646258504
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.6230568027210884
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.33368554421768715
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.6872853741496598
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.7011593537414965
},
"LDAP": {
"_n": 588.0,
"auroc": 0.33136301020408165
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.5099774659863945
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.6386501700680273
},
"Portmap": {
"_n": 588.0,
"auroc": 0.6296390306122449
},
"Syn": {
"_n": 588.0,
"auroc": 0.950218962585034
},
"TFTP": {
"_n": 588.0,
"auroc": 0.6960769557823131
},
"UDP": {
"_n": 588.0,
"auroc": 0.707658843537415
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.9208323979591837
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.3331678082191781
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed42.npz Normal file
View File

Binary file not shown.

320
artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed43.json Normal file
View File

@@ -0,0 +1,320 @@
{
"method": "anomaly_transformer",
"protocol": "forward_cross",
"seed": 43,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed43",
"n_train": 10000,
"n_val": 10000,
"n_atk": 9846,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 22.73,
"loss_first_last": [
0.1470018303658389,
0.0038443528826030185
],
"overall_by_agg": {
"mean": {
"auroc": 0.49693973694901483,
"auprc": 0.49237919507338757
},
"max": {
"auroc": 0.48891327950436725,
"auprc": 0.48587457299186
},
"median": {
"auroc": 0.6319110146252285,
"auprc": 0.6692675938817987
},
"p90": {
"auroc": 0.5400531383302865,
"auprc": 0.5541969096322785
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.3203674319727891
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.3218841836734694
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.5871074829931973
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.6272752551020409
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.564168537414966
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.33965544217687077
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.6149467687074831
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.6218015306122449
},
"LDAP": {
"_n": 588.0,
"auroc": 0.3331803571428571
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.5915131802721088
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.5796848639455783
},
"Portmap": {
"_n": 588.0,
"auroc": 0.5787874149659864
},
"Syn": {
"_n": 588.0,
"auroc": 0.34615850340136056
},
"TFTP": {
"_n": 588.0,
"auroc": 0.6246388605442177
},
"UDP": {
"_n": 588.0,
"auroc": 0.6211027210884353
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.30889481292517007
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.45648915525114153
}
},
"max": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.3108757653061225
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.31266122448979594
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.5745394557823129
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.6311168367346939
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.5498848639455782
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.32866666666666666
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.6105522959183673
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.6175331632653062
},
"LDAP": {
"_n": 588.0,
"auroc": 0.3237489795918368
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.5768287414965987
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.5674460884353741
},
"Portmap": {
"_n": 588.0,
"auroc": 0.5659906462585034
},
"Syn": {
"_n": 588.0,
"auroc": 0.3396127551020408
},
"TFTP": {
"_n": 588.0,
"auroc": 0.6206904761904761
},
"UDP": {
"_n": 588.0,
"auroc": 0.6166451530612246
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.30192755102040814
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.4538639269406393
}
},
"median": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.5342164115646258
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.5300934523809524
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.811029761904762
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.18806403061224486
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.8042083333333333
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.5495919217687075
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.7322251700680272
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.7518767006802721
},
"LDAP": {
"_n": 588.0,
"auroc": 0.5437292517006802
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.8123817176870749
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.8117076530612244
},
"Portmap": {
"_n": 588.0,
"auroc": 0.815749149659864
},
"Syn": {
"_n": 588.0,
"auroc": 0.48128273809523814
},
"TFTP": {
"_n": 588.0,
"auroc": 0.677136649659864
},
"UDP": {
"_n": 588.0,
"auroc": 0.7469124149659864
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.4573297619047618
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.448048401826484
}
},
"p90": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.3825764455782313
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.3856704931972789
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.6547069727891157
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.3307896258503401
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.6298654761904762
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.402787925170068
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.6847105442176871
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.6919932823129251
},
"LDAP": {
"_n": 588.0,
"auroc": 0.3971041666666666
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.6574113095238094
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.6470957482993198
},
"Portmap": {
"_n": 588.0,
"auroc": 0.645858843537415
},
"Syn": {
"_n": 588.0,
"auroc": 0.4046221088435374
},
"TFTP": {
"_n": 588.0,
"auroc": 0.6932554421768707
},
"UDP": {
"_n": 588.0,
"auroc": 0.6913091836734694
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.3709507653061225
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.49996963470319633
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed43.npz Normal file
View File

Binary file not shown.

320
artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed44.json Normal file
View File

@@ -0,0 +1,320 @@
{
"method": "anomaly_transformer",
"protocol": "forward_cross",
"seed": 44,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed44",
"n_train": 10000,
"n_val": 10000,
"n_atk": 9846,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 23.07,
"loss_first_last": [
0.1445360630750656,
0.0036873004643859556
],
"overall_by_agg": {
"mean": {
"auroc": 0.2586951960186878,
"auprc": 0.38725463725704035
},
"max": {
"auroc": 0.2539725878529352,
"auprc": 0.37013045453745935
},
"median": {
"auroc": 0.36782300426569164,
"auprc": 0.4000846941316927
},
"p90": {
"auroc": 0.3272018941702214,
"auprc": 0.409383950502048
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.12040178571428571
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.12222193877551019
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.1191062074829932
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.4970311224489796
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.10205935374149659
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.12043333333333331
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.4376086734693878
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.43346173469387755
},
"LDAP": {
"_n": 588.0,
"auroc": 0.12309634353741497
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.12141343537414964
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.10477653061224489
},
"Portmap": {
"_n": 588.0,
"auroc": 0.10042551020408164
},
"Syn": {
"_n": 588.0,
"auroc": 0.3379833333333333
},
"TFTP": {
"_n": 588.0,
"auroc": 0.5251631802721088
},
"UDP": {
"_n": 588.0,
"auroc": 0.4429054421768708
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.27171930272108846
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.47257134703196346
}
},
"max": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.12264251700680272
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.12220187074829933
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.11917806122448979
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.6788215986394558
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.10203945578231292
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.12052142857142857
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.37424438775510205
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.3764227891156462
},
"LDAP": {
"_n": 588.0,
"auroc": 0.12306598639455782
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.12105663265306123
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.10475527210884356
},
"Portmap": {
"_n": 588.0,
"auroc": 0.10019710884353741
},
"Syn": {
"_n": 588.0,
"auroc": 0.3209952380952381
},
"TFTP": {
"_n": 588.0,
"auroc": 0.4633068027210885
},
"UDP": {
"_n": 588.0,
"auroc": 0.3798562925170068
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.2553731292517007
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.49411666666666665
}
},
"median": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.3529809523809524
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.3595994897959184
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.3543705782312925
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.20132899659863943
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.32573061224489797
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.3585214285714286
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.5194945578231294
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.5092042517006803
},
"LDAP": {
"_n": 588.0,
"auroc": 0.36101071428571424
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.3558926870748299
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.3283911564625851
},
"Portmap": {
"_n": 588.0,
"auroc": 0.3172437074829932
},
"Syn": {
"_n": 588.0,
"auroc": 0.14559957482993197
},
"TFTP": {
"_n": 588.0,
"auroc": 0.5893117346938775
},
"UDP": {
"_n": 588.0,
"auroc": 0.5245889455782313
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.2092083333333333
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.46540730593607305
}
},
"p90": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.23454574829931973
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.23998571428571427
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.23522534013605445
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.33707559523809527
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.20624557823129253
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.2381486394557823
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.5053664965986395
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.5041442176870747
},
"LDAP": {
"_n": 588.0,
"auroc": 0.2412746598639456
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.23769591836734694
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.20951088435374152
},
"Portmap": {
"_n": 588.0,
"auroc": 0.2017103741496598
},
"Syn": {
"_n": 588.0,
"auroc": 0.34869957482993197
},
"TFTP": {
"_n": 588.0,
"auroc": 0.5825056122448979
},
"UDP": {
"_n": 588.0,
"auroc": 0.508903231292517
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.30893324829931973
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.4550844748858447
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed44.npz Normal file
View File

Binary file not shown.

64
artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed42.json Normal file
View File

@@ -0,0 +1,64 @@
{
"method": "anomaly_transformer",
"protocol": "iscxtor_within",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/iscxtor2016_lambda0p3_seed42",
"n_train": 10000,
"n_val": 10000,
"n_atk": 1312,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 21.2,
"loss_first_last": [
0.168125517100473,
0.005558558188591011
],
"overall_by_agg": {
"mean": {
"auroc": 0.49322682926829275,
"auprc": 0.17516424930113023
},
"max": {
"auroc": 0.5264396341463415,
"auprc": 0.22898976744241134
},
"median": {
"auroc": 0.47917290396341466,
"auprc": 0.18524469826822748
},
"p90": {
"auroc": 0.4372799923780488,
"auprc": 0.1486540527683511
}
},
"per_class_by_agg": {
"mean": {
"tor": {
"_n": 1312.0,
"auroc": 0.49322682926829275
}
},
"max": {
"tor": {
"_n": 1312.0,
"auroc": 0.5264396341463415
}
},
"median": {
"tor": {
"_n": 1312.0,
"auroc": 0.47917290396341466
}
},
"p90": {
"tor": {
"_n": 1312.0,
"auroc": 0.4372799923780488
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed42.npz Normal file
View File

Binary file not shown.

64
artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed43.json Normal file
View File

@@ -0,0 +1,64 @@
{
"method": "anomaly_transformer",
"protocol": "iscxtor_within",
"seed": 43,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/iscxtor2016_lambda0p3_seed43",
"n_train": 10000,
"n_val": 10000,
"n_atk": 1312,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 22.49,
"loss_first_last": [
0.1598462136108664,
0.003827647895469696
],
"overall_by_agg": {
"mean": {
"auroc": 0.41331875,
"auprc": 0.19291385713283932
},
"max": {
"auroc": 0.4134266768292683,
"auprc": 0.22173333659746425
},
"median": {
"auroc": 0.47364451219512194,
"auprc": 0.12669803046008787
},
"p90": {
"auroc": 0.35424939024390245,
"auprc": 0.11785068848066366
}
},
"per_class_by_agg": {
"mean": {
"tor": {
"_n": 1312.0,
"auroc": 0.41331875
}
},
"max": {
"tor": {
"_n": 1312.0,
"auroc": 0.4134266768292683
}
},
"median": {
"tor": {
"_n": 1312.0,
"auroc": 0.47364451219512194
}
},
"p90": {
"tor": {
"_n": 1312.0,
"auroc": 0.35424939024390245
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed43.npz Normal file
View File

Binary file not shown.

64
artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed44.json Normal file
View File

@@ -0,0 +1,64 @@
{
"method": "anomaly_transformer",
"protocol": "iscxtor_within",
"seed": 44,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/iscxtor2016_lambda0p3_seed44",
"n_train": 10000,
"n_val": 10000,
"n_atk": 1312,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 20.76,
"loss_first_last": [
0.15891090356096438,
0.006547442672750618
],
"overall_by_agg": {
"mean": {
"auroc": 0.48698502286585366,
"auprc": 0.1969998720953106
},
"max": {
"auroc": 0.5150377286585366,
"auprc": 0.22484916372486033
},
"median": {
"auroc": 0.5020458079268293,
"auprc": 0.1872525491455697
},
"p90": {
"auroc": 0.4449618902439024,
"auprc": 0.18297321346276874
}
},
"per_class_by_agg": {
"mean": {
"tor": {
"_n": 1312.0,
"auroc": 0.48698502286585366
}
},
"max": {
"tor": {
"_n": 1312.0,
"auroc": 0.5150377286585366
}
},
"median": {
"tor": {
"_n": 1312.0,
"auroc": 0.5020458079268293
}
},
"p90": {
"tor": {
"_n": 1312.0,
"auroc": 0.4449618902439024
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed44.npz Normal file
View File

Binary file not shown.

315
artifacts/baselines/anomaly_transformer_2026_04_29/master.log Normal file
View File

@@ -0,0 +1,315 @@
=== protocol=iscxtor_within seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=iscxtor_within seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/packets.npz
[data] using external flow features D=20
[data] rows total=103,079 keep len>=2: 66,189
[data] benign=64,877 attack=1,312 -> train=51,901 val=10,000
[data] train_flows=10,000 val=10,000 attack=1,312 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0274 (7.7s elapsed)
[epoch 10/15] rec_loss=0.0131 (14.3s elapsed)
[epoch 15/15] rec_loss=0.0056 (21.2s elapsed)
[train] 21.2s, final rec_loss=0.0056
[score] benign in 2.1s
[score] attack in 0.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed42.json
[best agg=max] AUROC=0.5264 AUPRC=0.2290
max AUROC=0.5264 AUPRC=0.2290
mean AUROC=0.4932 AUPRC=0.1752
median AUROC=0.4792 AUPRC=0.1852
p90 AUROC=0.4373 AUPRC=0.1487
[done] elapsed=33s artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed42.json
=== protocol=iscxtor_within seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=iscxtor_within seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/packets.npz
[data] using external flow features D=20
[data] rows total=103,079 keep len>=2: 66,189
[data] benign=64,877 attack=1,312 -> train=51,901 val=10,000
[data] train_flows=10,000 val=10,000 attack=1,312 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0250 (7.6s elapsed)
[epoch 10/15] rec_loss=0.0078 (14.9s elapsed)
[epoch 15/15] rec_loss=0.0038 (22.5s elapsed)
[train] 22.5s, final rec_loss=0.0038
[score] benign in 2.1s
[score] attack in 0.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed43.json
[best agg=median] AUROC=0.4736 AUPRC=0.1267
median AUROC=0.4736 AUPRC=0.1267
max AUROC=0.4134 AUPRC=0.2217
mean AUROC=0.4133 AUPRC=0.1929
p90 AUROC=0.3542 AUPRC=0.1179
[done] elapsed=34s artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed43.json
=== protocol=iscxtor_within seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=iscxtor_within seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/packets.npz
[data] using external flow features D=20
[data] rows total=103,079 keep len>=2: 66,189
[data] benign=64,877 attack=1,312 -> train=51,901 val=10,000
[data] train_flows=10,000 val=10,000 attack=1,312 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0230 (7.2s elapsed)
[epoch 10/15] rec_loss=0.0071 (14.0s elapsed)
[epoch 15/15] rec_loss=0.0065 (20.8s elapsed)
[train] 20.8s, final rec_loss=0.0065
[score] benign in 2.2s
[score] attack in 0.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed44.json
[best agg=max] AUROC=0.5150 AUPRC=0.2248
max AUROC=0.5150 AUPRC=0.2248
median AUROC=0.5020 AUPRC=0.1873
mean AUROC=0.4870 AUPRC=0.1970
p90 AUROC=0.4450 AUPRC=0.1830
[done] elapsed=33s artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed44.json
=== protocol=cicids_within seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicids_within seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=30,000 -> train=1,210,760 val=10,000
[data] train_flows=10,000 val=10,000 attack=30,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0256 (7.2s elapsed)
[epoch 10/15] rec_loss=0.0108 (14.6s elapsed)
[epoch 15/15] rec_loss=0.0041 (22.5s elapsed)
[train] 22.5s, final rec_loss=0.0041
[score] benign in 2.1s
[score] attack in 6.4s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed42.json
[best agg=mean] AUROC=0.6001 AUPRC=0.8383
mean AUROC=0.6001 AUPRC=0.8383
max AUROC=0.5478 AUPRC=0.7840
p90 AUROC=0.4774 AUPRC=0.7639
median AUROC=0.3624 AUPRC=0.7142
[done] elapsed=142s artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed42.json
=== protocol=cicids_within seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicids_within seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=30,000 -> train=1,210,760 val=10,000
[data] train_flows=10,000 val=10,000 attack=30,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0233 (8.2s elapsed)
[epoch 10/15] rec_loss=0.0081 (16.0s elapsed)
[epoch 15/15] rec_loss=0.0038 (23.7s elapsed)
[train] 23.7s, final rec_loss=0.0038
[score] benign in 2.1s
[score] attack in 6.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed43.json
[best agg=median] AUROC=0.2927 AUPRC=0.6398
median AUROC=0.2927 AUPRC=0.6398
p90 AUROC=0.2795 AUPRC=0.6818
mean AUROC=0.2588 AUPRC=0.6678
max AUROC=0.2573 AUPRC=0.6591
[done] elapsed=141s artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed43.json
=== protocol=cicids_within seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicids_within seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=30,000 -> train=1,210,760 val=10,000
[data] train_flows=10,000 val=10,000 attack=30,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0197 (8.3s elapsed)
[epoch 10/15] rec_loss=0.0097 (16.1s elapsed)
[epoch 15/15] rec_loss=0.0037 (23.8s elapsed)
[train] 23.8s, final rec_loss=0.0037
[score] benign in 2.2s
[score] attack in 6.4s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed44.json
[best agg=max] AUROC=0.6878 AUPRC=0.8658
max AUROC=0.6878 AUPRC=0.8658
mean AUROC=0.6436 AUPRC=0.8672
p90 AUROC=0.5559 AUPRC=0.8319
median AUROC=0.4988 AUPRC=0.8127
[done] elapsed=141s artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed44.json
=== protocol=cicddos_within seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicddos_within seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=20,000 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=20,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0306 (8.3s elapsed)
[epoch 10/15] rec_loss=0.0127 (15.9s elapsed)
[epoch 15/15] rec_loss=0.0056 (23.7s elapsed)
[train] 23.7s, final rec_loss=0.0056
[score] benign in 2.1s
[score] attack in 4.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed42.json
[best agg=median] AUROC=0.4061 AUPRC=0.7025
median AUROC=0.4061 AUPRC=0.7025
p90 AUROC=0.3688 AUPRC=0.6932
mean AUROC=0.3678 AUPRC=0.7023
max AUROC=0.3495 AUPRC=0.6605
[done] elapsed=45s artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed42.json
=== protocol=cicddos_within seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicddos_within seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=20,000 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=20,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0222 (8.2s elapsed)
[epoch 10/15] rec_loss=0.0079 (16.2s elapsed)
[epoch 15/15] rec_loss=0.0040 (24.5s elapsed)
[train] 24.5s, final rec_loss=0.0040
[score] benign in 2.0s
[score] attack in 4.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed43.json
[best agg=p90] AUROC=0.1995 AUPRC=0.6167
p90 AUROC=0.1995 AUPRC=0.6167
median AUROC=0.1869 AUPRC=0.5569
max AUROC=0.1673 AUPRC=0.6028
mean AUROC=0.1619 AUPRC=0.6078
[done] elapsed=46s artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed43.json
=== protocol=cicddos_within seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicddos_within seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=20,000 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=20,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0225 (8.0s elapsed)
[epoch 10/15] rec_loss=0.0078 (15.6s elapsed)
[epoch 15/15] rec_loss=0.0041 (23.1s elapsed)
[train] 23.1s, final rec_loss=0.0041
[score] benign in 2.1s
[score] attack in 4.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed44.json
[best agg=median] AUROC=0.8402 AUPRC=0.9045
median AUROC=0.8402 AUPRC=0.9045
p90 AUROC=0.6628 AUPRC=0.8249
mean AUROC=0.6323 AUPRC=0.8247
max AUROC=0.5918 AUPRC=0.7645
[done] elapsed=45s artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed44.json
=== protocol=forward_cross seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=forward_cross seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=503,730 -> train=1,210,760 val=302,690
[data] train_flows=10,000 val=10,000 attack=9,846 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0256 (7.7s elapsed)
[epoch 10/15] rec_loss=0.0107 (15.0s elapsed)
[epoch 15/15] rec_loss=0.0040 (22.5s elapsed)
[train] 22.5s, final rec_loss=0.0040
[score] benign in 2.1s
[score] attack in 2.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed42.json
[best agg=median] AUROC=0.6214 AUPRC=0.6244
median AUROC=0.6214 AUPRC=0.6244
p90 AUROC=0.5738 AUPRC=0.5601
mean AUROC=0.4892 AUPRC=0.5018
max AUROC=0.4702 AUPRC=0.4553
[done] elapsed=157s artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed42.json
=== protocol=forward_cross seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=forward_cross seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=503,730 -> train=1,210,760 val=302,690
[data] train_flows=10,000 val=10,000 attack=9,846 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0231 (7.8s elapsed)
[epoch 10/15] rec_loss=0.0081 (15.3s elapsed)
[epoch 15/15] rec_loss=0.0038 (22.7s elapsed)
[train] 22.7s, final rec_loss=0.0038
[score] benign in 2.1s
[score] attack in 2.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed43.json
[best agg=median] AUROC=0.6319 AUPRC=0.6693
median AUROC=0.6319 AUPRC=0.6693
p90 AUROC=0.5401 AUPRC=0.5542
mean AUROC=0.4969 AUPRC=0.4924
max AUROC=0.4889 AUPRC=0.4859
[done] elapsed=157s artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed43.json
=== protocol=forward_cross seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=forward_cross seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=503,730 -> train=1,210,760 val=302,690
[data] train_flows=10,000 val=10,000 attack=9,846 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0195 (8.1s elapsed)
[epoch 10/15] rec_loss=0.0100 (15.7s elapsed)
[epoch 15/15] rec_loss=0.0037 (23.1s elapsed)
[train] 23.1s, final rec_loss=0.0037
[score] benign in 2.2s
[score] attack in 2.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed44.json
[best agg=median] AUROC=0.3678 AUPRC=0.4001
median AUROC=0.3678 AUPRC=0.4001
p90 AUROC=0.3272 AUPRC=0.4094
mean AUROC=0.2587 AUPRC=0.3873
max AUROC=0.2540 AUPRC=0.3701
[done] elapsed=157s artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed44.json
=== protocol=reverse_cross seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=reverse_cross seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=8,893,668 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=6,772 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0299 (8.1s elapsed)
[epoch 10/15] rec_loss=0.0128 (15.7s elapsed)
[epoch 15/15] rec_loss=0.0056 (23.3s elapsed)
[train] 23.3s, final rec_loss=0.0056
[score] benign in 2.1s
[score] attack in 1.4s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed42.json
[best agg=mean] AUROC=0.8442 AUPRC=0.7504
mean AUROC=0.8442 AUPRC=0.7504
max AUROC=0.8172 AUPRC=0.7065
p90 AUROC=0.7700 AUPRC=0.6800
median AUROC=0.6507 AUPRC=0.6041
[done] elapsed=250s artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed42.json
=== protocol=reverse_cross seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=reverse_cross seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=8,893,668 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=6,772 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0222 (8.1s elapsed)
[epoch 10/15] rec_loss=0.0077 (15.8s elapsed)
[epoch 15/15] rec_loss=0.0040 (23.6s elapsed)
[train] 23.6s, final rec_loss=0.0040
[score] benign in 2.1s
[score] attack in 1.5s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed43.json
[best agg=max] AUROC=0.6797 AUPRC=0.5524
max AUROC=0.6797 AUPRC=0.5524
mean AUROC=0.4566 AUPRC=0.4307
p90 AUROC=0.3843 AUPRC=0.3849
median AUROC=0.3337 AUPRC=0.4061
[done] elapsed=247s artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed43.json
=== protocol=reverse_cross seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=reverse_cross seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=8,893,668 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=6,772 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0225 (7.9s elapsed)
[epoch 10/15] rec_loss=0.0077 (15.2s elapsed)
[epoch 15/15] rec_loss=0.0040 (22.7s elapsed)
[train] 22.7s, final rec_loss=0.0040
[score] benign in 2.1s
[score] attack in 1.5s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed44.json
[best agg=max] AUROC=0.5801 AUPRC=0.6040
max AUROC=0.5801 AUPRC=0.6040
median AUROC=0.4205 AUPRC=0.3476
mean AUROC=0.3775 AUPRC=0.4047
p90 AUROC=0.2758 AUPRC=0.3123
[done] elapsed=244s artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed44.json

316
artifacts/baselines/anomaly_transformer_2026_04_29/orchestrator.log Normal file
View File

@@ -0,0 +1,316 @@
=== protocol=iscxtor_within seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=iscxtor_within seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/packets.npz
[data] using external flow features D=20
[data] rows total=103,079 keep len>=2: 66,189
[data] benign=64,877 attack=1,312 -> train=51,901 val=10,000
[data] train_flows=10,000 val=10,000 attack=1,312 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0274 (7.7s elapsed)
[epoch 10/15] rec_loss=0.0131 (14.3s elapsed)
[epoch 15/15] rec_loss=0.0056 (21.2s elapsed)
[train] 21.2s, final rec_loss=0.0056
[score] benign in 2.1s
[score] attack in 0.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed42.json
[best agg=max] AUROC=0.5264 AUPRC=0.2290
max AUROC=0.5264 AUPRC=0.2290
mean AUROC=0.4932 AUPRC=0.1752
median AUROC=0.4792 AUPRC=0.1852
p90 AUROC=0.4373 AUPRC=0.1487
[done] elapsed=33s artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed42.json
=== protocol=iscxtor_within seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=iscxtor_within seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/packets.npz
[data] using external flow features D=20
[data] rows total=103,079 keep len>=2: 66,189
[data] benign=64,877 attack=1,312 -> train=51,901 val=10,000
[data] train_flows=10,000 val=10,000 attack=1,312 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0250 (7.6s elapsed)
[epoch 10/15] rec_loss=0.0078 (14.9s elapsed)
[epoch 15/15] rec_loss=0.0038 (22.5s elapsed)
[train] 22.5s, final rec_loss=0.0038
[score] benign in 2.1s
[score] attack in 0.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed43.json
[best agg=median] AUROC=0.4736 AUPRC=0.1267
median AUROC=0.4736 AUPRC=0.1267
max AUROC=0.4134 AUPRC=0.2217
mean AUROC=0.4133 AUPRC=0.1929
p90 AUROC=0.3542 AUPRC=0.1179
[done] elapsed=34s artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed43.json
=== protocol=iscxtor_within seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=iscxtor_within seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/iscxtor2016/processed/packets.npz
[data] using external flow features D=20
[data] rows total=103,079 keep len>=2: 66,189
[data] benign=64,877 attack=1,312 -> train=51,901 val=10,000
[data] train_flows=10,000 val=10,000 attack=1,312 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0230 (7.2s elapsed)
[epoch 10/15] rec_loss=0.0071 (14.0s elapsed)
[epoch 15/15] rec_loss=0.0065 (20.8s elapsed)
[train] 20.8s, final rec_loss=0.0065
[score] benign in 2.2s
[score] attack in 0.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed44.json
[best agg=max] AUROC=0.5150 AUPRC=0.2248
max AUROC=0.5150 AUPRC=0.2248
median AUROC=0.5020 AUPRC=0.1873
mean AUROC=0.4870 AUPRC=0.1970
p90 AUROC=0.4450 AUPRC=0.1830
[done] elapsed=33s artifacts/baselines/anomaly_transformer_2026_04_29/iscxtor_within_seed44.json
=== protocol=cicids_within seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicids_within seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=30,000 -> train=1,210,760 val=10,000
[data] train_flows=10,000 val=10,000 attack=30,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0256 (7.2s elapsed)
[epoch 10/15] rec_loss=0.0108 (14.6s elapsed)
[epoch 15/15] rec_loss=0.0041 (22.5s elapsed)
[train] 22.5s, final rec_loss=0.0041
[score] benign in 2.1s
[score] attack in 6.4s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed42.json
[best agg=mean] AUROC=0.6001 AUPRC=0.8383
mean AUROC=0.6001 AUPRC=0.8383
max AUROC=0.5478 AUPRC=0.7840
p90 AUROC=0.4774 AUPRC=0.7639
median AUROC=0.3624 AUPRC=0.7142
[done] elapsed=142s artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed42.json
=== protocol=cicids_within seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicids_within seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=30,000 -> train=1,210,760 val=10,000
[data] train_flows=10,000 val=10,000 attack=30,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0233 (8.2s elapsed)
[epoch 10/15] rec_loss=0.0081 (16.0s elapsed)
[epoch 15/15] rec_loss=0.0038 (23.7s elapsed)
[train] 23.7s, final rec_loss=0.0038
[score] benign in 2.1s
[score] attack in 6.3s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed43.json
[best agg=median] AUROC=0.2927 AUPRC=0.6398
median AUROC=0.2927 AUPRC=0.6398
p90 AUROC=0.2795 AUPRC=0.6818
mean AUROC=0.2588 AUPRC=0.6678
max AUROC=0.2573 AUPRC=0.6591
[done] elapsed=141s artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed43.json
=== protocol=cicids_within seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicids_within seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=30,000 -> train=1,210,760 val=10,000
[data] train_flows=10,000 val=10,000 attack=30,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0197 (8.3s elapsed)
[epoch 10/15] rec_loss=0.0097 (16.1s elapsed)
[epoch 15/15] rec_loss=0.0037 (23.8s elapsed)
[train] 23.8s, final rec_loss=0.0037
[score] benign in 2.2s
[score] attack in 6.4s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed44.json
[best agg=max] AUROC=0.6878 AUPRC=0.8658
max AUROC=0.6878 AUPRC=0.8658
mean AUROC=0.6436 AUPRC=0.8672
p90 AUROC=0.5559 AUPRC=0.8319
median AUROC=0.4988 AUPRC=0.8127
[done] elapsed=141s artifacts/baselines/anomaly_transformer_2026_04_29/cicids_within_seed44.json
=== protocol=cicddos_within seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicddos_within seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=20,000 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=20,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0306 (8.3s elapsed)
[epoch 10/15] rec_loss=0.0127 (15.9s elapsed)
[epoch 15/15] rec_loss=0.0056 (23.7s elapsed)
[train] 23.7s, final rec_loss=0.0056
[score] benign in 2.1s
[score] attack in 4.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed42.json
[best agg=median] AUROC=0.4061 AUPRC=0.7025
median AUROC=0.4061 AUPRC=0.7025
p90 AUROC=0.3688 AUPRC=0.6932
mean AUROC=0.3678 AUPRC=0.7023
max AUROC=0.3495 AUPRC=0.6605
[done] elapsed=45s artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed42.json
=== protocol=cicddos_within seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicddos_within seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=20,000 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=20,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0222 (8.2s elapsed)
[epoch 10/15] rec_loss=0.0079 (16.2s elapsed)
[epoch 15/15] rec_loss=0.0040 (24.5s elapsed)
[train] 24.5s, final rec_loss=0.0040
[score] benign in 2.0s
[score] attack in 4.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed43.json
[best agg=p90] AUROC=0.1995 AUPRC=0.6167
p90 AUROC=0.1995 AUPRC=0.6167
median AUROC=0.1869 AUPRC=0.5569
max AUROC=0.1673 AUPRC=0.6028
mean AUROC=0.1619 AUPRC=0.6078
[done] elapsed=46s artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed43.json
=== protocol=cicddos_within seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=cicddos_within seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=20,000 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=20,000 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0225 (8.0s elapsed)
[epoch 10/15] rec_loss=0.0078 (15.6s elapsed)
[epoch 15/15] rec_loss=0.0041 (23.1s elapsed)
[train] 23.1s, final rec_loss=0.0041
[score] benign in 2.1s
[score] attack in 4.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed44.json
[best agg=median] AUROC=0.8402 AUPRC=0.9045
median AUROC=0.8402 AUPRC=0.9045
p90 AUROC=0.6628 AUPRC=0.8249
mean AUROC=0.6323 AUPRC=0.8247
max AUROC=0.5918 AUPRC=0.7645
[done] elapsed=45s artifacts/baselines/anomaly_transformer_2026_04_29/cicddos_within_seed44.json
=== protocol=forward_cross seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=forward_cross seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=503,730 -> train=1,210,760 val=302,690
[data] train_flows=10,000 val=10,000 attack=9,846 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0256 (7.7s elapsed)
[epoch 10/15] rec_loss=0.0107 (15.0s elapsed)
[epoch 15/15] rec_loss=0.0040 (22.5s elapsed)
[train] 22.5s, final rec_loss=0.0040
[score] benign in 2.1s
[score] attack in 2.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed42.json
[best agg=median] AUROC=0.6214 AUPRC=0.6244
median AUROC=0.6214 AUPRC=0.6244
p90 AUROC=0.5738 AUPRC=0.5601
mean AUROC=0.4892 AUPRC=0.5018
max AUROC=0.4702 AUPRC=0.4553
[done] elapsed=157s artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed42.json
=== protocol=forward_cross seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=forward_cross seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=503,730 -> train=1,210,760 val=302,690
[data] train_flows=10,000 val=10,000 attack=9,846 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0231 (7.8s elapsed)
[epoch 10/15] rec_loss=0.0081 (15.3s elapsed)
[epoch 15/15] rec_loss=0.0038 (22.7s elapsed)
[train] 22.7s, final rec_loss=0.0038
[score] benign in 2.1s
[score] attack in 2.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed43.json
[best agg=median] AUROC=0.6319 AUPRC=0.6693
median AUROC=0.6319 AUPRC=0.6693
p90 AUROC=0.5401 AUPRC=0.5542
mean AUROC=0.4969 AUPRC=0.4924
max AUROC=0.4889 AUPRC=0.4859
[done] elapsed=157s artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed43.json
=== protocol=forward_cross seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=forward_cross seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicids2017/processed/packets.npz
[data] using external flow features D=20
[data] rows total=2,025,564 keep len>=2: 2,017,180
[data] benign=1,513,450 attack=503,730 -> train=1,210,760 val=302,690
[data] train_flows=10,000 val=10,000 attack=9,846 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0195 (8.1s elapsed)
[epoch 10/15] rec_loss=0.0100 (15.7s elapsed)
[epoch 15/15] rec_loss=0.0037 (23.1s elapsed)
[train] 23.1s, final rec_loss=0.0037
[score] benign in 2.2s
[score] attack in 2.1s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed44.json
[best agg=median] AUROC=0.3678 AUPRC=0.4001
median AUROC=0.3678 AUPRC=0.4001
p90 AUROC=0.3272 AUPRC=0.4094
mean AUROC=0.2587 AUPRC=0.3873
max AUROC=0.2540 AUPRC=0.3701
[done] elapsed=157s artifacts/baselines/anomaly_transformer_2026_04_29/forward_cross_seed44.json
=== protocol=reverse_cross seed=42 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=reverse_cross seed=42
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=8,893,668 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=6,772 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0299 (8.1s elapsed)
[epoch 10/15] rec_loss=0.0128 (15.7s elapsed)
[epoch 15/15] rec_loss=0.0056 (23.3s elapsed)
[train] 23.3s, final rec_loss=0.0056
[score] benign in 2.1s
[score] attack in 1.4s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed42.json
[best agg=mean] AUROC=0.8442 AUPRC=0.7504
mean AUROC=0.8442 AUPRC=0.7504
max AUROC=0.8172 AUPRC=0.7065
p90 AUROC=0.7700 AUPRC=0.6800
median AUROC=0.6507 AUPRC=0.6041
[done] elapsed=250s artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed42.json
=== protocol=reverse_cross seed=43 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=reverse_cross seed=43
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=8,893,668 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=6,772 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0222 (8.1s elapsed)
[epoch 10/15] rec_loss=0.0077 (15.8s elapsed)
[epoch 15/15] rec_loss=0.0040 (23.6s elapsed)
[train] 23.6s, final rec_loss=0.0040
[score] benign in 2.1s
[score] attack in 1.5s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed43.json
[best agg=max] AUROC=0.6797 AUPRC=0.5524
max AUROC=0.6797 AUPRC=0.5524
mean AUROC=0.4566 AUPRC=0.4307
p90 AUROC=0.3843 AUPRC=0.3849
median AUROC=0.3337 AUPRC=0.4061
[done] elapsed=247s artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed43.json
=== protocol=reverse_cross seed=44 epochs=15 batch=128 ===
[run] anomaly_transformer protocol=reverse_cross seed=44
[data] flows=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/flows.parquet packets_source=/home/chy/mambafortrafficmodeling/datasets/cicddos2019/processed/full_store
[data] using external flow features D=20
[data] rows total=8,993,376 keep len>=2: 8,986,875
[data] benign=93,207 attack=8,893,668 -> train=74,565 val=18,642
[data] train_flows=10,000 val=10,000 attack=6,772 D=9 device=cuda
[model] params=305,941
[epoch 5/15] rec_loss=0.0225 (7.9s elapsed)
[epoch 10/15] rec_loss=0.0077 (15.2s elapsed)
[epoch 15/15] rec_loss=0.0040 (22.7s elapsed)
[train] 22.7s, final rec_loss=0.0040
[score] benign in 2.1s
[score] attack in 1.5s
[saved] artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed44.json
[best agg=max] AUROC=0.5801 AUPRC=0.6040
max AUROC=0.5801 AUPRC=0.6040
median AUROC=0.4205 AUPRC=0.3476
mean AUROC=0.3775 AUPRC=0.4047
p90 AUROC=0.2758 AUPRC=0.3123
[done] elapsed=244s artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed44.json
ALL DONE

288
artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed42.json Normal file
View File

@@ -0,0 +1,288 @@
{
"method": "anomaly_transformer",
"protocol": "reverse_cross",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed42",
"n_train": 10000,
"n_val": 10000,
"n_atk": 6772,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 23.29,
"loss_first_last": [
0.14130105172531515,
0.005604498141409853
],
"overall_by_agg": {
"mean": {
"auroc": 0.8441902909037211,
"auprc": 0.7503811509240934
},
"max": {
"auroc": 0.8171573981098643,
"auprc": 0.7064730252974024
},
"median": {
"auroc": 0.6507296072061427,
"auprc": 0.6041436533714446
},
"p90": {
"auroc": 0.7699723641464855,
"auprc": 0.6800127316009327
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 666.0,
"auroc": 0.8937602102102102
},
"DDoS": {
"_n": 666.0,
"auroc": 0.9210349849849848
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.8860076576576577
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.8927644144144142
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.9019121621621621
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.7949579579579579
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.7788546546546548
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.7892
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.7397142857142857
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.8061200450450451
},
"Portscan": {
"_n": 666.0,
"auroc": 0.8833273273273273
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.698478828828829
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.7296356164383562
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.9339384615384616
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.7209722222222222
}
},
"max": {
"Botnet": {
"_n": 666.0,
"auroc": 0.780576951951952
},
"DDoS": {
"_n": 666.0,
"auroc": 0.8745903903903903
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.8431027027027026
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.8299033033033032
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.889171996996997
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.7773794294294294
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.7797936936936937
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.9548
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.8399714285714286
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.7701466966966968
},
"Portscan": {
"_n": 666.0,
"auroc": 0.8481421171171172
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.7707084834834834
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.8702712328767123
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.8619307692307692
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.85105
}
},
"median": {
"Botnet": {
"_n": 666.0,
"auroc": 0.7698609609609609
},
"DDoS": {
"_n": 666.0,
"auroc": 0.485801876876877
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.5896962462462464
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.6884997747747748
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.565258033033033
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.6203490240240241
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.5560197447447448
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.3437
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.40904285714285715
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.9600471471471471
},
"Portscan": {
"_n": 666.0,
"auroc": 0.9599916666666667
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.3526761261261261
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.3437
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.858123076923077
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.3437
}
},
"p90": {
"Botnet": {
"_n": 666.0,
"auroc": 0.9052775525525526
},
"DDoS": {
"_n": 666.0,
"auroc": 0.7179333333333333
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.7656078078078079
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.7565526276276275
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.805270870870871
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.7739398648648649
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.8422819819819819
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.1049
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.4063142857142857
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.8906704204204202
},
"Portscan": {
"_n": 666.0,
"auroc": 0.9116813813813812
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.42198918918918926
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.11386301369863014
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.9354615384615386
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.1049
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed42.npz Normal file
View File

Binary file not shown.

288
artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed43.json Normal file
View File

@@ -0,0 +1,288 @@
{
"method": "anomaly_transformer",
"protocol": "reverse_cross",
"seed": 43,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed43",
"n_train": 10000,
"n_val": 10000,
"n_atk": 6772,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 23.63,
"loss_first_last": [
0.14317506532880325,
0.003993322554079792
],
"overall_by_agg": {
"mean": {
"auroc": 0.4565637551683402,
"auprc": 0.43074154476845705
},
"max": {
"auroc": 0.6796632088009451,
"auprc": 0.552430354953035
},
"median": {
"auroc": 0.3336748154164205,
"auprc": 0.40610685321644757
},
"p90": {
"auroc": 0.3842701860602481,
"auprc": 0.3849137591979363
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 666.0,
"auroc": 0.4569912912912913
},
"DDoS": {
"_n": 666.0,
"auroc": 0.6351558558558559
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.42046006006006004
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.3436268768768769
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.45092192192192193
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.3478476726726727
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.17148513513513514
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.07469999999999999
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.12732857142857143
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.6950406906906906
},
"Portscan": {
"_n": 666.0,
"auroc": 0.9706375375375375
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.11286516516516515
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.18231780821917806
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.5425076923076922
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.1983611111111111
}
},
"max": {
"Botnet": {
"_n": 666.0,
"auroc": 0.6036139639639639
},
"DDoS": {
"_n": 666.0,
"auroc": 0.780161111111111
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.6608007507507507
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.6074217717717718
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.6471046546546547
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.5971674174174174
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.6830364864864865
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.8371000000000001
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.639357142857143
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.4951987987987988
},
"Portscan": {
"_n": 666.0,
"auroc": 0.9017625375375377
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.7886006006006006
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.9095986301369864
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.6941076923076923
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.9189111111111111
}
},
"median": {
"Botnet": {
"_n": 666.0,
"auroc": 0.250798048048048
},
"DDoS": {
"_n": 666.0,
"auroc": 0.26640420420420424
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.25962019519519514
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.16605758258258257
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.1765325825825826
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.2530582582582583
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.1632
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.1632
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.2160857142857143
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.6873358108108109
},
"Portscan": {
"_n": 666.0,
"auroc": 0.9778397897897897
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.1634536036036036
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.1632
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.19192307692307692
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.1632
}
},
"p90": {
"Botnet": {
"_n": 666.0,
"auroc": 0.4940503003003003
},
"DDoS": {
"_n": 666.0,
"auroc": 0.3986143393393393
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.2924507507507508
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.25072905405405405
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.31573280780780777
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.26408123123123123
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.11244279279279279
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.04904999999999998
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.14922142857142856
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.7294531531531532
},
"Portscan": {
"_n": 666.0,
"auroc": 0.9646893393393393
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.06833258258258255
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.04904999999999998
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.43051538461538463
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.04904999999999998
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed43.npz Normal file
View File

Binary file not shown.

288
artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed44.json Normal file
View File

@@ -0,0 +1,288 @@
{
"method": "anomaly_transformer",
"protocol": "reverse_cross",
"seed": 44,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed44",
"n_train": 10000,
"n_val": 10000,
"n_atk": 6772,
"D": 9,
"epochs": 15,
"lr": 0.0001,
"k_disc": 3.0,
"temperature": 50.0,
"d_model": 128,
"t_train_sec": 22.69,
"loss_first_last": [
0.14046703491218482,
0.003991496433869381
],
"overall_by_agg": {
"mean": {
"auroc": 0.37751623597164796,
"auprc": 0.4047034772236886
},
"max": {
"auroc": 0.5801241952155936,
"auprc": 0.6039728234817969
},
"median": {
"auroc": 0.42049935026580026,
"auprc": 0.3476409191499162
},
"p90": {
"auroc": 0.2757992690490254,
"auprc": 0.3122648050899628
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 666.0,
"auroc": 0.4738412162162162
},
"DDoS": {
"_n": 666.0,
"auroc": 0.7721778528528529
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.5258142642642643
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.711262912912913
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.5486725225225224
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.3832174924924925
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.30577192192192193
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.6284000000000001
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.19322857142857142
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.013867492492492497
},
"Portscan": {
"_n": 666.0,
"auroc": 0.009455630630630627
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.06311981981981982
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.1339671232876712
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.5136076923076923
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.13924999999999998
}
},
"max": {
"Botnet": {
"_n": 666.0,
"auroc": 0.626911036036036
},
"DDoS": {
"_n": 666.0,
"auroc": 0.9007268768768768
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.739242942942943
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.8433722222222222
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.7245804804804805
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.6317326576576576
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.7286666666666667
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.9992
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.6157285714285714
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.0187463963963964
},
"Portscan": {
"_n": 666.0,
"auroc": 0.009331831831831828
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.534770945945946
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.8766520547945206
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.6754769230769231
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.8684611111111111
}
},
"median": {
"Botnet": {
"_n": 666.0,
"auroc": 0.5296717717717717
},
"DDoS": {
"_n": 666.0,
"auroc": 0.3575990990990991
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.3952307807807808
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.3318165165165165
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.3367241741741741
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.35629737237237236
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.40581298798798804
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.30374999999999996
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.30374999999999996
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.522578978978979
},
"Portscan": {
"_n": 666.0,
"auroc": 0.6302899399399399
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.35813018018018017
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.30374999999999996
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.32809615384615387
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.30374999999999996
}
},
"p90": {
"Botnet": {
"_n": 666.0,
"auroc": 0.4574834084084084
},
"DDoS": {
"_n": 666.0,
"auroc": 0.43173205705705703
},
"DoS GoldenEye": {
"_n": 666.0,
"auroc": 0.32524474474474474
},
"DoS Hulk": {
"_n": 666.0,
"auroc": 0.29145908408408405
},
"DoS Slowhttptest": {
"_n": 666.0,
"auroc": 0.37425945945945943
},
"DoS Slowloris": {
"_n": 666.0,
"auroc": 0.21709729729729732
},
"FTP-Patator": {
"_n": 666.0,
"auroc": 0.21281051051051048
},
"Heartbleed": {
"_n": 1.0,
"auroc": 0.05625000000000002
},
"Infiltration": {
"_n": 7.0,
"auroc": 0.1657357142857143
},
"Infiltration - Portscan": {
"_n": 666.0,
"auroc": 0.1499222972972973
},
"Portscan": {
"_n": 666.0,
"auroc": 0.14283513513513513
},
"SSH-Patator": {
"_n": 666.0,
"auroc": 0.18387312312312312
},
"Web Attack - Brute Force": {
"_n": 73.0,
"auroc": 0.05786849315068495
},
"Web Attack - SQL Injection": {
"_n": 13.0,
"auroc": 0.4016461538461539
},
"Web Attack - XSS": {
"_n": 18.0,
"auroc": 0.06093611111111113
}
}
}
}

BIN
artifacts/baselines/anomaly_transformer_2026_04_29/reverse_cross_seed44.npz Normal file
View File

Binary file not shown.

724
artifacts/baselines/anomaly_transformer_2026_04_29/summary.json Normal file
View File

@@ -0,0 +1,724 @@
{
"rows": [
{
"protocol": "iscxtor_within",
"n_seeds": 3,
"best_agg": "p90",
"auroc_mean": 0.41216375762195123,
"auroc_std": 0.050302170433342654,
"abs_auroc_mean": 0.5878362423780489,
"abs_auroc_std": 0.050302170433342654,
"all_aggs": {
"mean": {
"auroc_mean": 0.46451020071138216,
"auroc_std": 0.04444281163746598,
"abs_auroc_mean": 0.5354897992886178,
"abs_auroc_std": 0.044442811637466016
},
"max": {
"auroc_mean": 0.4849680132113821,
"auroc_std": 0.062218349147918434,
"abs_auroc_mean": 0.54268356199187,
"abs_auroc_std": 0.03843480472750384
},
"median": {
"auroc_mean": 0.4849544080284553,
"auroc_std": 0.015057481255194563,
"abs_auroc_mean": 0.5164094639227642,
"abs_auroc_std": 0.012742713175109128
},
"p90": {
"auroc_mean": 0.41216375762195123,
"auroc_std": 0.050302170433342654,
"abs_auroc_mean": 0.5878362423780489,
"abs_auroc_std": 0.050302170433342654
}
}
},
{
"protocol": "cicids_within",
"n_seeds": 3,
"best_agg": "mean",
"auroc_mean": 0.5008647777777778,
"auroc_std": 0.2107434205204985,
"abs_auroc_mean": 0.6616516966666667,
"abs_auroc_std": 0.07222883427530209,
"all_aggs": {
"mean": {
"auroc_mean": 0.5008647777777778,
"auroc_std": 0.2107434205204985,
"abs_auroc_mean": 0.6616516966666667,
"abs_auroc_std": 0.07222883427530209
},
"max": {
"auroc_mean": 0.4976374927777778,
"auroc_std": 0.2196157413237703,
"abs_auroc_mean": 0.6594498772222223,
"abs_auroc_std": 0.10051393425032756
},
"median": {
"auroc_mean": 0.38462360944444446,
"auroc_std": 0.10480730142299796,
"abs_auroc_mean": 0.6153763905555555,
"abs_auroc_std": 0.10480730142299793
},
"p90": {
"auroc_mean": 0.43759324833333335,
"auroc_std": 0.14244768765655508,
"abs_auroc_mean": 0.5996638427777777,
"abs_auroc_std": 0.10599021352571453
}
}
},
{
"protocol": "cicddos_within",
"n_seeds": 3,
"best_agg": "median",
"auroc_mean": 0.47772612749999993,
"auroc_std": 0.3324922077261016,
"abs_auroc_mean": 0.7490622624999999,
"abs_auroc_std": 0.13508234670144054,
"all_aggs": {
"mean": {
"auroc_mean": 0.3873264733333334,
"auroc_std": 0.23584393356290714,
"abs_auroc_mean": 0.7009006683333333,
"abs_auroc_std": 0.11884329434390078
},
"max": {
"auroc_mean": 0.369551825,
"auroc_std": 0.2129503159588168,
"abs_auroc_mean": 0.69164826,
"abs_auroc_std": 0.12561585595244118
},
"median": {
"auroc_mean": 0.47772612749999993,
"auroc_std": 0.3324922077261016,
"abs_auroc_mean": 0.7490622624999999,
"abs_auroc_std": 0.13508234670144054
},
"p90": {
"auroc_mean": 0.4103730108333334,
"auroc_std": 0.23443402670235713,
"abs_auroc_mean": 0.6981660691666667,
"abs_auroc_std": 0.09002289821513176
}
}
},
{
"protocol": "forward_cross",
"n_seeds": 3,
"best_agg": "median",
"auroc_mean": 0.5403936403954228,
"auroc_std": 0.1495421102979965,
"abs_auroc_mean": 0.6285116375516284,
"abs_auroc_std": 0.00611968542235559,
"all_aggs": {
"mean": {
"auroc_mean": 0.41495291150382557,
"auroc_std": 0.1353781339481017,
"abs_auroc_mean": 0.5850470884961744,
"abs_auroc_std": 0.13537813394810166
},
"max": {
"auroc_mean": 0.40434917225269135,
"auroc_std": 0.13056700869549043,
"abs_auroc_mean": 0.5956508277473085,
"abs_auroc_std": 0.13056700869549043
},
"median": {
"auroc_mean": 0.5403936403954228,
"auroc_std": 0.1495421102979965,
"abs_auroc_mean": 0.6285116375516284,
"abs_auroc_std": 0.00611968542235559
},
"p90": {
"auroc_mean": 0.4803580371047465,
"auroc_std": 0.1337072839194574,
"abs_auroc_mean": 0.5955567743245989,
"abs_auroc_std": 0.06899059467567079
}
}
},
{
"protocol": "reverse_cross",
"n_seeds": 3,
"best_agg": "p90",
"auroc_mean": 0.4766806064185863,
"auroc_std": 0.2597239425287057,
"abs_auroc_mean": 0.7033009696790707,
"abs_auroc_std": 0.07921673490801118,
"all_aggs": {
"mean": {
"auroc_mean": 0.5594234273479031,
"auroc_std": 0.2497623920996739,
"abs_auroc_mean": 0.670036766587911,
"abs_auroc_std": 0.15591412731533005
},
"max": {
"auroc_mean": 0.6923149340421343,
"auroc_std": 0.11902199138084962,
"abs_auroc_mean": 0.6923149340421343,
"abs_auroc_std": 0.11902199138084962
},
"median": {
"auroc_mean": 0.4683012576294545,
"auroc_std": 0.1638435290449658,
"abs_auroc_mean": 0.6321851471746406,
"abs_auroc_std": 0.04628766262566922
},
"p90": {
"auroc_mean": 0.4766806064185863,
"auroc_std": 0.2597239425287057,
"abs_auroc_mean": 0.7033009696790707,
"abs_auroc_std": 0.07921673490801118
}
}
}
],
"per_class": {
"iscxtor_within": {
"tor": {
"n": 1312,
"aurocs": [
0.49322682926829275,
0.41331875,
0.48698502286585366
]
}
},
"cicids_within": {
"Botnet": {
"n": 46,
"aurocs": [
0.8872391304347825,
0.3123346153846153,
0.46004210526315786
]
},
"DDoS": {
"n": 5752,
"aurocs": [
0.8693338404033378,
0.6342495588494794,
0.5145650968544518
]
},
"DoS GoldenEye": {
"n": 464,
"aurocs": [
0.8588349137931033,
0.7240725672877847,
0.7046408296943232
]
},
"DoS Hulk": {
"n": 9358,
"aurocs": [
0.8672453141696944,
0.12875773550916603,
0.4318485673352435
]
},
"DoS Slowhttptest": {
"n": 78,
"aurocs": [
0.8991410256410255,
0.31539722222222216,
0.6630583333333333
]
},
"DoS Slowloris": {
"n": 185,
"aurocs": [
0.762110810810811,
0.15719341317365268,
0.5808367088607596
]
},
"FTP-Patator": {
"n": 236,
"aurocs": [
0.7410864406779661,
0.03223434579439251,
0.3321348214285714
]
},
"Infiltration": {
"n": 2,
"aurocs": [
0.46535,
0.9448
]
},
"Infiltration - Portscan": {
"n": 4295,
"aurocs": [
0.603313003492433,
0.4443001539554714,
0.5555688679245283
]
},
"Portscan": {
"n": 9425,
"aurocs": [
0.14459035543766577,
0.05020893327711605,
0.9853483215454448
]
},
"SSH-Patator": {
"n": 152,
"aurocs": [
0.6733217105263157,
0.9247592896174862,
0.19467950310559007
]
},
"Web Attack - Brute Force": {
"n": 5,
"aurocs": [
0.72576,
0.9488333333333333,
0.3236285714285714
]
},
"Web Attack - XSS": {
"n": 2,
"aurocs": [
0.7853,
0.9608599999999999
]
},
"Web Attack - SQL Injection": {
"n": 2,
"aurocs": [
0.8862,
0.19369999999999998
]
}
},
"cicddos_within": {
"DrDoS_DNS": {
"n": 1136,
"aurocs": [
0.19116588908450705,
0.017933482542524647,
0.38358046387154326
]
},
"DrDoS_LDAP": {
"n": 1152,
"aurocs": [
0.18985000000000002,
0.005402806563039751,
0.3384721048182587
]
},
"DrDoS_MSSQL": {
"n": 1135,
"aurocs": [
0.18985000000000002,
0.007452684859154955,
0.8784359464627152
]
},
"DrDoS_NTP": {
"n": 1171,
"aurocs": [
0.505333219470538,
0.27283804855275445,
0.42309863195057373
]
},
"DrDoS_NetBIOS": {
"n": 1166,
"aurocs": [
0.19054429674099488,
0.07414517657192077,
0.9832278733031674
]
},
"DrDoS_SNMP": {
"n": 1086,
"aurocs": [
0.18985000000000002,
0.009164126712328793,
0.37689410714285715
]
},
"DrDoS_SSDP": {
"n": 1092,
"aurocs": [
0.2037271978021978,
0.3974829535095715,
0.9443937277580071
]
},
"DrDoS_UDP": {
"n": 1109,
"aurocs": [
0.20567407574391344,
0.42633196573489635,
0.9635801412180053
]
},
"LDAP": {
"n": 1105,
"aurocs": [
0.18985000000000002,
0.005454378648874089,
0.34371287813310286
]
},
"MSSQL": {
"n": 1184,
"aurocs": [
0.18985000000000002,
0.007531764705882378,
0.8914378446115288
]
},
"NetBIOS": {
"n": 1539,
"aurocs": [
0.18985000000000002,
0.06480254614894973,
0.9836372795969773
]
},
"Portmap": {
"n": 417,
"aurocs": [
0.19371738609112713,
0.06535786240786241,
0.9855167865707434
]
},
"Syn": {
"n": 3361,
"aurocs": [
0.9442166468313002,
0.12263094156827128,
0.20813423054417787
]
},
"TFTP": {
"n": 1106,
"aurocs": [
0.23980361663652802,
0.5118133650519031,
0.9565173021925643
]
},
"UDP": {
"n": 1383,
"aurocs": [
0.20412762111352134,
0.43651510494752627,
0.9599579822616406
]
},
"UDPLag": {
"n": 857,
"aurocs": [
0.8218587514585765,
0.2210940389294404,
0.35599000000000003
]
},
"WebDDoS": {
"n": 1,
"aurocs": [
0.44220000000000004,
0.16269999999999996,
0.6828000000000001
]
}
},
"forward_cross": {
"DrDoS_DNS": {
"n": 588,
"aurocs": [
0.2418359693877551,
0.3203674319727891,
0.12040178571428571
]
},
"DrDoS_LDAP": {
"n": 588,
"aurocs": [
0.2234609693877551,
0.3218841836734694,
0.12222193877551019
]
},
"DrDoS_MSSQL": {
"n": 588,
"aurocs": [
0.37049260204081635,
0.5871074829931973,
0.1191062074829932
]
},
"DrDoS_NTP": {
"n": 588,
"aurocs": [
0.6877126700680272,
0.6272752551020409,
0.4970311224489796
]
},
"DrDoS_NetBIOS": {
"n": 588,
"aurocs": [
0.48235034013605443,
0.564168537414966,
0.10205935374149659
]
},
"DrDoS_SNMP": {
"n": 588,
"aurocs": [
0.22540357142857143,
0.33965544217687077,
0.12043333333333331
]
},
"DrDoS_SSDP": {
"n": 588,
"aurocs": [
0.5496690476190476,
0.6149467687074831,
0.4376086734693878
]
},
"DrDoS_UDP": {
"n": 588,
"aurocs": [
0.5629209183673469,
0.6218015306122449,
0.43346173469387755
]
},
"LDAP": {
"n": 588,
"aurocs": [
0.22110892857142855,
0.3331803571428571,
0.12309634353741497
]
},
"MSSQL": {
"n": 588,
"aurocs": [
0.38065323129251705,
0.5915131802721088,
0.12141343537414964
]
},
"NetBIOS": {
"n": 588,
"aurocs": [
0.4970219387755102,
0.5796848639455783,
0.10477653061224489
]
},
"Portmap": {
"n": 588,
"aurocs": [
0.48847789115646256,
0.5787874149659864,
0.10042551020408164
]
},
"Syn": {
"n": 588,
"aurocs": [
0.9474401360544217,
0.34615850340136056,
0.3379833333333333
]
},
"TFTP": {
"n": 588,
"aurocs": [
0.5612562925170068,
0.6246388605442177,
0.5251631802721088
]
},
"UDP": {
"n": 588,
"aurocs": [
0.5692301870748299,
0.6211027210884353,
0.4429054421768708
]
},
"UDPLag": {
"n": 588,
"aurocs": [
0.9008552721088435,
0.30889481292517007,
0.27171930272108846
]
},
"WebDDoS": {
"n": 438,
"aurocs": [
0.378726598173516,
0.45648915525114153,
0.47257134703196346
]
}
},
"reverse_cross": {
"Botnet": {
"n": 666,
"aurocs": [
0.8937602102102102,
0.4569912912912913,
0.4738412162162162
]
},
"DDoS": {
"n": 666,
"aurocs": [
0.9210349849849848,
0.6351558558558559,
0.7721778528528529
]
},
"DoS GoldenEye": {
"n": 666,
"aurocs": [
0.8860076576576577,
0.42046006006006004,
0.5258142642642643
]
},
"DoS Hulk": {
"n": 666,
"aurocs": [
0.8927644144144142,
0.3436268768768769,
0.711262912912913
]
},
"DoS Slowhttptest": {
"n": 666,
"aurocs": [
0.9019121621621621,
0.45092192192192193,
0.5486725225225224
]
},
"DoS Slowloris": {
"n": 666,
"aurocs": [
0.7949579579579579,
0.3478476726726727,
0.3832174924924925
]
},
"FTP-Patator": {
"n": 666,
"aurocs": [
0.7788546546546548,
0.17148513513513514,
0.30577192192192193
]
},
"Heartbleed": {
"n": 1,
"aurocs": [
0.7892,
0.07469999999999999,
0.6284000000000001
]
},
"Infiltration": {
"n": 7,
"aurocs": [
0.7397142857142857,
0.12732857142857143,
0.19322857142857142
]
},
"Infiltration - Portscan": {
"n": 666,
"aurocs": [
0.8061200450450451,
0.6950406906906906,
0.013867492492492497
]
},
"Portscan": {
"n": 666,
"aurocs": [
0.8833273273273273,
0.9706375375375375,
0.009455630630630627
]
},
"SSH-Patator": {
"n": 666,
"aurocs": [
0.698478828828829,
0.11286516516516515,
0.06311981981981982
]
},
"Web Attack - Brute Force": {
"n": 73,
"aurocs": [
0.7296356164383562,
0.18231780821917806,
0.1339671232876712
]
},
"Web Attack - SQL Injection": {
"n": 13,
"aurocs": [
0.9339384615384616,
0.5425076923076922,
0.5136076923076923
]
},
"Web Attack - XSS": {
"n": 18,
"aurocs": [
0.7209722222222222,
0.1983611111111111,
0.13924999999999998
]
}
}
},
"baselines": {
"terminal_norm": {
"iscxtor_within": [
0.9945,
0.0011
],
"cicids_within": [
0.9858,
0.0021
],
"cicddos_within": [
0.996,
0.001
],
"forward_cross": [
0.9109,
0.0032
],
"reverse_cross": [
0.5999,
null
]
}
}
}

69
artifacts/baselines/anomaly_transformer_2026_04_29/summary.md Normal file
View File

@@ -0,0 +1,69 @@
# Anomaly-Transformer (ICLR 2022) Baseline — On Our 5-Protocol Layout
Date: 2026-04-29
Method: ICLR 2022 Anomaly-Transformer (association-discrepancy minimax). Vendored model class from `baselines/Anomaly-Transformer/model/AnomalyTransformer.py`; training + scoring loop reimplemented to match our protocol (input shape [B, T=64, D=9] = our z-scored packet sequences, same train/val/attack splits as eval_new_scores.py).
Hyperparams: d_model=128, n_heads=4, e_layers=3, batch=128, lr=1e-4, k_disc=3.0, temperature=50.0, epochs=15.
Score: per-position softmax(-association_KL · T) · MSE(rec, x), then aggregated per flow (mean / max / median / p90).
## Headline AUROC (best aggregator per protocol, 3-seed mean ± std)
| Protocol | terminal_norm (Unified_CFM) | **AT (ours)** | abs AUROC | best agg | Δ vs terminal |
|---|---:|---:|---:|---|---:|
| ISCXTor2016 within | 0.9945 ± 0.0011 | **0.4122 ± 0.0503** | 0.5878 ± 0.0503 | `p90` | -0.5823 |
| CICIDS2017 within (σ=0.6) | 0.9858 ± 0.0021 | **0.5009 ± 0.2107** | 0.6617 ± 0.0722 | `mean` | -0.4849 |
| CICDDoS2019 within | 0.9960 ± 0.0010 | **0.4777 ± 0.3325** | 0.7491 ± 0.1351 | `median` | -0.5183 |
| IDS2017→DDoS2019 forward | 0.9109 ± 0.0032 | **0.5404 ± 0.1495** | 0.6285 ± 0.0061 | `median` | -0.3705 |
| DDoS2019→IDS2017 reverse | 0.5999 | **0.4767 ± 0.2597** | 0.7033 ± 0.0792 | `p90` | -0.1232 |
## All aggregators (3-seed mean ± std)
| Protocol | mean | max | median | p90 |
|---|---:|---:|---:|---:|
| ISCXTor2016 within | 0.4645 ± 0.0444 | 0.4850 ± 0.0622 | 0.4850 ± 0.0151 | 0.4122 ± 0.0503 |
| CICIDS2017 within (σ=0.6) | 0.5009 ± 0.2107 | 0.4976 ± 0.2196 | 0.3846 ± 0.1048 | 0.4376 ± 0.1424 |
| CICDDoS2019 within | 0.3873 ± 0.2358 | 0.3696 ± 0.2130 | 0.4777 ± 0.3325 | 0.4104 ± 0.2344 |
| IDS2017→DDoS2019 forward | 0.4150 ± 0.1354 | 0.4043 ± 0.1306 | 0.5404 ± 0.1495 | 0.4804 ± 0.1337 |
| DDoS2019→IDS2017 reverse | 0.5594 ± 0.2498 | 0.6923 ± 0.1190 | 0.4683 ± 0.1638 | 0.4767 ± 0.2597 |
## Per-attack (forward + reverse, mean aggregator)
### IDS2017→DDoS2019 forward
| attack | n | AT AUROC mean ± std |
|---|---:|---:|
| `DrDoS_DNS` | 588 | 0.2275 ± 0.1007 |
| `DrDoS_LDAP` | 588 | 0.2225 ± 0.0998 |
| `DrDoS_MSSQL` | 588 | 0.3589 ± 0.2342 |
| `DrDoS_NTP` | 588 | 0.6040 ± 0.0974 |
| `DrDoS_NetBIOS` | 588 | 0.3829 ± 0.2466 |
| `DrDoS_SNMP` | 588 | 0.2285 ± 0.1096 |
| `DrDoS_SSDP` | 588 | 0.5341 ± 0.0897 |
| `DrDoS_UDP` | 588 | 0.5394 ± 0.0963 |
| `LDAP` | 588 | 0.2258 ± 0.1051 |
| `MSSQL` | 588 | 0.3645 ± 0.2355 |
| `NetBIOS` | 588 | 0.3938 ± 0.2537 |
| `Portmap` | 588 | 0.3892 ± 0.2542 |
| `Syn` | 588 | 0.5439 ± 0.3495 |
| `TFTP` | 588 | 0.5704 ± 0.0504 |
| `UDP` | 588 | 0.5444 ± 0.0917 |
| `UDPLag` | 588 | 0.4938 ± 0.3530 |
| `WebDDoS` | 438 | 0.4359 ± 0.0502 |
### DDoS2019→IDS2017 reverse
| attack | n | AT AUROC mean ± std |
|---|---:|---:|
| `Botnet` | 666 | 0.6082 ± 0.2474 |
| `DDoS` | 666 | 0.7761 ± 0.1430 |
| `DoS GoldenEye` | 666 | 0.6108 ± 0.2441 |
| `DoS Hulk` | 666 | 0.6492 ± 0.2798 |
| `DoS Slowhttptest` | 666 | 0.6338 ± 0.2373 |
| `DoS Slowloris` | 666 | 0.5087 ± 0.2486 |
| `FTP-Patator` | 666 | 0.4187 ± 0.3190 |
| `Heartbleed` | 1 | 0.4974 ± 0.3748 |
| `Infiltration` | 7 | 0.3534 ± 0.3362 |
| `Infiltration - Portscan` | 666 | 0.5050 ± 0.4290 |
| `Portscan` | 666 | 0.6211 ± 0.5315 |
| `SSH-Patator` | 666 | 0.2915 ± 0.3533 |
| `Web Attack - Brute Force` | 73 | 0.3486 ± 0.3308 |
| `Web Attack - SQL Injection` | 13 | 0.6634 ± 0.2348 |
| `Web Attack - XSS` | 18 | 0.3529 ± 0.3202 |

85
artifacts/baselines/extract_ciciot2023.log Normal file
View File

@@ -0,0 +1,85 @@
[discover] 34 pcap files across 34 labels
backdoor_malware 1 pcap(s)
browserhijacking 1 pcap(s)
commandinjection 1 pcap(s)
ddos-ack_fragmentation 1 pcap(s)
ddos-http_flood 1 pcap(s)
ddos-icmp_flood 1 pcap(s)
ddos-icmp_fragmentation 1 pcap(s)
ddos-pshack_flood 1 pcap(s)
ddos-rstfinflood 1 pcap(s)
ddos-slowloris 1 pcap(s)
ddos-syn_flood 1 pcap(s)
ddos-synonymousip_flood 1 pcap(s)
ddos-tcp_flood 1 pcap(s)
ddos-udp_flood 1 pcap(s)
ddos-udp_fragmentation 1 pcap(s)
dictionarybruteforce 1 pcap(s)
dns_spoofing 1 pcap(s)
dos-http_flood 1 pcap(s)
dos-syn_flood 1 pcap(s)
dos-tcp_flood 1 pcap(s)
dos-udp_flood 1 pcap(s)
mirai-greeth_flood 1 pcap(s)
mirai-greip_flood 1 pcap(s)
mirai-udpplain 1 pcap(s)
mitm-arpspoofing 1 pcap(s)
normal 1 pcap(s)
recon-hostdiscovery 1 pcap(s)
recon-osscan 1 pcap(s)
recon-pingsweep 1 pcap(s)
recon-portscan 1 pcap(s)
sqlinjection 1 pcap(s)
uploading_attack 1 pcap(s)
vulnerabilityscan 1 pcap(s)
xss 1 pcap(s)
[extract_labeled_pcaps] n_pcaps=34 T_full=256 extra_cols=('class_folder',)
backdoor_malware Backdoor_Malware.pcap extra={'class_folder': 'Backdoor_Malware'}
normal BenignTraffic.pcap extra={'class_folder': 'Benign_Final'}
browserhijacking BrowserHijacking.pcap extra={'class_folder': 'BrowserHijacking'}
commandinjection CommandInjection.pcap extra={'class_folder': 'CommandInjection'}
ddos-ack_fragmentation DDoS-ACK_Fragmentation.pcap extra={'class_folder': 'DDoS-ACK_Fragmentation'}
ddos-http_flood DDoS-HTTP_Flood-.pcap extra={'class_folder': 'DDoS-HTTP_Flood'}
ddos-icmp_flood DDoS-ICMP_Flood.pcap extra={'class_folder': 'DDoS-ICMP_Flood'}
ddos-icmp_fragmentation DDoS-ICMP_Fragmentation.pcap extra={'class_folder': 'DDoS-ICMP_Fragmentation'}
ddos-pshack_flood DDoS-PSHACK_Flood.pcap extra={'class_folder': 'DDoS-PSHACK_Flood'}
ddos-rstfinflood DDoS-RSTFINFlood.pcap extra={'class_folder': 'DDoS-RSTFINFlood'}
... (24 more)
[extract_labeled_pcaps] running 34 pcap(s) with 4 worker(s)
[extract_labeled_pcaps] sharded output enabled: datasets/ciciot2023/processed/full_store shard_size=100,000
[extract_labeled_pcaps] worker spool=/home/chy/mambafortrafficmodeling/datasets/ciciot2023/processed/.full_store.spool._q0mmt_4 flush_size=10,000
[pcap:Backdoor_Malware.pcap] label=backdoor_malware 29,155 pkts → 2,325 flows in 0.6s (0.05M pkts/s)
[pcap:CommandInjection.pcap] label=commandinjection 49,515 pkts → 3,784 flows in 1.0s (0.05M pkts/s)
[pcap:BrowserHijacking.pcap] label=browserhijacking 55,181 pkts → 2,800 flows in 1.0s (0.05M pkts/s)
[pcap:BenignTraffic.pcap] label=normal 2,000,000 pkts → 130,266 flows in 36.7s (0.05M pkts/s)
[pcap:DDoS-HTTP_Flood-.pcap] label=ddos-http_flood 2,000,000 pkts → 424,632 flows in 51.0s (0.04M pkts/s)
[pcap:DDoS-ICMP_Fragmentation.pcap] label=ddos-icmp_fragmentation 91,881 pkts → 15,315 flows in 24.2s (0.00M pkts/s)
[pcap:DDoS-ACK_Fragmentation.pcap] label=ddos-ack_fragmentation 1,421,801 pkts → 1,199,853 flows in 72.1s (0.02M pkts/s)
[pcap:DDoS-PSHACK_Flood.pcap] label=ddos-pshack_flood 2,000,000 pkts → 472,916 flows in 47.0s (0.04M pkts/s)
[pcap:DDoS-SYN_Flood.pcap] label=ddos-syn_flood 2,000,000 pkts → 445,305 flows in 46.9s (0.04M pkts/s)
[pcap:DDoS-SlowLoris.pcap] label=ddos-slowloris 2,000,000 pkts → 170,596 flows in 40.9s (0.05M pkts/s)
[pcap:DDoS-RSTFINFlood.pcap] label=ddos-rstfinflood 2,000,000 pkts → 1,989,762 flows in 87.8s (0.02M pkts/s)
[pcap:DDoS-SynonymousIP_Flood.pcap] label=ddos-synonymousip_flood 2,000,000 pkts → 66,126 flows in 36.4s (0.05M pkts/s)
[pcap:DDoS-UDP_Flood.pcap] label=ddos-udp_flood 2,000,000 pkts → 3,021 flows in 22.8s (0.09M pkts/s)
[pcap:DDoS-UDP_Fragmentation.pcap] label=ddos-udp_fragmentation 1,141,302 pkts → 10,202 flows in 26.7s (0.04M pkts/s)
[pcap:DDoS-TCP_Flood.pcap] label=ddos-tcp_flood 2,000,000 pkts → 459,439 flows in 44.7s (0.04M pkts/s)
[pcap:DictionaryBruteForce.pcap] label=dictionarybruteforce 121,861 pkts → 7,910 flows in 2.3s (0.05M pkts/s)
[pcap:DNS_Spoofing.pcap] label=dns_spoofing 1,717,375 pkts → 83,761 flows in 29.3s (0.06M pkts/s)
[pcap:DoS-SYN_Flood.pcap] label=dos-syn_flood 2,000,000 pkts → 332,245 flows in 45.3s (0.04M pkts/s)
[pcap:DoS-HTTP_Flood.pcap] label=dos-http_flood 2,000,000 pkts → 426,432 flows in 51.0s (0.04M pkts/s)
[pcap:DoS-TCP_Flood.pcap] label=dos-tcp_flood 2,000,000 pkts → 404,258 flows in 46.6s (0.04M pkts/s)
[pcap:DoS-UDP_Flood.pcap] label=dos-udp_flood 2,000,000 pkts → 67,459 flows in 33.9s (0.06M pkts/s)
[pcap:DDoS-ICMP_Flood.pcap] label=ddos-icmp_flood 78,905 pkts → 12,441 flows in 267.1s (0.00M pkts/s)
[pcap:MITM-ArpSpoofing.pcap] label=mitm-arpspoofing 2,000,000 pkts → 55,312 flows in 32.1s (0.06M pkts/s)
[pcap:Mirai-udpplain.pcap] label=mirai-udpplain 2,000,000 pkts → 4,351 flows in 21.6s (0.09M pkts/s)
[pcap:Recon-HostDiscovery.pcap] label=recon-hostdiscovery 1,253,455 pkts → 663,947 flows in 39.1s (0.03M pkts/s)
[pcap:Recon-PingSweep.pcap] label=recon-pingsweep 19,361 pkts → 1,955 flows in 0.4s (0.05M pkts/s)
[pcap:Mirai-greeth_flood.pcap] label=mirai-greeth_flood 83,075 pkts → 5,465 flows in 60.5s (0.00M pkts/s)
[pcap:SqlInjection.pcap] label=sqlinjection 49,185 pkts → 6,693 flows in 1.0s (0.05M pkts/s)
[pcap:Uploading_Attack.pcap] label=uploading_attack 10,826 pkts → 1,338 flows in 0.2s (0.05M pkts/s)
[pcap:Recon-OSScan.pcap] label=recon-osscan 948,173 pkts → 193,983 flows in 22.2s (0.04M pkts/s)
[pcap:XSS.pcap] label=xss 34,617 pkts → 3,209 flows in 0.7s (0.05M pkts/s)
[pcap:Mirai-greip_flood.pcap] label=mirai-greip_flood 99,556 pkts → 10,526 flows in 51.7s (0.00M pkts/s)
[pcap:Recon-PortScan.pcap] label=recon-portscan 794,588 pkts → 225,917 flows in 20.2s (0.04M pkts/s)
[pcap:VulnerabilityScan.pcap] label=vulnerabilityscan 2,000,000 pkts → 290,077 flows in 42.2s (0.05M pkts/s)
[extract_labeled_pcaps] wrote sharded store datasets/ciciot2023/processed/full_store

315
artifacts/baselines/kitsune_2026_04_29/cicddos_within_seed42.json Normal file
View File

@@ -0,0 +1,315 @@
{
"method": "kitsune_path_b",
"protocol": "cicddos_within",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed42",
"n_train_flows": 5000,
"n_train_packets": 55918,
"n_val": 10000,
"n_atk": 20000,
"D": 9,
"fm_grace": 2000,
"ad_grace": 20000,
"max_ae_size": 10,
"t_train_sec": 2.34,
"overall_by_agg": {
"mean": {
"auroc": 0.4253447725,
"auprc": 0.6336203824974163
},
"max": {
"auroc": 0.3284644375,
"auprc": 0.5703262532476359
},
"median": {
"auroc": 0.474287345,
"auprc": 0.6604585372129211
},
"p90": {
"auroc": 0.345611965,
"auprc": 0.5911061098185502
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.37572706866197186
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.3659486545138889
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.3971305726872247
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.4506702391118702
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.39661989708404805
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.36687532228360953
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.4189297619047619
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.41150360685302073
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.36997108597285067
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.39456355574324325
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.4013035087719298
},
"Portmap": {
"_n": 417.0,
"auroc": 0.4142306954436451
},
"Syn": {
"_n": 3361.0,
"auroc": 0.5423447932163046
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.41400976491862573
},
"UDP": {
"_n": 1383.0,
"auroc": 0.41174555314533623
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.4536086931155192
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.08040000000000003
}
},
"max": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.23766461267605635
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.2267169704861111
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.2521197356828194
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.675849146029035
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.25728048885077187
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.2260081952117864
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.33097696886446887
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.3387283137962128
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.23371393665158371
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.24820565878378376
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.2602960688758934
},
"Portmap": {
"_n": 417.0,
"auroc": 0.2658227817745803
},
"Syn": {
"_n": 3361.0,
"auroc": 0.4394152038083904
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.3338644665461121
},
"UDP": {
"_n": 1383.0,
"auroc": 0.3339986623282719
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.35733786464410733
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.17759999999999998
}
},
"median": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.44823058978873237
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.43687092013888884
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.46721519823788543
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.5051565328778821
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.4637955403087478
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.4399157458563536
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.4803783882783883
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.466178088367899
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.4384012669683257
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.46593053209459456
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.46961754385964916
},
"Portmap": {
"_n": 417.0,
"auroc": 0.4834028776978418
},
"Syn": {
"_n": 3361.0,
"auroc": 0.5197994346920559
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.4705183092224231
},
"UDP": {
"_n": 1383.0,
"auroc": 0.47341887201735355
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.4768441073512252
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.1069
}
},
"p90": {
"DrDoS_DNS": {
"_n": 1136.0,
"auroc": 0.26284080105633806
},
"DrDoS_LDAP": {
"_n": 1152.0,
"auroc": 0.25305117187500004
},
"DrDoS_MSSQL": {
"_n": 1135.0,
"auroc": 0.2844314537444934
},
"DrDoS_NTP": {
"_n": 1171.0,
"auroc": 0.4862088385994876
},
"DrDoS_NetBIOS": {
"_n": 1166.0,
"auroc": 0.29204897084048026
},
"DrDoS_SNMP": {
"_n": 1086.0,
"auroc": 0.25303305709023943
},
"DrDoS_SSDP": {
"_n": 1092.0,
"auroc": 0.3609811813186813
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.3631856627592425
},
"LDAP": {
"_n": 1105.0,
"auroc": 0.26044470588235297
},
"MSSQL": {
"_n": 1184.0,
"auroc": 0.2799416385135135
},
"NetBIOS": {
"_n": 1539.0,
"auroc": 0.2939029564652372
},
"Portmap": {
"_n": 417.0,
"auroc": 0.30097362110311754
},
"Syn": {
"_n": 3361.0,
"auroc": 0.4718764207081226
},
"TFTP": {
"_n": 1106.0,
"auroc": 0.3635770795660036
},
"UDP": {
"_n": 1383.0,
"auroc": 0.3609580621836587
},
"UDPLag": {
"_n": 857.0,
"auroc": 0.3887544340723454
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.1442
}
}
}
}

BIN
artifacts/baselines/kitsune_2026_04_29/cicddos_within_seed42.npz Normal file
View File

Binary file not shown.

315
artifacts/baselines/kitsune_2026_04_29/cicddos_within_seed43.json Normal file
View File

@@ -0,0 +1,315 @@
{
"method": "kitsune_path_b",
"protocol": "cicddos_within",
"seed": 43,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed43",
"n_train_flows": 5000,
"n_train_packets": 55952,
"n_val": 10000,
"n_atk": 20000,
"D": 9,
"fm_grace": 2000,
"ad_grace": 20000,
"max_ae_size": 10,
"t_train_sec": 2.38,
"overall_by_agg": {
"mean": {
"auroc": 0.43168440249999995,
"auprc": 0.653248355032242
},
"max": {
"auroc": 0.34255154250000003,
"auprc": 0.5775747163392775
},
"median": {
"auroc": 0.4719932275,
"auprc": 0.6770389923462667
},
"p90": {
"auroc": 0.3623585325,
"auprc": 0.6097127275583607
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.4625233661593554
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.49788264248704667
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.3625894806338028
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.34427651727357605
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.3367791559000861
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.48163343321917806
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.3569629443938013
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.375805996393147
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.4901982485404504
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.34968621848739495
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.3650196053469128
},
"Portmap": {
"_n": 407.0,
"auroc": 0.35616412776412776
},
"Syn": {
"_n": 3303.0,
"auroc": 0.5789207841356343
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.39400925605536335
},
"UDP": {
"_n": 1334.0,
"auroc": 0.37381038230884556
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.49753053527980534
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.41180000000000005
}
},
"max": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.33061566696508504
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.3498836787564767
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.22813639964788732
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.5780242763772176
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.2146053402239449
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.33724122431506853
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.2981407474931632
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.30952894499549144
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.3542118015012511
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.2230166806722689
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.23675684277530235
},
"Portmap": {
"_n": 407.0,
"auroc": 0.23488280098280095
},
"Syn": {
"_n": 3303.0,
"auroc": 0.4851591583409022
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.32835947231833906
},
"UDP": {
"_n": 1334.0,
"auroc": 0.3146345952023988
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.39591076642335765
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.1008
}
},
"median": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.5223282452999105
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.5554525906735751
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.4286453345070423
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.431211858076564
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.3964294142980189
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.5432048801369863
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.4090533272561531
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.43014386834986473
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.5473794829024187
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.41311222689075633
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.42869287078294077
},
"Portmap": {
"_n": 407.0,
"auroc": 0.4212051597051597
},
"Syn": {
"_n": 3303.0,
"auroc": 0.5330334392976083
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.4276136678200692
},
"UDP": {
"_n": 1334.0,
"auroc": 0.4304715892053973
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.5127475669099757
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.4849
}
},
"p90": {
"DrDoS_DNS": {
"_n": 1117.0,
"auroc": 0.3723965085049239
},
"DrDoS_LDAP": {
"_n": 1158.0,
"auroc": 0.39827927461139895
},
"DrDoS_MSSQL": {
"_n": 1136.0,
"auroc": 0.2534990316901409
},
"DrDoS_NTP": {
"_n": 1071.0,
"auroc": 0.4119016339869281
},
"DrDoS_NetBIOS": {
"_n": 1161.0,
"auroc": 0.2365090439276486
},
"DrDoS_SNMP": {
"_n": 1168.0,
"auroc": 0.38031429794520544
},
"DrDoS_SSDP": {
"_n": 1097.0,
"auroc": 0.31528650865998176
},
"DrDoS_UDP": {
"_n": 1109.0,
"auroc": 0.328724526600541
},
"LDAP": {
"_n": 1199.0,
"auroc": 0.3967505838198499
},
"MSSQL": {
"_n": 1190.0,
"auroc": 0.24594655462184878
},
"NetBIOS": {
"_n": 1571.0,
"auroc": 0.2623697644812222
},
"Portmap": {
"_n": 407.0,
"auroc": 0.2580361179361179
},
"Syn": {
"_n": 3303.0,
"auroc": 0.5216696033908568
},
"TFTP": {
"_n": 1156.0,
"auroc": 0.3491068339100346
},
"UDP": {
"_n": 1334.0,
"auroc": 0.33177863568215893
},
"UDPLag": {
"_n": 822.0,
"auroc": 0.4339459245742092
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.13349999999999995
}
}
}
}

BIN
artifacts/baselines/kitsune_2026_04_29/cicddos_within_seed43.npz Normal file
View File

Binary file not shown.

315
artifacts/baselines/kitsune_2026_04_29/cicddos_within_seed44.json Normal file
View File

@@ -0,0 +1,315 @@
{
"method": "kitsune_path_b",
"protocol": "cicddos_within",
"seed": 44,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_multiseed_2026_04_25/cicddos2019_lambda0p3_seed44",
"n_train_flows": 5000,
"n_train_packets": 53770,
"n_val": 10000,
"n_atk": 20000,
"D": 9,
"fm_grace": 2000,
"ad_grace": 20000,
"max_ae_size": 10,
"t_train_sec": 2.33,
"overall_by_agg": {
"mean": {
"auroc": 0.4614600125,
"auprc": 0.6514813502115975
},
"max": {
"auroc": 0.336559935,
"auprc": 0.5586935467559199
},
"median": {
"auroc": 0.46671968499999994,
"auprc": 0.6562102115267414
},
"p90": {
"auroc": 0.35462059749999997,
"auprc": 0.5842039708602825
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.45824696699375556
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.4437374049027895
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.4331852772466539
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.4227395410414828
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.4768812669683258
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.44571107142857147
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.45909572953736655
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.4440355251544572
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.46051931719965433
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.4203869674185463
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.47621527078085646
},
"Portmap": {
"_n": 417.0,
"auroc": 0.4663419664268585
},
"Syn": {
"_n": 3418.0,
"auroc": 0.4984345669982446
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.4915897521448999
},
"UDP": {
"_n": 1353.0,
"auroc": 0.4453246858832225
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.4728106432748538
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.3994
}
},
"max": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.28695820695807317
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.2802118765849535
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.25612839388145314
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.6631459841129745
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.28098542986425334
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.27987794642857144
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.34742228647686835
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.34577197705207413
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.29160108038029386
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.2567668755221386
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.27886917506297226
},
"Portmap": {
"_n": 417.0,
"auroc": 0.273452757793765
},
"Syn": {
"_n": 3418.0,
"auroc": 0.3731526477472206
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.393523832221163
},
"UDP": {
"_n": 1353.0,
"auroc": 0.3437822246858832
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.34739397660818716
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.5508
}
},
"median": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.4651411239964317
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.4511319526627219
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.43998102294455066
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.47979249779346866
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.48298113122171943
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.45153102678571433
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.47963923487544485
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.46066866725507505
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.46814900605012966
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.426693567251462
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.4826334068010076
},
"Portmap": {
"_n": 417.0,
"auroc": 0.4743570743405276
},
"Syn": {
"_n": 3418.0,
"auroc": 0.472372279110591
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.4801701620591039
},
"UDP": {
"_n": 1353.0,
"auroc": 0.46612283813747224
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.47796064327485377
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.7785
}
},
"p90": {
"DrDoS_DNS": {
"_n": 1121.0,
"auroc": 0.3187443354148082
},
"DrDoS_LDAP": {
"_n": 1183.0,
"auroc": 0.3087689349112426
},
"DrDoS_MSSQL": {
"_n": 1046.0,
"auroc": 0.28197901529636715
},
"DrDoS_NTP": {
"_n": 1133.0,
"auroc": 0.5308290379523389
},
"DrDoS_NetBIOS": {
"_n": 1105.0,
"auroc": 0.30983904977375565
},
"DrDoS_SNMP": {
"_n": 1120.0,
"auroc": 0.3051109375
},
"DrDoS_SSDP": {
"_n": 1124.0,
"auroc": 0.3719600978647687
},
"DrDoS_UDP": {
"_n": 1133.0,
"auroc": 0.3729121800529568
},
"LDAP": {
"_n": 1157.0,
"auroc": 0.3238210025929127
},
"MSSQL": {
"_n": 1197.0,
"auroc": 0.2771423976608187
},
"NetBIOS": {
"_n": 1588.0,
"auroc": 0.30957616498740553
},
"Portmap": {
"_n": 417.0,
"auroc": 0.30309280575539566
},
"Syn": {
"_n": 3418.0,
"auroc": 0.39941252194265653
},
"TFTP": {
"_n": 1049.0,
"auroc": 0.42477235462345087
},
"UDP": {
"_n": 1353.0,
"auroc": 0.3666122320768662
},
"UDPLag": {
"_n": 855.0,
"auroc": 0.3708116374269006
},
"WebDDoS": {
"_n": 1.0,
"auroc": 0.512
}
}
}
}

BIN
artifacts/baselines/kitsune_2026_04_29/cicddos_within_seed44.npz Normal file
View File

Binary file not shown.

251
artifacts/baselines/kitsune_2026_04_29/cicids_within_seed42.json Normal file
View File

@@ -0,0 +1,251 @@
{
"method": "kitsune_path_b",
"protocol": "cicids_within",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed42",
"n_train_flows": 5000,
"n_train_packets": 60260,
"n_val": 10000,
"n_atk": 30000,
"D": 9,
"fm_grace": 2000,
"ad_grace": 20000,
"max_ae_size": 10,
"t_train_sec": 2.53,
"overall_by_agg": {
"mean": {
"auroc": 0.723987635,
"auprc": 0.8872871375309783
},
"max": {
"auroc": 0.70639805,
"auprc": 0.8641590977032279
},
"median": {
"auroc": 0.6683948383333334,
"auprc": 0.8659779046749252
},
"p90": {
"auroc": 0.7134654466666667,
"auprc": 0.8853790719337757
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 46.0,
"auroc": 0.4925913043478261
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.7028624217663422
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.6963890086206896
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.6402013784996794
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.5836448717948718
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.6075902702702703
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.6303199152542374
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.7437
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.6324439580908032
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.8726749496021221
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.5695197368421053
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.5733199999999999
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.5315
}
},
"max": {
"Botnet": {
"_n": 46.0,
"auroc": 0.5696065217391304
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.7740621957579972
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.7862476293103449
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.7299238352212012
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.6455884615384615
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.7176697297297296
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.8331372881355933
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.91735
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.505422549476135
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.7242967692307694
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.8722671052631579
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.9212400000000001
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.91505
}
},
"median": {
"Botnet": {
"_n": 46.0,
"auroc": 0.4132652173913043
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.5643121088317107
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.5558272629310346
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.5382328595853815
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.5202935897435899
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.528165945945946
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.5073029661016949
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.6384000000000001
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.6145343771827707
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.9034096233421751
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.49485723684210525
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.55864
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.42925
}
},
"p90": {
"Botnet": {
"_n": 46.0,
"auroc": 0.5577510869565218
},
"DDoS": {
"_n": 5752.0,
"auroc": 0.7295628911682893
},
"DoS GoldenEye": {
"_n": 464.0,
"auroc": 0.7360641163793102
},
"DoS Hulk": {
"_n": 9358.0,
"auroc": 0.6615358570207309
},
"DoS Slowhttptest": {
"_n": 78.0,
"auroc": 0.6040269230769231
},
"DoS Slowloris": {
"_n": 185.0,
"auroc": 0.6289054054054054
},
"FTP-Patator": {
"_n": 236.0,
"auroc": 0.7029288135593221
},
"Infiltration": {
"_n": 2.0,
"auroc": 0.76835
},
"Infiltration - Portscan": {
"_n": 4295.0,
"auroc": 0.574446717112922
},
"Portscan": {
"_n": 9425.0,
"auroc": 0.822038275862069
},
"SSH-Patator": {
"_n": 152.0,
"auroc": 0.6556690789473686
},
"Web Attack - Brute Force": {
"_n": 5.0,
"auroc": 0.57236
},
"Web Attack - XSS": {
"_n": 2.0,
"auroc": 0.6512
}
}
}
}

BIN
artifacts/baselines/kitsune_2026_04_29/cicids_within_seed42.npz Normal file
View File

Binary file not shown.

267
artifacts/baselines/kitsune_2026_04_29/cicids_within_seed43.json Normal file
View File

@@ -0,0 +1,267 @@
{
"method": "kitsune_path_b",
"protocol": "cicids_within",
"seed": 43,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed43",
"n_train_flows": 5000,
"n_train_packets": 59505,
"n_val": 10000,
"n_atk": 30000,
"D": 9,
"fm_grace": 2000,
"ad_grace": 20000,
"max_ae_size": 10,
"t_train_sec": 2.52,
"overall_by_agg": {
"mean": {
"auroc": 0.6668211533333334,
"auprc": 0.8607119185266181
},
"max": {
"auroc": 0.6554650116666667,
"auprc": 0.8277844345660434
},
"median": {
"auroc": 0.634465715,
"auprc": 0.8491222299021763
},
"p90": {
"auroc": 0.6561472166666666,
"auprc": 0.8497418251887944
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 39.0,
"auroc": 0.588597435897436
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.6416492323980942
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.6053505175983438
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.5833544717600931
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.5602988888888889
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.5421101796407185
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.6310906542056074
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.7455999999999999
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.6006448484130744
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.8038008748814167
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.5733109289617487
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.49396666666666667
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.476
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.45262
}
},
"max": {
"Botnet": {
"_n": 39.0,
"auroc": 0.5840461538461539
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.7347305540850538
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.720792132505176
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.6956247006463918
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.6544594444444445
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.6496095808383233
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.8215668224299066
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.9705
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.4487339175746092
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.6489923948561189
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.8829650273224042
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.8676
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.40935
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.7960200000000001
}
},
"median": {
"Botnet": {
"_n": 39.0,
"auroc": 0.5966423076923076
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.5571249250044115
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.5440619047619049
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.5237327169651372
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.5194622222222223
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.49491676646706584
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.5877266355140187
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.7531000000000001
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.5841906560871625
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.8246712712132391
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.5303459016393443
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.5329333333333333
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.5776
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.47639999999999993
}
},
"p90": {
"Botnet": {
"_n": 39.0,
"auroc": 0.6281205128205128
},
"DDoS": {
"_n": 5667.0,
"auroc": 0.6767522057526028
},
"DoS GoldenEye": {
"_n": 483.0,
"auroc": 0.6634209109730849
},
"DoS Hulk": {
"_n": 9437.0,
"auroc": 0.6218044187771539
},
"DoS Slowhttptest": {
"_n": 90.0,
"auroc": 0.5768083333333334
},
"DoS Slowloris": {
"_n": 167.0,
"auroc": 0.5709479041916168
},
"FTP-Patator": {
"_n": 214.0,
"auroc": 0.6819278037383177
},
"Infiltration": {
"_n": 1.0,
"auroc": 0.7867
},
"Infiltration - Portscan": {
"_n": 4222.0,
"auroc": 0.5129454168640455
},
"Portscan": {
"_n": 9487.0,
"auroc": 0.7434588858437863
},
"SSH-Patator": {
"_n": 183.0,
"auroc": 0.6450237704918033
},
"Web Attack - Brute Force": {
"_n": 3.0,
"auroc": 0.6121333333333333
},
"Web Attack - SQL Injection": {
"_n": 2.0,
"auroc": 0.4376
},
"Web Attack - XSS": {
"_n": 5.0,
"auroc": 0.55664
}
}
}
}

BIN
artifacts/baselines/kitsune_2026_04_29/cicids_within_seed43.npz Normal file
View File

Binary file not shown.

235
artifacts/baselines/kitsune_2026_04_29/cicids_within_seed44.json Normal file
View File

@@ -0,0 +1,235 @@
{
"method": "kitsune_path_b",
"protocol": "cicids_within",
"seed": 44,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed44",
"n_train_flows": 5000,
"n_train_packets": 60932,
"n_val": 10000,
"n_atk": 30000,
"D": 9,
"fm_grace": 2000,
"ad_grace": 20000,
"max_ae_size": 10,
"t_train_sec": 2.51,
"overall_by_agg": {
"mean": {
"auroc": 0.7161715483333334,
"auprc": 0.8881672941674507
},
"max": {
"auroc": 0.7170487166666666,
"auprc": 0.8756976844169416
},
"median": {
"auroc": 0.6483864266666667,
"auprc": 0.8600219764599565
},
"p90": {
"auroc": 0.7141619033333333,
"auprc": 0.8912745535785943
}
},
"per_class_by_agg": {
"mean": {
"Botnet": {
"_n": 38.0,
"auroc": 0.47882236842105264
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.7093726319530833
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.6815593886462883
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.6563557837206835
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.6089714285714286
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.5702905063291139
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.6069803571428571
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.5928810055223194
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.8490922833315739
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.4820925465838509
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.45777142857142855
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.1602
}
},
"max": {
"Botnet": {
"_n": 38.0,
"auroc": 0.5005552631578947
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.7984934956459926
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.7867799126637556
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.7623737928472885
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.7168559523809523
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.7038743670886076
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.8151049107142857
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.47648805798435345
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.7279311886414018
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.7982416149068323
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.8218714285714286
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.33325000000000005
}
},
"median": {
"Botnet": {
"_n": 38.0,
"auroc": 0.47616578947368426
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.5589718588946153
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.5580830786026201
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.5411474742650961
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.49092380952380954
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.5114113924050633
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.5382703125
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.5540403014265991
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.8657909479573525
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.4820391304347825
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.4935857142857143
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.15580000000000005
}
},
"p90": {
"Botnet": {
"_n": 38.0,
"auroc": 0.5257960526315789
},
"DDoS": {
"_n": 5627.0,
"auroc": 0.7506463657366269
},
"DoS GoldenEye": {
"_n": 458.0,
"auroc": 0.7347385371179039
},
"DoS Hulk": {
"_n": 9423.0,
"auroc": 0.6917450705720046
},
"DoS Slowhttptest": {
"_n": 84.0,
"auroc": 0.6745452380952381
},
"DoS Slowloris": {
"_n": 158.0,
"auroc": 0.5725
},
"FTP-Patator": {
"_n": 224.0,
"auroc": 0.6745756696428571
},
"Infiltration - Portscan": {
"_n": 4346.0,
"auroc": 0.5233837781868385
},
"Portscan": {
"_n": 9473.0,
"auroc": 0.8083549667475985
},
"SSH-Patator": {
"_n": 161.0,
"auroc": 0.5700437888198757
},
"Web Attack - Brute Force": {
"_n": 7.0,
"auroc": 0.5157428571428572
},
"Web Attack - SQL Injection": {
"_n": 1.0,
"auroc": 0.3842
}
}
}
}

BIN
artifacts/baselines/kitsune_2026_04_29/cicids_within_seed44.npz Normal file
View File

Binary file not shown.

315
artifacts/baselines/kitsune_2026_04_29/forward_cross_seed42.json Normal file
View File

@@ -0,0 +1,315 @@
{
"method": "kitsune_path_b",
"protocol": "forward_cross",
"seed": 42,
"model_dir": "/home/chy/mambafortrafficmodeling/artifacts/phase25_sigma06_multiseed_2026_04_25/cicids2017_lambda0p3_sigma0p6_seed42",
"n_train_flows": 5000,
"n_train_packets": 62210,
"n_val": 10000,
"n_atk": 9846,
"D": 9,
"fm_grace": 2000,
"ad_grace": 20000,
"max_ae_size": 10,
"t_train_sec": 2.53,
"overall_by_agg": {
"mean": {
"auroc": 0.422718281535649,
"auprc": 0.4437923569974027
},
"max": {
"auroc": 0.33296114665854154,
"auprc": 0.39356746933635084
},
"median": {
"auroc": 0.45944195612431443,
"auprc": 0.4715220579782502
},
"p90": {
"auroc": 0.35375587040422507,
"auprc": 0.40877349567832355
}
},
"per_class_by_agg": {
"mean": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.3616414115646258
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.3420397959183673
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.42484030612244894
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.5418039115646258
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.4732957482993197
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.3372551020408163
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.44126096938775505
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.44336471088435375
},
"LDAP": {
"_n": 588.0,
"auroc": 0.3339859693877551
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.3942460884353742
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.49183656462585035
},
"Portmap": {
"_n": 588.0,
"auroc": 0.46841309523809527
},
"Syn": {
"_n": 588.0,
"auroc": 0.4602437925170068
},
"TFTP": {
"_n": 588.0,
"auroc": 0.3960539115646259
},
"UDP": {
"_n": 588.0,
"auroc": 0.4473421768707483
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.4093687925170068
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.41801986301369864
}
},
"max": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.21539336734693879
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.2121749149659864
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.28435688775510204
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.7535377551020408
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.3294068027210884
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.21182831632653062
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.37969600340136056
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.368281037414966
},
"LDAP": {
"_n": 588.0,
"auroc": 0.2108563775510204
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.2722716836734694
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.3382204081632653
},
"Portmap": {
"_n": 588.0,
"auroc": 0.3190501700680272
},
"Syn": {
"_n": 588.0,
"auroc": 0.370610544217687
},
"TFTP": {
"_n": 588.0,
"auroc": 0.33109804421768707
},
"UDP": {
"_n": 588.0,
"auroc": 0.37657032312925176
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.30873630952380954
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.3937606164383562
}
},
"median": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.4222487244897959
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.3999406462585034
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.48004974489795915
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.5067812074829932
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.529956887755102
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.39267857142857143
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.46572057823129254
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.4678329081632653
},
"LDAP": {
"_n": 588.0,
"auroc": 0.38884914965986395
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.44794132653061225
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.5459670068027211
},
"Portmap": {
"_n": 588.0,
"auroc": 0.5235412414965986
},
"Syn": {
"_n": 588.0,
"auroc": 0.45838724489795923
},
"TFTP": {
"_n": 588.0,
"auroc": 0.4241635204081633
},
"UDP": {
"_n": 588.0,
"auroc": 0.47442329931972793
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.43052244897959185
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.448791894977169
}
},
"p90": {
"DrDoS_DNS": {
"_n": 588.0,
"auroc": 0.2390267006802721
},
"DrDoS_LDAP": {
"_n": 588.0,
"auroc": 0.2369531462585034
},
"DrDoS_MSSQL": {
"_n": 588.0,
"auroc": 0.32506156462585034
},
"DrDoS_NTP": {
"_n": 588.0,
"auroc": 0.6171353741496599
},
"DrDoS_NetBIOS": {
"_n": 588.0,
"auroc": 0.3765078231292517
},
"DrDoS_SNMP": {
"_n": 588.0,
"auroc": 0.23295960884353742
},
"DrDoS_SSDP": {
"_n": 588.0,
"auroc": 0.41379574829931975
},
"DrDoS_UDP": {
"_n": 588.0,
"auroc": 0.4043593537414966
},
"LDAP": {
"_n": 588.0,
"auroc": 0.23030076530612245
},
"MSSQL": {
"_n": 588.0,
"auroc": 0.30233843537414967
},
"NetBIOS": {
"_n": 588.0,
"auroc": 0.3889670068027211
},
"Portmap": {
"_n": 588.0,
"auroc": 0.3703044217687075
},
"Syn": {
"_n": 588.0,
"auroc": 0.3973332482993197
},
"TFTP": {
"_n": 588.0,
"auroc": 0.35951828231292515
},
"UDP": {
"_n": 588.0,
"auroc": 0.4113459183673469
},
"UDPLag": {
"_n": 588.0,
"auroc": 0.3376498299319728
},
"WebDDoS": {
"_n": 438.0,
"auroc": 0.3759558219178082
}
}
}
}

Some files were not shown because too many files have changed in this diff Show More