diff --git a/README.md b/README.md index ed8ab08..1363632 100644 --- a/README.md +++ b/README.md @@ -1,57 +1,151 @@ -# mambafortrafficmodeling +# JANUS -Network traffic anomaly detection with continuous flow matching (CFM). Three -sibling model packages over a shared canonical data contract. +**JANUS** (Joint Anomaly via Normalizing-flows of Unified States) — flow-matching unsupervised network anomaly detection over packet sequences. + +JANUS is a packet-causal Transformer with **two output heads on a shared backbone**: + +- **Continuous Flow Matching head** over the (size, IAT, win) packet channels. +- **Discrete Flow Matching head** over the 6 binary protocol-flag / direction channels. + +Trained jointly on benign traffic only (no attack labels at any stage). The deployable scalar score is a **Mahalanobis-OAS distance** over a 10-d per-flow score vector emitted by the trained model, with the aggregator fit on benign val only — entirely unsupervised end-to-end. + +JANUS is the first NIDS method to use Flow Matching as the training paradigm in mixed continuous–discrete state spaces over packet sequences. + +## Headline results + +3-seed mean ± std AUROC. Selection-bias-free Mahalanobis-OAS aggregator on the 10-d JANUS score vector, fit on benign val only. + +| Task | Shafir 2026 SOTA | **JANUS** | Δ | +|---|---|---|---| +| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9908 ± 0.0012** | **+0.118** | +| CICIDS2017 within | 0.9303 | **0.9845 ± 0.0030** | **+0.054** | +| CICDDoS2019 within | 0.93 | **0.9913 ± 0.0009** | **+0.061** | +| CICIDS2017 → CICDDoS2019 cross | 0.89 | **0.9594 ± 0.0046** | **+0.07** | +| CICDDoS2019 → CICIDS2017 reverse cross | 0.93 | **0.9301 ± 0.0122** | matches | + +3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIDS2017→CICDDoS2019 cross also beats; reverse direction matches. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table. ## Layout -- `common/data_contract.py` — single source of truth for the canonical - packet schema (9-d) and flow schema (20-d, packet-derived). All three - packages import constants and helpers from here. -- `Packet_CFM/` — packet-sequence OT-CFM with explicit σ-band benign - distribution learning. -- `Flow_CFM/` — flow-level CFM on the workspace-canonical 20-d packet-derived - `flow_features.parquet`. Legacy 61-d CICFlowMeter CSV caches are kept only - for paper reproduction (`--legacy-csv-features` flag). -- `Unified_CFM/` — unified packet+flow token CFM. **Current SOTA model** — - used for all main results (within-dataset SOTA on ISCXTor2016 / CICIDS2017 - / CICDDoS2019, near-SOTA cross-dataset). -- `datasets//processed/` — canonical artifact bundle: - - `packets.npz` (small/medium) or `full_store/` (large, sharded) - - `flows.parquet` (label + 5-tuple metadata) - - `flow_features.parquet` (20-d packet-derived, row-aligned) -- `scripts/` — workspace-level pcap → artifact extraction, CSV adapters, - cross-package eval tooling. `scripts/download/` is also here. -- `artifacts/` — run outputs (training checkpoints, eval JSONs, reports). - Phase 0 / 1 / 2 / 2.5 experiment summaries live under - `artifacts/phase{0,1,2}*` directories. -- `paper/` — paper PDFs we compare against (Shafir 2026 NF, ConMD 2026, - TIPSO-GAN 2026, Lipman 2210.02747 flow matching). +``` +common/ Data contract — single source of truth for the + 9-d packet schema, 20-d packet-derived flow schema, + label normalization, and packet preprocessing. +Mixed_CFM/ The JANUS model. Mixed continuous–discrete CFM + with two output heads on a shared causal Transformer. + configs/ Per-(dataset × seed) training configs. + model.py MixedTokenCFM + MixedVelocity. + train.py / eval_phase1.py / eval_cross.py +Unified_CFM/ Legacy unified token CFM. Mixed_CFM imports its + AdaLNBlock + sinusoidal time embedding for backbone + reuse. Kept as internal ablation reference. +scripts/ Workspace-level pcap → artifact pipeline, + CSV adapters, cross-package eval tooling. + download/ UNB/CIC dataset downloaders. + baselines/ Third-party baseline runners (Kitsune, Shafir-NF, + Anomaly-Transformer). +tests/ Data-contract unit tests. +``` -The root keeps only workspace-level files. All model/training/eval code -lives under one of the three packages. +The following directories are **gitignored** (live on the dev box, not in the repo): -## Current best results (Unified_CFM, λ=0.3, 3 seeds) +``` +artifacts/ All run outputs (checkpoints, eval JSONs, score npzs, + figures). Score-router aggregator at + artifacts/route_comparison/aggregate_score_router.py. +datasets/ Raw + processed datasets (~1 TB). +baselines/ Third-party baseline forks (Kitsune-py, + Anomaly-Transformer, ConMD, ganomaly, TIPSO-GAN, ...). +paper/ Paper sources & external PDFs (Shafir 2026, Lipman + 2210.02747, etc.). +.venv/ uv-managed Python 3.14 virtual env. +``` -Shafir baselines verified from paper PDF tables — see `artifacts/locked_baselines.md`. +## Data contract -| Task | Shafir 2026 SOTA | Our best | Δ | -|---|---|---|---| -| ISCXTor2016 (NonTor → Tor) | 0.8731 (Table VI) | 0.9945 ± 0.0011 (σ=0.1) | **+0.121** | -| CICIDS2017 within (10k/10k Shafir protocol) | 0.9303 (Table VII) | **0.9858 ± 0.0021** (σ=0.6) | **+0.055** | -| CICDDoS2019 within | 0.93 (Table IX) | **0.9958 ± 0.0010** (σ=0.1) | **+0.066** | -| CICIDS2017 → CICDDoS2019 cross (`terminal_norm`) | 0.89 (Table IX, IDS→DDoS row) | **0.9109 ± 0.0032** (σ=0.6) | **+0.021** | -| CICIDS2017 → CICDDoS2019 cross (`terminal_flow`) | 0.89 | **0.9197 ± 0.0036** | **+0.030** | +Every processed dataset under `datasets//processed/` ships an aligned triple, all with the same row order (`flow_id = arange(N)`): -**4 of 4 reported tasks achieve SOTA**. Cross-dataset baseline was previously misread as 0.93; the IDS→DDoS direction in Shafir Table IX is 0.89. +``` +packets.npz packet_tokens [N, T_full, 9], packet_lengths [N], flow_id [N] + (or full_store/ — sharded PacketShardStore — for large datasets) +flows.parquet flow_id + label + 5-tuple metadata (src_ip, dst_ip, ports, protocol) +flow_features.parquet flow_id + label + 20 canonical packet-derived features +``` -Plus an architectural contribution: a `flow_consistency` diagnostic score -that lifts from random (~0.6) to discriminative (~0.9) only when the model -is trained with the masked-prediction consistency loss. On SSH-Patator (the -hardest CICIDS2017 class for `terminal_norm` at 0.64) it reaches 0.94. +The 9-d packet schema and 20-d flow schema are FIXED in `common/data_contract.py`. Flow features are computed by `compute_flow_features_from_packets(packet_tokens, lens)` so row alignment is guaranteed. -Authoritative result tables live in `RESULTS.md` (root) and -`artifacts/locked_baselines.md` (Shafir baseline verification trail). -Thresholded F1 / Precision / Recall / TPR@FPR under unsupervised threshold -protocol: `RESULTS_THRESHOLDED.md`. -Per-attack-family multi-seed analysis: `artifacts/phase25_multiseed_2026_04_25/PER_ATTACK_TABLE.md`. +## Quick start + +```bash +# Train JANUS on CICIDS2017 (3 seeds available: 42, 43, 44) +cd Mixed_CFM +uv run --no-sync python train.py --config configs/cicids2017_ac_combo_seed42.yaml + +# Phase-1 evaluation: per-attack-class AUROC + 10-d score export +uv run --no-sync python eval_phase1.py \ + --model-dir --out-dir + +# Cross-dataset evaluation +uv run --no-sync python eval_cross.py \ + --src-model \ + --tgt-data datasets//processed/ \ + --out +``` + +JANUS hyper-parameters (locked in `Mixed_CFM/configs/_ac_combo_seed*.yaml`): + +```yaml +T: 64 # max packet sequence length +d_model: 128 +n_layers: 4 +n_heads: 4 +sigma: 0.1 # within-dataset; cross uses 0.6 +lambda_disc: 1.0 +use_ot: true # OT-CFM (Sinkhorn coupling on benign batch) +reference_mode: causal_packets # Route A: packet-causal attention +``` + +## Producing the deployable scalar score + +`eval_phase1.py` exports a 10-d per-flow score vector to `phase1_scores.npz`: + +``` +3 continuous-side scores : terminal_norm, terminal_flow, terminal_packet +7 discrete-side scores : disc_nll_total + disc_nll_ch{2,3,4,5,6,7} + (direction + 5 TCP flags) +``` + +The deployable scalar is the Mahalanobis-OAS distance: + +``` +d²(s) = (s − μ)ᵀ Σ⁻¹ (s − μ), where (μ, Σ) come from sklearn.covariance.OAS + fit on benign val ONLY (no attack labels). +``` + +Reference implementation: `artifacts/route_comparison/aggregate_score_router.py` (artifacts/ is gitignored; the script lives on the dev box). + +## Tests + +```bash +uv run --no-sync python -m pytest tests/ Mixed_CFM/tests/ Unified_CFM/tests/ +``` + +## Adding a new dataset + +Write one driver at `scripts/extract_.py` that calls `extract_lib.extract_dataset(...)` (see `scripts/extract_cicids2017.py` as the reference template). The driver hardcodes CSV column names, timestamp formats, benign aliases, and drop patterns as module constants, then feeds `extract_lib` a per-day `(canonical_key → [(row_idx, ts_epoch)])` mapping and a per-day pcap file map. The extract pipeline writes all three artifacts (packets.npz, flows.parquet, flow_features.parquet) row-aligned by `flow_id = arange(N)`. + +To upgrade an existing artifact pair that lacks `flow_features.parquet`, run `scripts/generate_flow_features.py --packets-npz ... --flows-parquet ... --out ...` (or `--source-store` for sharded stores). + +Common gotcha: if CSV timestamps and pcap epochs are in different time zones, `extract_lib` prints a diagnostic with the recommended `--time-offset`; rerun with that value. + +## Authoritative documents + +- `RESULTS.md` — full headline tables, ablations, per-attack analysis, JANUS configuration, thresholded operating-point metrics, what the experiments proved / disproved. +- `Mixed_CFM/model.py` and `common/data_contract.py` — model + data-contract source of truth. + +## Python environment + +- `requires-python = ">=3.14"`; PyTorch pinned to the `pytorch-cu128` index, plus `mamba-ssm`, `causal-conv1d`, `scapy`, `dpkt`, `pyarrow`, `sklearn` (for the OAS aggregator). +- Two `pyproject.toml` files exist: root and `Mixed_CFM/`; they are not declared as a uv workspace and resolve independently. Run `uv run ...` from whichever directory owns the entry point. +- `Unified_CFM/` has no `pyproject.toml`; it uses the root venv (`uv run --no-sync python `). +- Scripts under `scripts/download/` are pure stdlib — invoke with `python3`. diff --git a/RESULTS.md b/RESULTS.md index d1b0059..1122725 100644 --- a/RESULTS.md +++ b/RESULTS.md @@ -159,10 +159,10 @@ This is **+0.31 over our own legacy memory baseline of 0.62**. The "main attack direction" recorded in `reverse_cross_score_redirection_2026_04_25` is now substantially solved. -Thresholded F1 / Precision / Recall / TPR@FPR (unsupervised protocol, τ from -benign-val percentile) are reported separately in `RESULTS_THRESHOLDED.md`. -Headline thresholded numbers: CICDDoS2019 within `terminal_norm` F1=0.993 ± 0.001 -at τ=P95; cross `terminal_norm` F1=0.632 ± 0.051 at τ=P95 (precision ≈ 0.95, recall ≈ 0.47). +Thresholded F1 / Precision / Recall / TPR@FPR under the unsupervised threshold +protocol are reported in **section D** below. Headline thresholded numbers: +CICDDoS2019 within `terminal_norm` F1=0.993 ± 0.001 at τ=P95; cross `terminal_norm` +F1=0.632 ± 0.051 at τ=P95 (precision ≈ 0.95, recall ≈ 0.47). > **Note on cross-dataset baseline**: Shafir's Table IX is asymmetric. > The IDS2017→DDoS2019 direction (which we evaluate) reads **0.89**, not @@ -175,6 +175,52 @@ at τ=P95; cross `terminal_norm` F1=0.632 ± 0.051 at τ=P95 (precision ≈ 0.95 > Single-policy σ=0.6 also beats Shafir on 4/4. Full 4×2 sensitivity table > in `artifacts/sigma_validation.md`. +### D. Thresholded operating-point metrics + +⚠️ Numbers in this section are from the **Unified_CFM legacy** recipe (σ=0.1 +within, σ=0.6 cross, λ=0.3, single fixed score). Equivalent thresholded +numbers for current JANUS + Mahalanobis-OAS have not been recomputed yet; +the AUROC tables (A/B/C above) are the authoritative JANUS comparison. + +**Protocol**: τ is set from a benign-val half (A); F1 / Precision / Recall / +FPR are measured on benign-val half B + attack. AUROC / AUPRC use full +benign val + attack. TPR@FPR is measured on the test half. Both percentiles +τ ∈ {P95, P99} are reported because they correspond to different operating +points and F1 is sensitive to that choice. + +**CICDDoS2019 within** (σ=0.1, λ=0.3): + +| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR | +|---|---|---|---|---|---|---|---|---|---| +| `terminal_norm` | 0.9960 ± 0.0011 | 0.9975 ± 0.0008 | 0.9932 ± 0.0012 | 0.9881 ± 0.0015 | 0.9983 ± 0.0008 | 0.0481 ± 0.0061 | 0.9112 ± 0.0402 | 0.9013 ± 0.0540 | 0.9980 ± 0.0014 | +| `terminal_flow` | 0.9885 ± 0.0028 | 0.9918 ± 0.0017 | 0.9788 ± 0.0086 | 0.9868 ± 0.0009 | 0.9710 ± 0.0163 | 0.0517 ± 0.0030 | 0.7752 ± 0.0128 | 0.6052 ± 0.0347 | 0.9697 ± 0.0169 | + +**CICIDS2017 → CICDDoS2019 cross** (σ=0.6, λ=0.3): + +| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR | +|---|---|---|---|---|---|---|---|---|---| +| `terminal_norm` | 0.9109 ± 0.0032 | 0.8974 ± 0.0047 | 0.6321 ± 0.0513 | 0.9545 ± 0.0045 | 0.4745 ± 0.0550 | 0.0441 ± 0.0011 | 0.4202 ± 0.0171 | 0.2685 ± 0.0139 | 0.4940 ± 0.0399 | +| `terminal_flow` | 0.9197 ± 0.0036 | 0.8957 ± 0.0086 | 0.6324 ± 0.0585 | 0.9517 ± 0.0055 | 0.4762 ± 0.0639 | 0.0469 ± 0.0019 | 0.4028 ± 0.0049 | 0.2534 ± 0.0039 | 0.4776 ± 0.0636 | + +**Reading**: + +- *Within-dataset CICDDoS2019* saturates: at τ=P95 F1 ≈ 0.99 with balanced + precision and recall ≈ 0.99; at τ=P99 (≈1% FPR) F1 ≈ 0.91 with TPR@1%FPR + ≈ 0.90. The model is a working detector at fixed thresholds, not just an + AUROC artifact. +- *Cross-dataset CICIDS2017→CICDDoS2019* keeps AUROC ≈ 0.91 but at fixed τ + shows precision ≈ 0.95 / recall ≈ 0.50 at P95 and ≈0.27 at 1% FPR — the + cross-dataset domain shift compresses the score gap, so source-calibrated + thresholds are conservative on target. **AUROC alone overstates + deployability cross-dataset; thresholded numbers are the honest figure.** + +**TIPSO-GAN comparability**: TIPSO-GAN's CICDDoS2019 F1 ≈ 0.99 is reported +under a **supervised** protocol (model has seen attack examples). Our F1 +≈ 0.99 on CICDDoS2019 within is achieved under the **unsupervised** protocol +(benign-only training, threshold from benign val), which is the strictly +harder setting. Direct F1 numerical equivalence; protocol asymmetry is in +our favor. + ## Methodological contribution: `flow_consistency` diagnostic score Phase 2 masked-prediction consistency loss unlocks a new score that is @@ -319,9 +365,6 @@ artifacts. ## Source artifacts -- `RESULTS_THRESHOLDED.md` — F1 / Precision / Recall / TPR@FPR under unsupervised - threshold protocol (τ = benign-val P95/P99) for CICDDoS2019 within and - CICIDS2017→CICDDoS2019 cross. - `artifacts/locked_baselines.md` — verified Shafir baselines (PDF inspection trail). - `artifacts/sigma_validation.md` — full 4×2 σ-sensitivity table (σ ∈ {0.1, 0.6} × 4 tasks, 3 seeds each) and per-task σ-selection protocol. diff --git a/RESULTS_THRESHOLDED.md b/RESULTS_THRESHOLDED.md deleted file mode 100644 index ec0174a..0000000 --- a/RESULTS_THRESHOLDED.md +++ /dev/null @@ -1,34 +0,0 @@ -# Thresholded metrics — unsupervised AD protocol - -3-seed mean ± std. Threshold τ is set on benign-val half A; F1 / Precision / Recall / FPR are measured on benign-val half B + attack. AUROC/AUPRC use full benign val + attack. TPR@FPR is measured on the test half. - -Both percentiles are reported because P95 and P99 give different operating points; F1 numbers are sensitive to that choice. - -Primary score: `terminal_norm`. `terminal_flow` is reported on cross because RESULTS.md headlines both. - -## CICDDoS2019 within (σ=0.1, λ=0.3) - -| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR | -|---|---|---|---|---|---|---|---|---|---| -| `terminal_norm` | 0.9960 ± 0.0011 | 0.9975 ± 0.0008 | 0.9932 ± 0.0012 | 0.9881 ± 0.0015 | 0.9983 ± 0.0008 | 0.0481 ± 0.0061 | 0.9112 ± 0.0402 | 0.9013 ± 0.0540 | 0.9980 ± 0.0014 | -| `terminal_flow` | 0.9885 ± 0.0028 | 0.9918 ± 0.0017 | 0.9788 ± 0.0086 | 0.9868 ± 0.0009 | 0.9710 ± 0.0163 | 0.0517 ± 0.0030 | 0.7752 ± 0.0128 | 0.6052 ± 0.0347 | 0.9697 ± 0.0169 | - -## CICIDS2017 → CICDDoS2019 cross (σ=0.6, λ=0.3) - -| Score | AUROC | AUPRC | F1 (P95) | Prec (P95) | Recall (P95) | FPR (P95) | F1 (P99) | TPR@1%FPR | TPR@5%FPR | -|---|---|---|---|---|---|---|---|---|---| -| `terminal_norm` | 0.9109 ± 0.0032 | 0.8974 ± 0.0047 | 0.6321 ± 0.0513 | 0.9545 ± 0.0045 | 0.4745 ± 0.0550 | 0.0441 ± 0.0011 | 0.4202 ± 0.0171 | 0.2685 ± 0.0139 | 0.4940 ± 0.0399 | -| `terminal_flow` | 0.9197 ± 0.0036 | 0.8957 ± 0.0086 | 0.6324 ± 0.0585 | 0.9517 ± 0.0055 | 0.4762 ± 0.0639 | 0.0469 ± 0.0019 | 0.4028 ± 0.0049 | 0.2534 ± 0.0039 | 0.4776 ± 0.0636 | - -## Reading - -- **Within-dataset (CICDDoS2019)**: at τ=P95, `terminal_norm` reaches F1 ≈ 0.99 with precision ≈ 0.99 and recall ≈ 0.99 — saturation. At τ=P99 (≈1% FPR), F1 ≈ 0.91 / TPR@1%FPR ≈ 0.90. The model is a working detector at fixed thresholds, not just an AUROC artifact. -- **Cross-dataset (CICIDS2017 → CICDDoS2019)**: AUROC stays high (≈ 0.91) but at fixed thresholds Precision is high (≈0.95) and Recall drops to ≈0.50 at P95 / ≈0.27 at 1% FPR. The cross-dataset domain shift compresses the score gap, so a source-calibrated threshold is conservative on target — false positives stay low, but a substantial fraction of target-domain attacks score below the source benign P95. **AUROC alone overstates deployability cross-dataset; thresholded numbers are the honest figure.** -- TIPSO-GAN comparability: TIPSO-GAN's CIC-DDoS2019 F1 ≈ 0.99 is reported under a **supervised** protocol (model has seen attack examples). Our F1 ≈ 0.99 on CICDDoS2019 within is achieved under the **unsupervised** protocol (benign-only training, threshold from benign-val), which is the strictly harder setting. Direct F1 numerical equivalence; protocol asymmetry is in our favor. - -## Source artifacts - -- `artifacts/verify_2026_04_24/thresholded_metrics.py` — per-file metric tool. -- `artifacts/verify_2026_04_24/aggregate_thresholded.py` — this aggregator. -- Within: `artifacts/phase1_2026_04_25/cicddos2019_lambda0p3_seed*/thresholded_metrics.json` (computed from existing `phase1_scores.npz`). -- Cross: `artifacts/phase25_sigma06_cross_2026_04_25/with_scores/thresholded_seed*.json` (raw scores re-saved by patched `eval_phase2_cross_cicddos2019.py`).