Merge RESULTS_THRESHOLDED.md into RESULTS.md (section D); rewrite README.md for JANUS
This commit is contained in:
186
README.md
186
README.md
@@ -1,57 +1,151 @@
|
||||
# mambafortrafficmodeling
|
||||
# JANUS
|
||||
|
||||
Network traffic anomaly detection with continuous flow matching (CFM). Three
|
||||
sibling model packages over a shared canonical data contract.
|
||||
**JANUS** (Joint Anomaly via Normalizing-flows of Unified States) — flow-matching unsupervised network anomaly detection over packet sequences.
|
||||
|
||||
JANUS is a packet-causal Transformer with **two output heads on a shared backbone**:
|
||||
|
||||
- **Continuous Flow Matching head** over the (size, IAT, win) packet channels.
|
||||
- **Discrete Flow Matching head** over the 6 binary protocol-flag / direction channels.
|
||||
|
||||
Trained jointly on benign traffic only (no attack labels at any stage). The deployable scalar score is a **Mahalanobis-OAS distance** over a 10-d per-flow score vector emitted by the trained model, with the aggregator fit on benign val only — entirely unsupervised end-to-end.
|
||||
|
||||
JANUS is the first NIDS method to use Flow Matching as the training paradigm in mixed continuous–discrete state spaces over packet sequences.
|
||||
|
||||
## Headline results
|
||||
|
||||
3-seed mean ± std AUROC. Selection-bias-free Mahalanobis-OAS aggregator on the 10-d JANUS score vector, fit on benign val only.
|
||||
|
||||
| Task | Shafir 2026 SOTA | **JANUS** | Δ |
|
||||
|---|---|---|---|
|
||||
| ISCXTor2016 (NonTor → Tor) | 0.8731 | **0.9908 ± 0.0012** | **+0.118** |
|
||||
| CICIDS2017 within | 0.9303 | **0.9845 ± 0.0030** | **+0.054** |
|
||||
| CICDDoS2019 within | 0.93 | **0.9913 ± 0.0009** | **+0.061** |
|
||||
| CICIDS2017 → CICDDoS2019 cross | 0.89 | **0.9594 ± 0.0046** | **+0.07** |
|
||||
| CICDDoS2019 → CICIDS2017 reverse cross | 0.93 | **0.9301 ± 0.0122** | matches |
|
||||
|
||||
3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIDS2017→CICDDoS2019 cross also beats; reverse direction matches. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See `RESULTS.md` for caveats and the full headline table.
|
||||
|
||||
## Layout
|
||||
|
||||
- `common/data_contract.py` — single source of truth for the canonical
|
||||
packet schema (9-d) and flow schema (20-d, packet-derived). All three
|
||||
packages import constants and helpers from here.
|
||||
- `Packet_CFM/` — packet-sequence OT-CFM with explicit σ-band benign
|
||||
distribution learning.
|
||||
- `Flow_CFM/` — flow-level CFM on the workspace-canonical 20-d packet-derived
|
||||
`flow_features.parquet`. Legacy 61-d CICFlowMeter CSV caches are kept only
|
||||
for paper reproduction (`--legacy-csv-features` flag).
|
||||
- `Unified_CFM/` — unified packet+flow token CFM. **Current SOTA model** —
|
||||
used for all main results (within-dataset SOTA on ISCXTor2016 / CICIDS2017
|
||||
/ CICDDoS2019, near-SOTA cross-dataset).
|
||||
- `datasets/<name>/processed/` — canonical artifact bundle:
|
||||
- `packets.npz` (small/medium) or `full_store/` (large, sharded)
|
||||
- `flows.parquet` (label + 5-tuple metadata)
|
||||
- `flow_features.parquet` (20-d packet-derived, row-aligned)
|
||||
- `scripts/` — workspace-level pcap → artifact extraction, CSV adapters,
|
||||
cross-package eval tooling. `scripts/download/` is also here.
|
||||
- `artifacts/` — run outputs (training checkpoints, eval JSONs, reports).
|
||||
Phase 0 / 1 / 2 / 2.5 experiment summaries live under
|
||||
`artifacts/phase{0,1,2}*` directories.
|
||||
- `paper/` — paper PDFs we compare against (Shafir 2026 NF, ConMD 2026,
|
||||
TIPSO-GAN 2026, Lipman 2210.02747 flow matching).
|
||||
```
|
||||
common/ Data contract — single source of truth for the
|
||||
9-d packet schema, 20-d packet-derived flow schema,
|
||||
label normalization, and packet preprocessing.
|
||||
Mixed_CFM/ The JANUS model. Mixed continuous–discrete CFM
|
||||
with two output heads on a shared causal Transformer.
|
||||
configs/ Per-(dataset × seed) training configs.
|
||||
model.py MixedTokenCFM + MixedVelocity.
|
||||
train.py / eval_phase1.py / eval_cross.py
|
||||
Unified_CFM/ Legacy unified token CFM. Mixed_CFM imports its
|
||||
AdaLNBlock + sinusoidal time embedding for backbone
|
||||
reuse. Kept as internal ablation reference.
|
||||
scripts/ Workspace-level pcap → artifact pipeline,
|
||||
CSV adapters, cross-package eval tooling.
|
||||
download/ UNB/CIC dataset downloaders.
|
||||
baselines/ Third-party baseline runners (Kitsune, Shafir-NF,
|
||||
Anomaly-Transformer).
|
||||
tests/ Data-contract unit tests.
|
||||
```
|
||||
|
||||
The root keeps only workspace-level files. All model/training/eval code
|
||||
lives under one of the three packages.
|
||||
The following directories are **gitignored** (live on the dev box, not in the repo):
|
||||
|
||||
## Current best results (Unified_CFM, λ=0.3, 3 seeds)
|
||||
```
|
||||
artifacts/ All run outputs (checkpoints, eval JSONs, score npzs,
|
||||
figures). Score-router aggregator at
|
||||
artifacts/route_comparison/aggregate_score_router.py.
|
||||
datasets/ Raw + processed datasets (~1 TB).
|
||||
baselines/ Third-party baseline forks (Kitsune-py,
|
||||
Anomaly-Transformer, ConMD, ganomaly, TIPSO-GAN, ...).
|
||||
paper/ Paper sources & external PDFs (Shafir 2026, Lipman
|
||||
2210.02747, etc.).
|
||||
.venv/ uv-managed Python 3.14 virtual env.
|
||||
```
|
||||
|
||||
Shafir baselines verified from paper PDF tables — see `artifacts/locked_baselines.md`.
|
||||
## Data contract
|
||||
|
||||
| Task | Shafir 2026 SOTA | Our best | Δ |
|
||||
|---|---|---|---|
|
||||
| ISCXTor2016 (NonTor → Tor) | 0.8731 (Table VI) | 0.9945 ± 0.0011 (σ=0.1) | **+0.121** |
|
||||
| CICIDS2017 within (10k/10k Shafir protocol) | 0.9303 (Table VII) | **0.9858 ± 0.0021** (σ=0.6) | **+0.055** |
|
||||
| CICDDoS2019 within | 0.93 (Table IX) | **0.9958 ± 0.0010** (σ=0.1) | **+0.066** |
|
||||
| CICIDS2017 → CICDDoS2019 cross (`terminal_norm`) | 0.89 (Table IX, IDS→DDoS row) | **0.9109 ± 0.0032** (σ=0.6) | **+0.021** |
|
||||
| CICIDS2017 → CICDDoS2019 cross (`terminal_flow`) | 0.89 | **0.9197 ± 0.0036** | **+0.030** |
|
||||
Every processed dataset under `datasets/<name>/processed/` ships an aligned triple, all with the same row order (`flow_id = arange(N)`):
|
||||
|
||||
**4 of 4 reported tasks achieve SOTA**. Cross-dataset baseline was previously misread as 0.93; the IDS→DDoS direction in Shafir Table IX is 0.89.
|
||||
```
|
||||
packets.npz packet_tokens [N, T_full, 9], packet_lengths [N], flow_id [N]
|
||||
(or full_store/ — sharded PacketShardStore — for large datasets)
|
||||
flows.parquet flow_id + label + 5-tuple metadata (src_ip, dst_ip, ports, protocol)
|
||||
flow_features.parquet flow_id + label + 20 canonical packet-derived features
|
||||
```
|
||||
|
||||
Plus an architectural contribution: a `flow_consistency` diagnostic score
|
||||
that lifts from random (~0.6) to discriminative (~0.9) only when the model
|
||||
is trained with the masked-prediction consistency loss. On SSH-Patator (the
|
||||
hardest CICIDS2017 class for `terminal_norm` at 0.64) it reaches 0.94.
|
||||
The 9-d packet schema and 20-d flow schema are FIXED in `common/data_contract.py`. Flow features are computed by `compute_flow_features_from_packets(packet_tokens, lens)` so row alignment is guaranteed.
|
||||
|
||||
Authoritative result tables live in `RESULTS.md` (root) and
|
||||
`artifacts/locked_baselines.md` (Shafir baseline verification trail).
|
||||
Thresholded F1 / Precision / Recall / TPR@FPR under unsupervised threshold
|
||||
protocol: `RESULTS_THRESHOLDED.md`.
|
||||
Per-attack-family multi-seed analysis: `artifacts/phase25_multiseed_2026_04_25/PER_ATTACK_TABLE.md`.
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Train JANUS on CICIDS2017 (3 seeds available: 42, 43, 44)
|
||||
cd Mixed_CFM
|
||||
uv run --no-sync python train.py --config configs/cicids2017_ac_combo_seed42.yaml
|
||||
|
||||
# Phase-1 evaluation: per-attack-class AUROC + 10-d score export
|
||||
uv run --no-sync python eval_phase1.py \
|
||||
--model-dir <model_dir> --out-dir <eval_dir>
|
||||
|
||||
# Cross-dataset evaluation
|
||||
uv run --no-sync python eval_cross.py \
|
||||
--src-model <src_model_dir> \
|
||||
--tgt-data datasets/<tgt>/processed/ \
|
||||
--out <result.json>
|
||||
```
|
||||
|
||||
JANUS hyper-parameters (locked in `Mixed_CFM/configs/<dataset>_ac_combo_seed*.yaml`):
|
||||
|
||||
```yaml
|
||||
T: 64 # max packet sequence length
|
||||
d_model: 128
|
||||
n_layers: 4
|
||||
n_heads: 4
|
||||
sigma: 0.1 # within-dataset; cross uses 0.6
|
||||
lambda_disc: 1.0
|
||||
use_ot: true # OT-CFM (Sinkhorn coupling on benign batch)
|
||||
reference_mode: causal_packets # Route A: packet-causal attention
|
||||
```
|
||||
|
||||
## Producing the deployable scalar score
|
||||
|
||||
`eval_phase1.py` exports a 10-d per-flow score vector to `phase1_scores.npz`:
|
||||
|
||||
```
|
||||
3 continuous-side scores : terminal_norm, terminal_flow, terminal_packet
|
||||
7 discrete-side scores : disc_nll_total + disc_nll_ch{2,3,4,5,6,7}
|
||||
(direction + 5 TCP flags)
|
||||
```
|
||||
|
||||
The deployable scalar is the Mahalanobis-OAS distance:
|
||||
|
||||
```
|
||||
d²(s) = (s − μ)ᵀ Σ⁻¹ (s − μ), where (μ, Σ) come from sklearn.covariance.OAS
|
||||
fit on benign val ONLY (no attack labels).
|
||||
```
|
||||
|
||||
Reference implementation: `artifacts/route_comparison/aggregate_score_router.py` (artifacts/ is gitignored; the script lives on the dev box).
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
uv run --no-sync python -m pytest tests/ Mixed_CFM/tests/ Unified_CFM/tests/
|
||||
```
|
||||
|
||||
## Adding a new dataset
|
||||
|
||||
Write one driver at `scripts/extract_<name>.py` that calls `extract_lib.extract_dataset(...)` (see `scripts/extract_cicids2017.py` as the reference template). The driver hardcodes CSV column names, timestamp formats, benign aliases, and drop patterns as module constants, then feeds `extract_lib` a per-day `(canonical_key → [(row_idx, ts_epoch)])` mapping and a per-day pcap file map. The extract pipeline writes all three artifacts (packets.npz, flows.parquet, flow_features.parquet) row-aligned by `flow_id = arange(N)`.
|
||||
|
||||
To upgrade an existing artifact pair that lacks `flow_features.parquet`, run `scripts/generate_flow_features.py --packets-npz ... --flows-parquet ... --out ...` (or `--source-store` for sharded stores).
|
||||
|
||||
Common gotcha: if CSV timestamps and pcap epochs are in different time zones, `extract_lib` prints a diagnostic with the recommended `--time-offset`; rerun with that value.
|
||||
|
||||
## Authoritative documents
|
||||
|
||||
- `RESULTS.md` — full headline tables, ablations, per-attack analysis, JANUS configuration, thresholded operating-point metrics, what the experiments proved / disproved.
|
||||
- `Mixed_CFM/model.py` and `common/data_contract.py` — model + data-contract source of truth.
|
||||
|
||||
## Python environment
|
||||
|
||||
- `requires-python = ">=3.14"`; PyTorch pinned to the `pytorch-cu128` index, plus `mamba-ssm`, `causal-conv1d`, `scapy`, `dpkt`, `pyarrow`, `sklearn` (for the OAS aggregator).
|
||||
- Two `pyproject.toml` files exist: root and `Mixed_CFM/`; they are not declared as a uv workspace and resolve independently. Run `uv run ...` from whichever directory owns the entry point.
|
||||
- `Unified_CFM/` has no `pyproject.toml`; it uses the root venv (`uv run --no-sync python <script.py>`).
|
||||
- Scripts under `scripts/download/` are pure stdlib — invoke with `python3`.
|
||||
|
||||
Reference in New Issue
Block a user