Continuous Flow Matching head over the (size, IAT, win) packet channels.
Discrete Flow Matching head over the 6 binary protocol-flag / direction channels.

Trained jointly on benign traffic only (no attack labels at any stage). The deployable scalar score is a Mahalanobis-OAS distance over a 10-d per-flow score vector emitted by the trained model, with the aggregator fit on benign val only — entirely unsupervised end-to-end.

JANUS is the first NIDS method to use Flow Matching as the training paradigm in mixed continuous–discrete state spaces over packet sequences.

Headline results

3-seed mean ± std AUROC. Selection-bias-free Mahalanobis-OAS aggregator on the 10-d JANUS score vector, fit on benign val only.

Task	Shafir 2026 SOTA	JANUS	Δ
ISCXTor2016 (NonTor → Tor)	0.8731	0.9908 ± 0.0012	+0.118
CICIDS2017 within	0.9303	0.9845 ± 0.0030	+0.054
CICDDoS2019 within	0.93	0.9913 ± 0.0009	+0.061
CICIDS2017 → CICDDoS2019 cross	0.89	0.9594 ± 0.0046	+0.07
CICDDoS2019 → CICIDS2017 reverse cross	0.93	0.9301 ± 0.0122	matches

3/3 directly comparable within-dataset benchmarks beat external Shafir 2026 SOTA. CICIDS2017→CICDDoS2019 cross also beats; reverse direction matches. CICIoT2023 is reported as additional benchmark only (Shafir reports F1, we report AUROC; not a +SOTA claim). See RESULTS.md for caveats and the full headline table.

Layout

common/                    Data contract — single source of truth for the
                           9-d packet schema, 20-d packet-derived flow schema,
                           label normalization, and packet preprocessing.
Mixed_CFM/                 The JANUS model. Mixed continuous–discrete CFM
                           with two output heads on a shared causal Transformer.
                             configs/   Per-(dataset × seed) training configs.
                             model.py   MixedTokenCFM + MixedVelocity.
                             train.py / eval_phase1.py / eval_cross.py
Unified_CFM/               Legacy unified token CFM. Mixed_CFM imports its
                           AdaLNBlock + sinusoidal time embedding for backbone
                           reuse. Kept as internal ablation reference.
scripts/                   Workspace-level pcap → artifact pipeline,
                           CSV adapters, cross-package eval tooling.
  download/                UNB/CIC dataset downloaders.
  baselines/               Third-party baseline runners (Kitsune, Shafir-NF,
                           Anomaly-Transformer).
tests/                     Data-contract unit tests.

The following directories are gitignored (live on the dev box, not in the repo):

artifacts/                 All run outputs (checkpoints, eval JSONs, score npzs,
                           figures). Score-router aggregator at
                           artifacts/route_comparison/aggregate_score_router.py.
datasets/                  Raw + processed datasets (~1 TB).
baselines/                 Third-party baseline forks (Kitsune-py,
                           Anomaly-Transformer, ConMD, ganomaly, TIPSO-GAN, ...).
paper/                     Paper sources & external PDFs (Shafir 2026, Lipman
                           2210.02747, etc.).
.venv/                     uv-managed Python 3.14 virtual env.

Data contract

Every processed dataset under datasets/<name>/processed/ ships an aligned triple, all with the same row order (flow_id = arange(N)):

packets.npz            packet_tokens [N, T_full, 9], packet_lengths [N], flow_id [N]
                       (or full_store/ — sharded PacketShardStore — for large datasets)
flows.parquet          flow_id + label + 5-tuple metadata (src_ip, dst_ip, ports, protocol)
flow_features.parquet  flow_id + label + 20 canonical packet-derived features

The 9-d packet schema and 20-d flow schema are FIXED in common/data_contract.py. Flow features are computed by compute_flow_features_from_packets(packet_tokens, lens) so row alignment is guaranteed.

Quick start

# Train JANUS on CICIDS2017 (3 seeds available: 42, 43, 44)
cd Mixed_CFM
uv run --no-sync python train.py --config configs/cicids2017_seed42.yaml

# Phase-1 evaluation: per-attack-class AUROC + 10-d score export
uv run --no-sync python eval_phase1.py \
    --model-dir <model_dir> --out-dir <eval_dir>

# Cross-dataset evaluation
uv run --no-sync python eval_cross.py \
    --src-model <src_model_dir> \
    --tgt-data datasets/<tgt>/processed/ \
    --out <result.json>

JANUS hyper-parameters (locked in Mixed_CFM/configs/<dataset>_seed*.yaml):

T: 64                  # max packet sequence length
d_model: 128
n_layers: 4
n_heads: 4
sigma: 0.1             # within-dataset; cross uses 0.6
lambda_disc: 1.0
use_ot: true           # OT-CFM (Sinkhorn coupling on benign batch)
reference_mode: causal_packets    # Route A: packet-causal attention

Producing the deployable scalar score

eval_phase1.py exports a 10-d per-flow score vector to phase1_scores.npz:

3 continuous-side scores  : terminal_norm, terminal_flow, terminal_packet
7 discrete-side scores    : disc_nll_total + disc_nll_ch{2,3,4,5,6,7}
                            (direction + 5 TCP flags)

The deployable scalar is the Mahalanobis-OAS distance:

d²(s) = (s − μ)ᵀ Σ⁻¹ (s − μ),    where (μ, Σ) come from sklearn.covariance.OAS
                                 fit on benign val ONLY (no attack labels).

Reference implementation: artifacts/route_comparison/aggregate_score_router.py (artifacts/ is gitignored; the script lives on the dev box).

Tests

uv run --no-sync python -m pytest tests/ Mixed_CFM/tests/ Unified_CFM/tests/

Adding a new dataset

Write one driver at scripts/extract_<name>.py that calls extract_lib.extract_dataset(...) (see scripts/extract_cicids2017.py as the reference template). The driver hardcodes CSV column names, timestamp formats, benign aliases, and drop patterns as module constants, then feeds extract_lib a per-day (canonical_key → [(row_idx, ts_epoch)]) mapping and a per-day pcap file map. The extract pipeline writes all three artifacts (packets.npz, flows.parquet, flow_features.parquet) row-aligned by flow_id = arange(N).

To upgrade an existing artifact pair that lacks flow_features.parquet, run scripts/generate_flow_features.py --packets-npz ... --flows-parquet ... --out ... (or --source-store for sharded stores).

Common gotcha: if CSV timestamps and pcap epochs are in different time zones, extract_lib prints a diagnostic with the recommended --time-offset; rerun with that value.

Authoritative documents

RESULTS.md — full headline tables, ablations, per-attack analysis, JANUS configuration, thresholded operating-point metrics, what the experiments proved / disproved.
Mixed_CFM/model.py and common/data_contract.py — model + data-contract source of truth.

Python environment

requires-python = ">=3.14"; PyTorch pinned to the pytorch-cu128 index, plus mamba-ssm, causal-conv1d, scapy, dpkt, pyarrow, sklearn (for the OAS aggregator).
Two pyproject.toml files exist: root and Mixed_CFM/; they are not declared as a uv workspace and resolve independently. Run uv run ... from whichever directory owns the entry point.
Unified_CFM/ has no pyproject.toml; it uses the root venv (uv run --no-sync python <script.py>).
Scripts under scripts/download/ are pure stdlib — invoke with python3.

README.md Unescape Escape

JANUS