Initial commit: code, paper, small artifacts
This commit is contained in:
BIN
paper/2026-f3241-paper.pdf
Normal file
BIN
paper/2026-f3241-paper.pdf
Normal file
Binary file not shown.
BIN
paper/2210.02747v2.pdf
Normal file
BIN
paper/2210.02747v2.pdf
Normal file
Binary file not shown.
Binary file not shown.
Binary file not shown.
210
paper/SURVEY.md
Normal file
210
paper/SURVEY.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# JANUS Paper — Survey, Pain Points, and Outline
|
||||
|
||||
**Date**: 2026-05-04
|
||||
**Scope**: Field survey + framing for the JANUS paper (Mixed-CFM + DFM + causal-packet attention + Mahalanobis-OAS aggregator).
|
||||
**Target sections**: Introduction · Background · Methodology · Evaluation.
|
||||
|
||||
> Framing rule (from `RESULTS.md` caveats): the headline claim is **cross-dataset robustness + first FM/DFM in NIDS**, not "4/4 within SOTA". Within-dataset is saturated; the discriminating axis is cross-dataset.
|
||||
|
||||
---
|
||||
|
||||
## Part A. State of the field (2024–2026)
|
||||
|
||||
### A.1 Method families and where each one is stuck
|
||||
|
||||
| Family | Recent representative work | Core mechanism | Documented short-coming |
|
||||
|---|---|---|---|
|
||||
| **Normalizing Flows (NF)** | **Shafir et al. T-Netw 2026** (our main baseline), NF-NIDS (IAF/NSF), PrivFlow-NIDS (Springer 2025) | Explicit log-likelihood on benign; anomaly = low likelihood | Likelihood ≠ anomaly score; coupling-layer architecture brittle; categorical / flag fields handled crudely |
|
||||
| **Reconstruction (AE / VAE / MemAE)** | KitNET, MemAE, SparseMemAE | Reconstruction error as anomaly score | "Identity mapping trap" — OOD samples can be perfectly reconstructed (NeurIPS '24, OpenReview '25 multi-paper consensus) |
|
||||
| **Diffusion** | **ConMD (TIFS 2026)** (our main baseline), DMAD (IJCAI '25 survey), UnDiff (WWW '25), RDUAE | Denoising / score-based density | Slow inference; multi-step trace storage; training instability |
|
||||
| **GAN** | **TIPSO-GAN (NDSS 2026)** (our main baseline), DEGAN | Discriminator score / reconstruction | Mode collapse and training instability persist; TIPSO-GAN spends extra optimisation budget (PSO) to mitigate |
|
||||
| **Self-supervised contrastive** | Self-Supervised Transformer Contrastive Learning (NetSci '25), GraphIDS (NeurIPS 2025), SSGMHAN | Representation learning, downstream OCSVM / Mahal | Two-stage pipeline (rep + detector); no end-to-end anomaly score |
|
||||
| **Foundation models** | Traffic-MoE, ETC-IMC, Language-of-Network GBC | Pre-train / fine-tune over packet-byte tokens | Resource-heavy; primary task is encrypted-traffic classification, not AD |
|
||||
| **Knowledge distillation** | ConMD (TIFS 2026), Spatial-Temporal KD | Teacher → student alignment | Sensitive to teacher quality; two-stage |
|
||||
| **Flow Matching (FM)** ✨ | **TCCM (NeurIPS 2025) tabular**, rFM (image AD 2025), Lipman 2023 | Velocity-field regression, one-step deviation as score | **No application to NIDS yet — the gap JANUS fills** |
|
||||
| **Discrete FM (DFM)** ✨ | Gat et al. NeurIPS 2024, Fisher Flow Matching, FlowMol (molecules 2024–2025) | FM over discrete state space | **No application to NIDS yet** |
|
||||
|
||||
### A.2 Dataset / benchmark situation
|
||||
|
||||
- Within-dataset is saturated. Shafir 0.93 → JANUS 0.99 → TIPSO-GAN F1 = 0.99 on CICIDS2017 / CICDDoS2019 are all within seed noise of each other.
|
||||
- CICIDS2017 has documented benchmark bias (synthetic vs real-world traffic distribution mismatch — arXiv 2403.17458 "Expectations Versus Reality"; multiple 2024–2025 critiques).
|
||||
- Cross-dataset is now the field's chosen discriminating axis. HDSE-IDS, Transformer-IDS w/ calibration, Few-shot multi-domain fusion (PLOS One 2025), and the cross-dataset generalisation review (arXiv 2402.10974) all centre on the same problem.
|
||||
- AUROC alone is increasingly seen as inadequate for operational reporting; thresholded F1, Precision, Recall, and TPR @ FPR = 1% are demanded by the SOC community.
|
||||
|
||||
### A.3 Operational pain points (the "why this matters" stack)
|
||||
|
||||
| Pain point | Quantified evidence | Source |
|
||||
|---|---|---|
|
||||
| Operational FPR | ~99 % of NIDS alerts are FP; one OT refinery reported 27 000 alerts → 76 real; 51 % of SOC teams describe alert volume as unmanageable | Trend Micro 2024, OT IDS surveys 2024–2025 |
|
||||
| Cross-domain deployment collapse | Single-dataset trained IDSes typically drop 0.10–0.30 AUROC across environments | Tandfonline 2025; MDPI 2025 / 8466 |
|
||||
| Concept drift | Models become outdated post-deployment, requiring re-training | MDPI Future Internet 2025 / 328 |
|
||||
| IID-flow assumption mismatch | Multi-stage attacks broken into IID flows lose relational and temporal structure | DevSecOps NIDS guide 2026 |
|
||||
| Encrypted + heterogeneous protocols (IoT) | Packet-level features lose access to plaintext; protocol/device heterogeneity breaks unified models | Wiley 2025 IoT NIDS review |
|
||||
|
||||
---
|
||||
|
||||
## Part B. Pain points × JANUS capability mapping
|
||||
|
||||
> **P2 is our most distinctive observation** — the community talks about cross-domain failure, but no one has clearly characterised "density score implicitly learns source-likeness". This is the mechanism-level explanation that earns naming rights and serves as the introduction's hook.
|
||||
|
||||
| # | Pain point (community consensus) | Failure mode of current SOTA | JANUS mechanism | Evidence in our artifacts |
|
||||
|---|---|---|---|---|
|
||||
| **P1** | Cross-dataset / domain-shift collapse | Shafir NF: cross 0.89 (forward) / 0.93 (reverse, single-direction); legacy `terminal_norm` 0.62 (reverse) | DFM head emits `disc_nll` (protocol-flag distribution is transfer-stable); Mahalanobis-OAS re-weights on target benign | `RESULTS.md` §C: reverse 0.62 → 0.93 (+0.31); forward 0.89 → 0.96 (+0.07); 12 off-diagonal cells avg +0.175 |
|
||||
| **P2** | "Source-likeness collapse" of density scores (new framing) | terminal density score across **3** distinct backbones in reverse cross all ≤ 0.63 — effectively a source-domain classifier | DFM decouples protocol semantics; Mahalanobis fuses multiple complementary scores | 3-backbone × 16-score validation: terminal_norm 0.519–0.626 all collapse; `disc_nll` is only single-score that stays stable (0.903) |
|
||||
| **P3** | Continuous + discrete protocol fields squashed into one likelihood | NF / AE Gaussianise TCP flags / direction, losing semantics | **Mixed CFM**: continuous head over (size, IAT, win) + **DFM head** over 6 binary flag/direction channels, jointly trained | `sigma=0.1`, `λ_disc=1.0`; DFM head is the only transfer-stable single score on reverse cross (0.9191) |
|
||||
| **P4** | Reconstruction-based AD has identity-mapping trap | AE perfectly reconstructs OOD (NeurIPS '24, OpenReview '25) | FM is not reconstruction — velocity field, terminal point is source noise rather than the input | First FM application to packet-sequence NIDS |
|
||||
| **P5** | Multi-score selection bias (post-hoc best-fixed channel) | Shafir 5-feature ensemble is post-hoc; our best-fixed AUROC 0.99 also selection-biased | Mahalanobis-OAS fits once on benign val; **never sees attack labels** | `RESULTS.md` §A: OAS yields +0.054 to +0.118 over Shafir on 4 within-dataset benchmarks with one deployable scalar score |
|
||||
| **P6** | High operational FPR | 99 %-FP industrial reality | Thresholded F1 / Precision / Recall (τ = benign-val P95 / P99) | `RESULTS_THRESHOLDED.md`: CICDDoS2019 within τ=P95 F1 = 0.993; cross F1 = 0.632 (precision ≈ 0.95) |
|
||||
| **P7** | Run-to-run variance / poor reproducibility | Multi-seed std large in many baselines | Causal-packet attention shrinks std 1.6–8× | `RESULTS.md` "Stability": CICIoT2023 std 0.0017 → 0.0002 (8×) |
|
||||
| **P8** | NF likelihood is not necessarily anomaly | Discussed since NFAD 2021; Shafir bypasses with Shapley feature ensemble | We provide a 10-d score family + Mahalanobis routing; not dependent on a single likelihood | `CROSS_MATRIX.md` 12 cells across 4×4 matrix |
|
||||
|
||||
---
|
||||
|
||||
## Part C. Paper outline
|
||||
|
||||
### §1 Introduction (~1.5 pages)
|
||||
|
||||
Argument chain by paragraph:
|
||||
|
||||
1. **Hook**: 99 % FPR + cross-domain deployment collapse. One sentence with the alert-fatigue numbers makes the relevance immediate.
|
||||
2. **Common failure mode of current methods**: AE has identity trap; NF likelihood becomes a source-likeness classifier under cross-domain shift (this is P2; insert a teaser figure of `terminal_norm` 4×4 cross matrix where many off-diagonal cells are ≤ 0.55).
|
||||
3. **Missing tool**: FM / DFM are validated for image / molecule / tabular AD, but never applied to NIDS; protocol fields are intrinsically mixed continuous + discrete, exactly the setting mixed FM was designed for.
|
||||
4. **JANUS in one sentence**: jointly trained continuous CFM + discrete FM with causal-packet attention, aggregated by a benign-only Mahalanobis-OAS scalar.
|
||||
5. **Contributions** (3–4 bullets):
|
||||
- **(C1)** First Flow-Matching paradigm for packet-level NIDS; first DFM modelling of protocol flag / direction channels.
|
||||
- **(C2)** We characterise the *source-likeness collapse* of terminal density scores at architecture level (3 backbones × 16 scores), and show how DFM + Mahalanobis routing breaks it.
|
||||
- **(C3)** Mahalanobis-OAS as a benign-only single-scalar aggregator; the unsupervised contract is preserved (no attack labels at any step).
|
||||
- **(C4)** On a 4×4 cross-dataset matrix, JANUS averages **+0.175** AUROC over `terminal_norm`; reverse direction +0.31. Within-dataset matches or exceeds the NF SOTA (Shafir 2026) by 0.054–0.118 on 3/3 directly comparable benchmarks.
|
||||
6. **Outline + one-sentence takeaway**.
|
||||
|
||||
**Figure 1 placeholder** (end of §1): cross 4×4 heatmap, left `terminal_norm`, right `JANUS + Mahalanobis-OAS`, Δ in colour. The visceral hook.
|
||||
|
||||
---
|
||||
|
||||
### §2 Background & Related Work (~1.5 pages)
|
||||
|
||||
Four 1-paragraph subsections.
|
||||
|
||||
**§2.1 Unsupervised Network Anomaly Detection**
|
||||
- Reconstruction (AE / MemAE / Kitsune) — cite identity-mapping critiques as baseline limitation.
|
||||
- Density estimation with NF — Shafir 2026 + NF-NIDS as SOTA, but likelihood ≠ anomaly and categorical fields suffer.
|
||||
- GAN-based — TIPSO-GAN (mode collapse, optimisation cost).
|
||||
- Diffusion — DMAD survey + ConMD; high inference cost; AD remains image-centric.
|
||||
- Self-supervised contrastive — primarily representation learning, not direct anomaly scoring.
|
||||
|
||||
**§2.2 Flow Matching & Discrete Flow Matching**
|
||||
- Lipman 2023, OT-CFM (Tong et al. TMLR '24).
|
||||
- Discrete Flow Matching (Gat et al. NeurIPS '24).
|
||||
- FM for AD: TCCM NeurIPS '25 (tabular), rFM 2025 (image) — **highlight that NIDS remains untouched**.
|
||||
- Mixed continuous + discrete FM (FlowMol 2024–2025) — sets up the naming for our Mixed_CFM.
|
||||
|
||||
**§2.3 Cross-Dataset Generalisation in NIDS**
|
||||
- HDSE-IDS, Transformer-IDS w/ calibration, Few-shot multi-domain fusion (2025).
|
||||
- Cite arXiv 2402.10974 / 2403.17458 (Cross-Dataset Generalisation, "Expectations Versus Reality").
|
||||
- **Our framing**: prior work treats cross as a representation problem; nobody characterises the score-level source-likeness collapse.
|
||||
|
||||
**§2.4 Anomaly Score Aggregation**
|
||||
- Mahalanobis-based AD (MICCAI '24 brain MRI; M-SVDD 2025).
|
||||
- OAS / Ledoit-Wolf shrinkage covariance.
|
||||
- **Position**: Mahalanobis is widespread in image AD; never systematically applied as a benign-only aggregator over an FM score vector in NIDS.
|
||||
|
||||
---
|
||||
|
||||
### §3 Methodology (~3 pages)
|
||||
|
||||
**§3.1 Problem formulation**
|
||||
- Input: packet sequences `[N, T, 9]` + flow metadata; benign-only training; unsupervised inference returns one scalar.
|
||||
- 9-d packet schema (`common/data_contract.py`): 3 continuous (size, IAT, win) + 6 discrete (direction + 5 TCP flags).
|
||||
- The mixed-modality nature is *intrinsic* to the protocol — not a modelling choice.
|
||||
|
||||
**§3.2 The JANUS architecture**
|
||||
- (a) **Backbone**: causal-packet Transformer over `[FLOW_TOKEN, P_1, …, P_T]`. Figure 2 = model overview.
|
||||
- (b) **Continuous head (CFM)**: OT-CFM, σ=0.1, on the 3 continuous channels.
|
||||
- (c) **Discrete head (DFM)**: Discrete Flow Matching with linear interpolation probability path on the 6 binary channels; cross-entropy loss with `λ_disc = 1.0`.
|
||||
- (d) **Why mixed FM**: small ablation showing `λ_disc = 0` vs `λ_disc = 1.0`, demonstrating that flag fields cannot be Gaussianised.
|
||||
|
||||
**§3.3 Score family**
|
||||
- Enumerate the 10-d score vector: `terminal_norm`, `terminal_packet`, `terminal_flow`, `kinetic_*`, `disc_nll_total`, …
|
||||
- Each captures a different physical quantity (density / kinetic / discrete-flag distribution).
|
||||
- **Source-likeness collapse — formal observation**: under target-domain benign drift, `terminal_norm` degrades into a `1{x ∈ source distribution}` proxy and loses its anomaly signal. Evidence: 4×4 matrix and 3-backbone validation.
|
||||
|
||||
**§3.4 Mahalanobis-OAS aggregator**
|
||||
- Fit OAS-shrunk Mahalanobis on **target** benign val: `score = d²(s(x), µ_benign)`.
|
||||
- The aggregator never sees attack labels.
|
||||
- Selection: 5 benign-only aggregators evaluated (max-z, plain Mahalanobis, Ledoit-Wolf, OAS, score-subset variants); OAS performs best with sensitivity ≤ 0.005 vs Ledoit-Wolf (`SCORE_ROUTER.md`).
|
||||
|
||||
**§3.5 Causal-packet attention as a stabiliser**
|
||||
- Define the protocol-causal mask. Show std reduction 1.6–8× across 4 datasets (`RESULTS.md` "Stability").
|
||||
|
||||
---
|
||||
|
||||
### §4 Evaluation (~3–4 pages)
|
||||
|
||||
**§4.1 Datasets, baselines, protocol**
|
||||
- 4 datasets: ISCXTor2016, CICIDS2017, CICDDoS2019, CICIoT2023; canonical `packets.npz` 9-d schema.
|
||||
- Baselines (locked from PDFs in `paper/`): Shafir NF (T-Netw 2026), ConMD (TIFS 2026), TIPSO-GAN (NDSS 2026), Kitsune, AE / MemAE, OCSVM.
|
||||
- Protocol: 10K benign train (matches Shafir), 3 seeds, AUROC primary + thresholded F1 / Precision / Recall @ P95 / P99.
|
||||
|
||||
**§4.2 Within-dataset (Table 1)**
|
||||
- 4 datasets × {Shafir / ConMD / TIPSO-GAN / Kitsune / AE / **JANUS + Mahal-OAS** / JANUS best-fixed}.
|
||||
- Honest framing: "JANUS matches or exceeds the NF SOTA on 3/3 directly comparable benchmarks; CICIoT2023 reported as additional benchmark due to metric mismatch (Caveat 1, `RESULTS.md`)".
|
||||
- One sentence acknowledging that "within-dataset is saturated; the discriminating axis is cross-dataset (next section)".
|
||||
|
||||
**§4.3 Cross-dataset (Table 2 + Figure 3)**
|
||||
- The headline of the paper. Table = 4×4 matrix Mahal vs `terminal_norm`.
|
||||
- Figure 3 = detailed version of Figure 1.
|
||||
- Critical details:
|
||||
- Forward IDS17 → DDoS19: +0.07 over Shafir (genuine SOTA).
|
||||
- Reverse DDoS19 → IDS17: 0.93 = Shafir 0.93 (matches, does not exceed — Caveat 2).
|
||||
- 12 off-diagonal cells average +0.175 over `terminal_norm`.
|
||||
- 4 "collapse cells" (≤ 0.57) all recovered to ≥ 0.75.
|
||||
|
||||
**§4.4 Mechanism analysis (Table 3 + Figure 4)**
|
||||
- Source-likeness collapse: 3 backbones × 16 scores matrix.
|
||||
- DFM head ablation: `λ_disc ∈ {0, 0.5, 1.0, 2.0}` vs reverse-cross AUROC.
|
||||
- Mahalanobis aggregator ablation: max-z / plain Mahal / Ledoit-Wolf / OAS — sourced from `SCORE_ROUTER.md`.
|
||||
|
||||
**§4.5 Ablations & robustness**
|
||||
- σ sensitivity (`sigma_validation.md` 4×2 table).
|
||||
- Causal-packet attention contribution to std reduction (`RESULTS.md` Stability).
|
||||
- Per-attack-family table (`RESULTS.md` "Per-attack-family pattern" — SSH-Patator counter-example).
|
||||
|
||||
**§4.6 Thresholded metrics & operational impact**
|
||||
- `RESULTS_THRESHOLDED.md` F1 / Precision / Recall @ P95.
|
||||
- Direct dialogue with industry alert-fatigue numbers: "at the P95 threshold, our cross precision ≈ 0.95".
|
||||
|
||||
**§4.7 Discussion (sub-section, 1 paragraph)**
|
||||
- Limitations: aggregator post-hoc selection, target-benign-calibrated transfer (not zero-shot — Caveat 3), CICIoT2023 metric mismatch.
|
||||
- Honest reporting here closes the door on reviewer attacks.
|
||||
|
||||
---
|
||||
|
||||
## Part D. Writing red lines (from project memory)
|
||||
|
||||
1. **Never** write "zero-shot transfer" — write "calibrated cross-domain transfer" (Mahalanobis is fit on target benign).
|
||||
2. **Never** claim "+SOTA on CICIoT2023" — write "additional benchmark; metric mismatch (Shafir F1 vs our AUROC)".
|
||||
3. Reverse cross is "matches Shafir 0.93", not "beats". Our +0.31 is vs our own legacy.
|
||||
4. Best-fixed numbers are an ablation upper bound, never the SOTA claim.
|
||||
5. Mahalanobis-OAS was post-hoc-selected — write "we evaluated 5 benign-only aggregators; OAS performed best with sensitivity ≤ 0.005 vs Ledoit-Wolf".
|
||||
|
||||
---
|
||||
|
||||
## Part E. Sources
|
||||
|
||||
- Shafir, Giryes, Wool — *Explainable Anomaly Detection in Network Traffic Using Normalizing Flows*, IEEE T-Netw 2026 (PDF in `paper/`).
|
||||
- Lian et al. — *Contextual Masking Distillation for Network Traffic Anomaly Detection*, IEEE TIFS 2026.
|
||||
- *TIPSO-GAN: Malicious Network Traffic Detection*, NDSS 2026.
|
||||
- Gat et al. — *Discrete Flow Matching*, NeurIPS 2024.
|
||||
- *Scalable, Explainable and Provably Robust Anomaly Detection with One-Step Flow Matching*, NeurIPS 2025 (arXiv 2510.18328).
|
||||
- *How and Why: Taming Flow Matching for Unsupervised Anomaly Detection* (rFM), arXiv 2508.05461.
|
||||
- *On the Cross-Dataset Generalization of Machine Learning for Network Intrusion Detection*, arXiv 2402.10974.
|
||||
- *Expectations Versus Reality: Evaluating Intrusion Detection Systems in Practice*, arXiv 2403.17458.
|
||||
- HDSE-IDS — *Heterogeneous Deep Stacked Ensemble for Cross-Domain IDS*, Connection Science 2025.
|
||||
- *Self-Supervised Transformer-based Contrastive Learning for IDS*, arXiv 2505.08816.
|
||||
- *GraphIDS: Self-supervised GNN for Network Intrusion Detection*, NeurIPS 2025.
|
||||
- *Network traffic foundation models: A systematic review*, ScienceDirect 2026.
|
||||
- Tong et al. — *Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport*, TMLR 2024.
|
||||
- *DMAD: Diffusion Models for Anomaly Detection (survey)*, IJCAI 2025.
|
||||
- *Alert Fatigue in Security Operations Centres: Research Challenges and Opportunities*, ACM Computing Surveys 2024.
|
||||
- *Beyond the Norm: Unsupervised Anomaly Detection in Telecommunications with Mahalanobis Distance*, MDPI Computers 2025.
|
||||
- *Autoencoders for Anomaly Detection are Unreliable*, OpenReview 2025.
|
||||
125
paper/background_related.md
Normal file
125
paper/background_related.md
Normal file
@@ -0,0 +1,125 @@
|
||||
## 2 Background
|
||||
|
||||
### 2.1 Unsupervised network anomaly detection
|
||||
|
||||
We consider the standard unsupervised setting: a detector is trained only on
|
||||
benign traffic and, at inference time, must assign an anomaly score to each
|
||||
flow without access to attack labels at any stage of training. Public
|
||||
benchmarks (e.g., CIC-IDS2017, CIC-DDoS2019, ISCXTor2016) provide labelled
|
||||
attack traffic for evaluation only. Two granularities dominate the
|
||||
literature: flow-level detectors operate on per-flow aggregate features
|
||||
(byte counts, inter-arrival statistics, flag tallies), while packet-level
|
||||
detectors operate on the ordered sequence of per-packet features inside a
|
||||
flow and retain temporal structure that flow aggregates discard.
|
||||
|
||||
Within-dataset AUROC on the standard benchmarks has narrowed to within
|
||||
seed noise across recent recipes; the substantive evaluation axis is now
|
||||
cross-dataset transfer, in which a detector is trained on one environment
|
||||
and evaluated on traffic from another. Performance on this axis has not
|
||||
converged.
|
||||
|
||||
### 2.2 Continuous Flow Matching
|
||||
|
||||
Continuous Flow Matching (CFM) trains a time-dependent vector field
|
||||
$v_\theta(x, t)$ to transport a tractable source distribution (typically
|
||||
$\mathcal{N}(0, I)$) to the data distribution along an ODE
|
||||
$\mathrm{d}x_t = v_\theta(x_t, t)\,\mathrm{d}t$. The training objective
|
||||
regresses $v_\theta$ onto a target velocity defined along a chosen
|
||||
conditional probability path; for the linear (Gaussian) path this reduces
|
||||
to a simple least-squares loss, side-stepping the score-matching objective
|
||||
and stochastic sampler of diffusion models. OT-CFM straightens
|
||||
trajectories by pairing source and data samples through minibatch optimal
|
||||
transport, which lowers integration error and enables stable few-step
|
||||
inference.
|
||||
|
||||
A trained CFM model gives access not only to the learned density but to
|
||||
a family of geometric quantities along the trajectory: terminal velocity
|
||||
norm, divergence, curvature, and Jacobian-trace estimators. These can be
|
||||
read off the velocity field without retraining.
|
||||
|
||||
### 2.3 Discrete Flow Matching
|
||||
|
||||
Continuous FM does not apply to categorical state spaces, where adding
|
||||
Gaussian noise is undefined. Discrete Flow Matching (DFM) generalises
|
||||
the framework to finite alphabets through continuous-time Markov chains:
|
||||
the model parameterises token-level transition rates that interpolate
|
||||
between a source distribution (typically uniform) and the data
|
||||
distribution. The training objective remains a simple regression onto
|
||||
target rates derived from a chosen interpolation schedule. DFM has been
|
||||
validated on language and molecular generation; mixed
|
||||
continuous–discrete data, where each observation has both numerical and
|
||||
categorical channels, is the natural composition of CFM and DFM.
|
||||
|
||||
---
|
||||
|
||||
## 3 Related Work
|
||||
|
||||
### 3.1 Reconstruction-based detectors
|
||||
|
||||
Autoencoder-style detectors learn to reconstruct benign inputs and score
|
||||
anomalies by reconstruction error. Kitsune popularised the design for
|
||||
online NIDS using an ensemble of small autoencoders, and MemAE introduced
|
||||
a learned memory bank to constrain the latent representation to the
|
||||
benign manifold. The family suffers from a documented identity-mapping
|
||||
failure: sufficiently expressive autoencoders reconstruct out-of-
|
||||
distribution inputs near-perfectly, eroding the gap between benign and
|
||||
anomalous reconstruction error. Recent critiques argue that this
|
||||
behaviour is structural rather than a hyperparameter artefact, and that
|
||||
reconstruction error is therefore an unreliable anomaly score in
|
||||
general.
|
||||
|
||||
### 3.2 Density-based detectors
|
||||
|
||||
Three deep generative families currently hold the public SOTA on
|
||||
NIDS benchmarks. **Normalising flows** fit an explicit invertible
|
||||
density on benign traffic and score by negative log-likelihood; the
|
||||
strongest recent pipeline reports 0.93 within-dataset AUROC on
|
||||
CIC-DDoS2019 with cross-domain transfer in the 0.89–0.93 range.
|
||||
**Diffusion-based detectors** include contextual masking distillation
|
||||
schemes that compare a student denoiser against a benign-trained
|
||||
teacher, alongside a broader 2025 survey of diffusion AD variants.
|
||||
**GAN-based detectors**, exemplified by recent NDSS work that augments
|
||||
the optimisation with particle-swarm search, score by discriminator
|
||||
output or cycle-reconstruction error. All three families reduce a
|
||||
packet stream to a single scalar derived from one homogeneous
|
||||
probabilistic model fit to benign data, and the reported log-likelihood
|
||||
is known to dissociate from anomaly status once the benign distribution
|
||||
drifts.
|
||||
|
||||
A separate line of work uses self-supervised contrastive
|
||||
representations, graph neural networks, or pre-trained traffic
|
||||
foundation models, with anomaly scoring delegated to a downstream
|
||||
detector such as OCSVM or Mahalanobis distance. These pipelines are
|
||||
typically two-stage, are primarily evaluated on encrypted-traffic
|
||||
classification rather than open-set anomaly detection, and are not the
|
||||
focus of the cross-dataset robustness comparison we pursue.
|
||||
|
||||
### 3.3 Flow Matching for anomaly detection
|
||||
|
||||
Outside NIDS, two recent works adopt Flow Matching as the AD objective.
|
||||
A time-reversed FM detector for image anomaly detection couples
|
||||
worst-transport coupling with a high-dimensional latent, scoring by
|
||||
deviation from the learned velocity field. A tabular detector built on
|
||||
one-step FM offers explainability and provable robustness guarantees on
|
||||
heterogeneous structured data. Both validate FM-based scoring as
|
||||
competitive with reconstruction- and density-based baselines in their
|
||||
respective regimes. Discrete Flow Matching has been validated on
|
||||
language and molecular generation but not, to our knowledge, evaluated
|
||||
as an anomaly-detection objective. No prior work applies either
|
||||
continuous or discrete FM to packet-sequence NIDS.
|
||||
|
||||
### 3.4 Cross-dataset robustness in NIDS
|
||||
|
||||
As within-dataset metrics have saturated, cross-dataset evaluation has
|
||||
emerged as the field's discriminating axis. A 2024 systematic study
|
||||
measures the generalisation gap across the standard NIDS benchmarks
|
||||
under matched feature schemas and reports AUROC drops of 0.10–0.30
|
||||
when detectors trained on one environment are evaluated on another.
|
||||
Subsequent work on heterogeneous deep stacked ensembles, calibrated
|
||||
transformers, and few-shot multi-domain fusion targets the same gap
|
||||
through architectural or training-time interventions. The phenomenon is
|
||||
broadly observed and quantified; what is missing from the literature is
|
||||
a mechanism-level account of why density-based scores in particular
|
||||
degrade under domain shift, as opposed to an accumulation of empirical
|
||||
remedies. The pilot study in §X revisits this gap directly and frames
|
||||
the structural failure mode that the rest of the paper addresses.
|
||||
BIN
paper/figs/fig_data_efficiency.pdf
Normal file
BIN
paper/figs/fig_data_efficiency.pdf
Normal file
Binary file not shown.
BIN
paper/figures/figure1.pdf
Normal file
BIN
paper/figures/figure1.pdf
Normal file
Binary file not shown.
16
paper/figures/figure1_overview_v2.log
Normal file
16
paper/figures/figure1_overview_v2.log
Normal file
@@ -0,0 +1,16 @@
|
||||
This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2022/dev/Debian) (preloaded format=pdfetex 2026.1.31) 2 MAY 2026 16:36
|
||||
entering extended mode
|
||||
restricted \write18 enabled.
|
||||
%&-line parsing enabled.
|
||||
**&pdfetex figure1_overview_v2.tex
|
||||
(./figure1_overview_v2.tex
|
||||
! Undefined control sequence.
|
||||
l.24 \documentclass
|
||||
[border=6pt]{standalone}
|
||||
?
|
||||
! Emergency stop.
|
||||
l.24 \documentclass
|
||||
[border=6pt]{standalone}
|
||||
End of file on the terminal!
|
||||
|
||||
! ==> Fatal error occurred, no output PDF file produced!
|
||||
736
paper/figures/figure1_overview_v2.tex
Normal file
736
paper/figures/figure1_overview_v2.tex
Normal file
@@ -0,0 +1,736 @@
|
||||
% Figure 1 (v2): Mixed_CFM — system overview.
|
||||
% Compile: pdflatex figure1_overview_v2.tex
|
||||
%
|
||||
% Layout:
|
||||
% row 1: (a) Tokenization ──x_1──→ (b) Mixed-state path ★
|
||||
% │
|
||||
% x_t
|
||||
% ▼
|
||||
% row 2: (c) Causal-Packet Velocity Field + joint loss ★
|
||||
% │
|
||||
% frozen
|
||||
% ▼
|
||||
% row 3: (d) Inference: score → Mahalanobis-OAS router ★
|
||||
%
|
||||
% Coordinate hygiene:
|
||||
% - Each panel is its own sub-tikzpicture with a fixed bounding box
|
||||
% (\useasboundingbox 0..W x 0..H, in mm). Editing one panel cannot
|
||||
% disturb another.
|
||||
% - The OUTER tikzpicture only chains panels with `right=of` / `below=of`
|
||||
% and draws three inter-panel data-flow arrows.
|
||||
% - Numbers in the figure are architecture descriptors only (T, 21-d,
|
||||
% 10-d, lambda). No experimental result values.
|
||||
|
||||
\documentclass[border=6pt]{standalone}
|
||||
\usepackage{tikz}
|
||||
\usetikzlibrary{arrows.meta, positioning, calc}
|
||||
\usepackage{amsmath, amssymb}
|
||||
|
||||
% --- palette -----------------------------------------------------------------
|
||||
\definecolor{cOrange}{RGB}{230, 126, 34} % discrete (DFM)
|
||||
\definecolor{cBlue} {RGB}{ 52, 110, 180} % continuous (CFM)
|
||||
\definecolor{cPurple}{RGB}{142, 68, 173}
|
||||
\definecolor{cGray} {RGB}{170, 170, 170}
|
||||
\definecolor{cBgPanel}{RGB}{252, 252, 250}
|
||||
\definecolor{cBg} {RGB}{248, 248, 248}
|
||||
|
||||
\newcommand{\contrib}{\textcolor{cOrange}{\ensuremath{\bigstar}}}
|
||||
|
||||
\tikzset{
|
||||
panel/.style ={rectangle, draw=black!35, line width=0.5pt, rounded corners=2pt,
|
||||
fill=cBgPanel, inner sep=0pt},
|
||||
novelpanel/.style ={rectangle, draw=cOrange, line width=1.0pt, rounded corners=2pt,
|
||||
fill=cOrange!4, inner sep=0pt},
|
||||
paneltag/.style ={font=\sffamily\bfseries\footnotesize, anchor=north west,
|
||||
text=black!75, inner sep=0pt},
|
||||
paneltagN/.style ={font=\sffamily\bfseries\footnotesize, anchor=north west,
|
||||
text=cOrange!85!black, inner sep=0pt},
|
||||
archbox/.style ={rectangle, draw=black!70, thick, rounded corners=2pt,
|
||||
align=center, font=\scriptsize, fill=white,
|
||||
minimum height=8mm},
|
||||
novelbox/.style ={rectangle, draw=cOrange, line width=1.2pt, rounded corners=2pt,
|
||||
align=center, font=\scriptsize, fill=cOrange!10,
|
||||
minimum height=8mm},
|
||||
arrow/.style ={->, thick, black!65, >={Stealth[length=2.4mm]}},
|
||||
thinarrow/.style ={->, line width=0.5pt, black!55, >={Stealth[length=1.8mm]}},
|
||||
flowarrow/.style ={->, line width=1.2pt, black!70, >={Stealth[length=2.8mm]}},
|
||||
losseq/.style ={fill=cBg, draw=black!30, rounded corners=2pt,
|
||||
inner sep=4pt, font=\scriptsize, align=left},
|
||||
}
|
||||
|
||||
% =============================================================================
|
||||
% Panel (a): Tokenization (W=70mm, H=42mm)
|
||||
% =============================================================================
|
||||
\newcommand{\panelAcontent}{%
|
||||
\begin{tikzpicture}[x=1mm, y=1mm,
|
||||
panelarrow/.style={->, line width=0.5pt, black!55,
|
||||
>={Stealth[length=1.8mm]}}]
|
||||
\useasboundingbox (0,0) rectangle (70, 42);
|
||||
\node[paneltag] at (1, 41) {(a) Tokenization};
|
||||
|
||||
% --- pcap icon ------------------------------------------------------------
|
||||
\begin{scope}[shift={(1, 14)}]
|
||||
\fill[cBlue!12] (0,0) -- (0,18) -- (8,18) -- (10.8,15) -- (10.8,0) -- cycle;
|
||||
\draw[cBlue!80, line width=0.5pt]
|
||||
(0,0) -- (0,18) -- (8,18) -- (10.8,15) -- (10.8,0) -- cycle;
|
||||
\fill[cBlue!30] (8,18) -- (8,15) -- (10.8,15) -- cycle;
|
||||
\node[font=\bfseries\tiny, text=cBlue!90!black] at (5.4, 10) {flow.pcap};
|
||||
\foreach \y/\rl in {7/7, 5.5/5.5, 4/7, 2.5/4.5, 1/6} {
|
||||
\draw[cBlue!45, line width=0.25pt] (1, \y) -- (1+\rl, \y);
|
||||
}
|
||||
\end{scope}
|
||||
|
||||
% --- arrow: parse ---------------------------------------------------------
|
||||
\draw[panelarrow] (12.5, 23) -- (16.5, 23)
|
||||
node[midway, above=0.3mm, font=\tiny\itshape, text=black!65] {parse};
|
||||
|
||||
% --- packet stream (8 packets, each = 9-d feature vector) ----------------
|
||||
% Color encoding consistent with panel (b):
|
||||
% BLUE = 3 continuous channels (log_size, log_dt_ms, log_win)
|
||||
% ORANGE = 6 discrete channels (direction + 5 TCP flag bits S/F/R/P/A)
|
||||
\begin{scope}[shift={(17, 14)}]
|
||||
\foreach \i in {0,...,7} {
|
||||
\pgfmathsetmacro{\xx}{\i*3}
|
||||
\draw[fill=white, draw=black!50, line width=0.3pt, rounded corners=0.4pt]
|
||||
(\xx, 0) rectangle (\xx+2.5, 18);
|
||||
% --- top: 3 BLUE continuous bands (size, IAT, win) ------------------
|
||||
\fill[cBlue!65] (\xx+0.3, 14.3) rectangle (\xx+2.2, 16.6); % log_size
|
||||
\fill[cBlue!65] (\xx+0.3, 11.7) rectangle (\xx+2.2, 14.0); % log_dt_ms (IAT)
|
||||
\fill[cBlue!65] (\xx+0.3, 9.1) rectangle (\xx+2.2, 11.4); % log_win
|
||||
% thin separator between cont and disc sections
|
||||
\draw[black!25, line width=0.2pt] (\xx+0.3, 8.0) -- (\xx+2.2, 8.0);
|
||||
% --- bottom: 6 ORANGE discrete cells (dir + 5 TCP flags) ------------
|
||||
\foreach \k in {0,...,5} {
|
||||
\pgfmathsetmacro{\fx}{\xx + 0.3 + \k*0.32}
|
||||
\pgfmathsetmacro{\on}{int(mod(\i*3+\k,2))}
|
||||
\ifnum\on=1
|
||||
\fill[cOrange!75] (\fx, 1.5) rectangle (\fx+0.30, 7.0);
|
||||
\else
|
||||
\draw[cOrange!55, line width=0.2pt, fill=white]
|
||||
(\fx, 1.5) rectangle (\fx+0.30, 7.0);
|
||||
\fi
|
||||
}
|
||||
}
|
||||
\draw[->, line width=0.3pt, black!55] (-0.3, -2.5) -- (24.3, -2.5)
|
||||
node[anchor=west, font=\tiny, text=black!55, xshift=-0.5mm] {time};
|
||||
\end{scope}
|
||||
% --- inline mini-key showing what blue/orange mean inside one packet ----
|
||||
% Placed in the empty band between the panel tag (y=41) and the packet
|
||||
% stream top (y=32) so it does not crowd either.
|
||||
\node[font=\tiny, text=cBlue!85!black, anchor=west] at (17, 38)
|
||||
{\rule{1.6mm}{1.6mm}\,3 cont: size, IAT, win};
|
||||
\node[font=\tiny, text=cOrange!85!black, anchor=west] at (17, 35)
|
||||
{\rule{1.6mm}{1.6mm}\,6 disc: dir + S F R P A};
|
||||
|
||||
% --- arrow: tokenize ------------------------------------------------------
|
||||
\draw[panelarrow] (43, 23) -- (47, 23)
|
||||
node[midway, above=0.3mm, font=\tiny\itshape, text=black!65] {tokenize};
|
||||
|
||||
% --- token sequence: FLOW + 3 packet tokens + PAD ------------------------
|
||||
\begin{scope}[shift={(48, 11)}]
|
||||
\fill[cBlue!75] (0, 22) rectangle (3, 23.6);
|
||||
\foreach \i in {0,...,19} {
|
||||
\pgfmathsetmacro{\yp}{20.8 - \i*1.0}
|
||||
\fill[cBlue!25] (0, \yp) rectangle (3, \yp+0.9);
|
||||
}
|
||||
\node[font=\tiny, text=cBlue!90!black, anchor=north] at (1.5, 0.3) {FLOW};
|
||||
\foreach \k/\xoff/\lbl in {1/5/{$P_1$}, 2/9/{$P_2$}, 3/15/{$P_T$}} {
|
||||
\fill[cOrange!75] (\xoff, 22) rectangle (\xoff+3, 23.6);
|
||||
\foreach \i in {0,...,2} {
|
||||
\pgfmathsetmacro{\yp}{20.8 - \i*1.0}
|
||||
\fill[cBlue!25] (\xoff, \yp) rectangle (\xoff+3, \yp+0.9);
|
||||
}
|
||||
\foreach \i in {3,...,8} {
|
||||
\pgfmathsetmacro{\yp}{20.8 - \i*1.0}
|
||||
\fill[cOrange!30] (\xoff, \yp) rectangle (\xoff+3, \yp+0.9);
|
||||
}
|
||||
\foreach \i in {9,...,19} {
|
||||
\pgfmathsetmacro{\yp}{20.8 - \i*1.0}
|
||||
\fill[cGray!28] (\xoff, \yp) rectangle (\xoff+3, \yp+0.9);
|
||||
}
|
||||
\node[font=\tiny, anchor=north] at (\xoff+1.5, 0.3) {\lbl};
|
||||
}
|
||||
\node[font=\Large, text=black!55] at (13, 11) {$\cdots$};
|
||||
\draw[fill=cGray!22, draw=cGray, dash pattern=on 0.6pt off 0.6pt, line width=0.3pt]
|
||||
(19, 22) rectangle (22, 23.6);
|
||||
\foreach \i in {0,...,19} {
|
||||
\pgfmathsetmacro{\yp}{20.8 - \i*1.0}
|
||||
\fill[cGray!22] (19, \yp) rectangle (22, \yp+0.9);
|
||||
}
|
||||
\draw[draw=black!50, dashed, line width=0.4pt] (18.85, 0.3) rectangle (22.15, 23.7);
|
||||
\node[font=\tiny, anchor=north, text=black!65] at (20.5, 0.3) {PAD};
|
||||
\end{scope}
|
||||
\end{tikzpicture}%
|
||||
}
|
||||
|
||||
% =============================================================================
|
||||
% Panel (b): Mixed-state path (W=70mm, H=42mm)
|
||||
%
|
||||
% Conceptual layout (top → bottom):
|
||||
% row 1 — 3 column headers naming the 3 t snapshots:
|
||||
% x_0 (noise) | x_t (model input) | x_1 (data)
|
||||
% row 2 — continuous lane (blue): 3 particles linearly interpolating
|
||||
% from random Gaussian starts at t=0 to data values at t=1.
|
||||
% row 3 — discrete lane (orange): 3 snapshots of a 6-bit vector
|
||||
% (random at t=0, half kept at t=0.5, all kept at t=1).
|
||||
% row 4 — shared t-axis with ticks at the 3 column positions.
|
||||
%
|
||||
% Faint vertical dashed guides at x=8 / x=35 / x=62 tie the column headers,
|
||||
% both lanes, and the t-axis together so the eye can read top-to-bottom.
|
||||
% =============================================================================
|
||||
\newcommand{\panelBcontent}{%
|
||||
\begin{tikzpicture}[x=1mm, y=1mm]
|
||||
\useasboundingbox (0,0) rectangle (70, 42);
|
||||
\node[paneltagN] at (1, 41) {(b) Mixed-state path \contrib};
|
||||
|
||||
% --- column headers at the 3 t positions --------------------------------
|
||||
\node[font=\tiny\bfseries, anchor=south, text=black!70] at (8, 36)
|
||||
{$x_0$: noise};
|
||||
\node[font=\tiny\bfseries, anchor=south, text=cOrange!85!black] at (35, 36)
|
||||
{$x_t$ : \textit{model input}};
|
||||
\node[font=\tiny\bfseries, anchor=south, text=black!70] at (62, 36)
|
||||
{$x_1$: data};
|
||||
|
||||
% --- vertical guides tying the 3 columns to the t-axis ------------------
|
||||
\foreach \xc in {8, 35, 62} {
|
||||
\draw[black!22, dashed, line width=0.3pt] (\xc, 5) -- (\xc, 35.5);
|
||||
}
|
||||
|
||||
% ---- top: continuous track (linear interpolation x_0 → x_1) ------------
|
||||
\fill[cBlue!4] (1, 22) rectangle (69, 33);
|
||||
\draw[cBlue!30, line width=0.3pt] (1, 22) rectangle (69, 33);
|
||||
% Three independent particles, one per continuous channel.
|
||||
% Endpoint visual hierarchy: left small + faded (noise sample) →
|
||||
% mid medium →
|
||||
% right large + solid (data target).
|
||||
% Path 1
|
||||
\draw[cBlue, line width=1pt] (8, 30) -- (35, 28.25) -- (62, 26.5);
|
||||
\fill[cBlue, opacity=0.65] (8, 30) circle [radius=0.8];
|
||||
\fill[cBlue, opacity=0.9] (35, 28.25) circle [radius=0.9];
|
||||
\fill[cBlue] (62, 26.5) circle [radius=1.2];
|
||||
% Path 2
|
||||
\draw[cPurple!75!cBlue, line width=1pt] (8, 25) -- (35, 27.5) -- (62, 30);
|
||||
\fill[cPurple!75!cBlue, opacity=0.65] (8, 25) circle [radius=0.8];
|
||||
\fill[cPurple!75!cBlue, opacity=0.9] (35, 27.5) circle [radius=0.9];
|
||||
\fill[cPurple!75!cBlue] (62, 30) circle [radius=1.2];
|
||||
% Path 3
|
||||
\draw[cBlue!55!black, line width=1pt] (8, 28.5) -- (35, 26.7) -- (62, 24.9);
|
||||
\fill[cBlue!55!black, opacity=0.65] (8, 28.5) circle [radius=0.8];
|
||||
\fill[cBlue!55!black, opacity=0.9] (35, 26.7) circle [radius=0.9];
|
||||
\fill[cBlue!55!black] (62, 24.9) circle [radius=1.2];
|
||||
|
||||
\node[font=\tiny, text=cBlue!85!black, anchor=west] at (1, 20.5)
|
||||
{$x_t = (1{-}t)\,x_0 + t\,x_1, \quad x_0 \sim \mathcal{N}(0,I)$};
|
||||
|
||||
% ---- bottom: discrete track (uniform-corruption Bernoulli flip) -------
|
||||
\fill[cOrange!4] (1, 10) rectangle (69, 19);
|
||||
\draw[cOrange!35, line width=0.3pt] (1, 10) rectangle (69, 19);
|
||||
% Three time snapshots of a 6-bit vector, centered on the t-tick columns.
|
||||
% t=0 : 0 1 0 1 1 0 (random)
|
||||
% t=0.5 : 1 1 1 0 0 1 (about half kept)
|
||||
% t=1 : 1 0 1 0 1 1 (= data target)
|
||||
\foreach \pattern/\xc in {{0,1,0,1,1,0}/8,
|
||||
{1,1,1,0,0,1}/35,
|
||||
{1,0,1,0,1,1}/62} {
|
||||
\pgfmathsetmacro{\xs}{\xc - 6.5}
|
||||
\foreach \v [count=\k from 0] in \pattern {
|
||||
\pgfmathsetmacro{\fx}{\xs + \k*2.2}
|
||||
\ifnum\v=1
|
||||
\fill[cOrange!75] (\fx, 11.2) rectangle (\fx+1.8, 17.8);
|
||||
\node[font=\tiny, text=white] at (\fx+0.9, 14.5) {1};
|
||||
\else
|
||||
\draw[cOrange!75, line width=0.4pt, fill=white]
|
||||
(\fx, 11.2) rectangle (\fx+1.8, 17.8);
|
||||
\node[font=\tiny, text=cOrange!90!black] at (\fx+0.9, 14.5) {0};
|
||||
\fi
|
||||
}
|
||||
}
|
||||
\node[font=\tiny, text=cOrange!85!black, anchor=west] at (1, 8)
|
||||
{$x^{\mathrm{disc}}_t = x^{\mathrm{disc}}_1$ w.p.\ $t$, else $\mathrm{Unif}\{0,1\}$};
|
||||
|
||||
% ---- shared t-axis ----------------------------------------------------
|
||||
\draw[->, line width=0.5pt, black!60] (1, 4) -- (69, 4)
|
||||
node[anchor=west, font=\tiny, text=black!65, xshift=-0.5mm] {$t$};
|
||||
\foreach \tval/\xc in {0/8, 0.5/35, 1/62} {
|
||||
\draw[black!55, line width=0.4pt] (\xc, 3.6) -- (\xc, 4.4);
|
||||
\node[font=\tiny, anchor=north, text=black!70] at (\xc, 3.4) {$t{=}\tval$};
|
||||
}
|
||||
\end{tikzpicture}%
|
||||
}
|
||||
|
||||
% =============================================================================
|
||||
% Panel (c): Causal Velocity Field + heads + joint loss (W=152mm, H=82mm)
|
||||
%
|
||||
% Visual layout (route C + A combined per user spec):
|
||||
%
|
||||
% ────────────── t conditioning subsystem (y=66..78) ──────────────
|
||||
% t → sin embed (~) → MLP → cond cells [▓▒░] → cond_proj (Linear)
|
||||
% │
|
||||
% ┌──────────┼──────────┐
|
||||
% ▼γ1 ▼β1 ▼α1
|
||||
% ─────────── BACKBONE (y=2..62) ─────────────────────────────────
|
||||
%
|
||||
% LEFT col (tensor flow A): CENTER col (heatmap C):
|
||||
%
|
||||
% [tensor h_in] ▓▒░▓░▒ ★ MHSA + causal-packet
|
||||
% │ ┌─────────────────────┐
|
||||
% LN box │ █ █ █ █ █ █ █ █ █ █ │ ← FLOW row
|
||||
% │ │ █ █ │
|
||||
% [tensor h_LN] row-uniform │ █ █ █ │
|
||||
% │ │ █ █ █ █ │ large 12×12
|
||||
% ⊗γ_1 ◀──────── γ_1 (purple dashed) │ █ █ █ █ █ │ attention map
|
||||
% │ │ █ █ █ █ █ █ │ (FLOW row+col
|
||||
% [tensor h_γ] cols rescaled │ █ ... │ + lower-tri)
|
||||
% │ └──────────┬──────────┘
|
||||
% ⊕β_1 ◀──────── β_1 (green dashed) │
|
||||
% │ ⊗α_1 ◀── α_1 (orange dashed)
|
||||
% [tensor h_β] cells shifted │
|
||||
% │ ⊕ ◀── residual from h_in
|
||||
% └──────────────────────────────►──────────────┤
|
||||
% ▼
|
||||
% "MLP half (same)" → h'
|
||||
% ▼
|
||||
% [v_θ head]
|
||||
% [logits head ★]
|
||||
% [Joint loss eq]
|
||||
% =============================================================================
|
||||
\newcommand{\panelCcontent}{%
|
||||
\begin{tikzpicture}[x=1mm, y=1mm,
|
||||
opcirc/.style={circle, draw=black!70, fill=white,
|
||||
line width=0.5pt, inner sep=0pt,
|
||||
minimum size=2.6mm,
|
||||
font=\tiny\bfseries, text=black!75},
|
||||
rescirc/.style={circle, draw=black!70, fill=white,
|
||||
line width=0.7pt, inner sep=0pt,
|
||||
minimum size=3mm,
|
||||
font=\tiny\bfseries},
|
||||
pathline/.style={->, line width=0.5pt, black!65,
|
||||
>={Stealth[length=1.4mm]}},
|
||||
resarc/.style={->, line width=0.7pt, black!50,
|
||||
>={Stealth[length=1.6mm]}},
|
||||
gammaline/.style={->, dashed, line width=0.6pt, cPurple!85,
|
||||
>={Stealth[length=1.4mm]}},
|
||||
betaline/.style={->, dashed, line width=0.6pt, cGreen!75!black,
|
||||
>={Stealth[length=1.4mm]}},
|
||||
alphaline/.style={->, dashed, line width=0.6pt, cOrange!90,
|
||||
>={Stealth[length=1.4mm]}}]
|
||||
\useasboundingbox (0,0) rectangle (152, 82);
|
||||
\node[paneltagN] at (1, 81)
|
||||
{(c) Causal-Packet Velocity Field \contrib\ + joint loss};
|
||||
|
||||
% =========================================================================
|
||||
% T CONDITIONING SUBSYSTEM (y=66..78) — sin embed + MLP + cond + cond_proj
|
||||
% =========================================================================
|
||||
|
||||
% (1) t scalar
|
||||
\draw[fill=cBlue!15, draw=cBlue!80, line width=0.5pt]
|
||||
(8, 72) circle [radius=2.0];
|
||||
\node[font=\tiny\bfseries, text=cBlue!90!black] at (8, 72) {$t$};
|
||||
\node[font=\tiny, text=black!55, anchor=north] at (8, 69.5) {scalar};
|
||||
|
||||
\draw[pathline] (10.2, 72) -- (12, 72);
|
||||
|
||||
% (2) sin embed
|
||||
\draw[fill=white, draw=cBlue!80, line width=0.5pt, rounded corners=0.6pt]
|
||||
(12, 68.5) rectangle (28, 75.5);
|
||||
\draw[cBlue!80, line width=0.55pt]
|
||||
(13, 72) sin (14.5, 73.5) cos (16, 72) sin (17.5, 70.5) cos (19, 72)
|
||||
sin (20.5, 73.5) cos (22, 72) sin (23.5, 70.5) cos (25, 72)
|
||||
sin (26.5, 73.5) cos (27, 72.7);
|
||||
\node[font=\tiny, text=cBlue!90!black, anchor=north] at (20, 68.5)
|
||||
{sin embed $\to\mathbb{R}^{64}$};
|
||||
|
||||
\draw[pathline] (28.2, 72) -- (30, 72);
|
||||
|
||||
% (3) cond MLP trapezoid
|
||||
\fill[white] (30, 69) -- (30, 75) -- (37, 75.5) -- (37, 68.5) -- cycle;
|
||||
\draw[black!65, line width=0.5pt]
|
||||
(30, 69) -- (30, 75) -- (37, 75.5) -- (37, 68.5) -- cycle;
|
||||
\node[font=\tiny] at (33.5, 72) {MLP};
|
||||
|
||||
\draw[pathline] (37.2, 72) -- (39, 72);
|
||||
|
||||
% (4) cond vector cells
|
||||
\begin{scope}[shift={(39, 69)}]
|
||||
\foreach \k/\col in {0/cBlue, 1/cBlue!75!cPurple, 2/cPurple!75!cBlue, 3/cBlue!60!black,
|
||||
4/cBlue, 5/cPurple!85, 6/cBlue!75, 7/cBlue!85!black,
|
||||
8/cBlue!60!cPurple, 9/cBlue!70} {
|
||||
\pgfmathsetmacro{\fx}{\k * 1.4}
|
||||
\fill[\col] (\fx, 0) rectangle (\fx+1.2, 6);
|
||||
\draw[black!30, line width=0.15pt] (\fx, 0) rectangle (\fx+1.2, 6);
|
||||
}
|
||||
\end{scope}
|
||||
\node[font=\tiny, text=black!75, anchor=west] at (54, 72)
|
||||
{$\mathrm{cond}\!\in\!\mathbb{R}^{d}$};
|
||||
|
||||
\draw[pathline] (66, 72) -- (68, 72);
|
||||
|
||||
% (5) cond_proj — Linear that produces (γ, β, α) modulation parameters
|
||||
\node[rectangle, draw=black!65, fill=white, line width=0.5pt,
|
||||
rounded corners=0.6pt, minimum width=22mm, minimum height=8mm,
|
||||
inner sep=1pt, font=\tiny, align=center, anchor=west]
|
||||
(cproj) at (68, 72)
|
||||
{cond\_proj \\ $\mathbb{R}^d\!\to\!6d$};
|
||||
|
||||
% cond_proj output split: 3 colored mini-bars labelled γ_1, β_1, α_1
|
||||
% (γ_2, β_2, α_2 are implied — same fanout for the MLP half)
|
||||
\begin{scope}[shift={($(cproj.east)+(2mm, 4mm)$)}]
|
||||
\fill[cPurple!75] (0, 0) rectangle (10, 1.2);
|
||||
\draw[black!30, line width=0.15pt] (0, 0) rectangle (10, 1.2);
|
||||
\node[font=\tiny\bfseries, text=cPurple!75, anchor=west] at (10.5, 0.6) {$\gamma_1$};
|
||||
\end{scope}
|
||||
\begin{scope}[shift={($(cproj.east)+(2mm, 1mm)$)}]
|
||||
\fill[cGreen!75!black] (0, 0) rectangle (10, 1.2);
|
||||
\draw[black!30, line width=0.15pt] (0, 0) rectangle (10, 1.2);
|
||||
\node[font=\tiny\bfseries, text=cGreen!75!black, anchor=west] at (10.5, 0.6) {$\beta_1$};
|
||||
\end{scope}
|
||||
\begin{scope}[shift={($(cproj.east)+(2mm, -2mm)$)}]
|
||||
\fill[cOrange!90] (0, 0) rectangle (10, 1.2);
|
||||
\draw[black!30, line width=0.15pt] (0, 0) rectangle (10, 1.2);
|
||||
\node[font=\tiny\bfseries, text=cOrange!85!black, anchor=west] at (10.5, 0.6) {$\alpha_1$};
|
||||
\end{scope}
|
||||
\node[font=\tiny\itshape, text=black!55, anchor=west] at (cproj.south) [yshift=-0.5mm, xshift=-3mm]
|
||||
{(implied: $\gamma_2,\beta_2,\alpha_2$ for MLP half — same fanout)};
|
||||
|
||||
% =========================================================================
|
||||
% LEFT COLUMN: AdaLN modulation tensor flow visualization
|
||||
% Shows h transformed at each stage: h_in → LN → ⊗γ → ⊕β → h_β
|
||||
% =========================================================================
|
||||
|
||||
\def\gxc{32} % left column tensor center x
|
||||
\def\gw{1.4} % cell width
|
||||
\def\gh{1.0} % cell height
|
||||
|
||||
% token seq input box (far left)
|
||||
\node[archbox, minimum width=14mm, anchor=west] (toki) at (1, 50)
|
||||
{token seq\\$x_t$};
|
||||
\draw[pathline] (toki.east) -- ++(3mm, 0);
|
||||
|
||||
% --- (a) h_in tensor (top) ---
|
||||
\node[font=\tiny\bfseries, text=black!75, anchor=south] at (\gxc, 60)
|
||||
{$h$ (input)};
|
||||
\begin{scope}[shift={(\gxc - 4.2, 55.5)}]
|
||||
\foreach \op [count=\k from 0] in
|
||||
{30, 80, 50, 40, 90, 60,
|
||||
70, 20, 80, 50, 40, 90,
|
||||
60, 50, 90, 30, 80, 40,
|
||||
70, 30, 60, 80, 40, 70} {
|
||||
\pgfmathsetmacro{\i}{int(mod(\k, 6))}
|
||||
\pgfmathsetmacro{\j}{int(\k/6)}
|
||||
\fill[cBlue!\op] (\i*\gw, \j*\gh) rectangle (\i*\gw+\gw-0.05, \j*\gh+\gh-0.05);
|
||||
}
|
||||
\end{scope}
|
||||
\draw[pathline] (\gxc, 55.2) -- (\gxc, 54);
|
||||
|
||||
% LayerNorm box
|
||||
\node[archbox, minimum width=14mm, minimum height=2mm, inner sep=0.5pt,
|
||||
font=\tiny, anchor=north]
|
||||
(ln1) at (\gxc, 53.8) {LayerNorm};
|
||||
\draw[pathline] (\gxc, 51.7) -- (\gxc, 50.7);
|
||||
|
||||
% --- (b) h after LN (more uniform per col) ---
|
||||
\node[font=\tiny, text=cBlue!85!black, anchor=south] at (\gxc, 50.5)
|
||||
{$\mathrm{LN}(h)$};
|
||||
\begin{scope}[shift={(\gxc - 4.2, 45.5)}]
|
||||
\foreach \op [count=\k from 0] in
|
||||
{50, 30, 70, 50, 40, 60,
|
||||
40, 60, 50, 50, 40, 60,
|
||||
50, 50, 60, 40, 60, 40,
|
||||
50, 40, 50, 60, 40, 60} {
|
||||
\pgfmathsetmacro{\i}{int(mod(\k, 6))}
|
||||
\pgfmathsetmacro{\j}{int(\k/6)}
|
||||
\fill[cBlue!\op] (\i*\gw, \j*\gh) rectangle (\i*\gw+\gw-0.05, \j*\gh+\gh-0.05);
|
||||
}
|
||||
\end{scope}
|
||||
|
||||
% ⊗γ_1 operator
|
||||
\draw[pathline] (\gxc, 45.2) -- (\gxc, 44.2);
|
||||
\node[opcirc, draw=cPurple!75, text=cPurple!75] (gamma1) at (\gxc, 43.1) {\(\times\)};
|
||||
\node[font=\tiny, text=cPurple!75, anchor=west] at (gamma1.east) [xshift=0.3mm] {$\gamma_1$};
|
||||
\draw[pathline] (\gxc, 41.8) -- (\gxc, 40.8);
|
||||
|
||||
% --- (c) h after ⊗γ (per-column scaling visible) ---
|
||||
\node[font=\tiny, text=cPurple!75, anchor=south] at (\gxc, 40.5)
|
||||
{$\gamma_1\!\odot\!\mathrm{LN}(h)$};
|
||||
\begin{scope}[shift={(\gxc - 4.2, 35.5)}]
|
||||
\foreach \op [count=\k from 0] in
|
||||
{60, 12, 70, 25, 52, 60,
|
||||
48, 24, 50, 25, 52, 60,
|
||||
60, 20, 60, 20, 78, 40,
|
||||
60, 16, 50, 30, 52, 60} {
|
||||
\pgfmathsetmacro{\i}{int(mod(\k, 6))}
|
||||
\pgfmathsetmacro{\j}{int(\k/6)}
|
||||
\fill[cBlue!\op] (\i*\gw, \j*\gh) rectangle (\i*\gw+\gw-0.05, \j*\gh+\gh-0.05);
|
||||
}
|
||||
\end{scope}
|
||||
|
||||
% ⊕β_1 operator
|
||||
\draw[pathline] (\gxc, 35.2) -- (\gxc, 34.2);
|
||||
\node[opcirc, draw=cGreen!75!black, text=cGreen!75!black] (beta1) at (\gxc, 33.1) {\(+\)};
|
||||
\node[font=\tiny, text=cGreen!75!black, anchor=west] at (beta1.east) [xshift=0.3mm] {$\beta_1$};
|
||||
\draw[pathline] (\gxc, 31.8) -- (\gxc, 30.8);
|
||||
|
||||
% --- (d) h after ⊕β (uniform shift visible) ---
|
||||
\node[font=\tiny, text=cGreen!75!black, anchor=south] at (\gxc, 30.5)
|
||||
{$+\,\beta_1$};
|
||||
\begin{scope}[shift={(\gxc - 4.2, 25.5)}]
|
||||
\foreach \op [count=\k from 0] in
|
||||
{80, 32, 90, 45, 72, 80,
|
||||
68, 44, 70, 45, 72, 80,
|
||||
80, 40, 80, 40, 90, 60,
|
||||
80, 36, 70, 50, 72, 80} {
|
||||
\pgfmathsetmacro{\i}{int(mod(\k, 6))}
|
||||
\pgfmathsetmacro{\j}{int(\k/6)}
|
||||
\fill[cBlue!\op] (\i*\gw, \j*\gh) rectangle (\i*\gw+\gw-0.05, \j*\gh+\gh-0.05);
|
||||
}
|
||||
\end{scope}
|
||||
|
||||
% arrow → MHSA (going right toward heatmap)
|
||||
\draw[pathline] (\gxc + 4.5, 27.5) -- (62, 27.5);
|
||||
|
||||
% =========================================================================
|
||||
% CENTER COLUMN: large causal-packet attention heatmap
|
||||
% =========================================================================
|
||||
|
||||
\def\hxc{84} % heatmap center x
|
||||
\def\hcell{2.5} % heatmap cell size
|
||||
% heatmap occupies x = (\hxc - 15) .. (\hxc + 15) = 69..99, with 12 cells
|
||||
|
||||
% heatmap title
|
||||
\node[font=\tiny\bfseries, text=cOrange!85!black, anchor=south]
|
||||
at (\hxc, 60) {\contrib\ MHSA + causal-packet attention};
|
||||
|
||||
% heatmap top labels (FLOW + P_1 .. P_11) — only show subset for readability
|
||||
\foreach \i/\lbl in {0/F, 1/{$P_1$}, 5/{$P_5$}, 11/{$P_{11}$}} {
|
||||
\node[font=\tiny, text=black!60]
|
||||
at ({\hxc - 15 + \i*\hcell + \hcell/2}, 58.7) {\lbl};
|
||||
}
|
||||
% left labels
|
||||
\foreach \i/\lbl in {0/F, 1/{$P_1$}, 5/{$P_5$}, 11/{$P_{11}$}} {
|
||||
\node[font=\tiny, text=black!60, anchor=east]
|
||||
at ({\hxc - 15 - 0.4}, {57 - \i*\hcell - \hcell/2}) {\lbl};
|
||||
}
|
||||
|
||||
% heatmap cells
|
||||
\foreach \i in {0,...,11} {
|
||||
\foreach \j in {0,...,11} {
|
||||
\pgfmathsetmacro{\xx}{\hxc - 15 + \j * \hcell}
|
||||
\pgfmathsetmacro{\yy}{57 - (\i + 1) * \hcell}
|
||||
\ifnum\i=0
|
||||
\fill[cOrange!85] (\xx, \yy) rectangle (\xx+\hcell-0.1, \yy+\hcell-0.1);
|
||||
\else
|
||||
\ifnum\j=0
|
||||
\fill[cOrange!85] (\xx, \yy) rectangle (\xx+\hcell-0.1, \yy+\hcell-0.1);
|
||||
\else
|
||||
\ifnum\j>\i
|
||||
\draw[fill=white, draw=black!25, line width=0.15pt]
|
||||
(\xx, \yy) rectangle (\xx+\hcell-0.1, \yy+\hcell-0.1);
|
||||
\else
|
||||
\pgfmathsetmacro{\dist}{\i - \j}
|
||||
\pgfmathsetmacro{\opa}{int(75 - \dist*4)}
|
||||
\fill[cOrange!\opa] (\xx, \yy) rectangle (\xx+\hcell-0.1, \yy+\hcell-0.1);
|
||||
\fi
|
||||
\fi
|
||||
\fi
|
||||
}
|
||||
}
|
||||
|
||||
% heatmap legend below
|
||||
\node[font=\tiny\itshape, text=black!60, anchor=north, align=center]
|
||||
at (\hxc, 26)
|
||||
{row 0 / col 0 = FLOW token attends all\\
|
||||
lower-tri = packet $i$ attends pkt $\le i$};
|
||||
|
||||
% α_1 gate + ⊕ residual to the RIGHT of heatmap, vertical column
|
||||
\def\axc{106}
|
||||
\node[opcirc, draw=cOrange!90, text=cOrange!90] (alpha1) at (\axc, 27) {\(\times\)};
|
||||
\node[font=\tiny, text=cOrange!90, anchor=west] at (alpha1.east) [xshift=0.3mm] {$\alpha_1$};
|
||||
|
||||
% attention output → α gate
|
||||
\draw[pathline] (99, 27) -- (alpha1.west);
|
||||
|
||||
\draw[pathline] (\axc, 25.7) -- (\axc, 24.7);
|
||||
\node[rescirc] (res1) at (\axc, 23.5) {\(+\)};
|
||||
|
||||
% residual arc: from far-left (h_in) all the way over to res1 — long arc
|
||||
\draw[resarc] (\gxc, 60) to[out=90, in=180, looseness=0.7]
|
||||
($(res1.north) + (-3mm, 8mm)$) -- (res1.north);
|
||||
\node[font=\tiny\itshape, text=black!55, anchor=south]
|
||||
at ($(res1.north)+(0, 8mm)$) {residual from $h$};
|
||||
|
||||
% continuation → "MLP half" → output
|
||||
\draw[pathline] (\axc, 22) -- (\axc, 19);
|
||||
\node[archbox, dashed, minimum width=20mm, minimum height=4mm, inner sep=1pt,
|
||||
font=\tiny\itshape, anchor=north]
|
||||
(mlphalf) at (\axc, 18.5) {MLP half\\(same pattern)};
|
||||
\draw[pathline] (\axc, 12.7) -- (\axc, 11.5);
|
||||
\node[font=\tiny, text=black!75] at (\axc, 10.5) {$h'$};
|
||||
|
||||
% "× 4 layers" annotation at the bottom of the center area
|
||||
\node[font=\tiny\itshape, text=cOrange!75!black, anchor=center]
|
||||
at (\hxc, 7) {(this whole block stacks $\times\,4$ layers)};
|
||||
|
||||
% =========================================================================
|
||||
% CONDITIONING LINES — γ_1 / β_1 / α_1 from cond_proj output bars to ops
|
||||
% =========================================================================
|
||||
|
||||
% γ_1 (purple) → ⊗γ_1 in left column
|
||||
\draw[gammaline]
|
||||
($(cproj.east)+(12mm, 4.6mm)$) .. controls (90, 60) and (60, 55) .. (gamma1.north);
|
||||
|
||||
% β_1 (green) → ⊕β_1 in left column
|
||||
\draw[betaline]
|
||||
($(cproj.east)+(12mm, 1.6mm)$) .. controls (95, 55) and (55, 45) .. (beta1.north);
|
||||
|
||||
% α_1 (orange) → ⊗α_1 next to heatmap
|
||||
\draw[alphaline]
|
||||
($(cproj.east)+(12mm, -1.4mm)$) .. controls (105, 65) and (108, 35) .. (alpha1.north);
|
||||
|
||||
% =========================================================================
|
||||
% RIGHT: heads + joint loss
|
||||
% =========================================================================
|
||||
\node[archbox, minimum width=22mm, anchor=west]
|
||||
(vh) at (124, 28) {$v_\theta$ head\\(continuous)};
|
||||
\node[novelbox, minimum width=22mm, anchor=west]
|
||||
(dh) at (124, 17) {\contrib\ logits head\\(discrete)};
|
||||
|
||||
% from h' to heads (right-then-up branching)
|
||||
\coordinate (hsplit) at (118, 10.5);
|
||||
\draw[arrow] (\axc, 10.5) -- (hsplit);
|
||||
\draw[arrow] (hsplit) |- (vh.west);
|
||||
\draw[arrow] (hsplit) |- (dh.west);
|
||||
|
||||
\node[losseq, anchor=north west, minimum width=46mm]
|
||||
at (124, 50)
|
||||
{\textbf{Joint loss}\\[1pt]
|
||||
$\mathcal{L} =
|
||||
\underbrace{\lVert v_\theta - (x_1{-}x_0)\rVert^2}_{\text{\textcolor{cBlue!85!black}{cont CFM}}}$\\[2pt]
|
||||
$\quad\;\;\,+\;\lambda
|
||||
\underbrace{\mathrm{CE}(\mathrm{logits},\,x_1^{\text{disc}})}_{\text{\textcolor{cOrange!85!black}{discrete FM}}}$};
|
||||
\end{tikzpicture}%
|
||||
}
|
||||
|
||||
% =============================================================================
|
||||
% Panel (d): Inference + Mahalanobis-OAS router (W=152mm, H=44mm)
|
||||
% - "Why a router?" callout removed.
|
||||
% - Score-space scatter is now wider and centered under svec/router/auroc.
|
||||
% =============================================================================
|
||||
\newcommand{\panelDcontent}{%
|
||||
\begin{tikzpicture}[x=1mm, y=1mm]
|
||||
\useasboundingbox (0,0) rectangle (152, 44);
|
||||
\node[paneltagN] at (1, 43)
|
||||
{(d) Inference \& Mahalanobis-OAS router \contrib};
|
||||
|
||||
% --- linear chain (upper portion) ----------------------------------------
|
||||
\node[archbox, minimum width=16mm, fill=cGray!12, anchor=west]
|
||||
(testflow) at (1, 32) {test\\flow};
|
||||
\node[archbox, dashed, minimum width=22mm, anchor=west]
|
||||
(frozen) at ($(testflow.east)+(3mm,0)$)
|
||||
{\textsc{Frozen} backbone\\($v_\theta$, logits)};
|
||||
\node[archbox, minimum width=30mm, anchor=west]
|
||||
(svec) at ($(frozen.east)+(3mm,0)$)
|
||||
{\textsc{Score} $s\in\mathbb{R}^{10}$\\
|
||||
\scriptsize\texttt{terminal\_norm,}\\
|
||||
\scriptsize\texttt{terminal\_packet,}\\
|
||||
\scriptsize\texttt{disc\_nll\_total,}~$\dots$};
|
||||
\node[novelbox, minimum width=42mm, minimum height=18mm, anchor=west]
|
||||
(router) at ($(svec.east)+(3mm,0)$)
|
||||
{\contrib\ \textbf{Mahalanobis-OAS}\\[1pt]
|
||||
$D^2(s) = (s-\mu)^\top \Sigma_{\mathrm{OAS}}^{-1}(s-\mu)$\\[1pt]
|
||||
\scriptsize\itshape $\mu, \Sigma$ fit on benign val\\[-0.2pt]
|
||||
\scriptsize\itshape (no labels, no selection bias)};
|
||||
\node[archbox, minimum width=14mm, fill=cBlue!10, anchor=west]
|
||||
(auroc) at ($(router.east)+(3mm,0)$) {AUROC\\$s_{\text{anomaly}}$};
|
||||
\draw[arrow] (testflow.east) -- (frozen.west);
|
||||
\draw[arrow] (frozen.east) -- (svec.west);
|
||||
\draw[arrow] (svec.east) -- (router.west);
|
||||
\draw[arrow] (router.east) -- (auroc.west);
|
||||
|
||||
% --- score-space scatter (centered, sized to fill the lower band) -------
|
||||
% Center under svec..auroc span. svec.left=45, auroc.right≈137.
|
||||
% Center x ≈ (45+137)/2 = 91. Width = 90mm → x range 46..136.
|
||||
\node[draw=black!40, line width=0.3pt, fill=white,
|
||||
minimum width=90mm, minimum height=22mm, anchor=south,
|
||||
inner sep=0pt]
|
||||
(scatterbox) at (91, 1) {};
|
||||
\node[font=\tiny\bfseries, text=black!75, anchor=north west]
|
||||
at (scatterbox.north west) [xshift=1.5mm, yshift=-0.8mm]
|
||||
{score space (2 of 10 dims, illustrative)};
|
||||
% axes inside scatterbox
|
||||
\begin{scope}[shift={($(scatterbox.south west)+(7mm,3mm)$)}]
|
||||
\draw[->, line width=0.3pt, black!55] (0, 0) -- (76, 0);
|
||||
\draw[->, line width=0.3pt, black!55] (0, 0) -- (0, 14);
|
||||
\node[font=\tiny, text=black!60, anchor=west] at (60, -1.4)
|
||||
{\texttt{terminal\_norm}};
|
||||
\node[font=\tiny, text=black!60, rotate=90, anchor=south west] at (-1.6, 1)
|
||||
{\texttt{disc\_nll}};
|
||||
% benign cluster ellipse (Sigma_OAS) — wider since plot is wider
|
||||
\draw[cBlue!75, line width=0.6pt, rotate around={20:(33,6)}]
|
||||
(33,6) ellipse [x radius=14, y radius=4];
|
||||
\draw[cBlue!50, line width=0.4pt, dashed, rotate around={20:(33,6)}]
|
||||
(33,6) ellipse [x radius=21, y radius=6];
|
||||
% benign points
|
||||
\foreach \px/\py in {28/5.4, 32/6.6, 36/6.8, 30/4.8, 34/5.8,
|
||||
38/7, 26/6, 34/5.4, 37/6.4, 24/5.4,
|
||||
40/7.2, 22/4.8, 30/6.4, 35/7, 28/4.4} {
|
||||
\fill[cBlue!75] (\px, \py) circle [radius=0.5];
|
||||
}
|
||||
% attack points (outside the ellipse)
|
||||
\foreach \px/\py in {60/10, 56/4, 52/11, 64/8,
|
||||
10/9, 6/6, 68/5, 8/3,
|
||||
70/12, 14/2, 4/10, 66/2.5} {
|
||||
\fill[cOrange] (\px, \py) circle [radius=0.7];
|
||||
}
|
||||
% legend
|
||||
\fill[cBlue!75] (50, 12.5) circle [radius=0.5];
|
||||
\node[font=\tiny, text=cBlue!85!black, anchor=west] at (51.3, 12.5) {benign};
|
||||
\fill[cOrange] (62, 12.5) circle [radius=0.7];
|
||||
\node[font=\tiny, text=cOrange!85!black, anchor=west] at (63.3, 12.5) {attack};
|
||||
\end{scope}
|
||||
\end{tikzpicture}%
|
||||
}
|
||||
|
||||
% =============================================================================
|
||||
% OUTER FIGURE — chains the four panels with positioning, then draws three
|
||||
% inter-panel data-flow arrows that make the training/inference pipeline
|
||||
% explicit:
|
||||
%
|
||||
% (a) ──x_1──→ (b)
|
||||
% │ x_t
|
||||
% ▼
|
||||
% (c) [training]
|
||||
% │ frozen
|
||||
% ▼
|
||||
% (d) [inference]
|
||||
% =============================================================================
|
||||
\begin{document}
|
||||
\begin{tikzpicture}[node distance=8mm]
|
||||
|
||||
% row 1
|
||||
\node[panel] (panA) {\panelAcontent};
|
||||
\node[novelpanel, right=of panA] (panB) {\panelBcontent};
|
||||
|
||||
% row 2 — full width
|
||||
\node[novelpanel, below=of panA.south west, anchor=north west] (panC)
|
||||
{\panelCcontent};
|
||||
|
||||
% row 3 — full width
|
||||
\node[panel, below=of panC.south west, anchor=north west] (panD)
|
||||
{\panelDcontent};
|
||||
|
||||
% --- inter-panel data-flow arrows ---------------------------------------
|
||||
% (a) → (b): clean tokens x_1 produced by tokenization feed the corruption
|
||||
\draw[flowarrow] (panA.east) -- (panB.west)
|
||||
node[midway, above=0.5mm, font=\scriptsize\itshape, text=black!75]
|
||||
{$x_1$};
|
||||
% (b) → (c): mixed-state corrupted x_t at random t ∈ [0,1] feeds the model
|
||||
\draw[flowarrow] (panB.south) -- (panC.north -| panB.south)
|
||||
node[midway, right=0.5mm, font=\scriptsize\itshape, text=black!75]
|
||||
{$x_t$};
|
||||
% (c) → (d): trained backbone weights are frozen at test time
|
||||
\draw[flowarrow] (panC.south) -- (panD.north)
|
||||
node[midway, right=0.5mm, font=\scriptsize\itshape, text=black!75]
|
||||
{\textit{frozen}};
|
||||
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
338
paper/figures/figure1_pipeline.tex
Normal file
338
paper/figures/figure1_pipeline.tex
Normal file
@@ -0,0 +1,338 @@
|
||||
% Figure 1: A+C combo pipeline (full).
|
||||
% Compile: pdflatex figure1_pipeline.tex
|
||||
\documentclass[border=8pt]{standalone}
|
||||
\usepackage{tikz}
|
||||
\usetikzlibrary{arrows.meta, positioning, calc, decorations.pathreplacing}
|
||||
\usepackage{amsmath, amssymb}
|
||||
|
||||
\definecolor{myorange}{RGB}{230, 126, 34}
|
||||
\definecolor{myblue}{RGB}{52, 110, 180}
|
||||
\definecolor{mygreen}{RGB}{46, 139, 87}
|
||||
\definecolor{mypurple}{RGB}{142, 68, 173}
|
||||
\definecolor{mygray}{RGB}{170, 170, 170}
|
||||
\definecolor{mybg}{RGB}{248, 248, 248}
|
||||
|
||||
\newcommand{\contrib}{\textcolor{myorange}{\ensuremath{\bigstar}}}
|
||||
|
||||
\begin{document}
|
||||
\begin{tikzpicture}[
|
||||
font=\small,
|
||||
>={Stealth[length=2.5mm]},
|
||||
databox/.style={rectangle, draw=mygray, rounded corners=2pt,
|
||||
minimum height=12mm, minimum width=14mm, align=center, fill=mygray!20},
|
||||
box/.style={rectangle, draw=black!70, thick, rounded corners=2pt,
|
||||
minimum height=12mm, minimum width=22mm, align=center, fill=white},
|
||||
novelbox/.style={rectangle, draw=myorange, line width=1.4pt, rounded corners=2pt,
|
||||
minimum height=12mm, minimum width=22mm, align=center, fill=myorange!10},
|
||||
arrow/.style={->, thick},
|
||||
losseq/.style={fill=mybg, draw=black!30, rounded corners=2pt, inner sep=5pt}
|
||||
]
|
||||
|
||||
% =========================================================================
|
||||
% TRAINING ROW (no row title)
|
||||
% =========================================================================
|
||||
|
||||
% ----- (1) PCAP file icon -----
|
||||
\begin{scope}[shift={(0.0, -1.2)}]
|
||||
\fill[myblue!12] (0,0) -- (0,2.4) -- (1.05,2.4) -- (1.45,2.0) -- (1.45,0) -- cycle;
|
||||
\draw[myblue!80, line width=0.6pt]
|
||||
(0,0) -- (0,2.4) -- (1.05,2.4) -- (1.45,2.0) -- (1.45,0) -- cycle;
|
||||
\fill[myblue!30] (1.05,2.4) -- (1.05,2.0) -- (1.45,2.0) -- cycle;
|
||||
\draw[myblue!80, line width=0.4pt] (1.05,2.4) -- (1.05,2.0) -- (1.45,2.0);
|
||||
\node[font=\bfseries\scriptsize, text=myblue!90!black] at (0.7, 1.65) {flow.pcap};
|
||||
\foreach \y/\rl in {1.30/0.95, 1.10/0.78, 0.90/1.05, 0.70/0.65, 0.50/0.90, 0.30/0.55} {
|
||||
\draw[myblue!45, line width=0.3pt] (0.15, \y) -- (0.15+\rl, \y);
|
||||
}
|
||||
\end{scope}
|
||||
|
||||
% Arrow: parse (length = 1.4cm so label fits comfortably)
|
||||
\draw[arrow, black!60] (1.6, 0) -- (3.0, 0)
|
||||
node[midway, above=0.5mm, font=\scriptsize\itshape, text=black!70] {parse};
|
||||
|
||||
% ----- (2) Packet stream: pkt0, pkt1, pkt2, ..., pkt_T -----
|
||||
\begin{scope}[shift={(3.1, -1.1)}]
|
||||
% 4 packets at explicit x-offsets; first three are 0,1,2 then a gap with
|
||||
% ellipsis and the final one labeled pkt_T.
|
||||
\foreach \j/\xoff/\lbl in {0/0.00/0, 1/1.18/1, 2/2.36/2, 3/4.04/T} {
|
||||
\pgfmathsetmacro{\xx}{\xoff}
|
||||
\draw[fill=white, draw=black!55, line width=0.45pt, rounded corners=1pt]
|
||||
(\xx, 0.05) rectangle (\xx+1.05, 2.25);
|
||||
\draw[fill=myblue!35, draw=myblue, line width=0.3pt]
|
||||
(\xx+0.07, 2.00) rectangle (\xx+0.32, 2.15);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.34, 2.075) {size};
|
||||
\draw[fill=mygreen!35, draw=mygreen, line width=0.3pt]
|
||||
(\xx+0.07, 1.78) rectangle (\xx+0.32, 1.93);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.34, 1.855) {IAT};
|
||||
\draw[fill=mypurple!35, draw=mypurple, line width=0.3pt]
|
||||
(\xx+0.07, 1.56) rectangle (\xx+0.20, 1.71);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.22, 1.635) {dir};
|
||||
\ifcase\j
|
||||
\def\flagS{1}\def\flagF{0}\def\flagR{0}\def\flagP{0}\def\flagA{0}\or
|
||||
\def\flagS{1}\def\flagF{0}\def\flagR{0}\def\flagP{0}\def\flagA{1}\or
|
||||
\def\flagS{0}\def\flagF{0}\def\flagR{0}\def\flagP{0}\def\flagA{1}\or
|
||||
\def\flagS{0}\def\flagF{0}\def\flagR{0}\def\flagP{1}\def\flagA{1}\fi
|
||||
\foreach \fname/\fval/\k in {S/\flagS/0, F/\flagF/1, R/\flagR/2, P/\flagP/3, A/\flagA/4} {
|
||||
\pgfmathsetmacro{\fx}{\xx + 0.07 + \k*0.18}
|
||||
\ifnum\fval=1
|
||||
\draw[fill=myorange!70, draw=myorange, line width=0.3pt]
|
||||
(\fx, 1.30) rectangle (\fx+0.15, 1.46);
|
||||
\node[font=\tiny, white] at (\fx+0.075, 1.38) {\fname};
|
||||
\else
|
||||
\draw[fill=white, draw=myorange!60, line width=0.3pt]
|
||||
(\fx, 1.30) rectangle (\fx+0.15, 1.46);
|
||||
\node[font=\tiny, text=myorange!80!black] at (\fx+0.075, 1.38) {\fname};
|
||||
\fi
|
||||
}
|
||||
\draw[fill=myblue!20, draw=myblue!60, line width=0.3pt]
|
||||
(\xx+0.07, 1.05) rectangle (\xx+0.32, 1.20);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.34, 1.125) {win};
|
||||
\draw[black!30, line width=0.3pt] (\xx+0.07, 0.80) -- (\xx+0.97, 0.80);
|
||||
\draw[black!30, line width=0.3pt] (\xx+0.07, 0.65) -- (\xx+0.85, 0.65);
|
||||
\draw[black!30, line width=0.3pt] (\xx+0.07, 0.50) -- (\xx+0.95, 0.50);
|
||||
\draw[black!30, line width=0.3pt] (\xx+0.07, 0.35) -- (\xx+0.70, 0.35);
|
||||
\node[font=\scriptsize, text=black!70] at (\xx+0.525, -0.13) {pkt$_{\lbl}$};
|
||||
}
|
||||
% Ellipsis between pkt_2 (ends at x=3.41) and pkt_T (starts at x=4.04)
|
||||
\node[font=\Large, text=black!55] at (3.72, 1.15) {$\cdots$};
|
||||
% Time axis spans full row
|
||||
\draw[->, line width=0.35pt, black!55] (-0.05, -0.4) -- (5.20, -0.4);
|
||||
\node[font=\tiny, anchor=west, text=black!55] at (5.20, -0.4) {time};
|
||||
\end{scope}
|
||||
|
||||
% Arrow: tokenize (length = 1.4cm). Packet stream ends at x=3.1+5.09=8.19,
|
||||
% so arrow must start after that.
|
||||
\draw[arrow, black!60] (8.3, 0) -- (9.7, 0)
|
||||
node[midway, above=0.5mm, font=\scriptsize\itshape, text=black!70] {tokenize};
|
||||
|
||||
% ----- (3) Token sequence: [FLOW | P_1 | P_2 | ... | P_n | PAD] -----
|
||||
% All tokens are 21-d (1 type marker + 20 feature/pad cells). Two paddings:
|
||||
% • channel-padding inside packet tokens (cells 10..20, since pkt has 9 feats)
|
||||
% • sequence-padding tokens (entire token zeroed when actual flow length < T)
|
||||
\begin{scope}[shift={(9.8, -1.45)}]
|
||||
% --- helper macros for cell drawing inside this scope ---
|
||||
% Each token: 1 type cell on top + 20 feature/pad cells below.
|
||||
% Cell height 0.085, cell width 0.42.
|
||||
% Token x-offsets: FLOW=0, P_1=0.62, P_2=1.22, P_n=2.30, PAD=2.92.
|
||||
%
|
||||
% FLOW token: type (dark blue) + 20 cont features (light blue), no padding
|
||||
\draw[fill=myblue!75, draw=myblue, line width=0.25pt]
|
||||
(0.0, 2.55) rectangle (0.42, 2.64);
|
||||
\foreach \i in {0,...,19} {
|
||||
\pgfmathsetmacro{\yp}{2.46 - \i*0.085}
|
||||
\draw[fill=myblue!25, draw=myblue!80, line width=0.25pt]
|
||||
(0.0, \yp) rectangle (0.42, \yp+0.075);
|
||||
}
|
||||
\node[font=\tiny, anchor=north, text=myblue!90!black] at (0.21, 0.72) {\textbf{FLOW}};
|
||||
%
|
||||
% P_1, P_2, P_n: type (orange) + 3 cont + 6 disc + 11 channel-pad
|
||||
\foreach \subi/\xoff in {1/0.62, 2/1.22, n/2.30} {
|
||||
% type marker (orange)
|
||||
\draw[fill=myorange!75, draw=myorange, line width=0.25pt]
|
||||
(\xoff, 2.55) rectangle (\xoff+0.42, 2.64);
|
||||
% 3 cont (blue)
|
||||
\foreach \i in {0,...,2} {
|
||||
\pgfmathsetmacro{\yp}{2.46 - \i*0.085}
|
||||
\draw[fill=myblue!25, draw=myblue!80, line width=0.25pt]
|
||||
(\xoff, \yp) rectangle (\xoff+0.42, \yp+0.075);
|
||||
}
|
||||
% 6 disc (orange)
|
||||
\foreach \i in {3,...,8} {
|
||||
\pgfmathsetmacro{\yp}{2.46 - \i*0.085}
|
||||
\draw[fill=myorange!30, draw=myorange!80, line width=0.25pt]
|
||||
(\xoff, \yp) rectangle (\xoff+0.42, \yp+0.075);
|
||||
}
|
||||
% 11 channel-pad (gray hatched)
|
||||
\foreach \i in {9,...,19} {
|
||||
\pgfmathsetmacro{\yp}{2.46 - \i*0.085}
|
||||
\draw[fill=mygray!30, draw=mygray, line width=0.2pt, dash pattern=on 0.4pt off 0.4pt]
|
||||
(\xoff, \yp) rectangle (\xoff+0.42, \yp+0.075);
|
||||
}
|
||||
\node[font=\tiny, anchor=north] at (\xoff+0.21, 0.72) {$P_{\subi}$};
|
||||
}
|
||||
%
|
||||
% Ellipsis centered in gap between P_2 (ends x=1.64) and P_n (starts x=2.30),
|
||||
% vertically centered on token cell stack (top=2.64, bottom=0.86)
|
||||
\node[font=\Large, text=black!55] at (1.97, 1.75) {$\cdots$};
|
||||
%
|
||||
% PAD token (sequence-padding): entire token grayed + dashed outer border
|
||||
\draw[fill=mygray!20, draw=mygray, dash pattern=on 1pt off 1pt, line width=0.4pt]
|
||||
(2.92, 2.55) rectangle (3.34, 2.64);
|
||||
\foreach \i in {0,...,19} {
|
||||
\pgfmathsetmacro{\yp}{2.46 - \i*0.085}
|
||||
\draw[fill=mygray!25, draw=mygray, line width=0.2pt, dash pattern=on 0.5pt off 0.5pt]
|
||||
(2.92, \yp) rectangle (3.34, \yp+0.075);
|
||||
}
|
||||
% outer dashed wrap to emphasize "this whole token is padding"
|
||||
\draw[draw=black!50, dashed, line width=0.5pt]
|
||||
(2.90, 0.72) rectangle (3.36, 2.66);
|
||||
\node[font=\tiny, anchor=north, text=black!65] at (3.13, 0.72) {\textbf{PAD}};
|
||||
%
|
||||
% Bottom brace + sequence label (below the FLOW/P_i/PAD name labels)
|
||||
\draw[decorate, decoration={brace, amplitude=3pt, mirror}, line width=0.4pt, black!60]
|
||||
(-0.05, 0.30) -- (3.39, 0.30);
|
||||
\node[font=\scriptsize\itshape, text=black!70] at (1.67, -0.05)
|
||||
{token sequence (1 flow + $T$ packets, all 21-d)};
|
||||
\end{scope}
|
||||
|
||||
% Arrow: token → velocity (sequence ends at x=9.8+3.34=13.14)
|
||||
\draw[arrow, black!60] (13.3, 0) -- (14.0, 0);
|
||||
|
||||
% ----- (4) Velocity field (with detailed internals, DiT-style) -----
|
||||
% Outer box at x=14.0..19.0, y=-2.5..2.5 (5cm × 5cm)
|
||||
\begin{scope}[shift={(14.0, -2.5)}]
|
||||
% Outer rounded box
|
||||
\draw[fill=myorange!8, draw=myorange, line width=1.4pt, rounded corners=2pt]
|
||||
(0, 0) rectangle (5.0, 5.0);
|
||||
\node[font=\sffamily\bfseries\small, anchor=north, text=myorange!90!black]
|
||||
at (2.5, 4.85) {Velocity Field};
|
||||
%
|
||||
% Time embedding sub-box (top-left)
|
||||
\draw[fill=white, draw=black!55, line width=0.4pt, rounded corners=1pt]
|
||||
(0.20, 3.60) rectangle (1.30, 4.45);
|
||||
\node[font=\tiny\bfseries, anchor=north] at (0.75, 4.42) {time $t$};
|
||||
\node[font=\tiny, anchor=north] at (0.75, 4.20) {sinusoidal};
|
||||
\node[font=\tiny, anchor=north] at (0.75, 4.00) {emb + MLP};
|
||||
%
|
||||
% AdaLN-Zero block 1
|
||||
\draw[fill=white, draw=black!60, line width=0.4pt, rounded corners=1pt]
|
||||
(1.55, 3.95) rectangle (4.80, 4.45);
|
||||
\node[font=\scriptsize] at (3.18, 4.20)
|
||||
{AdaLN-Zero \,($\gamma_1, \beta_1, \alpha_1$)};
|
||||
% Conditioning arrow from time emb to AdaLN-1
|
||||
\draw[->, dashed, line width=0.4pt, black!50]
|
||||
(1.30, 4.20) -- (1.55, 4.20);
|
||||
%
|
||||
% Arrow down
|
||||
\draw[->, line width=0.4pt, black!60] (3.18, 3.95) -- (3.18, 3.65);
|
||||
%
|
||||
% MHA block with causal mask icon
|
||||
\draw[fill=myorange!15, draw=myorange, line width=0.7pt, rounded corners=1pt]
|
||||
(1.55, 2.05) rectangle (4.80, 3.65);
|
||||
\node[font=\scriptsize\bfseries, anchor=north, text=myorange!85!black]
|
||||
at (3.18, 3.55) {\contrib\ Multi-Head Self-Attn};
|
||||
\node[font=\tiny\itshape, anchor=north, text=myorange!85!black]
|
||||
at (3.18, 3.30) {causal-packet mask};
|
||||
% Mini 5×5 lower-triangular mask grid
|
||||
\begin{scope}[shift={(2.55, 2.30)}]
|
||||
\foreach \i in {0,...,4} {
|
||||
\foreach \j in {0,...,4} {
|
||||
\ifnum\j>\i
|
||||
\draw[fill=white, draw=black!40, line width=0.15pt]
|
||||
(\j*0.22, -\i*0.16) rectangle (\j*0.22+0.20, -\i*0.16-0.14);
|
||||
\else
|
||||
\draw[fill=myorange!55, draw=myorange, line width=0.15pt]
|
||||
(\j*0.22, -\i*0.16) rectangle (\j*0.22+0.20, -\i*0.16-0.14);
|
||||
\fi
|
||||
}
|
||||
}
|
||||
\end{scope}
|
||||
%
|
||||
% Arrow down
|
||||
\draw[->, line width=0.4pt, black!60] (3.18, 2.05) -- (3.18, 1.75);
|
||||
%
|
||||
% AdaLN-Zero block 2
|
||||
\draw[fill=white, draw=black!60, line width=0.4pt, rounded corners=1pt]
|
||||
(1.55, 1.25) rectangle (4.80, 1.75);
|
||||
\node[font=\scriptsize] at (3.18, 1.50)
|
||||
{AdaLN-Zero \,($\gamma_2, \beta_2, \alpha_2$)};
|
||||
% Curved conditioning arrow from time emb to AdaLN-2
|
||||
\draw[->, dashed, line width=0.4pt, black!50]
|
||||
(0.75, 3.60) .. controls (0.40, 2.30) and (0.80, 1.50) .. (1.55, 1.50);
|
||||
%
|
||||
% Arrow down
|
||||
\draw[->, line width=0.4pt, black!60] (3.18, 1.25) -- (3.18, 0.95);
|
||||
%
|
||||
% MLP block
|
||||
\draw[fill=white, draw=black!60, line width=0.4pt, rounded corners=1pt]
|
||||
(1.55, 0.45) rectangle (4.80, 0.95);
|
||||
\node[font=\scriptsize] at (3.18, 0.70)
|
||||
{MLP \,($d \to 4d \to d$)};
|
||||
%
|
||||
% "× 4 layers" stacking annotation at the bottom
|
||||
\node[font=\scriptsize\itshape, anchor=south, text=black!70]
|
||||
at (2.5, 0.05) {(stacked $\times\,4$ layers)};
|
||||
\end{scope}
|
||||
|
||||
% Helper coordinates for input/output of velocity field (so arrows still work)
|
||||
\coordinate (vel_west) at (14.0, 0);
|
||||
\coordinate (vel_east) at (19.0, 0);
|
||||
|
||||
% ----- (5) Two heads -----
|
||||
\node[box, minimum width=22mm] (vh) at (20.5, 0.7) {%
|
||||
$v_\theta$ head\\
|
||||
{\scriptsize (continuous)}%
|
||||
};
|
||||
\node[novelbox, minimum width=22mm] (dh) at (20.5, -0.7) {%
|
||||
\contrib\ logits head\\
|
||||
{\scriptsize (discrete)}%
|
||||
};
|
||||
|
||||
\draw[arrow] (vel_east) -- (vh.west);
|
||||
\draw[arrow] (vel_east) -- (dh.west);
|
||||
|
||||
% ----- Loss equation -----
|
||||
\node[losseq, minimum width=190mm, font=\small, align=center] (loss) at (9.85, -3.7) {%
|
||||
$\mathcal{L} \;=\;
|
||||
\underbrace{\| v_\theta(x_t,t) - (x_1 - x_0) \|^2}_{\text{continuous CFM}}
|
||||
\;+\; \lambda \cdot
|
||||
\underbrace{\mathrm{CE}(\mathrm{logits},\, x_{\mathrm{disc}})}_{\text{Discrete FM}\,\contrib}$%
|
||||
};
|
||||
|
||||
% =========================================================================
|
||||
% INFERENCE ROW (no row title)
|
||||
% =========================================================================
|
||||
|
||||
\node[databox, minimum width=18mm] (testflow) at (1.5, -5.7) {%
|
||||
{\scriptsize test}\\ flow%
|
||||
};
|
||||
|
||||
\node[box, dashed, minimum width=28mm] (frozen) at (4.7, -5.7) {%
|
||||
\textsc{Frozen}\\
|
||||
\textsc{Backbone}\\
|
||||
{\scriptsize $v_\theta$ + logits}%
|
||||
};
|
||||
|
||||
\node[box, minimum width=34mm] (svec) at (8.6, -5.7) {%
|
||||
\textsc{Score Vector} (10-d)\\[0.5mm]
|
||||
{\scriptsize\texttt{terminal\_norm}}\\
|
||||
{\scriptsize\texttt{terminal\_packet}}\\
|
||||
{\scriptsize\texttt{disc\_nll\_total}, \dots}%
|
||||
};
|
||||
|
||||
\node[novelbox, minimum width=46mm, minimum height=22mm] (mahal) at (13.7, -5.7) {%
|
||||
\contrib\ \textsc{Mahalanobis-OAS Router}\\[1mm]
|
||||
$D^2 = (s-\mu)^\top \Sigma^{-1}_{\mathrm{OAS}} (s-\mu)$\\[1mm]
|
||||
{\scriptsize\itshape fit on benign val (no labels)}%
|
||||
};
|
||||
|
||||
\node[databox, minimum width=18mm, fill=myblue!10] (out) at (18.5, -5.7) {%
|
||||
AUROC\\
|
||||
$s_{\text{anomaly}}$%
|
||||
};
|
||||
|
||||
\draw[arrow] (testflow) -- (frozen);
|
||||
\draw[arrow] (frozen) -- (svec);
|
||||
\draw[arrow] (svec) -- (mahal);
|
||||
\draw[arrow] (mahal) -- (out);
|
||||
|
||||
% =========================================================================
|
||||
% LEGEND
|
||||
% =========================================================================
|
||||
\node[draw=black!40, rounded corners=2pt, fill=white, inner sep=5pt, align=left,
|
||||
font=\scriptsize, anchor=north west] at (0, -7.3) {%
|
||||
\contrib\ \textbf{Our contributions} \quad
|
||||
\tikz\fill[myblue!25, draw=myblue, line width=0.4pt] (0,0) rectangle (0.22,0.16);
|
||||
\ continuous \quad
|
||||
\tikz\fill[myorange!30, draw=myorange, line width=0.4pt] (0,0) rectangle (0.22,0.16);
|
||||
\ discrete (DFM bits) \quad
|
||||
\tikz\fill[mygray!30, draw=mygray, line width=0.4pt, dash pattern=on 0.4pt off 0.4pt] (0,0) rectangle (0.22,0.16);
|
||||
\ channel-pad (9-d feat in 21-d slot) \quad
|
||||
\tikz\draw[dashed, draw=black!50, line width=0.4pt] (0,0) rectangle (0.22,0.16);
|
||||
\ sequence-pad (whole token, $n<T$)\\[2pt]
|
||||
\contrib\ Discrete-FM head on TCP flag bits + direction \quad
|
||||
\contrib\ Causal-packet attention (protocol-causal prior) \quad
|
||||
\contrib\ Mahalanobis-OAS auto-router (no selection bias)%
|
||||
};
|
||||
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
145
paper/figures/pcap_extraction.tex
Normal file
145
paper/figures/pcap_extraction.tex
Normal file
@@ -0,0 +1,145 @@
|
||||
% Pcap → packet stream → 9-d feature token extraction visual.
|
||||
% Compile: pdflatex pcap_extraction.tex
|
||||
\documentclass[border=8pt]{standalone}
|
||||
\usepackage{tikz}
|
||||
\usetikzlibrary{arrows.meta, positioning, calc, shapes.misc, decorations.pathreplacing}
|
||||
\usepackage{amsmath, amssymb}
|
||||
|
||||
\definecolor{myorange}{RGB}{230, 126, 34}
|
||||
\definecolor{myblue}{RGB}{52, 110, 180}
|
||||
\definecolor{mygreen}{RGB}{46, 139, 87}
|
||||
\definecolor{mypurple}{RGB}{142, 68, 173}
|
||||
\definecolor{mygray}{RGB}{170, 170, 170}
|
||||
|
||||
\begin{document}
|
||||
\begin{tikzpicture}[font=\small, >={Stealth[length=2.5mm]}]
|
||||
|
||||
% ============ 1. PCAP FILE ICON ============
|
||||
\begin{scope}[shift={(0,0)}]
|
||||
% file body with folded corner (drawn as polygon)
|
||||
\fill[myblue!12] (0,0) -- (0,3.0) -- (1.4,3.0) -- (1.9,2.5) -- (1.9,0) -- cycle;
|
||||
\draw[myblue!80, line width=0.7pt]
|
||||
(0,0) -- (0,3.0) -- (1.4,3.0) -- (1.9,2.5) -- (1.9,0) -- cycle;
|
||||
% the folded triangle on top-right corner
|
||||
\fill[myblue!30] (1.4,3.0) -- (1.4,2.5) -- (1.9,2.5) -- cycle;
|
||||
\draw[myblue!80, line width=0.5pt] (1.4,3.0) -- (1.4,2.5) -- (1.9,2.5);
|
||||
% file title
|
||||
\node[font=\bfseries\footnotesize, text=myblue!90!black] at (0.95, 2.1) {flow.pcap};
|
||||
% decorative "byte" lines (deterministic widths)
|
||||
\foreach \y/\rl in {1.65/1.30, 1.40/1.05, 1.15/1.40, 0.90/0.85, 0.65/1.20, 0.40/0.75} {
|
||||
\draw[myblue!45, line width=0.35pt] (0.18, \y) -- (0.18+\rl, \y);
|
||||
}
|
||||
\node[font=\scriptsize, anchor=north] at (0.95, -0.1) {raw bytes};
|
||||
\end{scope}
|
||||
|
||||
% Arrow 1: parse
|
||||
\draw[->, thick, black!60]
|
||||
(2.05, 1.5) -- (3.20, 1.5)
|
||||
node[midway, above, font=\scriptsize\itshape, text=black!70] {dpkt parse};
|
||||
|
||||
% ============ 2. DECODED PACKET STREAM ============
|
||||
\begin{scope}[shift={(3.4, 0)}]
|
||||
% four packets along time axis
|
||||
\foreach \j in {0,...,3} {
|
||||
\pgfmathsetmacro{\xx}{\j * 1.45}
|
||||
% packet frame
|
||||
\draw[fill=white, draw=black!55, line width=0.5pt, rounded corners=1pt]
|
||||
(\xx, 0.5) rectangle (\xx+1.25, 3.05);
|
||||
% size field (myblue)
|
||||
\draw[fill=myblue!35, draw=myblue, line width=0.4pt]
|
||||
(\xx+0.08, 2.78) rectangle (\xx+0.42, 2.93);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.45, 2.86) {size};
|
||||
% IAT field (mygreen)
|
||||
\draw[fill=mygreen!35, draw=mygreen, line width=0.4pt]
|
||||
(\xx+0.08, 2.55) rectangle (\xx+0.42, 2.70);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.45, 2.63) {IAT};
|
||||
% direction (mypurple)
|
||||
\draw[fill=mypurple!35, draw=mypurple, line width=0.4pt]
|
||||
(\xx+0.08, 2.32) rectangle (\xx+0.22, 2.47);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.25, 2.40) {dir};
|
||||
% flag bits row — show a real TCP 3-way handshake + data:
|
||||
% pkt0: SYN (client SYN)
|
||||
% pkt1: SYN+ACK (server SYN-ACK)
|
||||
% pkt2: ACK (client ACK, handshake done)
|
||||
% pkt3: PSH+ACK (data segment)
|
||||
\ifcase\j
|
||||
\def\flagS{1}\def\flagF{0}\def\flagR{0}\def\flagP{0}\def\flagA{0}\or
|
||||
\def\flagS{1}\def\flagF{0}\def\flagR{0}\def\flagP{0}\def\flagA{1}\or
|
||||
\def\flagS{0}\def\flagF{0}\def\flagR{0}\def\flagP{0}\def\flagA{1}\or
|
||||
\def\flagS{0}\def\flagF{0}\def\flagR{0}\def\flagP{1}\def\flagA{1}\fi
|
||||
\foreach \fname/\fval/\k in {S/\flagS/0, F/\flagF/1, R/\flagR/2, P/\flagP/3, A/\flagA/4} {
|
||||
\pgfmathsetmacro{\fx}{\xx + 0.08 + \k*0.22}
|
||||
\ifnum\fval=1
|
||||
\draw[fill=myorange!70, draw=myorange, line width=0.4pt]
|
||||
(\fx, 2.05) rectangle (\fx+0.18, 2.22);
|
||||
\node[font=\tiny, white] at (\fx+0.09, 2.135) {\fname};
|
||||
\else
|
||||
\draw[fill=white, draw=myorange!60, line width=0.4pt]
|
||||
(\fx, 2.05) rectangle (\fx+0.18, 2.22);
|
||||
\node[font=\tiny, text=myorange!80!black] at (\fx+0.09, 2.135) {\fname};
|
||||
\fi
|
||||
}
|
||||
% win
|
||||
\draw[fill=myblue!20, draw=myblue!60, line width=0.4pt]
|
||||
(\xx+0.08, 1.78) rectangle (\xx+0.42, 1.93);
|
||||
\node[font=\tiny, anchor=west] at (\xx+0.45, 1.86) {win};
|
||||
% decorative "payload" content
|
||||
\draw[black!30, line width=0.4pt] (\xx+0.08, 1.45) -- (\xx+1.17, 1.45);
|
||||
\draw[black!30, line width=0.4pt] (\xx+0.08, 1.25) -- (\xx+0.95, 1.25);
|
||||
\draw[black!30, line width=0.4pt] (\xx+0.08, 1.05) -- (\xx+1.10, 1.05);
|
||||
\draw[black!30, line width=0.4pt] (\xx+0.08, 0.85) -- (\xx+0.75, 0.85);
|
||||
% packet label below
|
||||
\node[font=\scriptsize, text=black!70] at (\xx+0.625, 0.32) {pkt$_\j$};
|
||||
}
|
||||
% time axis arrow
|
||||
\draw[->, line width=0.4pt, black!60] (-0.05, 0.05) -- (6.0, 0.05);
|
||||
\node[font=\tiny, anchor=west, text=black!60] at (6.05, 0.05) {time};
|
||||
\end{scope}
|
||||
|
||||
% Arrow 2: extract
|
||||
\draw[->, thick, black!60]
|
||||
(9.5, 1.7) -- (10.7, 1.7)
|
||||
node[midway, above, font=\scriptsize\itshape, text=black!70] {extract};
|
||||
|
||||
% ============ 3. 9-D FEATURE TOKEN ============
|
||||
\begin{scope}[shift={(10.85, -0.5)}]
|
||||
% 9 channels stacked vertically with type-grouped coloring
|
||||
% 0 = continuous (z-scored), 1 = discrete (DFM bit)
|
||||
\foreach \name/\g/\i in {%
|
||||
log\_size/0/0,
|
||||
log\_dt/0/1,
|
||||
direction/1/2,
|
||||
SYN/1/3,
|
||||
FIN/1/4,
|
||||
RST/1/5,
|
||||
PSH/1/6,
|
||||
ACK/1/7,
|
||||
log\_win/0/8} {
|
||||
\pgfmathsetmacro{\yp}{3.5 - \i*0.36}
|
||||
\ifnum\g=0
|
||||
\draw[fill=myblue!25, draw=myblue, line width=0.5pt]
|
||||
(0, \yp) rectangle (1.35, \yp+0.32);
|
||||
\else
|
||||
\draw[fill=myorange!30, draw=myorange, line width=0.5pt]
|
||||
(0, \yp) rectangle (1.35, \yp+0.32);
|
||||
\fi
|
||||
\node[font=\scriptsize] at (0.675, \yp+0.16) {\name};
|
||||
}
|
||||
% brace + label
|
||||
\draw[decorate, decoration={brace, amplitude=4pt}, line width=0.5pt, black!60]
|
||||
(1.45, 3.82) -- (1.45, 0.62)
|
||||
node[midway, right=4pt, font=\scriptsize, align=left, text=black!70]
|
||||
{9-d packet token\\[1pt] {[}cont (3) + disc (6){]}};
|
||||
\end{scope}
|
||||
|
||||
% Bottom legend
|
||||
\node[draw=black!30, rounded corners=2pt, fill=white, inner sep=4pt, font=\scriptsize,
|
||||
align=left, anchor=north]
|
||||
at (7.5, -1.1) {%
|
||||
\tikz\fill[myblue!25, draw=myblue, line width=0.4pt] (0,0) rectangle (0.25,0.18);
|
||||
\ continuous (z-scored) \quad
|
||||
\tikz\fill[myorange!30, draw=myorange, line width=0.4pt] (0,0) rectangle (0.25,0.18);
|
||||
\ discrete (kept as bits, fed to DFM head)};
|
||||
|
||||
\end{tikzpicture}
|
||||
\end{document}
|
||||
27
paper/figures/tensors/01_h_in.svg
Normal file
27
paper/figures/tensors/01_h_in.svg
Normal file
@@ -0,0 +1,27 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="306" height="206" viewBox="0 0 306 206">
|
||||
<rect width="306" height="206" fill="white"/>
|
||||
<rect x="3" y="3" width="50" height="50" fill="#C2D4E8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="3" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="3" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="3" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="3" width="50" height="50" fill="#487CBC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="3" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="53" width="50" height="50" fill="#719ACA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="53" width="50" height="50" fill="#D6E2F0" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="53" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="53" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="53" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="53" width="50" height="50" fill="#487CBC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="103" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="103" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="103" width="50" height="50" fill="#487CBC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="103" width="50" height="50" fill="#C2D4E8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="103" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="103" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="153" width="50" height="50" fill="#719ACA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="153" width="50" height="50" fill="#C2D4E8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="153" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="153" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="153" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="153" width="50" height="50" fill="#719ACA" stroke="#999" stroke-width="0.6"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
27
paper/figures/tensors/02_h_after_LN.svg
Normal file
27
paper/figures/tensors/02_h_after_LN.svg
Normal file
@@ -0,0 +1,27 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="306" height="206" viewBox="0 0 306 206">
|
||||
<rect width="306" height="206" fill="white"/>
|
||||
<rect x="3" y="3" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="3" width="50" height="50" fill="#C2D4E8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="3" width="50" height="50" fill="#719ACA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="3" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="3" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="3" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="53" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="53" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="53" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="53" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="53" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="53" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="103" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="103" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="103" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="103" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="103" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="103" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="153" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="153" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="153" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="153" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="153" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="153" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
27
paper/figures/tensors/03_h_after_gamma.svg
Normal file
27
paper/figures/tensors/03_h_after_gamma.svg
Normal file
@@ -0,0 +1,27 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="306" height="206" viewBox="0 0 306 206">
|
||||
<rect width="306" height="206" fill="white"/>
|
||||
<rect x="3" y="3" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="3" width="50" height="50" fill="#E7EEF6" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="3" width="50" height="50" fill="#719ACA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="3" width="50" height="50" fill="#CCDBEC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="3" width="50" height="50" fill="#95B4D8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="3" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="53" width="50" height="50" fill="#9EB9DB" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="53" width="50" height="50" fill="#CEDCED" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="53" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="53" width="50" height="50" fill="#CCDBEC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="53" width="50" height="50" fill="#95B4D8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="53" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="103" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="103" width="50" height="50" fill="#D6E2F0" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="103" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="103" width="50" height="50" fill="#D6E2F0" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="103" width="50" height="50" fill="#618EC4" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="103" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="153" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="153" width="50" height="50" fill="#DFE8F3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="153" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="153" width="50" height="50" fill="#C2D4E8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="153" width="50" height="50" fill="#95B4D8" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="153" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
27
paper/figures/tensors/04_h_after_beta.svg
Normal file
27
paper/figures/tensors/04_h_after_beta.svg
Normal file
@@ -0,0 +1,27 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="306" height="206" viewBox="0 0 306 206">
|
||||
<rect width="306" height="206" fill="white"/>
|
||||
<rect x="3" y="3" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="3" width="50" height="50" fill="#BED1E7" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="3" width="50" height="50" fill="#487CBC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="3" width="50" height="50" fill="#A4BEDD" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="3" width="50" height="50" fill="#6D97C9" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="3" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="53" width="50" height="50" fill="#759CCC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="53" width="50" height="50" fill="#A6BFDE" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="53" width="50" height="50" fill="#719ACA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="53" width="50" height="50" fill="#A4BEDD" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="53" width="50" height="50" fill="#6D97C9" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="53" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="103" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="103" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="103" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="103" width="50" height="50" fill="#AEC5E1" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="103" width="50" height="50" fill="#487CBC" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="103" width="50" height="50" fill="#85A8D2" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="3" y="153" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="53" y="153" width="50" height="50" fill="#B6CBE4" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="103" y="153" width="50" height="50" fill="#719ACA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="153" y="153" width="50" height="50" fill="#9AB6DA" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="203" y="153" width="50" height="50" fill="#6D97C9" stroke="#999" stroke-width="0.6"/>
|
||||
<rect x="253" y="153" width="50" height="50" fill="#5D8BC3" stroke="#999" stroke-width="0.6"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.4 KiB |
147
paper/figures/tensors/05_attention_heatmap.svg
Normal file
147
paper/figures/tensors/05_attention_heatmap.svg
Normal file
@@ -0,0 +1,147 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="394" height="394" viewBox="0 0 394 394">
|
||||
<rect width="394" height="394" fill="white"/>
|
||||
<rect x="6" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="70" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="102" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="134" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="166" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="198" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="230" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="262" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="294" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="326" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="358" y="6" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="6" y="38" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="38" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="102" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="134" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="166" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="198" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="230" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="262" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="294" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="38" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="70" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="70" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="70" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="134" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="166" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="198" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="230" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="262" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="294" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="70" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="102" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="102" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="102" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="102" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="166" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="198" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="230" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="262" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="294" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="102" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="134" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="134" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="134" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="134" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="134" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="134" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="198" y="134" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="230" y="134" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="262" y="134" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="294" y="134" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="134" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="134" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="166" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="166" width="30" height="30" rx="4" ry="4" fill="#F0B37D" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="166" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="166" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="166" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="166" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="198" y="166" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="230" y="166" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="262" y="166" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="294" y="166" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="166" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="166" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="198" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="198" width="30" height="30" rx="4" ry="4" fill="#F1B885" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="198" width="30" height="30" rx="4" ry="4" fill="#F0B37D" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="198" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="198" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="198" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="198" y="198" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="230" y="198" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="262" y="198" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="294" y="198" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="198" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="198" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="230" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="230" width="30" height="30" rx="4" ry="4" fill="#F2BD8E" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="230" width="30" height="30" rx="4" ry="4" fill="#F1B885" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="230" width="30" height="30" rx="4" ry="4" fill="#F0B37D" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="230" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="230" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="198" y="230" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="230" y="230" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="262" y="230" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="294" y="230" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="230" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="230" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="262" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="262" width="30" height="30" rx="4" ry="4" fill="#F3C297" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="262" width="30" height="30" rx="4" ry="4" fill="#F2BD8E" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="262" width="30" height="30" rx="4" ry="4" fill="#F1B885" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="262" width="30" height="30" rx="4" ry="4" fill="#F0B37D" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="262" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="198" y="262" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="230" y="262" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="262" y="262" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="294" y="262" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="326" y="262" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="262" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="294" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="294" width="30" height="30" rx="4" ry="4" fill="#F4C8A0" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="294" width="30" height="30" rx="4" ry="4" fill="#F3C297" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="294" width="30" height="30" rx="4" ry="4" fill="#F2BD8E" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="294" width="30" height="30" rx="4" ry="4" fill="#F1B885" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="294" width="30" height="30" rx="4" ry="4" fill="#F0B37D" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="198" y="294" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="230" y="294" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="262" y="294" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="294" y="294" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="326" y="294" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="358" y="294" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="326" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="326" width="30" height="30" rx="4" ry="4" fill="#F5CDA9" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="326" width="30" height="30" rx="4" ry="4" fill="#F4C8A0" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="326" width="30" height="30" rx="4" ry="4" fill="#F3C297" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="326" width="30" height="30" rx="4" ry="4" fill="#F2BD8E" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="326" width="30" height="30" rx="4" ry="4" fill="#F1B885" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="198" y="326" width="30" height="30" rx="4" ry="4" fill="#F0B37D" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="230" y="326" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="262" y="326" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="294" y="326" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="326" y="326" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="358" y="326" width="30" height="30" rx="4" ry="4" fill="#FFFFFF" stroke="#BBBBBB" stroke-width="0.5"/>
|
||||
<rect x="6" y="358" width="30" height="30" rx="4" ry="4" fill="#EA9143" stroke="#B8651F" stroke-width="0.6"/>
|
||||
<rect x="38" y="358" width="30" height="30" rx="4" ry="4" fill="#F6D2B2" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="70" y="358" width="30" height="30" rx="4" ry="4" fill="#F5CDA9" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="102" y="358" width="30" height="30" rx="4" ry="4" fill="#F4C8A0" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="134" y="358" width="30" height="30" rx="4" ry="4" fill="#F3C297" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="166" y="358" width="30" height="30" rx="4" ry="4" fill="#F2BD8E" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="198" y="358" width="30" height="30" rx="4" ry="4" fill="#F1B885" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="230" y="358" width="30" height="30" rx="4" ry="4" fill="#F0B37D" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="262" y="358" width="30" height="30" rx="4" ry="4" fill="#EFAE74" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="294" y="358" width="30" height="30" rx="4" ry="4" fill="#EEA96B" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="326" y="358" width="30" height="30" rx="4" ry="4" fill="#EDA362" stroke="#999999" stroke-width="0.3"/>
|
||||
<rect x="358" y="358" width="30" height="30" rx="4" ry="4" fill="#EC9E59" stroke="#999999" stroke-width="0.3"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 16 KiB |
146
paper/figures/tensors/06_attn_raw.svg
Normal file
146
paper/figures/tensors/06_attn_raw.svg
Normal file
@@ -0,0 +1,146 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="629" height="384" viewBox="0 0 629 384">
|
||||
<polygon points="314.98,24.00 339.23,38.00 314.98,52.00 290.74,38.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,38.00 363.48,52.00 339.23,66.00 314.98,52.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,52.00 387.73,66.00 363.48,80.00 339.23,66.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,66.00 411.98,80.00 387.73,94.00 363.48,80.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,80.00 436.23,94.00 411.98,108.00 387.73,94.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,94.00 460.48,108.00 436.23,122.00 411.98,108.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="460.48,108.00 484.73,122.00 460.48,136.00 436.23,122.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="484.73,122.00 508.97,136.00 484.73,150.00 460.48,136.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="508.97,136.00 533.22,150.00 508.97,164.00 484.73,150.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="533.22,150.00 557.47,164.00 533.22,178.00 508.97,164.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="557.47,164.00 581.72,178.00 557.47,192.00 533.22,178.00" fill="#F6D2B2" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="581.72,178.00 605.97,192.00 581.72,206.00 557.47,192.00" fill="#F7D7BA" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,38.00 314.98,52.00 290.74,66.00 266.49,52.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,52.00 339.23,66.00 314.98,80.00 290.74,66.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,66.00 363.48,80.00 339.23,94.00 314.98,80.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,80.00 387.73,94.00 363.48,108.00 339.23,94.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,94.00 411.98,108.00 387.73,122.00 363.48,108.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,108.00 436.23,122.00 411.98,136.00 387.73,122.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,122.00 460.48,136.00 436.23,150.00 411.98,136.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="460.48,136.00 484.73,150.00 460.48,164.00 436.23,150.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="484.73,150.00 508.97,164.00 484.73,178.00 460.48,164.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="508.97,164.00 533.22,178.00 508.97,192.00 484.73,178.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="533.22,178.00 557.47,192.00 533.22,206.00 508.97,192.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="557.47,192.00 581.72,206.00 557.47,220.00 533.22,206.00" fill="#F6D2B2" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,52.00 290.74,66.00 266.49,80.00 242.24,66.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,66.00 314.98,80.00 290.74,94.00 266.49,80.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,80.00 339.23,94.00 314.98,108.00 290.74,94.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,94.00 363.48,108.00 339.23,122.00 314.98,108.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,108.00 387.73,122.00 363.48,136.00 339.23,122.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,122.00 411.98,136.00 387.73,150.00 363.48,136.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,136.00 436.23,150.00 411.98,164.00 387.73,150.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,150.00 460.48,164.00 436.23,178.00 411.98,164.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="460.48,164.00 484.73,178.00 460.48,192.00 436.23,178.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="484.73,178.00 508.97,192.00 484.73,206.00 460.48,192.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="508.97,192.00 533.22,206.00 508.97,220.00 484.73,206.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="533.22,206.00 557.47,220.00 533.22,234.00 508.97,220.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,66.00 266.49,80.00 242.24,94.00 217.99,80.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,80.00 290.74,94.00 266.49,108.00 242.24,94.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,94.00 314.98,108.00 290.74,122.00 266.49,108.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,108.00 339.23,122.00 314.98,136.00 290.74,122.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,122.00 363.48,136.00 339.23,150.00 314.98,136.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,136.00 387.73,150.00 363.48,164.00 339.23,150.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,150.00 411.98,164.00 387.73,178.00 363.48,164.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,164.00 436.23,178.00 411.98,192.00 387.73,178.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,178.00 460.48,192.00 436.23,206.00 411.98,192.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="460.48,192.00 484.73,206.00 460.48,220.00 436.23,206.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="484.73,206.00 508.97,220.00 484.73,234.00 460.48,220.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="508.97,220.00 533.22,234.00 508.97,248.00 484.73,234.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,80.00 242.24,94.00 217.99,108.00 193.74,94.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,94.00 266.49,108.00 242.24,122.00 217.99,108.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,108.00 290.74,122.00 266.49,136.00 242.24,122.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,122.00 314.98,136.00 290.74,150.00 266.49,136.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,136.00 339.23,150.00 314.98,164.00 290.74,150.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,150.00 363.48,164.00 339.23,178.00 314.98,164.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,164.00 387.73,178.00 363.48,192.00 339.23,178.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,178.00 411.98,192.00 387.73,206.00 363.48,192.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,192.00 436.23,206.00 411.98,220.00 387.73,206.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,206.00 460.48,220.00 436.23,234.00 411.98,220.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="460.48,220.00 484.73,234.00 460.48,248.00 436.23,234.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="484.73,234.00 508.97,248.00 484.73,262.00 460.48,248.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,94.00 217.99,108.00 193.74,122.00 169.49,108.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,108.00 242.24,122.00 217.99,136.00 193.74,122.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,122.00 266.49,136.00 242.24,150.00 217.99,136.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,136.00 290.74,150.00 266.49,164.00 242.24,150.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,150.00 314.98,164.00 290.74,178.00 266.49,164.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,164.00 339.23,178.00 314.98,192.00 290.74,178.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,178.00 363.48,192.00 339.23,206.00 314.98,192.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,192.00 387.73,206.00 363.48,220.00 339.23,206.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,206.00 411.98,220.00 387.73,234.00 363.48,220.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,220.00 436.23,234.00 411.98,248.00 387.73,234.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,234.00 460.48,248.00 436.23,262.00 411.98,248.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="460.48,248.00 484.73,262.00 460.48,276.00 436.23,262.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,108.00 193.74,122.00 169.49,136.00 145.24,122.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,122.00 217.99,136.00 193.74,150.00 169.49,136.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,136.00 242.24,150.00 217.99,164.00 193.74,150.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,150.00 266.49,164.00 242.24,178.00 217.99,164.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,164.00 290.74,178.00 266.49,192.00 242.24,178.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,178.00 314.98,192.00 290.74,206.00 266.49,192.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,192.00 339.23,206.00 314.98,220.00 290.74,206.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,206.00 363.48,220.00 339.23,234.00 314.98,220.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,220.00 387.73,234.00 363.48,248.00 339.23,234.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,234.00 411.98,248.00 387.73,262.00 363.48,248.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,248.00 436.23,262.00 411.98,276.00 387.73,262.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,262.00 460.48,276.00 436.23,290.00 411.98,276.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,122.00 169.49,136.00 145.24,150.00 120.99,136.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,136.00 193.74,150.00 169.49,164.00 145.24,150.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,150.00 217.99,164.00 193.74,178.00 169.49,164.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,164.00 242.24,178.00 217.99,192.00 193.74,178.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,178.00 266.49,192.00 242.24,206.00 217.99,192.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,192.00 290.74,206.00 266.49,220.00 242.24,206.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,206.00 314.98,220.00 290.74,234.00 266.49,220.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,220.00 339.23,234.00 314.98,248.00 290.74,234.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,234.00 363.48,248.00 339.23,262.00 314.98,248.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,248.00 387.73,262.00 363.48,276.00 339.23,262.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,262.00 411.98,276.00 387.73,290.00 363.48,276.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,276.00 436.23,290.00 411.98,304.00 387.73,290.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="120.99,136.00 145.24,150.00 120.99,164.00 96.75,150.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,150.00 169.49,164.00 145.24,178.00 120.99,164.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,164.00 193.74,178.00 169.49,192.00 145.24,178.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,178.00 217.99,192.00 193.74,206.00 169.49,192.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,192.00 242.24,206.00 217.99,220.00 193.74,206.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,206.00 266.49,220.00 242.24,234.00 217.99,220.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,220.00 290.74,234.00 266.49,248.00 242.24,234.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,234.00 314.98,248.00 290.74,262.00 266.49,248.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,248.00 339.23,262.00 314.98,276.00 290.74,262.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,262.00 363.48,276.00 339.23,290.00 314.98,276.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,276.00 387.73,290.00 363.48,304.00 339.23,290.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,290.00 411.98,304.00 387.73,318.00 363.48,304.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="96.75,150.00 120.99,164.00 96.75,178.00 72.50,164.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="120.99,164.00 145.24,178.00 120.99,192.00 96.75,178.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,178.00 169.49,192.00 145.24,206.00 120.99,192.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,192.00 193.74,206.00 169.49,220.00 145.24,206.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,206.00 217.99,220.00 193.74,234.00 169.49,220.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,220.00 242.24,234.00 217.99,248.00 193.74,234.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,234.00 266.49,248.00 242.24,262.00 217.99,248.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,248.00 290.74,262.00 266.49,276.00 242.24,262.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,262.00 314.98,276.00 290.74,290.00 266.49,276.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,276.00 339.23,290.00 314.98,304.00 290.74,290.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,290.00 363.48,304.00 339.23,318.00 314.98,304.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,304.00 387.73,318.00 363.48,332.00 339.23,318.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="72.50,164.00 96.75,178.00 72.50,192.00 48.25,178.00" fill="#F6D2B2" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="96.75,178.00 120.99,192.00 96.75,206.00 72.50,192.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="120.99,192.00 145.24,206.00 120.99,220.00 96.75,206.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,206.00 169.49,220.00 145.24,234.00 120.99,220.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,220.00 193.74,234.00 169.49,248.00 145.24,234.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,234.00 217.99,248.00 193.74,262.00 169.49,248.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,248.00 242.24,262.00 217.99,276.00 193.74,262.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,262.00 266.49,276.00 242.24,290.00 217.99,276.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,276.00 290.74,290.00 266.49,304.00 242.24,290.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,290.00 314.98,304.00 290.74,318.00 266.49,304.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,304.00 339.23,318.00 314.98,332.00 290.74,318.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,318.00 363.48,332.00 339.23,346.00 314.98,332.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="48.25,178.00 72.50,192.00 48.25,206.00 24.00,192.00" fill="#F7D7BA" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="72.50,192.00 96.75,206.00 72.50,220.00 48.25,206.00" fill="#F6D2B2" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="96.75,206.00 120.99,220.00 96.75,234.00 72.50,220.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="120.99,220.00 145.24,234.00 120.99,248.00 96.75,234.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,234.00 169.49,248.00 145.24,262.00 120.99,248.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,248.00 193.74,262.00 169.49,276.00 145.24,262.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,262.00 217.99,276.00 193.74,290.00 169.49,276.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,276.00 242.24,290.00 217.99,304.00 193.74,290.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,290.00 266.49,304.00 242.24,318.00 217.99,304.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,304.00 290.74,318.00 266.49,332.00 242.24,318.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,318.00 314.98,332.00 290.74,346.00 266.49,332.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,332.00 339.23,346.00 314.98,360.00 290.74,346.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 18 KiB |
146
paper/figures/tensors/07_attn_mask.svg
Normal file
146
paper/figures/tensors/07_attn_mask.svg
Normal file
@@ -0,0 +1,146 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="629" height="384" viewBox="0 0 629 384">
|
||||
<polygon points="314.98,24.00 339.23,38.00 314.98,52.00 290.74,38.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,38.00 363.48,52.00 339.23,66.00 314.98,52.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="363.48,52.00 387.73,66.00 363.48,80.00 339.23,66.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="387.73,66.00 411.98,80.00 387.73,94.00 363.48,80.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="411.98,80.00 436.23,94.00 411.98,108.00 387.73,94.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="436.23,94.00 460.48,108.00 436.23,122.00 411.98,108.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="460.48,108.00 484.73,122.00 460.48,136.00 436.23,122.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="484.73,122.00 508.97,136.00 484.73,150.00 460.48,136.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="508.97,136.00 533.22,150.00 508.97,164.00 484.73,150.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="533.22,150.00 557.47,164.00 533.22,178.00 508.97,164.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="557.47,164.00 581.72,178.00 557.47,192.00 533.22,178.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="581.72,178.00 605.97,192.00 581.72,206.00 557.47,192.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,38.00 314.98,52.00 290.74,66.00 266.49,52.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,52.00 339.23,66.00 314.98,80.00 290.74,66.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,66.00 363.48,80.00 339.23,94.00 314.98,80.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,80.00 387.73,94.00 363.48,108.00 339.23,94.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,94.00 411.98,108.00 387.73,122.00 363.48,108.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="411.98,108.00 436.23,122.00 411.98,136.00 387.73,122.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="436.23,122.00 460.48,136.00 436.23,150.00 411.98,136.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="460.48,136.00 484.73,150.00 460.48,164.00 436.23,150.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="484.73,150.00 508.97,164.00 484.73,178.00 460.48,164.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="508.97,164.00 533.22,178.00 508.97,192.00 484.73,178.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="533.22,178.00 557.47,192.00 533.22,206.00 508.97,192.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="557.47,192.00 581.72,206.00 557.47,220.00 533.22,206.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="266.49,52.00 290.74,66.00 266.49,80.00 242.24,66.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,66.00 314.98,80.00 290.74,94.00 266.49,80.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,80.00 339.23,94.00 314.98,108.00 290.74,94.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,94.00 363.48,108.00 339.23,122.00 314.98,108.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,108.00 387.73,122.00 363.48,136.00 339.23,122.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,122.00 411.98,136.00 387.73,150.00 363.48,136.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="411.98,136.00 436.23,150.00 411.98,164.00 387.73,150.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="436.23,150.00 460.48,164.00 436.23,178.00 411.98,164.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="460.48,164.00 484.73,178.00 460.48,192.00 436.23,178.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="484.73,178.00 508.97,192.00 484.73,206.00 460.48,192.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="508.97,192.00 533.22,206.00 508.97,220.00 484.73,206.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="533.22,206.00 557.47,220.00 533.22,234.00 508.97,220.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="242.24,66.00 266.49,80.00 242.24,94.00 217.99,80.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,80.00 290.74,94.00 266.49,108.00 242.24,94.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,94.00 314.98,108.00 290.74,122.00 266.49,108.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,108.00 339.23,122.00 314.98,136.00 290.74,122.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,122.00 363.48,136.00 339.23,150.00 314.98,136.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,136.00 387.73,150.00 363.48,164.00 339.23,150.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,150.00 411.98,164.00 387.73,178.00 363.48,164.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="411.98,164.00 436.23,178.00 411.98,192.00 387.73,178.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="436.23,178.00 460.48,192.00 436.23,206.00 411.98,192.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="460.48,192.00 484.73,206.00 460.48,220.00 436.23,206.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="484.73,206.00 508.97,220.00 484.73,234.00 460.48,220.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="508.97,220.00 533.22,234.00 508.97,248.00 484.73,234.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="217.99,80.00 242.24,94.00 217.99,108.00 193.74,94.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,94.00 266.49,108.00 242.24,122.00 217.99,108.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,108.00 290.74,122.00 266.49,136.00 242.24,122.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,122.00 314.98,136.00 290.74,150.00 266.49,136.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,136.00 339.23,150.00 314.98,164.00 290.74,150.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,150.00 363.48,164.00 339.23,178.00 314.98,164.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,164.00 387.73,178.00 363.48,192.00 339.23,178.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,178.00 411.98,192.00 387.73,206.00 363.48,192.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="411.98,192.00 436.23,206.00 411.98,220.00 387.73,206.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="436.23,206.00 460.48,220.00 436.23,234.00 411.98,220.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="460.48,220.00 484.73,234.00 460.48,248.00 436.23,234.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="484.73,234.00 508.97,248.00 484.73,262.00 460.48,248.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="193.74,94.00 217.99,108.00 193.74,122.00 169.49,108.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="217.99,108.00 242.24,122.00 217.99,136.00 193.74,122.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,122.00 266.49,136.00 242.24,150.00 217.99,136.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,136.00 290.74,150.00 266.49,164.00 242.24,150.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,150.00 314.98,164.00 290.74,178.00 266.49,164.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,164.00 339.23,178.00 314.98,192.00 290.74,178.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,178.00 363.48,192.00 339.23,206.00 314.98,192.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,192.00 387.73,206.00 363.48,220.00 339.23,206.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,206.00 411.98,220.00 387.73,234.00 363.48,220.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="411.98,220.00 436.23,234.00 411.98,248.00 387.73,234.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="436.23,234.00 460.48,248.00 436.23,262.00 411.98,248.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="460.48,248.00 484.73,262.00 460.48,276.00 436.23,262.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="169.49,108.00 193.74,122.00 169.49,136.00 145.24,122.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="193.74,122.00 217.99,136.00 193.74,150.00 169.49,136.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="217.99,136.00 242.24,150.00 217.99,164.00 193.74,150.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,150.00 266.49,164.00 242.24,178.00 217.99,164.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,164.00 290.74,178.00 266.49,192.00 242.24,178.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,178.00 314.98,192.00 290.74,206.00 266.49,192.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,192.00 339.23,206.00 314.98,220.00 290.74,206.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,206.00 363.48,220.00 339.23,234.00 314.98,220.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,220.00 387.73,234.00 363.48,248.00 339.23,234.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,234.00 411.98,248.00 387.73,262.00 363.48,248.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="411.98,248.00 436.23,262.00 411.98,276.00 387.73,262.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="436.23,262.00 460.48,276.00 436.23,290.00 411.98,276.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="145.24,122.00 169.49,136.00 145.24,150.00 120.99,136.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="169.49,136.00 193.74,150.00 169.49,164.00 145.24,150.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="193.74,150.00 217.99,164.00 193.74,178.00 169.49,164.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="217.99,164.00 242.24,178.00 217.99,192.00 193.74,178.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,178.00 266.49,192.00 242.24,206.00 217.99,192.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,192.00 290.74,206.00 266.49,220.00 242.24,206.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,206.00 314.98,220.00 290.74,234.00 266.49,220.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,220.00 339.23,234.00 314.98,248.00 290.74,234.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,234.00 363.48,248.00 339.23,262.00 314.98,248.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,248.00 387.73,262.00 363.48,276.00 339.23,262.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,262.00 411.98,276.00 387.73,290.00 363.48,276.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="411.98,276.00 436.23,290.00 411.98,304.00 387.73,290.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="120.99,136.00 145.24,150.00 120.99,164.00 96.75,150.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="145.24,150.00 169.49,164.00 145.24,178.00 120.99,164.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="169.49,164.00 193.74,178.00 169.49,192.00 145.24,178.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="193.74,178.00 217.99,192.00 193.74,206.00 169.49,192.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="217.99,192.00 242.24,206.00 217.99,220.00 193.74,206.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,206.00 266.49,220.00 242.24,234.00 217.99,220.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,220.00 290.74,234.00 266.49,248.00 242.24,234.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,234.00 314.98,248.00 290.74,262.00 266.49,248.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,248.00 339.23,262.00 314.98,276.00 290.74,262.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,262.00 363.48,276.00 339.23,290.00 314.98,276.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,276.00 387.73,290.00 363.48,304.00 339.23,290.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="387.73,290.00 411.98,304.00 387.73,318.00 363.48,304.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="96.75,150.00 120.99,164.00 96.75,178.00 72.50,164.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="120.99,164.00 145.24,178.00 120.99,192.00 96.75,178.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="145.24,178.00 169.49,192.00 145.24,206.00 120.99,192.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="169.49,192.00 193.74,206.00 169.49,220.00 145.24,206.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="193.74,206.00 217.99,220.00 193.74,234.00 169.49,220.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="217.99,220.00 242.24,234.00 217.99,248.00 193.74,234.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,234.00 266.49,248.00 242.24,262.00 217.99,248.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,248.00 290.74,262.00 266.49,276.00 242.24,262.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,262.00 314.98,276.00 290.74,290.00 266.49,276.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,276.00 339.23,290.00 314.98,304.00 290.74,290.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,290.00 363.48,304.00 339.23,318.00 314.98,304.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="363.48,304.00 387.73,318.00 363.48,332.00 339.23,318.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="72.50,164.00 96.75,178.00 72.50,192.00 48.25,178.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="96.75,178.00 120.99,192.00 96.75,206.00 72.50,192.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="120.99,192.00 145.24,206.00 120.99,220.00 96.75,206.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="145.24,206.00 169.49,220.00 145.24,234.00 120.99,220.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="169.49,220.00 193.74,234.00 169.49,248.00 145.24,234.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="193.74,234.00 217.99,248.00 193.74,262.00 169.49,248.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="217.99,248.00 242.24,262.00 217.99,276.00 193.74,262.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,262.00 266.49,276.00 242.24,290.00 217.99,276.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,276.00 290.74,290.00 266.49,304.00 242.24,290.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,290.00 314.98,304.00 290.74,318.00 266.49,304.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,304.00 339.23,318.00 314.98,332.00 290.74,318.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="339.23,318.00 363.48,332.00 339.23,346.00 314.98,332.00" fill="#3A3A3A" stroke="#1F1F1F" stroke-width="0.5"/>
|
||||
<polygon points="48.25,178.00 72.50,192.00 48.25,206.00 24.00,192.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="72.50,192.00 96.75,206.00 72.50,220.00 48.25,206.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="96.75,206.00 120.99,220.00 96.75,234.00 72.50,220.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="120.99,220.00 145.24,234.00 120.99,248.00 96.75,234.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="145.24,234.00 169.49,248.00 145.24,262.00 120.99,248.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="169.49,248.00 193.74,262.00 169.49,276.00 145.24,262.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="193.74,262.00 217.99,276.00 193.74,290.00 169.49,276.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="217.99,276.00 242.24,290.00 217.99,304.00 193.74,290.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="242.24,290.00 266.49,304.00 242.24,318.00 217.99,304.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="266.49,304.00 290.74,318.00 266.49,332.00 242.24,318.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="290.74,318.00 314.98,332.00 290.74,346.00 266.49,332.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
<polygon points="314.98,332.00 339.23,346.00 314.98,360.00 290.74,346.00" fill="#F0B27A" stroke="#B8651F" stroke-width="0.4"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 18 KiB |
146
paper/figures/tensors/08_attn_masked.svg
Normal file
146
paper/figures/tensors/08_attn_masked.svg
Normal file
@@ -0,0 +1,146 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="629" height="384" viewBox="0 0 629 384">
|
||||
<polygon points="314.98,24.00 339.23,38.00 314.98,52.00 290.74,38.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,38.00 363.48,52.00 339.23,66.00 314.98,52.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="363.48,52.00 387.73,66.00 363.48,80.00 339.23,66.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="387.73,66.00 411.98,80.00 387.73,94.00 363.48,80.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="411.98,80.00 436.23,94.00 411.98,108.00 387.73,94.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="436.23,94.00 460.48,108.00 436.23,122.00 411.98,108.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="460.48,108.00 484.73,122.00 460.48,136.00 436.23,122.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="484.73,122.00 508.97,136.00 484.73,150.00 460.48,136.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="508.97,136.00 533.22,150.00 508.97,164.00 484.73,150.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="533.22,150.00 557.47,164.00 533.22,178.00 508.97,164.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="557.47,164.00 581.72,178.00 557.47,192.00 533.22,178.00" fill="#F6D2B2" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="581.72,178.00 605.97,192.00 581.72,206.00 557.47,192.00" fill="#F7D7BA" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,38.00 314.98,52.00 290.74,66.00 266.49,52.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,52.00 339.23,66.00 314.98,80.00 290.74,66.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,66.00 363.48,80.00 339.23,94.00 314.98,80.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,80.00 387.73,94.00 363.48,108.00 339.23,94.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,94.00 411.98,108.00 387.73,122.00 363.48,108.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="411.98,108.00 436.23,122.00 411.98,136.00 387.73,122.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="436.23,122.00 460.48,136.00 436.23,150.00 411.98,136.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="460.48,136.00 484.73,150.00 460.48,164.00 436.23,150.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="484.73,150.00 508.97,164.00 484.73,178.00 460.48,164.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="508.97,164.00 533.22,178.00 508.97,192.00 484.73,178.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="533.22,178.00 557.47,192.00 533.22,206.00 508.97,192.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="557.47,192.00 581.72,206.00 557.47,220.00 533.22,206.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="266.49,52.00 290.74,66.00 266.49,80.00 242.24,66.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,66.00 314.98,80.00 290.74,94.00 266.49,80.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,80.00 339.23,94.00 314.98,108.00 290.74,94.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,94.00 363.48,108.00 339.23,122.00 314.98,108.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,108.00 387.73,122.00 363.48,136.00 339.23,122.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,122.00 411.98,136.00 387.73,150.00 363.48,136.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="411.98,136.00 436.23,150.00 411.98,164.00 387.73,150.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="436.23,150.00 460.48,164.00 436.23,178.00 411.98,164.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="460.48,164.00 484.73,178.00 460.48,192.00 436.23,178.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="484.73,178.00 508.97,192.00 484.73,206.00 460.48,192.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="508.97,192.00 533.22,206.00 508.97,220.00 484.73,206.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="533.22,206.00 557.47,220.00 533.22,234.00 508.97,220.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="242.24,66.00 266.49,80.00 242.24,94.00 217.99,80.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,80.00 290.74,94.00 266.49,108.00 242.24,94.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,94.00 314.98,108.00 290.74,122.00 266.49,108.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,108.00 339.23,122.00 314.98,136.00 290.74,122.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,122.00 363.48,136.00 339.23,150.00 314.98,136.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,136.00 387.73,150.00 363.48,164.00 339.23,150.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,150.00 411.98,164.00 387.73,178.00 363.48,164.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="411.98,164.00 436.23,178.00 411.98,192.00 387.73,178.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="436.23,178.00 460.48,192.00 436.23,206.00 411.98,192.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="460.48,192.00 484.73,206.00 460.48,220.00 436.23,206.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="484.73,206.00 508.97,220.00 484.73,234.00 460.48,220.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="508.97,220.00 533.22,234.00 508.97,248.00 484.73,234.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="217.99,80.00 242.24,94.00 217.99,108.00 193.74,94.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,94.00 266.49,108.00 242.24,122.00 217.99,108.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,108.00 290.74,122.00 266.49,136.00 242.24,122.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,122.00 314.98,136.00 290.74,150.00 266.49,136.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,136.00 339.23,150.00 314.98,164.00 290.74,150.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,150.00 363.48,164.00 339.23,178.00 314.98,164.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,164.00 387.73,178.00 363.48,192.00 339.23,178.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,178.00 411.98,192.00 387.73,206.00 363.48,192.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="411.98,192.00 436.23,206.00 411.98,220.00 387.73,206.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="436.23,206.00 460.48,220.00 436.23,234.00 411.98,220.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="460.48,220.00 484.73,234.00 460.48,248.00 436.23,234.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="484.73,234.00 508.97,248.00 484.73,262.00 460.48,248.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="193.74,94.00 217.99,108.00 193.74,122.00 169.49,108.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,108.00 242.24,122.00 217.99,136.00 193.74,122.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,122.00 266.49,136.00 242.24,150.00 217.99,136.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,136.00 290.74,150.00 266.49,164.00 242.24,150.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,150.00 314.98,164.00 290.74,178.00 266.49,164.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,164.00 339.23,178.00 314.98,192.00 290.74,178.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,178.00 363.48,192.00 339.23,206.00 314.98,192.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,192.00 387.73,206.00 363.48,220.00 339.23,206.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,206.00 411.98,220.00 387.73,234.00 363.48,220.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="411.98,220.00 436.23,234.00 411.98,248.00 387.73,234.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="436.23,234.00 460.48,248.00 436.23,262.00 411.98,248.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="460.48,248.00 484.73,262.00 460.48,276.00 436.23,262.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="169.49,108.00 193.74,122.00 169.49,136.00 145.24,122.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,122.00 217.99,136.00 193.74,150.00 169.49,136.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,136.00 242.24,150.00 217.99,164.00 193.74,150.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,150.00 266.49,164.00 242.24,178.00 217.99,164.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,164.00 290.74,178.00 266.49,192.00 242.24,178.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,178.00 314.98,192.00 290.74,206.00 266.49,192.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,192.00 339.23,206.00 314.98,220.00 290.74,206.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,206.00 363.48,220.00 339.23,234.00 314.98,220.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,220.00 387.73,234.00 363.48,248.00 339.23,234.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,234.00 411.98,248.00 387.73,262.00 363.48,248.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="411.98,248.00 436.23,262.00 411.98,276.00 387.73,262.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="436.23,262.00 460.48,276.00 436.23,290.00 411.98,276.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="145.24,122.00 169.49,136.00 145.24,150.00 120.99,136.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,136.00 193.74,150.00 169.49,164.00 145.24,150.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,150.00 217.99,164.00 193.74,178.00 169.49,164.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,164.00 242.24,178.00 217.99,192.00 193.74,178.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,178.00 266.49,192.00 242.24,206.00 217.99,192.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,192.00 290.74,206.00 266.49,220.00 242.24,206.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,206.00 314.98,220.00 290.74,234.00 266.49,220.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,220.00 339.23,234.00 314.98,248.00 290.74,234.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,234.00 363.48,248.00 339.23,262.00 314.98,248.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,248.00 387.73,262.00 363.48,276.00 339.23,262.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,262.00 411.98,276.00 387.73,290.00 363.48,276.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="411.98,276.00 436.23,290.00 411.98,304.00 387.73,290.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="120.99,136.00 145.24,150.00 120.99,164.00 96.75,150.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,150.00 169.49,164.00 145.24,178.00 120.99,164.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,164.00 193.74,178.00 169.49,192.00 145.24,178.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,178.00 217.99,192.00 193.74,206.00 169.49,192.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,192.00 242.24,206.00 217.99,220.00 193.74,206.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,206.00 266.49,220.00 242.24,234.00 217.99,220.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,220.00 290.74,234.00 266.49,248.00 242.24,234.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,234.00 314.98,248.00 290.74,262.00 266.49,248.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,248.00 339.23,262.00 314.98,276.00 290.74,262.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,262.00 363.48,276.00 339.23,290.00 314.98,276.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,276.00 387.73,290.00 363.48,304.00 339.23,290.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="387.73,290.00 411.98,304.00 387.73,318.00 363.48,304.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="96.75,150.00 120.99,164.00 96.75,178.00 72.50,164.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="120.99,164.00 145.24,178.00 120.99,192.00 96.75,178.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,178.00 169.49,192.00 145.24,206.00 120.99,192.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,192.00 193.74,206.00 169.49,220.00 145.24,206.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,206.00 217.99,220.00 193.74,234.00 169.49,220.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,220.00 242.24,234.00 217.99,248.00 193.74,234.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,234.00 266.49,248.00 242.24,262.00 217.99,248.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,248.00 290.74,262.00 266.49,276.00 242.24,262.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,262.00 314.98,276.00 290.74,290.00 266.49,276.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,276.00 339.23,290.00 314.98,304.00 290.74,290.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,290.00 363.48,304.00 339.23,318.00 314.98,304.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="363.48,304.00 387.73,318.00 363.48,332.00 339.23,318.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="72.50,164.00 96.75,178.00 72.50,192.00 48.25,178.00" fill="#F6D2B2" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="96.75,178.00 120.99,192.00 96.75,206.00 72.50,192.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="120.99,192.00 145.24,206.00 120.99,220.00 96.75,206.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,206.00 169.49,220.00 145.24,234.00 120.99,220.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,220.00 193.74,234.00 169.49,248.00 145.24,234.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,234.00 217.99,248.00 193.74,262.00 169.49,248.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,248.00 242.24,262.00 217.99,276.00 193.74,262.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,262.00 266.49,276.00 242.24,290.00 217.99,276.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,276.00 290.74,290.00 266.49,304.00 242.24,290.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,290.00 314.98,304.00 290.74,318.00 266.49,304.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,304.00 339.23,318.00 314.98,332.00 290.74,318.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="339.23,318.00 363.48,332.00 339.23,346.00 314.98,332.00" fill="#FFFFFF" stroke="#CCCCCC" stroke-width="0.4"/>
|
||||
<polygon points="48.25,178.00 72.50,192.00 48.25,206.00 24.00,192.00" fill="#F7D7BA" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="72.50,192.00 96.75,206.00 72.50,220.00 48.25,206.00" fill="#F6D2B2" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="96.75,206.00 120.99,220.00 96.75,234.00 72.50,220.00" fill="#F5CDA9" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="120.99,220.00 145.24,234.00 120.99,248.00 96.75,234.00" fill="#F4C8A0" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="145.24,234.00 169.49,248.00 145.24,262.00 120.99,248.00" fill="#F3C297" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="169.49,248.00 193.74,262.00 169.49,276.00 145.24,262.00" fill="#F2BD8E" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="193.74,262.00 217.99,276.00 193.74,290.00 169.49,276.00" fill="#F1B885" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="217.99,276.00 242.24,290.00 217.99,304.00 193.74,290.00" fill="#F0B37D" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="242.24,290.00 266.49,304.00 242.24,318.00 217.99,304.00" fill="#EFAE74" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="266.49,304.00 290.74,318.00 266.49,332.00 242.24,318.00" fill="#EEA96B" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="290.74,318.00 314.98,332.00 290.74,346.00 266.49,332.00" fill="#EDA362" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
<polygon points="314.98,332.00 339.23,346.00 314.98,360.00 290.74,346.00" fill="#EC9E59" stroke="#9C9C9C" stroke-width="0.4"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 18 KiB |
68
paper/figures/tensors/gen_attn_heatmap.py
Normal file
68
paper/figures/tensors/gen_attn_heatmap.py
Normal file
@@ -0,0 +1,68 @@
|
||||
"""Generate the causal-packet MHSA attention heatmap SVG.
|
||||
|
||||
12 x 12 attention matrix, rounded-rectangle cells, no text labels:
|
||||
- row 0 and col 0 (FLOW token): cOrange!85 (full attention)
|
||||
- lower triangular within [1:, 1:]: cOrange decaying with distance
|
||||
(cOrange!{75 - 4*dist})
|
||||
- upper triangular (blocked): white + light grey border
|
||||
|
||||
Run:
|
||||
python gen_attn_heatmap.py
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
CORANGE = (230, 126, 34) # RGB of cOrange palette
|
||||
WHITE = (255, 255, 255)
|
||||
|
||||
CELL = 30 # px per cell (square)
|
||||
GAP = 2 # px between cells
|
||||
PAD = 6 # px outer padding
|
||||
RADIUS = 4 # px corner radius for cells
|
||||
N = 12 # 12 x 12 grid
|
||||
|
||||
STRIDE = CELL + GAP
|
||||
SIZE = N * CELL + (N - 1) * GAP + 2 * PAD
|
||||
|
||||
|
||||
def mix_orange(p: int) -> str:
|
||||
"""cOrange mixed with white at p% (matches TikZ cOrange!p)."""
|
||||
f = p / 100
|
||||
r = round(CORANGE[0] * f + WHITE[0] * (1 - f))
|
||||
g = round(CORANGE[1] * f + WHITE[1] * (1 - f))
|
||||
b = round(CORANGE[2] * f + WHITE[2] * (1 - f))
|
||||
return f"#{r:02X}{g:02X}{b:02X}"
|
||||
|
||||
|
||||
def cell_style(i: int, j: int) -> tuple[str, str, float]:
|
||||
"""(fill, stroke, stroke_width) for cell (row=i, col=j)."""
|
||||
if i == 0 or j == 0: # FLOW row / col
|
||||
return mix_orange(85), "#B8651F", 0.6
|
||||
if j > i: # blocked (upper-tri)
|
||||
return "#FFFFFF", "#BBBBBB", 0.5
|
||||
opa = max(15, 75 - (i - j) * 4) # lower-tri decay
|
||||
return mix_orange(opa), "#999999", 0.3
|
||||
|
||||
|
||||
def make_svg() -> str:
|
||||
parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" '
|
||||
f'width="{SIZE}" height="{SIZE}" viewBox="0 0 {SIZE} {SIZE}">',
|
||||
f' <rect width="{SIZE}" height="{SIZE}" fill="white"/>',
|
||||
]
|
||||
for i in range(N):
|
||||
for j in range(N):
|
||||
x = PAD + j * STRIDE
|
||||
y = PAD + i * STRIDE
|
||||
fill, stroke, sw = cell_style(i, j)
|
||||
parts.append(
|
||||
f' <rect x="{x}" y="{y}" width="{CELL}" height="{CELL}" '
|
||||
f'rx="{RADIUS}" ry="{RADIUS}" '
|
||||
f'fill="{fill}" stroke="{stroke}" stroke-width="{sw}"/>'
|
||||
)
|
||||
parts.append('</svg>')
|
||||
return '\n'.join(parts) + '\n'
|
||||
|
||||
|
||||
OUT = Path(__file__).parent
|
||||
(OUT / "05_attention_heatmap.svg").write_text(make_svg())
|
||||
print(f"wrote 05_attention_heatmap.svg ({SIZE}x{SIZE})")
|
||||
136
paper/figures/tensors/gen_attn_perspective.py
Normal file
136
paper/figures/tensors/gen_attn_perspective.py
Normal file
@@ -0,0 +1,136 @@
|
||||
"""Generate 3 perspective (isometric) SVGs for the MHSA attention story:
|
||||
|
||||
06_attn_raw.svg — raw attention pattern, no mask
|
||||
07_attn_mask.svg — the causal-packet mask itself (binary shape)
|
||||
08_attn_masked.svg — raw attention with the mask applied
|
||||
|
||||
Each cell is rendered as a parallelogram via a 30°/30° isometric projection.
|
||||
All three SVGs share the same dimensions and the same projection so they can
|
||||
be composed / cross-faded in drawio.
|
||||
"""
|
||||
from pathlib import Path
|
||||
import math
|
||||
|
||||
# --- palette ----------------------------------------------------------------
|
||||
CORANGE = (230, 126, 34)
|
||||
WHITE = (255, 255, 255)
|
||||
|
||||
# --- geometry ---------------------------------------------------------------
|
||||
CELL = 28 # local cell size (before projection)
|
||||
N = 12 # 12 x 12 grid
|
||||
PAD = 24
|
||||
|
||||
COS30 = math.sqrt(3) / 2 # ≈ 0.866
|
||||
SIN30 = 0.5
|
||||
|
||||
|
||||
def project(x: float, y: float) -> tuple[float, float]:
|
||||
"""Map local (x, y) to screen (sx, sy) via 30°/30° isometric projection.
|
||||
|
||||
Local x-axis (cols) goes down-right, local y-axis (rows) goes down-left.
|
||||
Origin (0, 0) ends up at the TOP of the projected diamond.
|
||||
"""
|
||||
sx = x * COS30 - y * COS30
|
||||
sy = x * SIN30 + y * SIN30
|
||||
return sx, sy
|
||||
|
||||
|
||||
def _bbox() -> tuple[float, float, float, float]:
|
||||
corners = [
|
||||
project(0, 0),
|
||||
project(N * CELL, 0),
|
||||
project(0, N * CELL),
|
||||
project(N * CELL, N * CELL),
|
||||
]
|
||||
xs, ys = [c[0] for c in corners], [c[1] for c in corners]
|
||||
return min(xs), max(xs), min(ys), max(ys)
|
||||
|
||||
|
||||
_xmin, _xmax, _ymin, _ymax = _bbox()
|
||||
W = int(_xmax - _xmin + 2 * PAD)
|
||||
H = int(_ymax - _ymin + 2 * PAD)
|
||||
OX = -_xmin + PAD
|
||||
OY = -_ymin + PAD
|
||||
|
||||
|
||||
def mix_orange(p: int) -> str:
|
||||
f = p / 100
|
||||
r = round(CORANGE[0] * f + WHITE[0] * (1 - f))
|
||||
g = round(CORANGE[1] * f + WHITE[1] * (1 - f))
|
||||
b = round(CORANGE[2] * f + WHITE[2] * (1 - f))
|
||||
return f"#{r:02X}{g:02X}{b:02X}"
|
||||
|
||||
|
||||
def cell_poly(i: int, j: int) -> str:
|
||||
"""Return SVG points string for the parallelogram at row i, col j."""
|
||||
pts = []
|
||||
for di, dj in [(0, 0), (0, 1), (1, 1), (1, 0)]:
|
||||
x = (j + dj) * CELL
|
||||
y = (i + di) * CELL
|
||||
sx, sy = project(x, y)
|
||||
pts.append(f"{sx + OX:.2f},{sy + OY:.2f}")
|
||||
return " ".join(pts)
|
||||
|
||||
|
||||
def make_svg(style_fn) -> str:
|
||||
# No background rect — SVGs are transparent so they can be cleanly
|
||||
# overlaid / placed on any drawio canvas color.
|
||||
parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" '
|
||||
f'width="{W}" height="{H}" viewBox="0 0 {W} {H}">',
|
||||
]
|
||||
# draw row-by-row, far rows first so near rows overlay (depth order).
|
||||
for i in range(N):
|
||||
for j in range(N):
|
||||
fill, stroke, sw = style_fn(i, j)
|
||||
parts.append(
|
||||
f' <polygon points="{cell_poly(i, j)}" '
|
||||
f'fill="{fill}" stroke="{stroke}" stroke-width="{sw}"/>'
|
||||
)
|
||||
parts.append('</svg>')
|
||||
return '\n'.join(parts) + '\n'
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Style functions for each of the 3 SVGs
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
def style_raw(i: int, j: int) -> tuple[str, str, float]:
|
||||
"""Raw attention before any mask: band-matrix-like soft pattern.
|
||||
|
||||
Brightness peaks on the diagonal and decays with |i - j|, giving a
|
||||
realistic-looking dense attention map.
|
||||
"""
|
||||
dist = abs(i - j)
|
||||
opa = max(20, 75 - dist * 4)
|
||||
return mix_orange(opa), "#9C9C9C", 0.4
|
||||
|
||||
|
||||
def style_mask(i: int, j: int) -> tuple[str, str, float]:
|
||||
"""The causal-packet mask itself, as a binary visual:
|
||||
|
||||
- allowed cells (FLOW row/col + lower-tri): orange-light
|
||||
- blocked cells (upper-tri except FLOW): dark gray
|
||||
"""
|
||||
blocked = (i != 0) and (j != 0) and (j > i)
|
||||
if blocked:
|
||||
return "#3A3A3A", "#1F1F1F", 0.5
|
||||
return mix_orange(60), "#B8651F", 0.4
|
||||
|
||||
|
||||
def style_masked(i: int, j: int) -> tuple[str, str, float]:
|
||||
"""Raw attention AFTER applying the causal-packet mask: blocked cells
|
||||
are whited out, allowed cells keep EXACTLY their raw attention intensity
|
||||
(i.e. masked == raw × mask, nothing else changes).
|
||||
"""
|
||||
blocked = (i != 0) and (j != 0) and (j > i)
|
||||
if blocked:
|
||||
return "#FFFFFF", "#CCCCCC", 0.4
|
||||
return style_raw(i, j)
|
||||
|
||||
|
||||
OUT = Path(__file__).parent
|
||||
(OUT / "06_attn_raw.svg").write_text(make_svg(style_raw))
|
||||
(OUT / "07_attn_mask.svg").write_text(make_svg(style_mask))
|
||||
(OUT / "08_attn_masked.svg").write_text(make_svg(style_masked))
|
||||
print(f"wrote 06_attn_raw.svg / 07_attn_mask.svg / 08_attn_masked.svg ({W}x{H})")
|
||||
71
paper/figures/tensors/gen_tensors.py
Normal file
71
paper/figures/tensors/gen_tensors.py
Normal file
@@ -0,0 +1,71 @@
|
||||
"""Generate 4 SVG tensor visualisations for the AdaLN modulation flow.
|
||||
|
||||
Each SVG is a 6 col x 4 row grid where cell color = cBlue mixed with white at
|
||||
the per-cell opacity (0..100). Same color values used in the TikZ panel (c).
|
||||
|
||||
Run:
|
||||
python gen_tensors.py
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
CBLUE = (52, 110, 180) # RGB of cBlue palette in the TikZ figure
|
||||
WHITE = (255, 255, 255)
|
||||
CELL = 50 # px per cell
|
||||
BORDER = 3 # outer padding (px)
|
||||
COLS, ROWS = 6, 4
|
||||
W = COLS * CELL + 2 * BORDER
|
||||
H = ROWS * CELL + 2 * BORDER
|
||||
|
||||
|
||||
def mix(p: int) -> str:
|
||||
"""Mix cBlue with white at p% (matches TikZ cBlue!p)."""
|
||||
f = p / 100
|
||||
r = round(CBLUE[0] * f + WHITE[0] * (1 - f))
|
||||
g = round(CBLUE[1] * f + WHITE[1] * (1 - f))
|
||||
b = round(CBLUE[2] * f + WHITE[2] * (1 - f))
|
||||
return f"#{r:02X}{g:02X}{b:02X}"
|
||||
|
||||
|
||||
def make_svg(opacities: list[int]) -> str:
|
||||
parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" '
|
||||
f'width="{W}" height="{H}" viewBox="0 0 {W} {H}">',
|
||||
f' <rect width="{W}" height="{H}" fill="white"/>',
|
||||
]
|
||||
for k, op in enumerate(opacities):
|
||||
i, j = k % COLS, k // COLS
|
||||
x = BORDER + i * CELL
|
||||
y = BORDER + j * CELL
|
||||
parts.append(
|
||||
f' <rect x="{x}" y="{y}" width="{CELL}" height="{CELL}" '
|
||||
f'fill="{mix(op)}" stroke="#999" stroke-width="0.6"/>'
|
||||
)
|
||||
parts.append('</svg>')
|
||||
return '\n'.join(parts) + '\n'
|
||||
|
||||
|
||||
# Same opacity arrays used in the TikZ panel (c). Row-major: 6 cols * 4 rows.
|
||||
TENSORS = {
|
||||
"01_h_in": [30, 80, 50, 40, 90, 60,
|
||||
70, 20, 80, 50, 40, 90,
|
||||
60, 50, 90, 30, 80, 40,
|
||||
70, 30, 60, 80, 40, 70],
|
||||
"02_h_after_LN": [50, 30, 70, 50, 40, 60,
|
||||
40, 60, 50, 50, 40, 60,
|
||||
50, 50, 60, 40, 60, 40,
|
||||
50, 40, 50, 60, 40, 60],
|
||||
"03_h_after_gamma":[60, 12, 70, 25, 52, 60,
|
||||
48, 24, 50, 25, 52, 60,
|
||||
60, 20, 60, 20, 78, 40,
|
||||
60, 16, 50, 30, 52, 60],
|
||||
"04_h_after_beta": [80, 32, 90, 45, 72, 80,
|
||||
68, 44, 70, 45, 72, 80,
|
||||
80, 40, 80, 40, 90, 60,
|
||||
80, 36, 70, 50, 72, 80],
|
||||
}
|
||||
|
||||
OUT = Path(__file__).parent
|
||||
for name, vals in TENSORS.items():
|
||||
assert len(vals) == COLS * ROWS, f"{name}: need {COLS*ROWS} values"
|
||||
(OUT / f"{name}.svg").write_text(make_svg(vals))
|
||||
print(f"wrote {name}.svg ({W}x{H})")
|
||||
18
paper/intro.md
Normal file
18
paper/intro.md
Normal file
@@ -0,0 +1,18 @@
|
||||
Network intrusion detection systems (NIDS) in production are dogged by two persistent failures. Alert volume overwhelms downstream triage: industry surveys and recent reviews report false-positive rates that frequently exceed 90%, and at the upper end approach 99% [Trend2024; ACM-CSur-2024]. Detectors that score well in one environment also lose a substantial fraction of that performance once evaluated on traffic from a different deployment [Cross2402.10974; Tand2025]. Modern NIDS research has largely converged on unsupervised anomaly detection, but neither failure has a settled answer within that paradigm. With within-dataset scores on the standard public benchmarks now narrowed to within reporting noise, the substantive evaluation axis has shifted to cross-dataset robustness, on which the field is far from converged.
|
||||
|
||||
The unsupervised NIDS toolkit has converged on three families of methods, all of which reduce a packet stream to a single anomaly score. Reconstruction-based detectors such as autoencoders, KitNET, and MemAE [Kitsune; MemAE] score by reconstruction error and exhibit a documented identity-mapping failure in which anomalies far from the benign manifold can still be reconstructed near-perfectly, undermining the core assumption [AE-Unreliable-2025; NeurIPS24-Reconstruction]. Density-based detectors built on normalising flows (NF) are the current public SOTA; the strongest recent pipeline reports 0.93 AUROC within-dataset on CIC-DDoS2019, with cross-domain transfer ranging from 0.89 to 0.93 depending on direction [Shafir2026]. The log-likelihood these methods rely on, however, is known to dissociate from anomaly status once the benign distribution drifts [NFAD2021]. Diffusion-based detectors [ConMD2026; DMAD2025] and optimised GAN variants [TIPSO-GAN-NDSS2026] have arrived in 2025–2026 with strong within-dataset numbers but share the same underlying object: a single scalar derived from a homogeneous probabilistic model fit to benign traffic.
|
||||
|
||||
Why these density-based scores transfer poorly has gone uncharacterised. We identify a structural failure mode we call source-likeness collapse. Under target-domain drift, the log-likelihood emitted by a benign-fit generative model no longer discriminates "x is benign vs malicious" but rather "x lies in the source benign distribution vs not"; the two coincide only when there is no shift, and diverge as drift grows. Empirically, across three independent Continuous Flow-Matching backbones (CFM; framework introduced below) and 16 candidate score channels, the canonical density-based score for these models (the terminal-norm of the velocity field) drops to AUROC ≤ 0.63 when trained on CIC-DDoS2019 and evaluated on CIC-IDS2017, with four of the twelve off-diagonal cells of our 4×4 cross-dataset matrix falling below 0.57 (near-random). The collapse persists across recipes, ruling out hyperparameter artefacts and indicating a structural property of likelihood as an anomaly proxy.
|
||||
|
||||
Two recent generative frameworks point to a way out. Continuous Flow Matching [Lipman2023; OT-CFM-Tong2024] learns a velocity field rather than a reconstruction, side-stepping the identity-mapping trap of reconstruction-based detectors. Discrete Flow Matching [Gat-NeurIPS2024] extends the same machinery to categorical state spaces. Network packets sit naturally in both regimes: each packet contributes three continuous channels (size, inter-arrival time, TCP window) and six binary channels (direction and five TCP flags). To our knowledge, neither paradigm has been applied to packet-sequence NIDS, although Flow Matching has been validated for image [rFM2025] and tabular [TCCM-NeurIPS2025] anomaly detection. Mixed continuous–discrete modelling emits a family of complementary scores rather than the single homogeneous likelihood under which source-likeness collapse occurs, and provides a structural path to the discrete protocol semantics that prior NF / autoencoder / GAN approaches must either Gaussianise away or ignore.
|
||||
|
||||
We present JANUS, an unsupervised packet-sequence anomaly detector with three components.
|
||||
1. A causal-packet Transformer backbone that produces a temporally-ordered representation of each flow.
|
||||
2. Two jointly-trained Flow-Matching heads on benign traffic, one over the continuous packet channels and one over the discrete protocol channels. Together they emit a family of complementary scores rather than a single likelihood.
|
||||
3. A benign-only aggregator that compresses the score family into a single deployable scalar, fit on target-domain benign validation data and never on attack labels.
|
||||
Together, the discrete head supplies a transfer-stable signal that survives source-likeness collapse, and the aggregator combines it with the residual information in the continuous-head scores rather than discarding them. The unsupervised contract holds end-to-end.
|
||||
We make four contributions:
|
||||
- (C1) First Flow-Matching detector for NIDS. To the best of our knowledge, JANUS is the first network anomaly detector to use Flow Matching as its training objective. It also combines continuous and discrete FM heads, a configuration not present in prior FM anomaly-detection work on image [rFM2025] or tabular [TCCM-NeurIPS2025] data.
|
||||
- (C2) Characterisation of source-likeness collapse. We name and analyse a structural failure mode in which density-based anomaly scores degrade into source-domain membership classifiers under cross-dataset shift. The phenomenon persists across three independent CFM backbones and all 16 candidate score channels we evaluate, identifying it as a property of density-based scoring rather than of any specific backbone. This explains a cross-domain failure mode that prior work observed but did not name.
|
||||
- (C3) A benign-only Mahalanobis aggregator with Oracle-Approximating-Shrinkage (OAS) covariance that compresses the score family into a single deployable scalar without consuming attack labels. We compare five aggregators (max-z, plain Mahalanobis, Ledoit–Wolf, OAS, and score-subset variants) and observe sensitivity ≤ 0.005 in AUROC across them, supporting the design as robust rather than hyperparameter-tuned.
|
||||
- (C4) Cross-dataset robustness with within-dataset competitiveness. On a 4×4 cross-dataset matrix (12 off-diagonal directions, three seeds per cell), JANUS averages +0.175 AUROC over the terminal-norm baseline and recovers all four collapse cells (terminal-norm < 0.57) to ≥ 0.75. It exceeds the Shafir NF baseline [Shafir2026] by +0.07 AUROC (0.96 vs 0.89) when trained on CIC-IDS2017 and evaluated on CIC-DDoS2019, and matches it (0.93) when the direction is reversed. Within-dataset, it exceeds the NF SOTA on three benchmarks by margins of +0.054 to +0.118, all exceeding the standard deviation across three seeds.
|
||||
293
paper/references.bib
Normal file
293
paper/references.bib
Normal file
@@ -0,0 +1,293 @@
|
||||
% =============================================================================
|
||||
% JANUS — Verified BibTeX for intro.md
|
||||
% Cite-key spelling matches the keys used in paper/intro.md.
|
||||
% Each entry includes a `url` field linking to the canonical source page so the
|
||||
% reference can be re-checked without re-searching.
|
||||
%
|
||||
% IMPORTANT NOTES (please review before submitting):
|
||||
%
|
||||
% * Trend2024: The Trend Micro 2024 "World Tour Survey" reports 51% of
|
||||
% SOC teams feel overwhelmed by alert volume but does NOT
|
||||
% state ">90% / 99%" false-positive rates. The 99% figure
|
||||
% traces to Alahmadi et al., USENIX Security 2022, which
|
||||
% is included below as @Alahmadi2022. Consider citing
|
||||
% [Alahmadi2022; Trend2024] together, or replacing.
|
||||
%
|
||||
% * ACM-CSur-2024: Tariq et al. is published in ACM Computing Surveys
|
||||
% Vol. 57(9), April 2025 — not 2024. The cite key is
|
||||
% preserved per intro.md, but @year is 2025.
|
||||
%
|
||||
% * Shafir2026: Venue is IEEE/ACM Transactions on Networking (ToN),
|
||||
% not IEEE TNSM. Verified via DOI 10.1109/TON.2025.3617580.
|
||||
%
|
||||
% * NFAD2021: Kirichenko et al. is NeurIPS 2020 (arXiv 2006.08545),
|
||||
% not 2021. Cite key preserved per intro.md.
|
||||
%
|
||||
% * AE-Unreliable-2025: Bouman & Heskes was *withdrawn* from ICLR 2025;
|
||||
% cited here as an arXiv preprint (2501.13864).
|
||||
%
|
||||
% * NeurIPS24-Reconstruction: The closest NeurIPS 2024 paper on the
|
||||
% reconstruction-AD identity-mapping limitation is Kim
|
||||
% et al., "Rethinking Reconstruction-based Graph-Level
|
||||
% Anomaly Detection". It is graph-level, not generic
|
||||
% image/tabular. Verify the citation matches your intent.
|
||||
%
|
||||
% * Tand2025: Best match for a Taylor & Francis 2025 cross-dataset
|
||||
% NIDS paper is Connection Science 2025 (HDSE-IDS).
|
||||
% The "0.10–0.30 AUROC drop" framing in intro.md is
|
||||
% primarily supported by Cross2402.10974, not by
|
||||
% Tand2025 directly.
|
||||
%
|
||||
% * rFM2025: arXiv 2508.05461's actual title is "Time-reversed Flow
|
||||
% Matching with Worst Transport in High-dimensional Latent
|
||||
% Space for Image Anomaly Detection". Earlier survey
|
||||
% notes called it "How and Why: Taming Flow Matching..."
|
||||
% — that title is incorrect. Updated below.
|
||||
% =============================================================================
|
||||
|
||||
|
||||
% --- Operational pain points (FP rates, alert fatigue) -----------------------
|
||||
|
||||
@misc{Trend2024,
|
||||
author = {{Trend Micro}},
|
||||
title = {{SOC Around the Clock: World Tour Survey Findings}},
|
||||
year = {2024},
|
||||
howpublished = {Trend Micro Research Report},
|
||||
url = {https://www.trendmicro.com/en_us/research/24/k/world-tour-survey-results.html},
|
||||
note = {Survey of 2,303 IT security/SOC decision makers; 51\% report
|
||||
feeling overwhelmed by alert volume.}
|
||||
}
|
||||
|
||||
@inproceedings{Alahmadi2022,
|
||||
author = {Bushra A. Alahmadi and Louise Axon and Ivan Martinovic},
|
||||
title = {99\% False Positives: A Qualitative Study of {SOC} Analysts'
|
||||
Perspectives on Security Alarms},
|
||||
booktitle = {31st USENIX Security Symposium (USENIX Security 22)},
|
||||
year = {2022},
|
||||
pages = {2783--2800},
|
||||
publisher = {USENIX Association},
|
||||
url = {https://www.usenix.org/conference/usenixsecurity22/presentation/alahmadi}
|
||||
}
|
||||
|
||||
@article{ACM-CSur-2024,
|
||||
author = {Shahroz Tariq and Mohan Baruwal Chhetri and Surya Nepal and
|
||||
C{\'e}cile Paris},
|
||||
title = {Alert Fatigue in Security Operations Centres:
|
||||
Research Challenges and Opportunities},
|
||||
journal = {ACM Computing Surveys},
|
||||
volume = {57},
|
||||
number = {9},
|
||||
articleno = {224},
|
||||
year = {2025},
|
||||
doi = {10.1145/3723158},
|
||||
url = {https://dl.acm.org/doi/10.1145/3723158}
|
||||
}
|
||||
|
||||
|
||||
% --- Cross-dataset NIDS robustness -------------------------------------------
|
||||
|
||||
@article{Cross2402.10974,
|
||||
author = {Marco Cantone and Claudio Marrocco and Alessandro Bria},
|
||||
title = {On the Cross-Dataset Generalization of Machine Learning
|
||||
for Network Intrusion Detection},
|
||||
journal = {arXiv preprint arXiv:2402.10974},
|
||||
year = {2024},
|
||||
eprint = {2402.10974},
|
||||
archivePrefix = {arXiv},
|
||||
primaryClass = {cs.CR},
|
||||
url = {https://arxiv.org/abs/2402.10974}
|
||||
}
|
||||
|
||||
@article{Tand2025,
|
||||
title = {Enhancing generalization of cross-domain intrusion detection:
|
||||
a heterogeneous deep stacked ensemble approach},
|
||||
journal = {Connection Science},
|
||||
publisher = {Taylor \& Francis},
|
||||
year = {2025},
|
||||
doi = {10.1080/09540091.2025.2599708},
|
||||
url = {https://www.tandfonline.com/doi/full/10.1080/09540091.2025.2599708},
|
||||
note = {Author list to be confirmed from publisher page (publisher
|
||||
returned 403 to automated fetch).}
|
||||
}
|
||||
|
||||
|
||||
% --- Reconstruction-based detectors ------------------------------------------
|
||||
|
||||
@inproceedings{Kitsune,
|
||||
author = {Yisroel Mirsky and Tomer Doitshman and Yuval Elovici and
|
||||
Asaf Shabtai},
|
||||
title = {{Kitsune}: An Ensemble of Autoencoders for Online Network
|
||||
Intrusion Detection},
|
||||
booktitle = {Network and Distributed System Security Symposium (NDSS)},
|
||||
year = {2018},
|
||||
eprint = {1802.09089},
|
||||
archivePrefix = {arXiv},
|
||||
url = {https://arxiv.org/abs/1802.09089}
|
||||
}
|
||||
|
||||
@inproceedings{MemAE,
|
||||
author = {Dong Gong and Lingqiao Liu and Vuong Le and Budhaditya Saha and
|
||||
Moussa Reda Mansour and Svetha Venkatesh and
|
||||
Anton {van den Hengel}},
|
||||
title = {Memorizing Normality to Detect Anomaly: Memory-Augmented Deep
|
||||
Autoencoder for Unsupervised Anomaly Detection},
|
||||
booktitle = {Proceedings of the IEEE/CVF International Conference on
|
||||
Computer Vision (ICCV)},
|
||||
year = {2019},
|
||||
pages = {1705--1714},
|
||||
eprint = {1904.02639},
|
||||
archivePrefix = {arXiv},
|
||||
url = {https://openaccess.thecvf.com/content_ICCV_2019/html/Gong_Memorizing_Normality_to_Detect_Anomaly_Memory-Augmented_Deep_Autoencoder_for_Unsupervised_ICCV_2019_paper.html}
|
||||
}
|
||||
|
||||
@article{AE-Unreliable-2025,
|
||||
author = {Roel Bouman and Tom Heskes},
|
||||
title = {Autoencoders for Anomaly Detection are Unreliable},
|
||||
journal = {arXiv preprint arXiv:2501.13864},
|
||||
year = {2025},
|
||||
eprint = {2501.13864},
|
||||
archivePrefix = {arXiv},
|
||||
primaryClass = {cs.LG},
|
||||
url = {https://arxiv.org/abs/2501.13864},
|
||||
note = {Withdrawn ICLR 2025 submission;
|
||||
OpenReview: https://openreview.net/forum?id=X8XQOLjLX6}
|
||||
}
|
||||
|
||||
@inproceedings{NeurIPS24-Reconstruction,
|
||||
author = {Sunwoo Kim and Soo Yong Lee and Fanchen Bu and Shinhwan Kang and
|
||||
Kyungho Kim and Jaemin Yoo and Kijung Shin},
|
||||
title = {Rethinking Reconstruction-based Graph-Level Anomaly Detection:
|
||||
Limitations and a Simple Remedy},
|
||||
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
|
||||
year = {2024},
|
||||
url = {https://openreview.net/forum?id=e2INndPINB}
|
||||
}
|
||||
|
||||
|
||||
% --- Density-based detectors (NF / Diffusion / GAN) --------------------------
|
||||
|
||||
@article{Shafir2026,
|
||||
author = {Lior Shafir and Raja Giryes and Avishai Wool},
|
||||
title = {Explainable Anomaly Detection in Network Traffic Using
|
||||
Normalizing Flows},
|
||||
journal = {IEEE/ACM Transactions on Networking},
|
||||
volume = {34},
|
||||
year = {2026},
|
||||
doi = {10.1109/TON.2025.3617580},
|
||||
url = {https://doi.org/10.1109/TON.2025.3617580}
|
||||
}
|
||||
|
||||
@inproceedings{NFAD2021,
|
||||
author = {Polina Kirichenko and Pavel Izmailov and Andrew Gordon Wilson},
|
||||
title = {Why Normalizing Flows Fail to Detect Out-of-Distribution Data},
|
||||
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
|
||||
year = {2020},
|
||||
eprint = {2006.08545},
|
||||
archivePrefix = {arXiv},
|
||||
url = {https://arxiv.org/abs/2006.08545},
|
||||
note = {NeurIPS 2020 (cite key NFAD2021 retained per intro.md).}
|
||||
}
|
||||
|
||||
@article{ConMD2026,
|
||||
author = {Xinglin Lian and Yu Zheng and Yan Liu and Fan Zhou and
|
||||
Chunlei Peng and Xinbo Gao},
|
||||
title = {Contextual Masking Distillation for Network Traffic Anomaly
|
||||
Detection},
|
||||
journal = {IEEE Transactions on Information Forensics and Security},
|
||||
volume = {21},
|
||||
pages = {1273--1286},
|
||||
year = {2026},
|
||||
doi = {10.1109/TIFS.2026.3655514},
|
||||
url = {https://ieeexplore.ieee.org/document/11358423/}
|
||||
}
|
||||
|
||||
@article{DMAD2025,
|
||||
author = {Hui Liu and others},
|
||||
title = {A Survey on Diffusion Models for Anomaly Detection},
|
||||
journal = {arXiv preprint arXiv:2501.11430},
|
||||
year = {2025},
|
||||
eprint = {2501.11430},
|
||||
archivePrefix = {arXiv},
|
||||
primaryClass = {cs.LG},
|
||||
url = {https://arxiv.org/abs/2501.11430},
|
||||
note = {Submitted to IJCAI 2025 (per associated GitHub repository);
|
||||
verify final IJCAI proceedings entry before publication.}
|
||||
}
|
||||
|
||||
@inproceedings{TIPSO-GAN-NDSS2026,
|
||||
author = {Ernest Akpaku and Jinfu Chen and Joshua Ofoeda},
|
||||
title = {{TIPSO-GAN}: Malicious Network Traffic Detection Using a Novel
|
||||
Optimized Generative Adversarial Network},
|
||||
booktitle = {Network and Distributed System Security Symposium (NDSS)},
|
||||
year = {2026},
|
||||
url = {https://www.ndss-symposium.org/ndss-paper/tipso-gan-malicious-network-traffic-detection-using-a-novel-optimized-generative-adversarial-network/}
|
||||
}
|
||||
|
||||
|
||||
% --- Flow Matching foundations -----------------------------------------------
|
||||
|
||||
@inproceedings{Lipman2023,
|
||||
author = {Yaron Lipman and Ricky T. Q. Chen and Heli Ben-Hamu and
|
||||
Maximilian Nickel and Matt Le},
|
||||
title = {Flow Matching for Generative Modeling},
|
||||
booktitle = {International Conference on Learning Representations (ICLR)},
|
||||
year = {2023},
|
||||
eprint = {2210.02747},
|
||||
archivePrefix = {arXiv},
|
||||
url = {https://arxiv.org/abs/2210.02747}
|
||||
}
|
||||
|
||||
@article{OT-CFM-Tong2024,
|
||||
author = {Alexander Tong and Kilian Fatras and Nikolay Malkin and
|
||||
Guillaume Huguet and Yanlei Zhang and Jarrid Rector-Brooks and
|
||||
Guy Wolf and Yoshua Bengio},
|
||||
title = {Improving and Generalizing Flow-Based Generative Models with
|
||||
Minibatch Optimal Transport},
|
||||
journal = {Transactions on Machine Learning Research (TMLR)},
|
||||
year = {2024},
|
||||
eprint = {2302.00482},
|
||||
archivePrefix = {arXiv},
|
||||
url = {https://openreview.net/forum?id=CD9Snc73AW}
|
||||
}
|
||||
|
||||
@inproceedings{Gat-NeurIPS2024,
|
||||
author = {Itai Gat and Tal Remez and Neta Shaul and Felix Kreuk and
|
||||
Ricky T. Q. Chen and Gabriel Synnaeve and Yossi Adi and
|
||||
Yaron Lipman},
|
||||
title = {Discrete Flow Matching},
|
||||
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
|
||||
year = {2024},
|
||||
eprint = {2407.15595},
|
||||
archivePrefix = {arXiv},
|
||||
url = {https://openreview.net/forum?id=GTDKo3Sv9p}
|
||||
}
|
||||
|
||||
|
||||
% --- Flow-Matching anomaly detection (image / tabular) -----------------------
|
||||
|
||||
@article{rFM2025,
|
||||
author = {Liangwei Li and Lin Liu and Hanzhe Liang and Juanxiu Liu and
|
||||
Jing Zhang and Ruqian Hao and Xiaohui Du and Yong Liu and
|
||||
Pan Li},
|
||||
title = {Time-reversed Flow Matching with Worst Transport in
|
||||
High-dimensional Latent Space for Image Anomaly Detection},
|
||||
journal = {arXiv preprint arXiv:2508.05461},
|
||||
year = {2025},
|
||||
eprint = {2508.05461},
|
||||
archivePrefix = {arXiv},
|
||||
primaryClass = {cs.CV},
|
||||
url = {https://arxiv.org/abs/2508.05461}
|
||||
}
|
||||
|
||||
@inproceedings{TCCM-NeurIPS2025,
|
||||
author = {Zhong Li and Qi Huang and Yuxuan Zhu and Lincen Yang and
|
||||
Mohammad Mohammadi Amiri and Niki van Stein and
|
||||
Matthijs van Leeuwen},
|
||||
title = {Scalable, Explainable and Provably Robust Anomaly Detection
|
||||
with One-Step Flow Matching},
|
||||
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
|
||||
year = {2025},
|
||||
eprint = {2510.18328},
|
||||
archivePrefix = {arXiv},
|
||||
url = {https://arxiv.org/abs/2510.18328}
|
||||
}
|
||||
21
paper/tables/tab_data_efficiency.tex
Normal file
21
paper/tables/tab_data_efficiency.tex
Normal file
@@ -0,0 +1,21 @@
|
||||
\begin{table}[htbp]
|
||||
\centering
|
||||
\caption{Data-efficiency sweep on CICIDS2017. Both architectures evaluated on the full 1.2M benign + 500K attack set; only training data size varies.}
|
||||
\label{tab:data_efficiency}
|
||||
\begin{tabular}{lrrrrr}
|
||||
\toprule
|
||||
arch & n\_train & tnorm\_auroc & tnorm\_auprc & nll\_auroc & nll\_auprc \\
|
||||
\midrule
|
||||
A1 & 1000 & 0.9671 & 0.8985 & 0.9575 & 0.8871 \\
|
||||
A1 & 3000 & 0.9791 & 0.9199 & 0.9584 & 0.8862 \\
|
||||
A1 & 10000 & 0.9920 & 0.9653 & 0.9856 & 0.9533 \\
|
||||
A1 & 30000 & 0.9942 & 0.9772 & 0.9890 & 0.9674 \\
|
||||
A1 & 100000 & 0.9948 & 0.9791 & 0.9902 & 0.9686 \\
|
||||
A9 & 1000 & 0.9573 & 0.8785 & 0.9458 & 0.8647 \\
|
||||
A9 & 3000 & 0.9857 & 0.9380 & 0.9766 & 0.9229 \\
|
||||
A9 & 10000 & 0.9931 & 0.9672 & 0.9901 & 0.9612 \\
|
||||
A9 & 30000 & 0.9938 & 0.9746 & 0.9906 & 0.9686 \\
|
||||
A9 & 100000 & 0.9950 & 0.9809 & 0.9929 & 0.9756 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
Reference in New Issue
Block a user