llmiotsafe/EGPv2_1/miss_checklist_diag60.md

# EGPv2.1 MISS Checklist on `qwen36_35B_egpv2_1_1diag60`

## Snapshot

- Total `MISS`: 16 / 28 TP
- By SQ:
  - `SQ1`: 4
  - `SQ2`: 2
  - `SQ3`: 5
  - `SQ4`: 5
  - `SQ5`: 0
- By workflow behavior:
  - `primary_task_profile=device-health`: 9 / 16
  - `supervisor_action=refine_investigation`: 14 / 16
  - `evidence_sufficient=false`: 14 / 16
  - Final prediction: all 16 were collapsed to `is_anomaly=false`, `threat_type=none`

## Cluster 1: SQ1 real device faults are still over-suppressed

Affected episodes:
- `SQ1_TP_A_0004` / `DF-06` / `actuator_stuck`
- `SQ1_TP_A_0006` / `DF-02` / `sensor_drift`
- `SQ1_TP_B_0011` / `DF-05` / `safety_device_failure`
- `SQ1_TP_C_0005` / `DF-01` / `sensor_stuck`

Observed pattern:
- The new anti-false-alarm rules are working, but in these 4 cases they over-regularize real faults.
- The model often explains away the anomaly as:
  - transient communication issue
  - missing logs
  - normal continuous readings in other sensors
  - incomplete evidence, therefore normal
- Focus selection is still too narrow in some cases:
  - `C00 + C13` only
  - `C00 + C20` only

Diagnosis:
- The current prompt now rejects weak fault claims well, but it still lacks a strong positive rule for:
  - persistent stuck behavior
  - slow drift across chunks
  - safety device failing in the presence of correlated hazard context
  - actuator not changing despite command or expected physical consequence

Next patch target:
- Strengthen positive triggers for real `device-health` anomalies.
- Especially add explicit language for:
  - cross-chunk persistence
  - monotonic drift without return
  - commanded actuator state not physically progressing
  - safety-device silence/failure when hazard-side evidence is present

## Cluster 2: SQ2/SQ3 still drift into wrong task understanding

Affected episodes:
- `SQ2_TP_B_0192` / `INS-02` / `intrusion`
- `SQ2_TP_B_0220` / `WD-03` / `water_leak`
- `SQ3_TP_A_0433` / `INS-05` / `credential_theft`
- `SQ3_TP_B_0452` / `BA-01` / `behavioral_anomaly`
- `SQ3_TP_B_0457` / `INS-01` / `intrusion`
- `SQ3_TP_C_0444` / `CH-04` / `child_safety`

Observed pattern:
- These are not being rejected because the model saw no anomaly at all.
- They are being rejected because the pipeline often frames them as:
  - `device-health`
  - normal cooking
  - transient telemetry noise
  - ordinary evening routine

Strong symptom:
- `SQ3` misses are dominated by task-profile drift:
  - 4 / 5 `SQ3` misses have `primary_task_profile=device-health`

Diagnosis:
- The workflow is still too attracted to local sensor irregularities (`None`, spike, dropout, lock flip).
- Once triage drifts there, later stages keep discussing telemetry quality instead of:
  - access-path inconsistency
  - identity / credential mismatch
  - subject-specific risk
  - behavior-sequence abnormality

Next patch target:
- In triage, if the query is about behavior, intrusion, vulnerable groups, or security events:
  - do not let isolated telemetry oddities dominate `suspected_patterns`
  - prioritize entry path, human sequence, device-use context, and room-to-room progression
- In investigator/verifier:
  - explicitly demote “normal cooking” as a default explanation unless it actually explains the queried abnormal event

## Cluster 3: SQ4 composite-safety is now closer, but retrieval/evidence assembly still leaks recall

Affected episodes:
- `SQ4_TP_A_0720` / `FG-01` / `unattended_cooking`
- `SQ4_TP_B_0721` / `FG-02` / `fire_risk`
- `SQ4_TP_B_0722` / `BA-01` / `behavioral_anomaly`
- `SQ4_TP_B_0768` / `BA-03` / `behavioral_anomaly`
- `SQ4_TP_D_0752` / `EL-02` / `possible_fall`

Observed pattern:
- Triage is at least choosing `composite-safety`, so the high-level framing is better than before.
- But the final answer still collapses to:
  - “no definitive safety anomaly”
  - “transient telemetry issues”
  - “incomplete logs, low confidence”

Most important concrete failure:
- `SQ4_TP_A_0720`:
  - supervisor explicitly says the investigator identified the right anomaly region
  - but the focused chunks did not actually include the needed chunks
  - so the final stage had no solid evidence to commit

Diagnosis:
- The current SQ4 weakness is less about task-profile drift and more about:
  - wrong chunk retrieval
  - incomplete hazard-context assembly
  - verifier collapsing `refine_investigation` into `none`

Next patch target:
- When `primary_task_profile=composite-safety` and a hazard candidate exists:
  - enforce inclusion of the hazard chunk itself
  - plus one chunk for human/vulnerability context
  - plus one chunk for mitigation/outcome
- If supervisor says `refine_investigation` and names missing critical chunks, the second round should bias harder toward those chunks instead of reusing safe chunks

## Cluster 4: Workflow-level over-conservatism after supervisor uncertainty

Evidence:
- `supervisor_action=refine_investigation`: 14 / 16 misses
- `evidence_sufficient=false`: 14 / 16 misses
- final verifier still returned `none` for all 16 misses

Diagnosis:
- The intended rule was:
  - `evidence_sufficient=false` does not mean `no anomaly`
- In practice, the verifier still behaves as if:
  - uncertain = reject anomaly

Interpretation:
- The workflow has solved much of the false-alarm problem.
- The remaining recall gap is now largely a “decision policy under uncertainty” problem.

Next patch target:
- Prevent the verifier from collapsing every plausible-but-incomplete TP into `none`.
- For non-device-fault cases especially:
  - if there is a coherent anomaly hypothesis,
  - and supervisor did not `abstain`,
  - and the anomaly directly answers the query,
  - prefer a low/medium-confidence anomaly over a blanket normal verdict.

## Priority Order for Next Fix

1. Fix `SQ3` / `SQ2` task-profile drift away from telemetry-noise narratives.
2. Fix `SQ4` second-round retrieval so supervisor-requested hazard chunks are actually pulled in.
3. Add stronger positive rules for real `SQ1` faults so anti-false-alarm logic does not suppress true persistence/drift/stuck cases.
4. Tighten verifier policy so `refine_investigation != normal`.