SafeHome Benchmark: Diversity & Hardness Report
1. Overall Scale
| Metric |
Value |
| Total episodes |
1200 |
| True Positive (TP) |
550 |
| False Positive (FP) |
400 |
| True Negative (TN) |
250 |
| Unique anomaly scenarios |
33 |
| Unique FP scenarios |
35 |
| Anomaly categories |
7 |
| Query types (SQ) |
5 |
| Home layouts |
4 |
| Resident profiles |
3 |
| Difficulty levels |
3 (L1/L2/L3) |
2. Distribution by Query Type
| SQ Type |
Description |
TP |
FP |
TN |
Total |
| SQ1 |
Device Health Diagnosis |
80 |
55 |
55 |
190 |
| SQ2 |
Single-Event Safety Judgment |
105 |
80 |
55 |
240 |
| SQ3 |
Behavioral Sequence Analysis |
130 |
105 |
55 |
290 |
| SQ4 |
Composite Safety Reasoning |
130 |
105 |
55 |
290 |
| SQ5 |
Emergency Response Planning |
105 |
55 |
30 |
190 |
3. Distribution by Anomaly Category
| Category |
TP episodes |
FP episodes |
Unique scenarios |
| Intrusion |
147 |
93 |
5 |
| Fire/Gas |
119 |
73 |
4 |
| Water Damage |
42 |
51 |
3 |
| Device Fault |
80 |
55 |
6 |
| Elderly-Specific |
45 |
23 |
7 |
| Child-Specific |
29 |
24 |
4 |
| Behavioral Anomaly |
88 |
81 |
4 |
4. Distribution by Home Layout
| Layout |
Type |
Devices |
Episodes |
Avg Events/Episode |
| A |
Studio/1BR |
24 |
250 |
1210 |
| B |
2-Bedroom |
34 |
290 |
1691 |
| C |
3-Bedroom |
48 |
373 |
2106 |
| D |
Elderly Living Alone |
26 |
287 |
1855 |
5. Distribution by Resident Profile
| Profile |
Episodes |
Applicable Layouts |
| Young Professional |
400 |
A, B |
| Family with Children |
513 |
B, C |
| Elderly Living Alone |
287 |
D |
6. Event Count Statistics
| Metric |
All |
Layout A |
Layout B |
Layout C |
Layout D |
| Min |
1114 |
1114 |
1392 |
1488 |
1844 |
| Max |
2475 |
1354 |
1849 |
2475 |
1894 |
| Mean |
1759 |
1210 |
1691 |
2106 |
1855 |
| Median |
1840 |
1130 |
1660 |
2295 |
1854 |
| Std |
391 |
99 |
142 |
371 |
7 |
7. Difficulty Distribution
7.1 By Level
| Level |
Label |
Score Range |
Description |
Episodes (TP+FP) |
% |
| L1 |
基础检测 |
5-7 |
Direct alarm signals, single device |
111 |
11% |
| L2 |
推理检测 |
8-10 |
State correlation, cross-device reasoning |
394 |
41% |
| L3 |
复合推理 |
11-15 |
Temporal analysis, absence reasoning, high FP similarity |
445 |
46% |
7.2 Five-Dimensional Difficulty Breakdown
| Dimension |
Description |
Mean |
Distribution (1/2/3) |
| D1_evidence_count |
Number of evidence pieces needed |
2.19 |
145/479/326 |
| D2_signal_directness |
Signal directness (1=alarm, 3=absence) |
1.98 |
214/542/194 |
| D3_cross_device |
Cross-device reasoning required |
1.91 |
304/429/217 |
| D4_temporal_span |
Temporal span of evidence |
1.71 |
497/235/218 |
| D5_fp_similarity |
TP/FP discrimination difficulty |
2.25 |
0/715/235 |
7.3 Difficulty Score Distribution (TP+FP)
Score 5: 0
Score 6: 111 ███████████████████████████████████████████████████████████████████████████████████████████████████████████████
Score 7: 0
Score 8: 93 █████████████████████████████████████████████████████████████████████████████████████████████
Score 9: 137 █████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Score 10: 164 ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Score 11: 217 █████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Score 12: 104 ████████████████████████████████████████████████████████████████████████████████████████████████████████
Score 13: 124 ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Score 14: 0
Score 15: 0
8. Anomaly Injection Characteristics (TP only)
8.1 Injection Time-of-Day
| Hour |
Count |
|
| 00:00 |
24 |
████████████ |
| 01:00 |
41 |
█████████████████████ |
| 02:00 |
55 |
████████████████████████████ |
| 03:00 |
21 |
███████████ |
| 04:00 |
14 |
███████ |
| 05:00 |
17 |
█████████ |
| 06:00 |
12 |
██████ |
| 07:00 |
30 |
███████████████ |
| 08:00 |
45 |
███████████████████████ |
| 09:00 |
24 |
████████████ |
| 10:00 |
24 |
████████████ |
| 11:00 |
14 |
███████ |
| 12:00 |
15 |
████████ |
| 13:00 |
12 |
██████ |
| 14:00 |
19 |
██████████ |
| 15:00 |
10 |
█████ |
| 16:00 |
15 |
████████ |
| 17:00 |
24 |
████████████ |
| 18:00 |
30 |
███████████████ |
| 19:00 |
26 |
█████████████ |
| 20:00 |
15 |
████████ |
| 21:00 |
24 |
████████████ |
| 22:00 |
13 |
███████ |
| 23:00 |
26 |
█████████████ |
8.2 Anomaly Event Statistics
| Metric |
Value |
| Anomaly events per TP episode (mean) |
4.7 |
| Anomaly events per TP episode (range) |
[1, 11] |
| Anomaly devices per TP episode (mean) |
2.4 |
| Anomaly events as % of total events |
0.26% |
9. Temporal Characteristics
| Metric |
Value |
| Event log time span (mean) |
24.7 hours |
| Event log time span (range) |
[23.9, 60.0] hours |
| Sampling interval (temperature) |
5 minutes |
| Sampling interval (occupancy, occupied) |
5 minutes |
| Sampling interval (occupancy, unoccupied) |
30 minutes |
10. SQ Type × Difficulty Level Cross-tabulation
|
L1 |
L2 |
L3 |
TN |
Total |
| SQ1 |
20 |
75 |
40 |
55 |
190 |
| SQ2 |
39 |
77 |
69 |
55 |
240 |
| SQ3 |
0 |
105 |
130 |
55 |
290 |
| SQ4 |
25 |
79 |
131 |
55 |
290 |
| SQ5 |
27 |
58 |
75 |
30 |
190 |
11. Comparison with SimuHome
| Dimension |
SimuHome (ICLR 2026) |
SafeHome (Ours) |
| Total episodes |
600 |
1,200 |
| Query/task types |
6 (QT1-QT4) |
5 (SQ1-SQ5) |
| Evaluation categories |
12 (6×F/IF) |
15 (5×TP/FP/TN) |
| Anomaly scenarios |
N/A (task execution) |
35 |
| FP near-miss scenarios |
N/A |
35 (1:1 paired) |
| Anomaly categories |
N/A |
7 |
| Home layouts |
Random config |
4 fixed archetypes |
| Resident profiles |
N/A |
3 |
| Difficulty levels |
Implicit (QT complexity) |
3 levels, 5-dim scored |
| Avg events per episode |
~50 (API calls) |
~1,750 (sensor logs) |
| Device protocol |
Matter (17 types) |
Matter 1.5.1 (13 types) |
| Evaluation focus |
Proactive task execution |
Reactive anomaly reasoning |
