TL;DR — MemER scene-mem v2 LoRA, held-out val: final-query 0.57% (1/174); true compressed-memory recall (chunked) 0.99%
0.57%
PRIMARY final-query
1/174
0.54%
memory-turn match
24/4465
0.99%
CHUNKED (true mem)
1/101
0.0%
NOT-chunked floor
0/73
Eval set (honest split)
Held-out val only: 189 test samples across
57 scenarios (split-val total
57). Train leak:
NONE. MemER did NOT train on these scenarios.
base = Qwen/Qwen3-VL-4B-Instruct
adapter = /home/nas_main/taewoongkang/repos/Robotics/memer/runs/ckpts/scene_mem_v2_lora_260516_125653/v0-20260516-125746/checkpoint-123
max_new_tokens = 128
(2) Per-memory-turn narration match rate
| bucket | match rate | count |
|---|
| overall | 0.54% |
24/4465 |
| fine segments | 0.56% |
24/4250 |
| composite segments (verbatim) | 0.0% |
0/215 |
(3) Keyframe selection quality
keyframe-count distribution:
{0: 11324, 1: 2890}
mean keyframes/turn: 0.203
final-query emitted-a-keyframe rate:
6.9%
(12/174)
(4) Chunked-vs-not memory-recall decomposition
| bucket | final-query acc | count |
|---|
| CHUNKED (true compressed-memory recall) |
0.99% |
1/101 |
| NOT-chunked (learned-prior floor) |
0.0% |
0/73 |
chunked = the test's target task instruction matches the _PNP pick-and-place regex reused verbatim from scripts/analyze_tier3b_vs_tier3a.py (object genuinely entered the bounded FIFO). chunked = true compressed-memory recall; notchunked = learned-prior floor.
Run
parse failures: 21 ·
duration: 21531.7s