B1 eval visual viewer — scene-mem v2 MemER (held-out val)

TL;DR

Watch what the B1 MemER model actually saw, turn by turn, and what it predicted vs GT — 14 curated annotated videos below (balanced: correct AND wrong finals, chunked AND not-chunked, §7-composite AND fine-only turn sequences).
m1 final-query accuracy 88.36% (167/189); m4 chunked = true compressed-memory recall 97.2% (103/106), not-chunked prior floor 77.1%.
Believability: a prior broken per-frame-stride rollout scored ~0.6% on the same model; the jump to ~89% is the eval contract being fixed to match training (builder-faithful, no teacher forcing), not the model changing. See methodology note for the full 5-iteration journey and metric definitions.
Incremental: predictions.jsonl is written per-scenario by the running job 4192 (189/189 done). Re-run scripts/render_b1_eval_video.py --curated then this builder to refresh once 4192 completes.

1Current metrics (metrics.json, 189/189 samples)

88.4%

m1 final-query
167/189

97.2%

m4 chunked
(true mem recall)

77.1%

m4 not-chunked
(prior floor)

95.2%

m2 narration
fine 96/comp 83

100%

m3 final-kf rate
prec 0.94

parse
failures

Metric definitions & the 5-iteration believability journey are in claude/260517/analysis-b1_eval_methodology.md (repo). Short version: m1 = model's final-turn current_subtask vs the verified fine_subtasks.json target, scored by the deterministic rule-based score_subtask (no LLM judge); m4 splits chunked (target object genuinely passed through the bounded FIFO → true compressed-memory recall) vs not-chunked (answerable from learned prior only → guessing floor). Prior tier3b: overall 64.9% / chunked 100%; here not-chunked rose 17% → ~74% — the quantified value of the V2 fine-subtask redesign.

2How to read the videos

Each video is one scenario+test. The left camera stack is exactly the pixels B1 received (wrist robot0_eye_in_hand top, robot0_agentview_left bottom). It plays the ~8 evenly-spaced recent-window frames the builder fed the model for that turn (we deliberately do not interpolate frames the model never saw — the turn→frame mapping is recovered by re-running the same builder machinery the eval used, byte-identical to predictions.jsonl). The right panel shows that turn's instruction, the builder/GT subtask, the model's predicted subtask, the match verdict + reason, model-vs-GT keyframe positions, the running memory-turn tally, and the prominent FINAL-QUERY verdict. The bottom strip is the size-8 memory FIFO (frames MemER carries forward); a green border marks the keyframe pushed into memory.

►combo_011_L33_S36_0218 · MicrowaveThawingFridge_meat

FINAL WRONG not-chunked (prior floor) fine-only task=MicrowaveThawingFridge turns=21 (memory 20 + final 1) memory-turn match 17/20

camera stack = exactly what B1 saw (wrist robot0_eye_in_hand top + robot0_agentview_left bottom); plays the ~8 recent-window frames the model received per turn, overlay shows that turn's GT vs prediction, match, keyframes, and the running tally. Green border = keyframe pushed into the FIFO.

FINAL-QUERY VERDICT: instruction Task: Show me where the steak is placed. → target GT Place the steak in the microwave; model said Place the steak on the toaster oven — MISMATCH (wrong recall) | target_chunked=False.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
1	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
2	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
3	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
4	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
6	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
7	fine	Task: Pick up the bagel from the plate, place it on the toas	Pick up the bagel from the plate	Pick up the bagel from the plate	✓	m=[] g=[]
8	fine	Task: Pick up the bagel from the plate, place it on the toas	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[8] g=[8]
9	fine	Task: Pick up the bagel from the plate, place it on the toas	Close the toaster oven door	Close the toaster oven door	✓	m=[] g=[]
10	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
11	fine	Task: Transport the steak from the fridge to the microwave a	Pick up the steak from the fridge	Transport the steak from the fridge to the microwave and turn on the microwave.	✗	m=[8] g=[]
12	fine	Task: Transport the steak from the fridge to the microwave a	Place the steak in the microwave	Turn on the microwave	✗	m=[] g=[8]
13	fine/transit	Task: Approach the coffee machine for the next task.	Navigate to the coffee machine	Navigate to the coffee machine	✓	m=[] g=[]
14	fine	Task: Pick the mug from the cabinet, place it under the coff	Pick up the mug from the cabinet	Pick up the mug from the cabinet	✓	m=[] g=[]
15	fine	Task: Pick the mug from the cabinet, place it under the coff	Place the mug under the coffee machine	Place the mug under the coffee machine	✓	m=[8] g=[8]
16	fine	Task: Pick the mug from the cabinet, place it under the coff	Press the start button on the coffee machine	Pick up the mug from the cabinet	✗	m=[] g=[]
17	fine/transit	Task: Approach the drawer for the next task.	Navigate to the drawer	Navigate to the drawer	✓	m=[] g=[]
18	fine	Task: Take the pizza cutter from the drawer on the left and	Pick up the pizza from the drawer	Pick up the pizza cutter from the drawer	✓	m=[] g=[]
19	fine	Task: Take the pizza cutter from the drawer on the left and	Place the pizza on the dining counter	Place the pizza cutter in the dining counter	✓	m=[8] g=[8]
20 (final)	FINAL	Task: Show me where the steak is placed.	Place the steak in the microwave	Place the steak on the toaster oven	✗	m=[2] g=[3]

►combo_011_L33_S36_0145 · MicrowaveThawingFridge_meat

FINAL WRONG not-chunked (prior floor) has §7 composite turn task=MicrowaveThawingFridge turns=21 (memory 20 + final 1) memory-turn match 17/20

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
1	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
2	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
3	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
4	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
6	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
7	fine	Task: Pick up the bagel from the plate, place it on the toas	Pick up the bagel from the plate	Pick up the bagel from the plate	✓	m=[] g=[]
8	fine	Task: Pick up the bagel from the plate, place it on the toas	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[8] g=[8]
9	fine	Task: Pick up the bagel from the plate, place it on the toas	Close the toaster oven door	Close the toaster oven door	✓	m=[] g=[]
10	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
11	fine	Task: Transport the steak from the fridge to the microwave a	Pick up the steak from the fridge	Transport the steak from the fridge to the microwave and turn on the microwave.	✗	m=[8] g=[]
12	fine	Task: Transport the steak from the fridge to the microwave a	Place the steak in the microwave	Turn on the microwave	✗	m=[] g=[8]
13	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
14	composite	Task: Place the chicken drumstick on the plate and place the	Place the chicken drumstick on the plate and place the onion on the same plate.	Pick up the chicken drumstick from the counter	✗	m=[] g=[8]
15	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
16	composite	Task: Collect the paprika, ketchup, and shaker from the cabi	Collect the paprika, ketchup, and shaker from the cabinet and place them next to the stove.	Collect the paprika, ketchup, and shaker from the cabinet and place them next to the stove.	✓	m=[8] g=[8]
17	fine/transit	Task: Approach the next location for the next task.	Navigate to the next location	Navigate to the next location	✓	m=[] g=[]
18	fine	Task: Pick the mug from the counter and place it under the c	Pick up the mug from the counter	Pick up the mug from the counter	✓	m=[] g=[]
19	fine	Task: Pick the mug from the counter and place it under the c	Place the mug under the coffee machine dispenser	Place the mug under the coffee machine dispenser	✓	m=[8] g=[8]
20 (final)	FINAL	Task: Show me where the steak is placed.	Place the steak in the microwave	Place the steak on the toaster oven	✗	m=[2] g=[3]

►combo_072_L56_S42_1109 · LoadDishwasher_dish0

FINAL WRONG chunked (true memory recall) fine-only task=LoadDishwasher turns=34 (memory 33 + final 1) memory-turn match 31/33

FINAL-QUERY VERDICT: instruction Task: Show me where the cup is placed. → target GT Place the bowl in the dishwasher; model said Place the cup in the dishwasher — MISMATCH (wrong recall) | target_chunked=True.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Pick up the cup and bowl from the counter, place them	Pick up the cup from the counter	Pick up the cup from the counter	✓	m=[] g=[]
1	fine	Task: Pick up the cup and bowl from the counter, place them	Place the cup in the dishwasher	Place the cup in the dishwasher	✓	m=[] g=[]
2	fine	Task: Pick up the cup and bowl from the counter, place them	Pick up the bowl from the counter	Pick up the bowl from the counter	✓	m=[] g=[]
3	fine	Task: Pick up the cup and bowl from the counter, place them	Place the bowl in the dishwasher	Place the bowl in the dishwasher	✓	m=[8] g=[8]
4	fine	Task: Pick up the cup and bowl from the counter, place them	Close the dishwasher door	Close the dishwasher door	✓	m=[] g=[]
5	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
6	fine	Task: Pick the bell pepper from the sink and place it in the	Pick up the bell pepper from the sink	Pick up the bell pepper from the sink	✓	m=[] g=[]
7	fine	Task: Pick the bell pepper from the sink and place it in the	Place the bell pepper in the bowl	Place the bell pepper in the bowl	✓	m=[] g=[]
8	fine	Task: Pick the bell pepper from the sink and place it in the	Pick up the bowl from the counter	Pick up the bowl from the counter	✓	m=[] g=[]
9	fine	Task: Pick the bell pepper from the sink and place it in the	Place the bowl inside the microwave	Place the bowl inside the microwave	✓	m=[8] g=[8]
10	fine	Task: Pick the bell pepper from the sink and place it in the	Close the microwave door	Close the microwave door	✓	m=[] g=[]
11	fine	Task: Pick the bell pepper from the sink and place it in the	Press the start button on the microwave	Press the start button on the microwave	✓	m=[] g=[]
12	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
13	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Pull the toaster oven rack outwards	Pull the toaster oven rack outwards	✓	m=[] g=[]
14	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Pick up the baguette from the plate on the counter	Pick up the baguette from the plate on the counter	✓	m=[] g=[]
15	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Place the baguette on the toaster oven rack	Place the baguette on the toaster oven rack	✓	m=[8] g=[8]
16	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Close the toaster oven door fully	Close the toaster oven door fully	✓	m=[] g=[]
17	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Set the toaster oven temperature to the lowest setting	Set the toaster oven temperature to the lowest setting	✓	m=[] g=[]
18	fine/transit	Task: Approach the drawer for the next task.	Navigate to the drawer	Navigate to the drawer	✓	m=[] g=[]
19	fine	Task: Pick up the straw from the left drawer and put it insi	Pick up the straw from the drawer	Pick up the straw from the drawer	✓	m=[] g=[]
20	fine	Task: Pick up the straw from the left drawer and put it insi	Place the straw in the smoothie cup	Place the straw in the smoothie cup	✓	m=[8] g=[8]
21	fine/transit	Task: Approach the cabinet for the next task.	Navigate to the cabinet	Navigate to the cabinet	✓	m=[] g=[]
22	fine	Task: Pick the chicken drumstick from the plate and place it	Pick up the chicken drumstick from the cabinet	Pick up the chicken drumstick from the cabinet	✓	m=[] g=[]
23	fine	Task: Pick the chicken drumstick from the plate and place it	Place the chicken drumstick on the cutting board	Place the chicken drumstick on the cutting board	✓	m=[] g=[]
24	fine	Task: Pick the chicken drumstick from the plate and place it	Pick up the chicken drumstick from the cabinet	Pick up the condiment bottle from the cabinet	✗	m=[] g=[]
25	fine	Task: Pick the chicken drumstick from the plate and place it	Place the chicken drumstick in the cutting board	Place the condiment bottle in the cabinet	✗	m=[8] g=[8]
26	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
27	fine	Task: Open the cabinet and move the ladle to the pot. Then c	Close the cabinet door	Close the cabinet door	✓	m=[8] g=[8]
28	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
29	fine	Task: Pick up the potato and potato from the counter and pla	Pick up the potato from the counter	Pick up the potato from the counter	✓	m=[] g=[]
30	fine	Task: Pick up the potato and potato from the counter and pla	Place the potato in the pan	Place the potato in the pan	✓	m=[] g=[]
31	fine	Task: Pick up the potato and potato from the counter and pla	Pick up the potato from the counter	Pick up the potato from the counter	✓	m=[] g=[]
32	fine	Task: Pick up the potato and potato from the counter and pla	Place the potato in the pan	Place the potato in the pan	✓	m=[8] g=[8]
33 (final)	FINAL	Task: Show me where the cup is placed.	Place the bowl in the dishwasher	Place the cup in the dishwasher	✗	m=[1] g=[1]

►combo_039_L42_S25_0784 · FoodCleanup_food0

FINAL WRONG chunked (true memory recall) has §7 composite turn task=FoodCleanup turns=19 (memory 18 + final 1) memory-turn match 16/18

FINAL-QUERY VERDICT: instruction Task: Show me where the kiwi is placed. → target GT Place the kiwi in the cabinet; model said Place the cup inside the bowl on the dining table and move the bowl to any counter. — MISMATCH (wrong recall) | target_chunked=True.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	composite	Task: Transport the fish from the fridge to the microwave an	Transport the fish from the fridge to the microwave and turn on the microwave.	Place the fish in the microwave	✗	m=[8] g=[8]
1	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
2	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Pull the toaster oven rack outwards	Pull the toaster oven rack outwards	✓	m=[] g=[]
3	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Pick up the baguette from the plate on the counter	Pick up the baguette from the plate on the counter	✓	m=[] g=[]
4	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Place the baguette on the toaster oven rack	Place the baguette on the toaster oven rack	✓	m=[8] g=[8]
5	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Close the toaster oven door fully	Close the toaster oven door fully	✓	m=[] g=[]
6	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Set the toaster oven temperature to the lowest setting	Set the toaster oven temperature to the lowest setting	✓	m=[] g=[]
7	fine/transit	Task: Approach the cabinet for the next task.	Navigate to the cabinet	Navigate to the cabinet	✓	m=[] g=[]
8	fine	Task: Pick the kiwi from the counter and place it in the cab	Pick up the kiwi from the counter	Pick up the kiwi from the counter	✓	m=[] g=[]
9	fine	Task: Pick the kiwi from the counter and place it in the cab	Place the kiwi in the cabinet	Place the kiwi in the cabinet	✓	m=[8] g=[8]
10	fine	Task: Pick the kiwi from the counter and place it in the cab	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
11	fine/transit	Task: Approach the fixture for the next task.	Navigate to the fixture	Navigate to the fixture	✓	m=[] g=[]
12	fine	Task: Take the fish from the oven and place it on the plate	Pick up the fish from the oven	Pick up the fish from the oven	✓	m=[] g=[]
13	fine	Task: Take the fish from the oven and place it on the plate	Place the fish on the counter	Place the fish on the counter	✓	m=[8] g=[8]
14	fine/transit	Task: Approach the fixture for the next task.	Navigate to the fixture	Navigate to the fixture	✓	m=[] g=[]
15	fine	Task: Close the oven door.	Close the oven door	Close the oven door	✓	m=[8] g=[8]
16	fine/transit	Task: Approach the stool for the next task.	Navigate to the stool	Navigate to the stool	✓	m=[] g=[]
17	composite	Task: Place the cup inside the bowl on the dining table and	Place the cup inside the bowl on the dining table and move the bowl to any counter.	Pick up the cup from the counter	✗	m=[] g=[8]
18 (final)	FINAL	Task: Show me where the kiwi is placed.	Place the kiwi in the cabinet	Place the cup inside the bowl on the dining table and move the bowl to any counter.	✗	m=[8] g=[3]

►combo_014_L29_S48_0328 · PlaceMicrowaveSafeItem_microwave_safe_item

FINAL CORRECT not-chunked (prior floor) fine-only task=PlaceMicrowaveSafeItem turns=35 (memory 34 + final 1) memory-turn match 34/34

FINAL-QUERY VERDICT: instruction Task: Show me where the cup with liquid is placed. → target GT Place the cup in the microwave; model said Place the cup in the microwave — MATCH (correct recall) | target_chunked=False.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: There's a steak in the fridge that needs to be moved t	Pick up the steak from the fridge	Pick up the steak from the fridge	✓	m=[] g=[]
1	fine	Task: There's a steak in the fridge that needs to be moved t	Place the steak on a shelf in the freezer	Place the steak on a shelf in the freezer	✓	m=[8] g=[8]
2	fine	Task: There's a steak in the fridge that needs to be moved t	Close the fridge door	Close the fridge door	✓	m=[] g=[]
3	fine	Task: There's a steak in the fridge that needs to be moved t	Close the freezer door	Close the freezer door	✓	m=[] g=[]
4	fine/transit	Task: Approach the cabinet for the next task.	Navigate to the cabinet	Navigate to the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
6	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
7	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
8	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
9	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
10	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
11	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
12	fine	Task: Place the microwavable item in the microwave. Then, cl	Pick up the cup from the counter	Pick up the cup from the counter	✓	m=[] g=[]
13	fine	Task: Place the microwavable item in the microwave. Then, cl	Place the cup in the microwave	Place the cup in the microwave	✓	m=[8] g=[8]
14	fine	Task: Place the microwavable item in the microwave. Then, cl	Pick up the cup from the counter	Pick up the cup from the counter	✓	m=[] g=[]
15	fine	Task: Place the microwavable item in the microwave. Then, cl	Place the cup in the microwave	Place the cup in the microwave	✓	m=[] g=[]
16	fine	Task: Place the microwavable item in the microwave. Then, cl	Close the microwave door	Close the microwave door	✓	m=[] g=[]
17	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
18	fine	Task: Open the cabinet. Pick the spray and place it next to	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
19	fine	Task: Open the cabinet. Pick the spray and place it next to	Pick up the spray from the cabinet	Pick up the spray from the cabinet	✓	m=[] g=[]
20	fine	Task: Open the cabinet. Pick the spray and place it next to	Place the spray in the sink	Place the spray in the sink	✓	m=[8] g=[8]
21	fine	Task: Open the cabinet. Pick the spray and place it next to	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
22	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
23	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the pan from the counter	Pick up the pan from the counter	✓	m=[] g=[]
24	fine	Task: Place the pan on the front left burner on the stove. T	Place the pan on the stove	Place the pan on the stove	✓	m=[] g=[]
25	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the tomato from the counter	Pick up the tomato from the counter	✓	m=[] g=[]
26	fine	Task: Place the pan on the front left burner on the stove. T	Place the tomato in the pan	Place the tomato in the pan	✓	m=[] g=[]
27	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the onion from the counter	Pick up the onion from the counter	✓	m=[] g=[]
28	fine	Task: Place the pan on the front left burner on the stove. T	Place the onion in the pan	Place the onion in the pan	✓	m=[8] g=[8]
29	fine/transit	Task: Approach the counter for the next task.	Navigate to the counter	Navigate to the counter	✓	m=[] g=[]
30	fine	Task: Place the bowl with cake and the cupcake onto the tray	Pick up the bowl from the counter	Pick up the bowl from the counter	✓	m=[] g=[]
31	fine	Task: Place the bowl with cake and the cupcake onto the tray	Place the bowl on the tray	Place the bowl in the tray	✓	m=[] g=[]
32	fine	Task: Place the bowl with cake and the cupcake onto the tray	Pick up the cupcake from the counter	Pick up the cupcake from the counter	✓	m=[] g=[]
33	fine	Task: Place the bowl with cake and the cupcake onto the tray	Place the cupcake on the tray	Place the cupcake in the tray	✓	m=[8] g=[8]
34 (final)	FINAL	Task: Show me where the cup with liquid is placed.	Place the cup in the microwave	Place the cup in the microwave	✓	m=[2] g=[3]

►combo_002_L29_S48_0089 · OvenBroilFish_obj

FINAL CORRECT not-chunked (prior floor) has §7 composite turn task=OvenBroilFish turns=31 (memory 30 + final 1) memory-turn match 30/30

FINAL-QUERY VERDICT: instruction Task: Show me where the fish is placed. → target GT Place the fish on the bottom rack of the oven; model said Place the fish on the bottom rack of the oven — MATCH (correct recall) | target_chunked=False.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: There's a steak in the fridge that needs to be moved t	Pick up the steak from the fridge	Pick up the steak from the fridge	✓	m=[] g=[]
1	fine	Task: There's a steak in the fridge that needs to be moved t	Place the steak on a shelf in the freezer	Place the steak on a shelf in the freezer	✓	m=[8] g=[8]
2	fine	Task: There's a steak in the fridge that needs to be moved t	Close the fridge door	Close the fridge door	✓	m=[] g=[]
3	fine	Task: There's a steak in the fridge that needs to be moved t	Close the freezer door	Close the freezer door	✓	m=[] g=[]
4	fine/transit	Task: Approach the cabinet for the next task.	Navigate to the cabinet	Navigate to the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
6	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
7	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
8	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
9	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
10	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
11	fine/transit	Task: Approach the oven for the next task.	Navigate to the oven	Navigate to the oven	✓	m=[] g=[]
12	fine	Task: Place the fish on the bottom rack of the oven, and sli	Pick up the fish from the counter	Pick up the fish from the counter	✓	m=[] g=[]
13	fine	Task: Place the fish on the bottom rack of the oven, and sli	Place the fish on the bottom rack of the oven	Place the fish on the bottom rack of the oven	✓	m=[8] g=[8]
14	fine	Task: Place the fish on the bottom rack of the oven, and sli	Slide the oven rack back inside	Slide the oven rack back inside	✓	m=[] g=[]
15	fine	Task: Place the fish on the bottom rack of the oven, and sli	Close the oven door	Close the oven door	✓	m=[] g=[]
16	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
17	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the pan from the counter	Pick up the pan from the counter	✓	m=[] g=[]
18	fine	Task: Place the pan on the front left burner on the stove. T	Place the pan on the stove	Place the pan on the stove	✓	m=[] g=[]
19	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the tomato from the counter	Pick up the tomato from the counter	✓	m=[] g=[]
20	fine	Task: Place the pan on the front left burner on the stove. T	Place the tomato in the pan	Place the tomato in the pan	✓	m=[] g=[]
21	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the onion from the counter	Pick up the onion from the counter	✓	m=[] g=[]
22	fine	Task: Place the pan on the front left burner on the stove. T	Place the onion in the pan	Place the onion in the pan	✓	m=[8] g=[8]
23	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
24	composite	Task: Remove the onion and tangerine from the sink. Then mov	Remove the onion and tangerine from the sink. Then move the bowl and cup from the counter to the sink for washing.	Remove the onion and tangerine from the sink. Then move the bowl and cup from the counter to the sink for washing.	✓	m=[8] g=[8]
25	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
26	fine	Task: Move the pot and pan from the stove into the sink.	Pick up the pot from the stove	Pick up the pot from the stove	✓	m=[] g=[]
27	fine	Task: Move the pot and pan from the stove into the sink.	Place the pot in the sink	Place the pot in the sink	✓	m=[] g=[]
28	fine	Task: Move the pot and pan from the stove into the sink.	Pick up the pan from the stove	Pick up the pan from the stove	✓	m=[] g=[]
29	fine	Task: Move the pot and pan from the stove into the sink.	Place the pan in the sink	Place the pan in the sink	✓	m=[8] g=[8]
30 (final)	FINAL	Task: Show me where the fish is placed.	Place the fish on the bottom rack of the oven	Place the fish on the bottom rack of the oven	✓	m=[3] g=[3]

►combo_015_L50_S40_0435 · RestockBowls_obj1

FINAL CORRECT chunked (true memory recall) fine-only task=RestockBowls turns=30 (memory 29 + final 1) memory-turn match 29/29

FINAL-QUERY VERDICT: instruction Task: Show me where the bowl is placed. → target GT Place a bowl in the cabinet; model said Place a bowl in the cabinet — MATCH (correct recall) | target_chunked=True.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
1	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
2	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
3	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
4	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
6	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
7	fine	Task: Remove the pear from the bowl and place it on the smal	Pick up the bowl from the bowl	Pick up the bowl from the bowl	✓	m=[] g=[]
8	fine	Task: Remove the pear from the bowl and place it on the smal	Place the bowl in the small plate	Place the bowl in the small plate	✓	m=[8] g=[8]
9	fine	Task: Remove the pear from the bowl and place it on the smal	Close the microwave door	Close the microwave door	✓	m=[] g=[]
10	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
11	fine	Task: Pick up the bagel from the plate, place it on the toas	Pick up the bagel from the plate	Pick up the bagel from the plate	✓	m=[] g=[]
12	fine	Task: Pick up the bagel from the plate, place it on the toas	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[8] g=[8]
13	fine	Task: Pick up the bagel from the plate, place it on the toas	Close the toaster oven door	Close the toaster oven door	✓	m=[] g=[]
14	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
15	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
16	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[] g=[]
17	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
18	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[] g=[]
19	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
20	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[8] g=[8]
21	fine/transit	Task: Approach the next location for the next task.	Navigate to the next location	Navigate to the next location	✓	m=[] g=[]
22	fine	Task: Pick the mug from the counter and place it under the c	Pick up the mug from the counter	Pick up the mug from the counter	✓	m=[] g=[]
23	fine	Task: Pick the mug from the counter and place it under the c	Place the mug under the coffee machine dispenser	Place the mug under the coffee machine dispenser	✓	m=[8] g=[8]
24	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
25	fine	Task: Soak the sponge with water for a few seconds and then	Pick up the sponge from the counter	Pick up the sponge from the counter	✓	m=[] g=[]
26	fine	Task: Soak the sponge with water for a few seconds and then	Place the sponge inside the sink	Place the sponge inside the sink	✓	m=[8] g=[8]
27	fine/transit	Task: Approach the drawer for the next task.	Navigate to the drawer	Navigate to the drawer	✓	m=[] g=[]
28	fine	Task: Close the left drawer.	Close the left drawer	Close the left drawer	✓	m=[8] g=[8]
29 (final)	FINAL	Task: Show me where the bowl is placed.	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[1] g=[1]

►combo_002_L29_S48_0089 · MoveFridgeToFreezer_refrigerated_item

FINAL CORRECT chunked (true memory recall) has §7 composite turn task=MoveFridgeToFreezer turns=31 (memory 30 + final 1) memory-turn match 30/30

FINAL-QUERY VERDICT: instruction Task: Show me where the steak is placed. → target GT Place the steak on a shelf in the freezer; model said Place the steak on a shelf in the freezer — MATCH (correct recall) | target_chunked=True.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: There's a steak in the fridge that needs to be moved t	Pick up the steak from the fridge	Pick up the steak from the fridge	✓	m=[] g=[]
1	fine	Task: There's a steak in the fridge that needs to be moved t	Place the steak on a shelf in the freezer	Place the steak on a shelf in the freezer	✓	m=[8] g=[8]
2	fine	Task: There's a steak in the fridge that needs to be moved t	Close the fridge door	Close the fridge door	✓	m=[] g=[]
3	fine	Task: There's a steak in the fridge that needs to be moved t	Close the freezer door	Close the freezer door	✓	m=[] g=[]
4	fine/transit	Task: Approach the cabinet for the next task.	Navigate to the cabinet	Navigate to the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
6	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
7	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
8	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
9	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
10	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
11	fine/transit	Task: Approach the oven for the next task.	Navigate to the oven	Navigate to the oven	✓	m=[] g=[]
12	fine	Task: Place the fish on the bottom rack of the oven, and sli	Pick up the fish from the counter	Pick up the fish from the counter	✓	m=[] g=[]
13	fine	Task: Place the fish on the bottom rack of the oven, and sli	Place the fish on the bottom rack of the oven	Place the fish on the bottom rack of the oven	✓	m=[8] g=[8]
14	fine	Task: Place the fish on the bottom rack of the oven, and sli	Slide the oven rack back inside	Slide the oven rack back inside	✓	m=[] g=[]
15	fine	Task: Place the fish on the bottom rack of the oven, and sli	Close the oven door	Close the oven door	✓	m=[] g=[]
16	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
17	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the pan from the counter	Pick up the pan from the counter	✓	m=[] g=[]
18	fine	Task: Place the pan on the front left burner on the stove. T	Place the pan on the stove	Place the pan on the stove	✓	m=[] g=[]
19	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the tomato from the counter	Pick up the tomato from the counter	✓	m=[] g=[]
20	fine	Task: Place the pan on the front left burner on the stove. T	Place the tomato in the pan	Place the tomato in the pan	✓	m=[] g=[]
21	fine	Task: Place the pan on the front left burner on the stove. T	Pick up the onion from the counter	Pick up the onion from the counter	✓	m=[] g=[]
22	fine	Task: Place the pan on the front left burner on the stove. T	Place the onion in the pan	Place the onion in the pan	✓	m=[8] g=[8]
23	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
24	composite	Task: Remove the onion and tangerine from the sink. Then mov	Remove the onion and tangerine from the sink. Then move the bowl and cup from the counter to the sink for washing.	Remove the onion and tangerine from the sink. Then move the bowl and cup from the counter to the sink for washing.	✓	m=[8] g=[8]
25	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
26	fine	Task: Move the pot and pan from the stove into the sink.	Pick up the pot from the stove	Pick up the pot from the stove	✓	m=[] g=[]
27	fine	Task: Move the pot and pan from the stove into the sink.	Place the pot in the sink	Place the pot in the sink	✓	m=[] g=[]
28	fine	Task: Move the pot and pan from the stove into the sink.	Pick up the pan from the stove	Pick up the pan from the stove	✓	m=[] g=[]
29	fine	Task: Move the pot and pan from the stove into the sink.	Place the pan in the sink	Place the pan in the sink	✓	m=[8] g=[8]
30 (final)	FINAL	Task: Show me where the steak is placed.	Place the steak on a shelf in the freezer	Place the steak on a shelf in the freezer	✓	m=[1] g=[1]

►combo_015_L50_S40_0435 · FilterMicrowavableItem_meat

FINAL WRONG not-chunked (prior floor) fine-only task=FilterMicrowavableItem turns=30 (memory 29 + final 1) memory-turn match 28/29

FINAL-QUERY VERDICT: instruction Task: Show me where the shrimp is placed. → target GT Place the bowl in the microwave; model said Place the bowl in the small plate — MISMATCH (wrong recall) | target_chunked=False.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
1	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
2	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
3	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
4	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
6	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
7	fine	Task: Remove the pear from the bowl and place it on the smal	Pick up the bowl from the bowl	Pick up the bowl from the bowl	✓	m=[] g=[]
8	fine	Task: Remove the pear from the bowl and place it on the smal	Place the bowl in the microwave	Place the bowl in the small plate	✗	m=[8] g=[8]
9	fine	Task: Remove the pear from the bowl and place it on the smal	Close the microwave door	Close the microwave door	✓	m=[] g=[]
10	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
11	fine	Task: Pick up the bagel from the plate, place it on the toas	Pick up the bagel from the plate	Pick up the bagel from the plate	✓	m=[] g=[]
12	fine	Task: Pick up the bagel from the plate, place it on the toas	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[8] g=[8]
13	fine	Task: Pick up the bagel from the plate, place it on the toas	Close the toaster oven door	Close the toaster oven door	✓	m=[] g=[]
14	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
15	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
16	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[] g=[]
17	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
18	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[] g=[]
19	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
20	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[8] g=[8]
21	fine/transit	Task: Approach the next location for the next task.	Navigate to the next location	Navigate to the next location	✓	m=[] g=[]
22	fine	Task: Pick the mug from the counter and place it under the c	Pick up the mug from the counter	Pick up the mug from the counter	✓	m=[] g=[]
23	fine	Task: Pick the mug from the counter and place it under the c	Place the mug under the coffee machine dispenser	Place the mug under the coffee machine dispenser	✓	m=[8] g=[8]
24	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
25	fine	Task: Soak the sponge with water for a few seconds and then	Pick up the sponge from the counter	Pick up the sponge from the counter	✓	m=[] g=[]
26	fine	Task: Soak the sponge with water for a few seconds and then	Place the sponge inside the sink	Place the sponge inside the sink	✓	m=[8] g=[8]
27	fine/transit	Task: Approach the drawer for the next task.	Navigate to the drawer	Navigate to the drawer	✓	m=[] g=[]
28	fine	Task: Close the left drawer.	Close the left drawer	Close the left drawer	✓	m=[8] g=[8]
29 (final)	FINAL	Task: Show me where the shrimp is placed.	Place the bowl in the microwave	Place the bowl in the small plate	✗	m=[2] g=[2]

►combo_011_L33_S36_0154 · MicrowaveThawingFridge_meat

FINAL WRONG not-chunked (prior floor) has §7 composite turn task=MicrowaveThawingFridge turns=24 (memory 23 + final 1) memory-turn match 17/23

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
1	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
2	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
3	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
4	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
6	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
7	fine	Task: Pick up the bagel from the plate, place it on the toas	Pick up the bagel from the plate	Pick up the bagel from the plate	✓	m=[] g=[]
8	fine	Task: Pick up the bagel from the plate, place it on the toas	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[8] g=[8]
9	fine	Task: Pick up the bagel from the plate, place it on the toas	Close the toaster oven door	Close the toaster oven door	✓	m=[] g=[]
10	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
11	fine	Task: Transport the steak from the fridge to the microwave a	Pick up the steak from the fridge	Transport the steak from the fridge to the microwave and turn on the microwave.	✗	m=[8] g=[]
12	fine	Task: Transport the steak from the fridge to the microwave a	Place the steak in the microwave	Turn on the microwave	✗	m=[] g=[8]
13	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
14	composite	Task: Place the chicken drumstick on the plate and place the	Place the chicken drumstick on the plate and place the onion on the same plate.	Place the chicken drumstick on the plate and place the onion on the same plate.	✓	m=[8] g=[8]
15	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
16	fine	Task: Stack the bowls in the sink.	Pick up a bowl from the counter	Stack the bowls in the sink.	✗	m=[8] g=[]
17	fine	Task: Stack the bowls in the sink.	Place a bowl in the sink	Stack the bowls in the sink.	✗	m=[] g=[]
18	fine	Task: Stack the bowls in the sink.	Pick up a bowl from the counter	Stack the bowls in the sink.	✗	m=[8] g=[]
19	fine	Task: Stack the bowls in the sink.	Place a bowl in the sink	Stack the bowls in the sink.	✗	m=[] g=[8]
20	fine/transit	Task: Approach the drawer for the next task.	Navigate to the drawer	Navigate to the drawer	✓	m=[] g=[]
21	fine	Task: Take the pizza cutter from the drawer on the left and	Pick up the pizza from the drawer	Pick up the pizza cutter from the drawer	✓	m=[] g=[]
22	fine	Task: Take the pizza cutter from the drawer on the left and	Place the pizza on the dining counter	Place the pizza cutter in the dining counter	✓	m=[8] g=[8]
23 (final)	FINAL	Task: Show me where the steak is placed.	Place the steak in the microwave	Place the steak on the toaster oven	✗	m=[2] g=[3]

►combo_072_L56_S42_1067 · LoadDishwasher_dish1

FINAL WRONG chunked (true memory recall) has §7 composite turn task=LoadDishwasher turns=32 (memory 31 + final 1) memory-turn match 31/31

FINAL-QUERY VERDICT: instruction Task: Show me where the bowl is placed. → target GT Place the bowl in the dishwasher; model said Place the bowl inside the microwave — MISMATCH (wrong recall) | target_chunked=True.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Pick up the cup and bowl from the counter, place them	Pick up the cup from the counter	Pick up the cup from the counter	✓	m=[] g=[]
1	fine	Task: Pick up the cup and bowl from the counter, place them	Place the cup in the dishwasher	Place the cup in the dishwasher	✓	m=[] g=[]
2	fine	Task: Pick up the cup and bowl from the counter, place them	Pick up the bowl from the counter	Pick up the bowl from the counter	✓	m=[] g=[]
3	fine	Task: Pick up the cup and bowl from the counter, place them	Place the bowl in the dishwasher	Place the bowl in the dishwasher	✓	m=[8] g=[8]
4	fine	Task: Pick up the cup and bowl from the counter, place them	Close the dishwasher door	Close the dishwasher door	✓	m=[] g=[]
5	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
6	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Pull the toaster oven rack outwards	Pull the toaster oven rack outwards	✓	m=[] g=[]
7	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Pick up the baguette from the plate on the counter	Pick up the baguette from the plate on the counter	✓	m=[] g=[]
8	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Place the baguette on the toaster oven rack	Place the baguette on the toaster oven rack	✓	m=[8] g=[8]
9	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Close the toaster oven door fully	Close the toaster oven door fully	✓	m=[] g=[]
10	fine	Task: Pull the toaster oven rack outwards. Pick up the bague	Set the toaster oven temperature to the lowest setting	Set the toaster oven temperature to the lowest setting	✓	m=[] g=[]
11	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
12	fine	Task: Pick the bell pepper from the sink and place it in the	Pick up the bell pepper from the sink	Pick up the bell pepper from the sink	✓	m=[] g=[]
13	fine	Task: Pick the bell pepper from the sink and place it in the	Place the bell pepper in the bowl	Place the bell pepper in the bowl	✓	m=[] g=[]
14	fine	Task: Pick the bell pepper from the sink and place it in the	Pick up the bowl from the counter	Pick up the bowl from the counter	✓	m=[] g=[]
15	fine	Task: Pick the bell pepper from the sink and place it in the	Place the bowl inside the microwave	Place the bowl inside the microwave	✓	m=[8] g=[8]
16	fine	Task: Pick the bell pepper from the sink and place it in the	Close the microwave door	Close the microwave door	✓	m=[] g=[]
17	fine	Task: Pick the bell pepper from the sink and place it in the	Press the start button on the microwave	Press the start button on the microwave	✓	m=[] g=[]
18	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
19	fine	Task: Pick up the butter stick and place it in the pan. Then	Pick up the butter stick from the counter	Pick up the butter stick from the counter	✓	m=[] g=[]
20	fine	Task: Pick up the butter stick and place it in the pan. Then	Place the butter stick in the pan	Place the butter stick in the pan	✓	m=[8] g=[8]
21	fine/transit	Task: Approach the fridge for the next task.	Navigate to the fridge	Navigate to the fridge	✓	m=[] g=[]
22	composite	Task: Place the bowl with the broccoli and lemon on the seco	Place the bowl with the broccoli and lemon on the second highest shelf of the fridge. Place the bowl with the fish and steak on the top shelf of the fridge.	Place the bowl with the broccoli and lemon on the second highest shelf of the fridge. Place the bowl with the fish and steak on the top shelf of the fridge.	✓	m=[8] g=[8]
23	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
24	fine	Task: Pick up the potato and potato from the counter and pla	Pick up the potato from the counter	Pick up the potato from the counter	✓	m=[] g=[]
25	fine	Task: Pick up the potato and potato from the counter and pla	Place the potato in the pan	Place the potato in the pan	✓	m=[] g=[]
26	fine	Task: Pick up the potato and potato from the counter and pla	Pick up the potato from the counter	Pick up the potato from the counter	✓	m=[] g=[]
27	fine	Task: Pick up the potato and potato from the counter and pla	Place the potato in the pan	Place the potato in the pan	✓	m=[8] g=[8]
28	fine/transit	Task: Approach the drawer for the next task.	Navigate to the drawer	Navigate to the drawer	✓	m=[] g=[]
29	fine	Task: Pick up the straw from the left drawer and put it insi	Pick up the straw from the drawer	Pick up the straw from the drawer	✓	m=[] g=[]
30	fine	Task: Pick up the straw from the left drawer and put it insi	Place the straw in the smoothie cup	Place the straw in the smoothie cup	✓	m=[8] g=[8]
31 (final)	FINAL	Task: Show me where the bowl is placed.	Place the bowl in the dishwasher	Place the bowl inside the microwave	✗	m=[3] g=[1]

►combo_024_L39_S40_0621 · OvenBroilFish_obj

FINAL CORRECT not-chunked (prior floor) fine-only task=OvenBroilFish turns=34 (memory 33 + final 1) memory-turn match 28/33

FINAL-QUERY VERDICT: instruction Task: Show me where the fish is placed. → target GT Place the fish on the rack of the oven; model said Place the fish on the rack of the oven — MATCH (correct recall) | target_chunked=False.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Place the fish on the rack of the oven, and slide the	Pick up the fish from the counter	Pick up the fish from the counter	✓	m=[] g=[]
1	fine	Task: Place the fish on the rack of the oven, and slide the	Place the fish on the rack of the oven	Place the fish on the rack of the oven	✓	m=[8] g=[8]
2	fine	Task: Place the fish on the rack of the oven, and slide the	Slide the oven rack back inside	Slide the oven rack back inside	✓	m=[] g=[]
3	fine	Task: Place the fish on the rack of the oven, and slide the	Close the oven door	Close the oven door	✓	m=[] g=[]
4	fine/transit	Task: Approach the cabinet for the next task.	Navigate to the cabinet	Navigate to the cabinet	✓	m=[] g=[]
5	fine	Task: Pick the tangerine from the counter and place it in th	Pick up the tangerine from the counter	Pick up the tangerine from the counter	✓	m=[] g=[]
6	fine	Task: Pick the tangerine from the counter and place it in th	Place the tangerine in the cabinet	Place the tangerine in the cabinet	✓	m=[8] g=[8]
7	fine	Task: Pick the tangerine from the counter and place it in th	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
8	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
9	fine	Task: Pick the carrot from the sink and place it in the bowl	Pick up the carrot from the sink	Pick up the carrot from the sink	✓	m=[] g=[]
10	fine	Task: Pick the carrot from the sink and place it in the bowl	Place the carrot in the bowl	Place the carrot in the bowl	✓	m=[] g=[]
11	fine	Task: Pick the carrot from the sink and place it in the bowl	Pick up the bowl from the counter	Pick up the bowl from the counter	✓	m=[] g=[]
12	fine	Task: Pick the carrot from the sink and place it in the bowl	Place the bowl inside the microwave	Place the bowl inside the microwave	✓	m=[8] g=[8]
13	fine	Task: Pick the carrot from the sink and place it in the bowl	Close the microwave door	Close the microwave door	✓	m=[] g=[]
14	fine	Task: Pick the carrot from the sink and place it in the bowl	Press the start button on the microwave	Press the start button on the microwave	✓	m=[] g=[]
15	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
16	fine	Task: Place the pan on the center burner on the stove. Then	Pick up the pan from the counter	Pick up the pan from the counter	✓	m=[] g=[]
17	fine	Task: Place the pan on the center burner on the stove. Then	Place the pan on the stove	Place the pan on the stove	✓	m=[] g=[]
18	fine	Task: Place the pan on the center burner on the stove. Then	Pick up the tomato from the counter	Pick up the tomato from the counter	✓	m=[] g=[]
19	fine	Task: Place the pan on the center burner on the stove. Then	Place the tomato in the pan	Place the tomato in the pan	✓	m=[] g=[]
20	fine	Task: Place the pan on the center burner on the stove. Then	Pick up the onion from the counter	Pick up the onion from the counter	✓	m=[] g=[]
21	fine	Task: Place the pan on the center burner on the stove. Then	Place the onion in the pan	Place the onion in the pan	✓	m=[8] g=[8]
22	fine/transit	Task: Approach the stove for the next task.	Navigate to the stove	Navigate to the stove	✓	m=[] g=[]
23	fine	Task: Put the corn and eggplant in the pan. Then turn on the	Pick up the pan from the stove	Pick up the corn from the counter	✗	m=[] g=[]
24	fine	Task: Put the corn and eggplant in the pan. Then turn on the	Place the pan in the pan	Place the corn in the pan	✓	m=[] g=[]
25	fine	Task: Put the corn and eggplant in the pan. Then turn on the	Pick up the corn from the counter	Pick up the eggplant from the counter	✗	m=[] g=[]
26	fine	Task: Put the corn and eggplant in the pan. Then turn on the	Place the corn in the pan	Place the eggplant in the pan	✗	m=[8] g=[]
27	fine	Task: Put the corn and eggplant in the pan. Then turn on the	Pick up the eggplant from the counter	Turn on the front right burner	✗	m=[] g=[]
28	fine	Task: Put the corn and eggplant in the pan. Then turn on the	Place the eggplant in the pan	Set the front right burner to medium	✗	m=[] g=[8]
29	fine/transit	Task: Approach the fixture for the next task.	Navigate to the fixture	Navigate to the fixture	✓	m=[] g=[]
30	fine	Task: Close the cabinet door.	Close the cabinet door	Close the cabinet door	✓	m=[8] g=[8]
31	fine/transit	Task: Approach the fixture for the next task.	Navigate to the fixture	Navigate to the fixture	✓	m=[] g=[]
32	fine	Task: Open the oven door.	Open the oven door	Open the oven door	✓	m=[8] g=[8]
33 (final)	FINAL	Task: Show me where the fish is placed.	Place the fish on the rack of the oven	Place the fish on the rack of the oven	✓	m=[1] g=[1]

►combo_005_L42_S17_0130 · OvenBroilFish_obj

FINAL CORRECT not-chunked (prior floor) has §7 composite turn task=OvenBroilFish turns=28 (memory 27 + final 1) memory-turn match 23/27

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Pick up the bagel from the plate, place it on the toas	Pick up the bagel from the plate	Pick up the bagel from the plate	✓	m=[] g=[]
1	fine	Task: Pick up the bagel from the plate, place it on the toas	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[8] g=[8]
2	fine	Task: Pick up the bagel from the plate, place it on the toas	Close the toaster oven door	Close the toaster oven door	✓	m=[] g=[]
3	fine/transit	Task: Approach the cabinet for the next task.	Navigate to the cabinet	Navigate to the cabinet	✓	m=[] g=[]
4	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
6	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
7	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
8	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
9	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
10	fine/transit	Task: Approach the oven for the next task.	Navigate to the oven	Navigate to the oven	✓	m=[] g=[]
11	fine	Task: Place the fish on the bottom rack of the oven, and sli	Pick up the fish from the counter	Pick up the fish from the counter	✓	m=[] g=[]
12	fine	Task: Place the fish on the bottom rack of the oven, and sli	Place the fish on the bottom rack of the oven	Place the fish on the bottom rack of the oven	✓	m=[8] g=[8]
13	fine	Task: Place the fish on the bottom rack of the oven, and sli	Slide the oven rack back inside	Slide the oven rack back inside	✓	m=[] g=[]
14	fine	Task: Place the fish on the bottom rack of the oven, and sli	Close the oven door	Close the oven door	✓	m=[] g=[]
15	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
16	composite	Task: Pick the cup of tea, navigate to the dining table, and	Pick the cup of tea, navigate to the dining table, and place it on the saucer.	Pick up the cup from the microwave	✗	m=[] g=[8]
17	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
18	composite	Task: Pick the objects in the sink and place them on the cou	Pick the objects in the sink and place them on the counter to clear sink space for washing produce.	Pick up the cup from the sink	✗	m=[] g=[8]
19	fine/transit	Task: Approach the fridge for the next task.	Navigate to the fridge	Navigate to the fridge	✓	m=[] g=[]
20	fine	Task: Pick the apple and banana from the fridge and place th	Pick up the apple from the fridge	Pick up the apple from the fridge	✓	m=[] g=[]
21	fine	Task: Pick the apple and banana from the fridge and place th	Place the apple on the counter	Place the apple in the counter	✓	m=[] g=[]
22	fine	Task: Pick the apple and banana from the fridge and place th	Pick up the banana from the fridge	Pick up the banana from the fridge	✓	m=[] g=[]
23	fine	Task: Pick the apple and banana from the fridge and place th	Place the banana on the counter	Place the banana in the counter	✓	m=[8] g=[8]
24	fine/transit	Task: Approach the fridge for the next task.	Navigate to the fridge	Navigate to the fridge	✓	m=[] g=[]
25	fine	Task: Move the chicken drumstick and burrito from the second	Pick up the pitcher from the counter	Pick up the chicken drumstick from the second highest shelf	✗	m=[] g=[]
26	fine	Task: Move the chicken drumstick and burrito from the second	Place the pitcher in the top shelf	Place the chicken drumstick on the top shelf	✗	m=[] g=[8]
27 (final)	FINAL	Task: Show me where the fish is placed.	Place the fish on the bottom rack of the oven	Place the fish on the bottom rack of the oven	✓	m=[3] g=[3]

►combo_015_L50_S40_0435 · ToastBagel_bagel

FINAL CORRECT chunked (true memory recall) fine-only task=ToastBagel turns=30 (memory 29 + final 1) memory-turn match 29/29

FINAL-QUERY VERDICT: instruction Task: Show me where the bagel is placed. → target GT Place the bagel on the toaster oven; model said Place the bagel on the toaster oven — MATCH (correct recall) | target_chunked=True.

turn	type	instruction (first 60)	GT subtask	MemER predicted	match	keyframes
0	fine	Task: Open the cabinet. Pick the bowls from the counter and	Open the cabinet door	Open the cabinet door	✓	m=[] g=[]
1	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
2	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[8] g=[8]
3	fine	Task: Open the cabinet. Pick the bowls from the counter and	Pick up a bowl from the counter	Pick up a bowl from the counter	✓	m=[] g=[]
4	fine	Task: Open the cabinet. Pick the bowls from the counter and	Place a bowl in the cabinet	Place a bowl in the cabinet	✓	m=[] g=[]
5	fine	Task: Open the cabinet. Pick the bowls from the counter and	Close the cabinet door	Close the cabinet door	✓	m=[] g=[]
6	fine/transit	Task: Approach the microwave for the next task.	Navigate to the microwave	Navigate to the microwave	✓	m=[] g=[]
7	fine	Task: Remove the pear from the bowl and place it on the smal	Pick up the bowl from the bowl	Pick up the bowl from the bowl	✓	m=[] g=[]
8	fine	Task: Remove the pear from the bowl and place it on the smal	Place the bowl in the small plate	Place the bowl in the small plate	✓	m=[8] g=[8]
9	fine	Task: Remove the pear from the bowl and place it on the smal	Close the microwave door	Close the microwave door	✓	m=[] g=[]
10	fine/transit	Task: Approach the toaster oven for the next task.	Navigate to the toaster oven	Navigate to the toaster oven	✓	m=[] g=[]
11	fine	Task: Pick up the bagel from the plate, place it on the toas	Pick up the bagel from the plate	Pick up the bagel from the plate	✓	m=[] g=[]
12	fine	Task: Pick up the bagel from the plate, place it on the toas	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[8] g=[8]
13	fine	Task: Pick up the bagel from the plate, place it on the toas	Close the toaster oven door	Close the toaster oven door	✓	m=[] g=[]
14	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
15	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
16	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[] g=[]
17	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
18	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[] g=[]
19	fine	Task: Pick the onions from the sink and place them in one bo	Pick up the onions from the sink	Pick up the onions from the sink	✓	m=[] g=[]
20	fine	Task: Pick the onions from the sink and place them in one bo	Place the onions in the one bowl	Place the onions in the one bowl	✓	m=[8] g=[8]
21	fine/transit	Task: Approach the next location for the next task.	Navigate to the next location	Navigate to the next location	✓	m=[] g=[]
22	fine	Task: Pick the mug from the counter and place it under the c	Pick up the mug from the counter	Pick up the mug from the counter	✓	m=[] g=[]
23	fine	Task: Pick the mug from the counter and place it under the c	Place the mug under the coffee machine dispenser	Place the mug under the coffee machine dispenser	✓	m=[8] g=[8]
24	fine/transit	Task: Approach the sink for the next task.	Navigate to the sink	Navigate to the sink	✓	m=[] g=[]
25	fine	Task: Soak the sponge with water for a few seconds and then	Pick up the sponge from the counter	Pick up the sponge from the counter	✓	m=[] g=[]
26	fine	Task: Soak the sponge with water for a few seconds and then	Place the sponge inside the sink	Place the sponge inside the sink	✓	m=[8] g=[8]
27	fine/transit	Task: Approach the drawer for the next task.	Navigate to the drawer	Navigate to the drawer	✓	m=[] g=[]
28	fine	Task: Close the left drawer.	Close the left drawer	Close the left drawer	✓	m=[8] g=[8]
29 (final)	FINAL	Task: Show me where the bagel is placed.	Place the bagel on the toaster oven	Place the bagel on the toaster oven	✓	m=[3] g=[3]

scene-mem v2 MemER B1 trainstruct eval · generated 2026-05-17 · videos under claude_reports/memer/260511-260517/videos/260517-b1/ (source of truth: eval_outputs/scene_mem_v2_b1_trainstruct_fullval/)