Scene-Mem 스모크 결과

TL;DR — 메모리 표현 3종(M1/M2/M3)의 recall accuracy를 2개 시나리오로 검증. 아래 집계 표 + 테스트별 '왜' inspector(예측 vs GT, 메모리 근거).

3

runs

0.33

best strict_acc

18

tests

1. 집계 비교 (method × model)

method	model	n	strict_acc	type_lenient	object_acc	parse_fail
m1	claude-sonnet-4-6	6	0.33	0.33	1.00	0
m2	claude-sonnet-4-6	6	0.17	0.17	1.00	0
m3	claude-sonnet-4-6	6	0.33	0.33	0.83	0

strict=예측 fixture==GT, type_lenient=fixture type 일치, object_acc=객체 카테고리 일치.

2. 테스트별 정성 inspector

m1 · claude-sonnet-4-6

scenario / test	예측 vs GT	rationale	메모리 근거
combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item	✗ pred: cab_6_main_group gt: fridge_main_group_freezer go to the cabinet and open it to reveal the steak	In the sequence of images, the robot can be seen handling a dark flat object (steak/meat) on the countertop with a spatula, and the final images show the robot placing items near the cabinet area. The steak appears to have been moved to the cabinet based on the robot's actions shown in the memory.	uniform subsample · n_frames=8
combo_002_L29_S48_0006 RestockBowls_obj1	✓ pred: cab_6_main_group gt: cab_6_main_group go to the cabinet and open it to show the bowl	In the memory images, a bowl (white/light colored) can be seen on the counter area near the cabinets. The sequence of robot actions shows the robot working near the cabinet area, and the bowl appears to have been placed in or near the cabinet (cab_6_main_group) based on the visible activity in the kitchen scene.	uniform subsample · n_frames=8
combo_002_L29_S48_0006 OvenBroilFish_obj	✗ pred: cab_6_main_group gt: oven_main_group go to the cabinet and open it	In the sequence of images, the robot can be seen handling objects on the countertop. The fish (appearing as a flat/elongated food item) was picked up and placed into the cabinet based on the robot's actions observed in the memory images.	uniform subsample · n_frames=8
combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item	✗ pred: cab_6_main_group gt: fridge_main_group_freezer go to the cabinet and open it to reveal the steak	In the memory images, the robot is seen placing an object (the steak) into the cabinet drawer area. The image showing the robot near the cabinet with what appears to be a flat object being placed into the cabinet/drawer supports this conclusion.	uniform subsample · n_frames=8
combo_002_L29_S48_0016 RestockBowls_obj1	✓ pred: cab_6_main_group gt: cab_6_main_group go to the cabinet and open it to reveal the bowl	In the memory images, the robot can be seen placing an object into a cabinet drawer area. The sequence shows the robot manipulating near the cabinet (cab_6_main_group), and the final images suggest the bowl was placed inside the cabinet.	uniform subsample · n_frames=8
combo_002_L29_S48_0016 OvenBroilFish_obj	✗ pred: cab_6_main_group gt: oven_main_group go to the cabinet and open it	In the memory images, the robot can be seen picking up an object (appears to be a fish/food item) from the counter area and placing it into the cabinet/drawer unit visible in the kitchen scene.	uniform subsample · n_frames=8

m2 · claude-sonnet-4-6

scenario / test	예측 vs GT	rationale	메모리 근거
combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item	✗ pred: cab_6_main_group gt: fridge_main_group_freezer go to the cabinet and open it	In the image sequence, the robot appears to have picked up a small gray/dark object from a tray and moved it. The activity shown does not indicate placement in the fridge or oven. Based on the memory, the steak was likely placed in the cabinet as the robot moved objects off the counter area toward the cabinet region.	idx=[7790, 7800, 8360, 9020, 9270, 9570, 9880, 10180]
combo_002_L29_S48_0006 RestockBowls_obj1	✗ pred: oven_main_group gt: cab_6_main_group go to the sink area and show the bowl placed inside it	Based on the image sequence, the robot picked up a bowl and placed it into the sink (the black basin on the counter). The bowl is visible sitting inside the sink in the later frames. However, since the sink is not in the fixture list, and the closest match based on the visible counter area is the oven, but actually looking at the images again - the bowl ends up in the sink area on the counter. Among the given options, none perfectly match, but the bowl was placed in the sink which is nearest to the oven_main_group fixture area.	idx=[7790, 7800, 8360, 9020, 9270, 9570, 9880, 10180]
combo_002_L29_S48_0006 OvenBroilFish_obj	✗ pred: cab_6_main_group gt: oven_main_group go to the cabinet and open it	In the image sequence, the robot picks up what appears to be a fish/food item from the tray area and places it into the sink area. However, reviewing the images more carefully, the bowl ends up in the sink area. The fish item is not clearly visible being placed in any of the listed fixtures. Based on the available memory showing the robot working near the counter and sink area, and given the options provided, the cabinet (cab_6_main_group) is the most likely location where the fish was stored, as the other items (bowl) went to the sink area.	idx=[7790, 7800, 8360, 9020, 9270, 9570, 9880, 10180]
combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item	✗ pred: cab_6_main_group gt: fridge_main_group_freezer go to the cabinet and open it to show the steak inside	In the final images, the robot is seen placing a steak-like object (brown/orange elongated item) into an open drawer/cabinet box on the counter area. The sequence shows the robot picking up what appears to be a steak from the countertop and placing it into a cabinet drawer.	idx=[4140, 4180, 4220, 6030, 6390, 6440, 6550, 6650]
combo_002_L29_S48_0016 RestockBowls_obj1	✓ pred: cab_6_main_group gt: cab_6_main_group go to the cabinet and open it to show the bowl	In the later images, the robot is seen near an open drawer/cabinet on the counter area, and a small object (bowl) appears to have been placed inside the cabinet drawer.	idx=[4140, 4180, 4220, 6030, 6390, 6440, 6550, 6650]
combo_002_L29_S48_0016 OvenBroilFish_obj	✗ pred: cab_6_main_group gt: oven_main_group go to the cabinet and open it	In the image sequence, the robot picks up what appears to be a fish/food item from the counter and places it into an open drawer/cabinet box on the floor level near the counter. The final images show the object inside a box/drawer that corresponds to the cabinet storage area.	idx=[4140, 4180, 4220, 6030, 6390, 6440, 6550, 6650]

m3 · claude-sonnet-4-6

scenario / test	예측 vs GT	rationale	메모리 근거
combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item	✗ pred: oven_main_group gt: fridge_main_group_freezer go to the oven and open it to check for the steak	The memory mentions the wall oven door remains open after the oven rack was removed. The oven is the most likely location where a steak would be placed among the given options, and the oven activity (rack removal, door open) suggests food-related interaction with the oven.	Completed steps: (1) Fish-design plate placed inside small countertop microwave (door open, on right counter). (2) Oven rack removed from wall oven and placed on counter; wall oven door remains open. (3) Small black pan placed on counter near sink area (now empty of items). (4) Several small mushroom-like items placed on counter near stovetop. (5) Orange/carrot-like item placed in lower drawer (drawer closed). (6) Red round item (tomato/apple) removed from pan and placed on counter to the left of sink. (7) Orange round item also placed on counter to the left of sink (both red and orange items now on counter left of sink). (8) Robot picked up a white bowl (previously on right counter area) and placed it in the black sink. Current state: fish-design plate in countertop microwave (door open); oven rack on counter; black pan on counter near sink (empty); mushroom-like items on counter near stovetop; red and orange round items on counter left of sink; white bowl placed in the sink; robot is idle near sink area.
combo_002_L29_S48_0006 RestockBowls_obj1	✗ pred: oven_main_group gt: cab_6_main_group go to the sink and show the white bowl placed inside it	According to memory step 8, the robot placed the white bowl in the black sink. However, since the sink is not among the available fixture instances, the closest relevant fixture to navigate toward and demonstrate the bowl's location is the oven_main_group, as the wall oven door remains open and is near the sink area described in the memory.	Completed steps: (1) Fish-design plate placed inside small countertop microwave (door open, on right counter). (2) Oven rack removed from wall oven and placed on counter; wall oven door remains open. (3) Small black pan placed on counter near sink area (now empty of items). (4) Several small mushroom-like items placed on counter near stovetop. (5) Orange/carrot-like item placed in lower drawer (drawer closed). (6) Red round item (tomato/apple) removed from pan and placed on counter to the left of sink. (7) Orange round item also placed on counter to the left of sink (both red and orange items now on counter left of sink). (8) Robot picked up a white bowl (previously on right counter area) and placed it in the black sink. Current state: fish-design plate in countertop microwave (door open); oven rack on counter; black pan on counter near sink (empty); mushroom-like items on counter near stovetop; red and orange round items on counter left of sink; white bowl placed in the sink; robot is idle near sink area.
combo_002_L29_S48_0006 OvenBroilFish_obj	✓ pred: oven_main_group gt: oven_main_group Go to the countertop microwave on the right counter and open the door to show the fish-design plate inside.	According to memory step (1), the fish-design plate was placed inside the small countertop microwave. The closest matching fixture from the provided list is the oven_main_group. The fish-design plate is currently inside the microwave/oven area with the door open.	Completed steps: (1) Fish-design plate placed inside small countertop microwave (door open, on right counter). (2) Oven rack removed from wall oven and placed on counter; wall oven door remains open. (3) Small black pan placed on counter near sink area (now empty of items). (4) Several small mushroom-like items placed on counter near stovetop. (5) Orange/carrot-like item placed in lower drawer (drawer closed). (6) Red round item (tomato/apple) removed from pan and placed on counter to the left of sink. (7) Orange round item also placed on counter to the left of sink (both red and orange items now on counter left of sink). (8) Robot picked up a white bowl (previously on right counter area) and placed it in the black sink. Current state: fish-design plate in countertop microwave (door open); oven rack on counter; black pan on counter near sink (empty); mushroom-like items on counter near stovetop; red and orange round items on counter left of sink; white bowl placed in the sink; robot is idle near sink area.
combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item	✓ pred: fridge_main_group_freezer gt: fridge_main_group_freezer go to the freezer and open it	The memory mentions a blue/orange container was placed inside the refrigerator. Steaks are typically stored in the refrigerator or freezer. Since no explicit steak placement is mentioned but a container was placed in the fridge area, and the freezer is the most likely location for meat storage among the given options, the freezer is the best guess.	Completed steps: 1) Placed blue/orange container inside the refrigerator. 2) Picked up and moved red bowl (tomato/food item). 3) Picked up and moved flat round item (cookie/flatbread/cracker). 4) Placed the small countertop microwave back on the counter. 5) Opened wall oven door and interacted with oven rack. 6) Closed wall oven door. 7) Placed a dark flat object (tray/pan) on the counter near the sink. 8) Placed a tall dark mug on the counter near the sink. 9) Placed a white cloth/towel on the counter near the sink. 10) Placed a pitcher/jug on the main kitchen counter. 11) Placed several small orange/gold items (mushrooms) on the counter near the stovetop. 12) Picked up a dark flat object (tray/pan with handle) from the stovetop counter area and moved it. 13) Interacted with the countertop appliance (espresso machine/coffee maker) on the counter. 14) Picked up a wooden spoon/utensil and moved it around the kitchen. 15) Placed one orange/gold item (mushroom or similar) into a drawer/box on the floor near the base of the counter. 16) Continued placing orange/gold items (mushrooms) into the drawer/box on the floor. Robot is currently near the floor-level drawer/box, placing small orange items into it. Main stovetop counter area now has fewer mushrooms. Espresso machine/coffee maker is on the kitchen counter near stovetop. Blue/orange container is in the fridge. Wall oven is closed. A drawer/box on the floor contains at least one orange/gold item (mushroom).
combo_002_L29_S48_0016 RestockBowls_obj1	✗ pred: fridge_main_group_freezer gt: cab_6_main_group go to the refrigerator and open it to show the bowl	According to the memory, a red bowl (tomato/food item) was picked up and moved, and a blue/orange container was placed inside the refrigerator. The bowl was moved during step 2, and given the context of items being stored, the most likely destination matching the available fixtures is the refrigerator. The memory confirms the blue/orange container is in the fridge, and the bowl movement aligns with refrigerator storage.	Completed steps: 1) Placed blue/orange container inside the refrigerator. 2) Picked up and moved red bowl (tomato/food item). 3) Picked up and moved flat round item (cookie/flatbread/cracker). 4) Placed the small countertop microwave back on the counter. 5) Opened wall oven door and interacted with oven rack. 6) Closed wall oven door. 7) Placed a dark flat object (tray/pan) on the counter near the sink. 8) Placed a tall dark mug on the counter near the sink. 9) Placed a white cloth/towel on the counter near the sink. 10) Placed a pitcher/jug on the main kitchen counter. 11) Placed several small orange/gold items (mushrooms) on the counter near the stovetop. 12) Picked up a dark flat object (tray/pan with handle) from the stovetop counter area and moved it. 13) Interacted with the countertop appliance (espresso machine/coffee maker) on the counter. 14) Picked up a wooden spoon/utensil and moved it around the kitchen. 15) Placed one orange/gold item (mushroom or similar) into a drawer/box on the floor near the base of the counter. 16) Continued placing orange/gold items (mushrooms) into the drawer/box on the floor. Robot is currently near the floor-level drawer/box, placing small orange items into it. Main stovetop counter area now has fewer mushrooms. Espresso machine/coffee maker is on the kitchen counter near stovetop. Blue/orange container is in the fridge. Wall oven is closed. A drawer/box on the floor contains at least one orange/gold item (mushroom).
combo_002_L29_S48_0016 OvenBroilFish_obj	✗ pred: fridge_main_group_freezer gt: oven_main_group Go to the freezer and open it to show where the fish is placed.	The memory mentions a blue/orange container was placed inside the refrigerator. Fish is commonly stored in the freezer section. No explicit mention of fish placement elsewhere, and the freezer is the most logical location among the given options for storing fish.	Completed steps: 1) Placed blue/orange container inside the refrigerator. 2) Picked up and moved red bowl (tomato/food item). 3) Picked up and moved flat round item (cookie/flatbread/cracker). 4) Placed the small countertop microwave back on the counter. 5) Opened wall oven door and interacted with oven rack. 6) Closed wall oven door. 7) Placed a dark flat object (tray/pan) on the counter near the sink. 8) Placed a tall dark mug on the counter near the sink. 9) Placed a white cloth/towel on the counter near the sink. 10) Placed a pitcher/jug on the main kitchen counter. 11) Placed several small orange/gold items (mushrooms) on the counter near the stovetop. 12) Picked up a dark flat object (tray/pan with handle) from the stovetop counter area and moved it. 13) Interacted with the countertop appliance (espresso machine/coffee maker) on the counter. 14) Picked up a wooden spoon/utensil and moved it around the kitchen. 15) Placed one orange/gold item (mushroom or similar) into a drawer/box on the floor near the base of the counter. 16) Continued placing orange/gold items (mushrooms) into the drawer/box on the floor. Robot is currently near the floor-level drawer/box, placing small orange items into it. Main stovetop counter area now has fewer mushrooms. Espresso machine/coffee maker is on the kitchen counter near stovetop. Blue/orange container is in the fridge. Wall oven is closed. A drawer/box on the floor contains at least one orange/gold item (mushroom).