2026-05-19 — Analysis
Scene-Mem API Track — 2-ep 스모크 결과
SceneMem | Letsur gateway · claude-sonnet-4-6 · dev --limit 2
TL;DR — control/M1/M2/M3 4종 메모리 메커니즘의 next-subtask slot-match accuracy를 검증. 아래 집계 표 + 테스트별 '왜' inspector(예측 sub-task vs GT, 메모리 근거).
16
runs
0.17
best slot_acc
96
tests
1. 집계 비교 (method × model)
| method | model | n | slot_acc | place_acc | action_acc | exact_acc | parse_fail |
|---|---|---|---|---|---|---|---|
| control | claude-sonnet-4-6 | 6 | 0.00 | 0.33 | 0.00 | 0.00 | 0 |
| m1 | claude-sonnet-4-6 | 6 | 0.00 | 0.50 | 0.00 | 0.00 | 0 |
| m2 | claude-sonnet-4-6 | 6 | 0.00 | 0.33 | 0.00 | 0.00 | 0 |
| m3 | claude-sonnet-4-6 | 6 | 0.17 | 0.33 | 0.17 | 0.17 | 0 |
| control | gemini-2.0-flash | 6 | 0.00 | 0.33 | 0.00 | 0.00 | 0 |
| m1 | gemini-2.0-flash | 6 | 0.17 | 0.67 | 0.33 | 0.00 | 0 |
| m2 | gemini-2.0-flash | 6 | 0.17 | 0.50 | 0.17 | 0.00 | 0 |
| m3 | gemini-2.0-flash | 6 | 0.00 | 0.00 | 0.00 | 0.00 | 0 |
| control | gemini-2.5-flash | 6 | 0.17 | 0.33 | 0.17 | 0.17 | 0 |
| m1 | gemini-2.5-flash | 6 | 0.00 | 0.17 | 0.17 | 0.00 | 0 |
| m2 | gemini-2.5-flash | 6 | 0.00 | 0.17 | 0.17 | 0.00 | 0 |
| m3 | gemini-2.5-flash | 6 | 0.00 | 0.67 | 0.33 | 0.00 | 0 |
| control | gemini-2.5-pro | 6 | 0.00 | 0.17 | 0.17 | 0.00 | 0 |
| m1 | gemini-2.5-pro | 6 | 0.00 | 0.33 | 0.17 | 0.00 | 0 |
| m2 | gemini-2.5-pro | 6 | 0.00 | 0.17 | 0.00 | 0.00 | 0 |
| m3 | gemini-2.5-pro | 6 | 0.00 | 0.17 | 0.00 | 0.00 | 0 |
slot_acc=action&object&place 슬롯 일치, place_acc=place 일치, action_acc=action 일치, exact_acc=문자열 완전 일치.
2. 테스트별 정성 inspector
control · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray back inside the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the tray with the fish back inside the oven onto the bottom rack gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=64 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=56 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=64 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Pick up the fish from the counter gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | uniform history · n_history=56 | target window 146.2–158.3s (STATE 위치) |
m2 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[170]  | target window 9.1–18.1s (STATE 위치) | KF@ 8.5s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[270, 360]   | target window 99.0–111.2s (STATE 위치) | KF@ 13.5s, 18.0s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Pick up the fish from the floor gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[270, 640, 2900]   | target window 146.1–158.2s (STATE 위치) | KF@ 13.5s, 32.0s, 145.0s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[170]  | target window 9.1–18.1s (STATE 위치) | KF@ 8.5s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[360, 640]   | target window 99.2–111.4s (STATE 위치) | KF@ 18.0s, 32.0s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Pick up the fish from the counter gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[550, 640, 2900]   | target window 146.2–158.3s (STATE 위치) | KF@ 27.5s, 32.0s, 145.0s |
m3 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Place the steak in/on the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | Robot picked up steak from fridge. Robot is now at the fridge/freezer unit with doors closed. Need to open the freezer door to place the steak inside. After placement, need to close both fridge and freezer doors. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Place the red bowl in/on the cabinet gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | Red bowl placed in cabinet. Now attempting to pick up the beige/tan bowl from the counter. Cabinet is open. After placing this bowl in the cabinet, need to close the cabinet. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray with the fish back inside the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | Fish is on the tray/rack. The tray needs to be slid fully back inside the oven. In the latest frames the robot appears to be pushing the tray back in. Next step after sliding tray in: close the oven door. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | The robot has navigated to the fridge/freezer unit. Both fridge and freezer doors appear closed now. The steak (red item) was previously visible on a fridge shelf. Need to open the fridge door to access the steak. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Place the bowl in/on the cabinet gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | Cabinet is open. Red bowl was being placed in the cabinet. In the latest frames (frame 8), the robot arm is near the open cabinet. A second bowl (tan/plate-like) is visible on the counter to the left. Need to confirm red bowl placement, then pick up the second bowl from the counter and place it in the cabinet, then close the cabinet. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | Fish has been placed on the bottom rack/tray of the oven. Tray needs to be slid back inside, then close the oven door. | target window 146.2–158.3s (STATE 위치) |
control · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge. gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray back inside the oven. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge. gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the tray back inside. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge. gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=64 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Place the tray in the oven. gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | uniform history · n_history=56 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=64 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish in the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | uniform history · n_history=56 | target window 146.2–158.3s (STATE 위치) |
m2 · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[20, 140, 230]    | target window 9.1–18.1s (STATE 위치) | KF@ 1.0s, 7.0s, 11.5s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[1010, 1070, 1470, 1740, 1770, 1990, 2140, 2230]         | target window 99.0–111.2s (STATE 위치) | KF@ 50.5s, 53.5s, 73.5s, 87.0s, 88.5s, 99.5s, 107.0s, 111.5s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish in the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | idx=[2630, 2800, 2840, 2870, 3080, 3110, 3140, 3230]         | target window 146.1–158.2s (STATE 위치) | KF@ 131.5s, 140.0s, 142.0s, 143.5s, 154.0s, 155.5s, 157.0s, 161.5s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge. gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[20, 140, 230]   | target window 9.1–18.1s (STATE 위치) | KF@ 1.0s, 7.0s, 11.5s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[1010, 1040, 1070, 1100, 1130, 1710, 1740, 1990]         | target window 99.2–111.4s (STATE 위치) | KF@ 50.5s, 52.0s, 53.5s, 55.0s, 56.5s, 85.5s, 87.0s, 99.5s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Pick up the fish from the counter gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[1190, 1380, 2640, 2750, 2780, 2990, 3170, 3230]         | target window 146.2–158.3s (STATE 위치) | KF@ 59.5s, 69.0s, 132.0s, 137.5s, 139.0s, 149.5s, 158.5s, 161.5s |
m3 · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | The steak is currently in the fridge. The next step is to pick up the steak from the fridge, place it in the freezer, and close both the fridge and freezer doors. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | Opening the cabinet, picking the bowls from the counter and placing them in the cabinet. Then close the cabinet. Cabinet is open. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray back inside. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | Opening cabinet. Pick the bowls from the counter and place them in the cabinet. Then close the cabinet. Now approaching the oven. The oven is open. Placed fish on oven rack. Tray is still out and need to be slid back. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | The steak is in the fridge and needs to be moved to the freezer. Picked up the steak from the fridge. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | Open the cabinet. Pick the bowls from the counter and place them in the cabinet. Then close the cabinet. The cabinet is open. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the tray back inside the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | The fish is now on the bottom rack of the oven. Now, slide the tray back inside. | target window 146.2–158.3s (STATE 위치) |
control · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the oven tray back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=64 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Place the fish on the oven rack gt: Place the fish on the bottom rack of the oven action=✓ place=✗ exact=✗ | ok | uniform history · n_history=56 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Open the freezer door gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=64 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the tray back inside the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | uniform history · n_history=56 | target window 146.2–158.3s (STATE 위치) |
m2 · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[170] | target window 9.1–18.1s (STATE 위치) | KF@ 8.5s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Place the red bowl in the cabinet gt: Place a bowl in the cabinet action=✓ place=✓ exact=✗ | ok | idx=[1250, 1280, 1310, 1340, 1370, 1410, 1440, 1470]        | target window 99.0–111.2s (STATE 위치) | KF@ 62.5s, 64.0s, 65.5s, 67.0s, 68.5s, 70.5s, 72.0s, 73.5s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[730, 970, 2290, 2900, 3080]    | target window 146.1–158.2s (STATE 위치) | KF@ 36.5s, 48.5s, 114.5s, 145.0s, 154.0s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[140, 170] | target window 9.1–18.1s (STATE 위치) | KF@ 7.0s, 8.5s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | idx=[1190, 1220, 1960]   | target window 99.2–111.4s (STATE 위치) | KF@ 59.5s, 61.0s, 98.0s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the tray back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[910, 940, 970, 1010, 1190, 3110, 3140]      | target window 146.2–158.3s (STATE 위치) | KF@ 45.5s, 47.0s, 48.5s, 50.5s, 59.5s, 155.5s, 157.0s |
m3 · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | The robot has picked up the steak from the fridge. Now, close the fridge door. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | The robot needs to open the cabinet, then pick up the bowls from the counter and place them inside, and finally close the cabinet. The cabinet door is currently closed, so the next step is to open it. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the rack back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | The robot has placed the fish on the bottom rack of the oven. The oven door needs to be closed. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | The steak has been picked up from the fridge and is in the robot's gripper. The fridge door is almost closed, or in the process of closing. The next step is to move the steak to the freezer. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Place the red bowl in the cabinet gt: Place a bowl in the cabinet action=✓ place=✓ exact=✗ | ok | The cabinet has been opened. The red bowl has been picked from the counter. The robot is placing the red bowl in the cabinet. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Place the fish on the rack gt: Place the fish on the bottom rack of the oven action=✓ place=✗ exact=✗ | ok | The robot is at the oven, which is open. It has placed the fish on the bottom rack of the oven. Now it needs to slide the tray back in. | target window 146.2–158.3s (STATE 위치) |
control · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge shelf. gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Place the fish on the oven rack gt: Place the fish on the bottom rack of the oven action=✓ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the oven rack back into the oven. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge. gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter. gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=64 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Place the fish on the oven rack gt: Place the fish on the bottom rack of the oven action=✓ place=✗ exact=✗ | ok | uniform history · n_history=56 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=32 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door. gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=64 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=56 | target window 146.2–158.3s (STATE 위치) |
m2 · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[170, 710]  | target window 9.1–18.1s (STATE 위치) | KF@ 8.5s, 35.5s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[640, 970, 1220, 1280]  | target window 99.0–111.2s (STATE 위치) | KF@ 32.0s, 48.5s, 61.0s, 64.0s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the oven tray back inside the oven. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[640, 870, 910, 940, 970, 2680, 3140]     | target window 146.1–158.2s (STATE 위치) | KF@ 32.0s, 43.5s, 45.5s, 47.0s, 48.5s, 134.0s, 157.0s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[170] | target window 9.1–18.1s (STATE 위치) | KF@ 8.5s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | idx=[360, 970, 1470, 2390]   | target window 99.2–111.4s (STATE 위치) | KF@ 18.0s, 48.5s, 73.5s, 119.5s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the oven rack back into the oven. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[640, 970, 1010, 2640, 2900, 3140] | target window 146.2–158.3s (STATE 위치) | KF@ 32.0s, 48.5s, 50.5s, 132.0s, 145.0s, 157.0s |
m3 · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Open the freezer door gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | I am holding the steak. The fridge and freezer doors are both open. I will now place the steak in the freezer. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the second bowl from the counter. gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | Both bowls are now in the cabinet. Next, I need to close the cabinet. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the oven tray back inside the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | The fish is on the oven tray, which has been pulled out. The next step is to slide the tray back in. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | I am holding the steak and have closed the fridge. Now I will open the freezer to put the steak inside. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the beige bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | The cabinet is open. I have placed the red bowl inside and am now placing the beige bowl inside. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the oven tray back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | The fish is on the oven rack, which has now been slid back into the oven. The next step is to close the oven door. | target window 146.2–158.3s (STATE 위치) |