2026-05-19 — Analysis
Scene-Mem API Track — 2-ep 스모크 결과
SceneMem | Letsur gateway · claude-sonnet-4-6 · dev --limit 2
TL;DR — control/M1/M2/M3 4종 메모리 메커니즘의 next-subtask slot-match accuracy를 검증. 아래 집계 표 + 테스트별 '왜' inspector(예측 sub-task vs GT, 메모리 근거).
16
runs
0.33
best slot_acc
96
tests
1. 집계 비교 (method × model)
| method | model | n | slot_acc | place_acc | action_acc | exact_acc | parse_fail |
|---|---|---|---|---|---|---|---|
| control | claude-sonnet-4-6 | 6 | 0.17 | 0.67 | 0.17 | 0.17 | 0 |
| m1 | claude-sonnet-4-6 | 6 | 0.00 | 0.33 | 0.00 | 0.00 | 0 |
| m2 | claude-sonnet-4-6 | 6 | 0.17 | 0.83 | 0.17 | 0.17 | 0 |
| m3 | claude-sonnet-4-6 | 6 | 0.17 | 0.50 | 0.17 | 0.17 | 0 |
| control | gemini-2.0-flash | 6 | 0.17 | 0.17 | 0.17 | 0.00 | 0 |
| m1 | gemini-2.0-flash | 6 | 0.33 | 0.67 | 0.33 | 0.00 | 0 |
| m2 | gemini-2.0-flash | 6 | 0.33 | 0.33 | 0.33 | 0.00 | 0 |
| m3 | gemini-2.0-flash | 6 | 0.33 | 0.33 | 0.33 | 0.00 | 0 |
| control | gemini-2.5-flash | 6 | 0.00 | 0.33 | 0.17 | 0.00 | 0 |
| m1 | gemini-2.5-flash | 6 | 0.17 | 0.33 | 0.17 | 0.17 | 0 |
| m2 | gemini-2.5-flash | 6 | 0.00 | 0.33 | 0.00 | 0.00 | 0 |
| m3 | gemini-2.5-flash | 6 | 0.00 | 0.17 | 0.50 | 0.00 | 0 |
| control | gemini-2.5-pro | 6 | 0.00 | 0.50 | 0.00 | 0.00 | 0 |
| m1 | gemini-2.5-pro | 6 | 0.00 | 0.50 | 0.00 | 0.00 | 0 |
| m2 | gemini-2.5-pro | 6 | 0.00 | 0.33 | 0.17 | 0.00 | 0 |
| m3 | gemini-2.5-pro | 6 | 0.00 | 0.17 | 0.33 | 0.00 | 0 |
slot_acc=action&object&place 슬롯 일치, place_acc=place 일치, action_acc=action 일치, exact_acc=문자열 완전 일치.
2. 테스트별 정성 inspector
control · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Close the oven door gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=222 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=218 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=223 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=217 | target window 146.2–158.3s (STATE 위치) |
m2 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | idx=[150, 160, 180]    | target window 9.1–18.1s (STATE 위치) | KF@ 7.5s, 8.0s, 9.0s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[670, 680, 880, 890, 960, 1730]       | target window 99.0–111.2s (STATE 위치) | KF@ 33.5s, 34.0s, 44.0s, 44.5s, 48.0s, 86.5s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | idx=[650, 680, 960, 970, 2440, 2880, 2890, 2920]       | target window 146.1–158.2s (STATE 위치) | KF@ 32.5s, 34.0s, 48.0s, 48.5s, 122.0s, 144.0s, 144.5s, 146.0s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | idx=[150, 160, 170, 180]     | target window 9.1–18.1s (STATE 위치) | KF@ 7.5s, 8.0s, 8.5s, 9.0s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Navigate to the cabinet gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[650, 680, 880, 970, 1730]      | target window 99.2–111.4s (STATE 위치) | KF@ 32.5s, 34.0s, 44.0s, 48.5s, 86.5s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Navigate to the counter to pick up the fish gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[680, 700, 970, 980, 2880, 2920]     | target window 146.2–158.3s (STATE 위치) | KF@ 34.0s, 35.0s, 48.5s, 49.0s, 144.0s, 146.0s |
m3 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | Fridge door appears closed now - robot closed it or it was being opened. The fridge door handle is being grabbed in recent frames. Need to open the fridge door to access the steak inside. Steak is on top shelf of fridge. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Navigate to the cabinet gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | Robot has navigated to the cabinet area. Need to open the cabinet door before placing the red bowl inside. Still need to: open cabinet, place red bowl in cabinet, pick up second bowl (tan/beige) from counter, place second bowl in cabinet, close cabinet. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Pick up the fish from the counter gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | Robot is at the oven area. Oven door is open, rack is pulled out. Robot has picked up the fish and needs to place it on the bottom rack of the oven. After placing, need to slide the tray back in and close the oven door. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | Fridge door appears closed now. Robot arm is near the fridge/freezer unit. Need to open the freezer door (top or side compartment) to place steak inside. Steak status unclear - may still need to be picked from fridge. Need to open freezer door first, then pick steak from fridge, place in freezer, close both doors. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | Cabinet door needs to be opened. Robot is near the cabinet with the red bowl previously held (now possibly set down or still in hand). Need to open cabinet, then pick bowls from counter and place them inside, then close cabinet. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | The oven door is open and the rack is pulled out. The robot has been holding the fish and is now placing it on the bottom rack of the oven. After placing the fish, next steps are: slide the tray/rack back inside, then close the oven door. | target window 146.2–158.3s (STATE 위치) |
control · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray back inside the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish in the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=222 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven. gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | uniform history · n_history=218 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=223 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish in the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | uniform history · n_history=217 | target window 146.2–158.3s (STATE 위치) |
m2 · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[130, 300, 370, 430, 720]      | target window 9.1–18.1s (STATE 위치) | KF@ 6.5s, 15.0s, 18.5s, 21.5s, 36.0s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the bowls from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | idx=[1640, 1700, 1900, 1970, 2210, 2220, 2430, 2440]        | target window 99.0–111.2s (STATE 위치) | KF@ 82.0s, 85.0s, 95.0s, 98.5s, 110.5s, 111.0s, 121.5s, 122.0s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish in the oven. gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | idx=[170, 180, 940, 1100, 1440, 2580, 2680, 2770]        | target window 146.1–158.2s (STATE 위치) | KF@ 8.5s, 9.0s, 47.0s, 55.0s, 72.0s, 129.0s, 134.0s, 138.5s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[250, 260, 270, 280, 290, 300, 510, 710]         | target window 9.1–18.1s (STATE 위치) | KF@ 12.5s, 13.0s, 13.5s, 14.0s, 14.5s, 15.0s, 25.5s, 35.5s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the bowls from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | idx=[1650, 1880, 1930, 2140, 2180, 2210, 2390, 2400]         | target window 99.2–111.4s (STATE 위치) | KF@ 82.5s, 94.0s, 96.5s, 107.0s, 109.0s, 110.5s, 119.5s, 120.0s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish in the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | idx=[2800, 2810, 2820, 2830, 2840, 2850, 3170, 3210]         | target window 146.2–158.3s (STATE 위치) | KF@ 140.0s, 140.5s, 141.0s, 141.5s, 142.0s, 142.5s, 158.5s, 160.5s |
m3 · gemini-2.0-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | The steak is in the fridge. The goal is to move the steak from the fridge to the freezer and close the doors. The robot needs to pick up the steak. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | The bowls are on the counter. The cabinet is open. Next pick and place the bowls into the cabinet, and close the cabinet. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven. gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✗ | ok | The fish is in the oven. The oven door needs to be closed. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | The steak needs to be moved from the fridge to the freezer. The robot has picked up the steak from the fridge and will now close the fridge door. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✓ pred: Place the bowl in the cabinet gt: Place a bowl in the cabinet action=✓ place=✓ exact=✗ | ok | The robot has to pick the bowls from the counter and place them in the cabinet, and close the cabinet when done. The cabinet is open and the robot has a bowl. Placed bowl in cabinet. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Pick up the fish from the counter gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | The oven door is open. Need to pick the fish and place it in the oven. The fish is on the counter. | target window 146.2–158.3s (STATE 위치) |
control · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Place the red bowl in the cabinet gt: Place a bowl in the cabinet action=✓ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Navigate to the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Close the oven door gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=222 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | uniform history · n_history=218 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=223 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Open the oven door gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=217 | target window 146.2–158.3s (STATE 위치) |
m2 · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[310, 320, 330, 340, 350, 360, 370, 720]       | target window 9.1–18.1s (STATE 위치) | KF@ 15.5s, 16.0s, 16.5s, 17.0s, 17.5s, 18.0s, 18.5s, 36.0s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[200, 210, 730, 1490, 1990, 2000, 2010, 2020]         | target window 99.0–111.2s (STATE 위치) | KF@ 10.0s, 10.5s, 36.5s, 74.5s, 99.5s, 100.0s, 100.5s, 101.0s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | idx=[490, 500, 520, 1890, 2580, 2630, 2680, 2780]       | target window 146.1–158.2s (STATE 위치) | KF@ 24.5s, 25.0s, 26.0s, 94.5s, 129.0s, 131.5s, 134.0s, 139.0s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[250, 260, 270, 280, 290, 300, 310, 370]   | target window 9.1–18.1s (STATE 위치) | KF@ 12.5s, 13.0s, 13.5s, 14.0s, 14.5s, 15.0s, 15.5s, 18.5s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | idx=[1200, 1210, 1220, 1230, 1240, 1460, 1730, 2210]       | target window 99.2–111.4s (STATE 위치) | KF@ 60.0s, 60.5s, 61.0s, 61.5s, 62.0s, 73.0s, 86.5s, 110.5s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the oven rack completely back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[2610, 2680, 2780, 3120, 3130, 3140, 3150, 3160]         | target window 146.2–158.3s (STATE 위치) | KF@ 130.5s, 134.0s, 139.0s, 156.0s, 156.5s, 157.0s, 157.5s, 158.0s |
m3 · gemini-2.5-flash
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Place the bottle in the fridge gt: Place the steak on a shelf in the freezer action=✓ place=✗ exact=✗ | ok | The robot is holding a bottle and needs to place it on the counter before picking up the steak from the fridge to move it to the freezer. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Place the red bowl in the cabinet gt: Place a bowl in the cabinet action=✓ place=✓ exact=✗ | ok | The robot has opened the cabinet and picked up the red bowl from the counter. Now it needs to place the red bowl in the cabinet. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the tray back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | The robot has placed the fish on the bottom rack of the oven. Now it needs to slide the tray back inside the oven and then close the oven door. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Place the bottle in/on the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | The robot is currently holding a bottle and needs to place it before picking up the steak. The next step is to place the bottle. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Place the screwdriver on the counter gt: Place a bowl in the cabinet action=✓ place=✗ exact=✗ | ok | The cabinet is open. The robot is currently holding a screwdriver and needs to place it on the counter before picking up a bowl. The goal is to pick bowls from the counter and place them in the cabinet, then close the cabinet. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the tray back into the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | Approached the cabinet. Opened the cabinet. Navigated to the oven. Picked up the fish from the plate. The oven door is open and the rack is pulled out. Placed the fish on the bottom oven rack. The tray has been slid back into the oven. | target window 146.2–158.3s (STATE 위치) |
control · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Close the oven door gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Slide the oven rack back into the oven. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) | target window 146.2–158.3s (STATE 위치) |
m1 · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge. gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter. gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=222 | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=218 | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door. gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | uniform history · n_history=223 | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=217 | target window 146.2–158.3s (STATE 위치) |
m2 · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[160, 440]  | target window 9.1–18.1s (STATE 위치) | KF@ 8.0s, 22.0s |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | idx=[1000, 1080]   | target window 99.0–111.2s (STATE 위치) | KF@ 50.0s, 54.0s |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Close the oven door. gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | idx=[730, 1000, 1010, 2920, 3150]   | target window 146.1–158.2s (STATE 위치) | KF@ 36.5s, 50.0s, 50.5s, 146.0s, 157.5s |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | idx=[160, 550]  | target window 9.1–18.1s (STATE 위치) | KF@ 8.0s, 27.5s |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[360, 710, 970]  | target window 99.2–111.4s (STATE 위치) | KF@ 18.0s, 35.5s, 48.5s |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Place the fish on the oven rack. gt: Place the fish on the bottom rack of the oven action=✓ place=✗ exact=✗ | ok | idx=[710, 980, 2800, 2920] | target window 146.2–158.3s (STATE 위치) | KF@ 35.5s, 49.0s, 140.0s, 146.0s |
m3 · gemini-2.5-pro
| scenario / test | 예측 vs GT | status | 메모리 근거 | 영상(타깃 구간) |
|---|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | I am holding the steak. I have closed the fridge door. Now I need to open the freezer. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the red bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | I have opened the cabinet and placed both bowls inside. The next step is to close the cabinet. | target window 99.0–111.2s (STATE 위치) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Slide the oven rack back into the oven. gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | The fish has been placed on the rack inside the oven. The oven door is now closed. The goal is complete. | target window 146.1–158.2s (STATE 위치) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Close the fridge door gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | I have picked up the steak and closed the fridge. Now I need to navigate to the freezer, open it, and place the steak inside. | target window 9.1–18.1s (STATE 위치) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Place the tan bowl in the cabinet. gt: Place a bowl in the cabinet action=✓ place=✓ exact=✗ | ok | The cabinet is open and both the red and tan bowls are inside. The final step is to close the cabinet. | target window 99.2–111.4s (STATE 위치) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Place the fish on the oven rack gt: Place the fish on the bottom rack of the oven action=✓ place=✗ exact=✗ | ok | I have placed the fish on the oven rack and slid the rack inside. Now I am closing the oven door. | target window 146.2–158.3s (STATE 위치) |