2026-05-19 — Analysis
Scene-Mem API Track — 2-ep 스모크 결과
SceneMem | Letsur gateway · claude-sonnet-4-6 · dev --limit 2
TL;DR — control/M1/M2/M3 4종 메모리 메커니즘의 next-subtask slot-match accuracy를 검증. 아래 집계 표 + 테스트별 '왜' inspector(예측 sub-task vs GT, 메모리 근거).
4
runs
0.17
best slot_acc
24
tests
1. 집계 비교 (method × model)
| method | model | n | slot_acc | place_acc | action_acc | exact_acc | parse_fail |
|---|---|---|---|---|---|---|---|
| control | claude-sonnet-4-6 | 6 | 0.17 | 0.67 | 0.17 | 0.17 | 0 |
| m1 | claude-sonnet-4-6 | 6 | 0.00 | 0.33 | 0.00 | 0.00 | 0 |
| m2 | claude-sonnet-4-6 | 6 | 0.17 | 0.83 | 0.17 | 0.17 | 0 |
| m3 | claude-sonnet-4-6 | 6 | 0.17 | 0.50 | 0.17 | 0.17 | 0 |
slot_acc=action&object&place 슬롯 일치, place_acc=place 일치, action_acc=action 일치, exact_acc=문자열 완전 일치.
2. 테스트별 정성 inspector
control · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 |
|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Close the oven door gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | (no long-term memory — control) |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | (no long-term memory — control) |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | (no long-term memory — control) |
m1 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 |
|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=222 |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=218 |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | uniform history · n_history=73 |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Pick up the bowl from the counter gt: Place a bowl in the cabinet action=✗ place=✗ exact=✗ | ok | uniform history · n_history=223 |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Navigate to the oven gt: Place the fish on the bottom rack of the oven action=✗ place=✓ exact=✗ | ok | uniform history · n_history=217 |
m2 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 |
|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | idx=[150, 160, 180]    |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[670, 680, 880, 890, 960, 1730]       |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | idx=[650, 680, 960, 970, 2440, 2880, 2890, 2920]       |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Navigate to the freezer gt: Place the steak on a shelf in the freezer action=✗ place=✓ exact=✗ | ok | idx=[150, 160, 170, 180]     |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Navigate to the cabinet gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | idx=[650, 680, 880, 970, 1730]      |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✗ pred: Navigate to the counter to pick up the fish gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | idx=[680, 700, 970, 980, 2880, 2920]     |
m3 · claude-sonnet-4-6
| scenario / test | 예측 vs GT | status | 메모리 근거 |
|---|---|---|---|
| combo_002_L29_S48_0006 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | Fridge door appears closed now - robot closed it or it was being opened. The fridge door handle is being grabbed in recent frames. Need to open the fridge door to access the steak inside. Steak is on top shelf of fridge. |
| combo_002_L29_S48_0006 RestockBowls_obj1 | ✗ pred: Navigate to the cabinet gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | Robot has navigated to the cabinet area. Need to open the cabinet door before placing the red bowl inside. Still need to: open cabinet, place red bowl in cabinet, pick up second bowl (tan/beige) from counter, place second bowl in cabinet, close cabinet. |
| combo_002_L29_S48_0006 OvenBroilFish_obj | ✗ pred: Pick up the fish from the counter gt: Place the fish on the bottom rack of the oven action=✗ place=✗ exact=✗ | ok | Robot is at the oven area. Oven door is open, rack is pulled out. Robot has picked up the fish and needs to place it on the bottom rack of the oven. After placing, need to slide the tray back in and close the oven door. |
| combo_002_L29_S48_0016 MoveFridgeToFreezer_refrigerated_item | ✗ pred: Pick up the steak from the fridge gt: Place the steak on a shelf in the freezer action=✗ place=✗ exact=✗ | ok | Fridge door appears closed now. Robot arm is near the fridge/freezer unit. Need to open the freezer door (top or side compartment) to place steak inside. Steak status unclear - may still need to be picked from fridge. Need to open freezer door first, then pick steak from fridge, place in freezer, close both doors. |
| combo_002_L29_S48_0016 RestockBowls_obj1 | ✗ pred: Open the cabinet door gt: Place a bowl in the cabinet action=✗ place=✓ exact=✗ | ok | Cabinet door needs to be opened. Robot is near the cabinet with the red bowl previously held (now possibly set down or still in hand). Need to open cabinet, then pick bowls from counter and place them inside, then close cabinet. |
| combo_002_L29_S48_0016 OvenBroilFish_obj | ✓ pred: Place the fish on the bottom rack of the oven gt: Place the fish on the bottom rack of the oven action=✓ place=✓ exact=✓ | ok | The oven door is open and the rack is pulled out. The robot has been holding the fish and is now placing it on the bottom rack of the oven. After placing the fish, next steps are: slide the tray/rack back inside, then close the oven door. |