2026-08-21 — Experiment (중간 결과 — G3 진행 중)
E2E 실행 SR 결과 — VLM eval 답 재생: Stage B(접지·도달) → Stage C(VLA 실조작)
scene_bench | 353질의 × 5전략 × 2모델 + oracle-loc · 고유 플랜 1,216 · Stage C 81 플랜 실행 · VLM 재호출 0
TL;DR
- 하네스 ceiling(oracle-loc): 접지·도달 SR_B 0.830(플랜 단위 v1 222/353 → v2 285/353 → v3 289/353), 조작 포함 SR_C 0.239, P(lift|reached, lifted-only) 0.166 (n=247) — 조작이 병목.
- 셀 격자: SR_C 상위 = oracle-loc 0.239, frames-uniform@gemini-3.6-flash 0.178, rollforward-vis-pure@gemini-3.6-flash 0.167. 기권(abstain)은 0점으로 집계(A5 원칙과 별개로 실행 SR이므로).
- 분해식 SR_C = P(reached) × P(lift|reached): 셀 간 차이는 거의 전부 P(reached)(= 메모리 답의 위치 정확도 + 접지)에서 나고, P(lift|reached)는 ceiling 공통.
- Stage C 실행 81 플랜, 성공 44, 평균 292s/플랜(sparse).
0.83
oracle SR_B
0.24
oracle SR_C
0.17
P(lift|reached)
81
Stage C 플랜
1 무엇을 어떻게 쟀나
VLM eval(S3) 답 {category, room, surface}을 재생: 벤치마크 종료 상태(final_state.npz 25 pose + 로봇 base) 복원 → 답의 (room, surface)를 접지 v3(답 granularity까지 메모리, 그 가구 위 인스턴스 찾기는 하네스, 후보 ≤2) → 텔레포트 → Stage B reached(GT ≤0.85 m) → Stage C VLA(mixv2-20k, zed2 입력) "Pick up the {category}" 600스텝 × 조건부 재시도 2, 판정 GT body any-of lift(프로브 실행 전 생성). 집계는 질의 단위(기권·미접지·미도달 = 0), reached_only 질의는 Stage B가 최종.
접지 규칙 v1→v3 진화(oracle 0.63→0.82)와 Stage C 재시도·프로브 버그 수정(스모크에서 발견)은
claude/260821/exp-e2e_exec_stage_b.md, exp-e2e_exec_stage_c.md 참조.2 셀별 SR_B / SR_C (질의 단위, n=gradable)
| cell | n | abstain | SR_B | SR_C | P(lift|reached, lifted-only) | n_lift_eval |
|---|---|---|---|---|---|---|
| oracle-loc | 348 | 0.00 | 0.830 | 0.239 | 0.166 | 247 |
| frames-uniform@gemini-3.6-flash | 348 | 0.03 | 0.589 | 0.178 | 0.192 | 177 |
| rollforward-vis-pure@gemini-3.6-flash | 348 | 0.02 | 0.612 | 0.167 | 0.171 | 187 |
| textlog@gemini-3.6-flash | 348 | 0.38 | 0.405 | 0.109 | 0.149 | 121 |
| textlog@claude-sonnet-5 | 348 | 0.41 | 0.374 | 0.101 | 0.144 | 111 |
| frames-uniform@claude-sonnet-5 | 348 | 0.05 | 0.362 | 0.095 | 0.184 | 114 |
| rollforward-vis-pure@claude-sonnet-5 | 348 | 0.05 | 0.379 | 0.095 | 0.161 | 118 |
| rollforward-v2@claude-sonnet-5 | 348 | 0.32 | 0.296 | 0.078 | 0.146 | 89 |
| blind@gemini-3.6-flash | 348 | 0.41 | 0.216 | 0.066 | 0.175 | 63 |
| rollforward-v2@gemini-3.6-flash | 348 | 0.37 | 0.244 | 0.066 | 0.162 | 74 |
| blind@claude-sonnet-5 | 348 | 0.53 | 0.152 | 0.032 | 0.125 | 48 |
플랜 단위 Stage C 성공(소스별): blind@claude-sonnet-5 6/12, blind@gemini-3.6-flash 11/16, frames-uniform@claude-sonnet-5 21/28, frames-uniform@gemini-3.6-flash 34/52, oracle-loc 41/69, rollforward-v2@claude-sonnet-5 13/19, rollforward-v2@gemini-3.6-flash 12/22, rollforward-vis-pure@claude-sonnet-5 19/29, rollforward-vis-pure@gemini-3.6-flash 32/48, textlog@claude-sonnet-5 16/32, textlog@gemini-3.6-flash 18/34
3 층별 분해 (SR_B / SR_C)
질의 종류
| cell | find_distractor | find_target |
|---|---|---|
| oracle-loc | 0.91 / 0.21 (n=173) | 0.75 / 0.27 (n=175) |
| frames-uniform@gemini-3.6-flash | 0.65 / 0.18 (n=165) | 0.57 / 0.19 (n=172) |
| rollforward-vis-pure@gemini-3.6-flash | 0.71 / 0.15 (n=165) | 0.55 / 0.19 (n=175) |
| textlog@gemini-3.6-flash | 0.62 / 0.14 (n=42) | 0.66 / 0.18 (n=174) |
| textlog@claude-sonnet-5 | 0.50 / 0.12 (n=32) | 0.66 / 0.18 (n=174) |
| frames-uniform@claude-sonnet-5 | 0.50 / 0.13 (n=160) | 0.27 / 0.07 (n=170) |
| rollforward-vis-pure@claude-sonnet-5 | 0.58 / 0.13 (n=157) | 0.24 / 0.07 (n=172) |
| rollforward-v2@claude-sonnet-5 | 0.54 / 0.10 (n=98) | 0.36 / 0.12 (n=139) |
| blind@gemini-3.6-flash | 0.42 / 0.11 (n=97) | 0.31 / 0.11 (n=110) |
| rollforward-v2@gemini-3.6-flash | 0.37 / 0.09 (n=67) | 0.39 / 0.11 (n=153) |
| blind@claude-sonnet-5 | 0.41 / 0.07 (n=76) | 0.25 / 0.07 (n=88) |
채점 방식
| cell | lifted | reached_only |
|---|---|---|
| oracle-loc | 0.84 / 0.14 (n=293) | 0.76 / 0.76 (n=55) |
| frames-uniform@gemini-3.6-flash | 0.63 / 0.12 (n=283) | 0.52 / 0.52 (n=54) |
| rollforward-vis-pure@gemini-3.6-flash | 0.66 / 0.11 (n=285) | 0.47 / 0.47 (n=55) |
| textlog@gemini-3.6-flash | 0.72 / 0.11 (n=168) | 0.42 / 0.42 (n=48) |
| textlog@claude-sonnet-5 | 0.70 / 0.10 (n=159) | 0.40 / 0.40 (n=47) |
| frames-uniform@claude-sonnet-5 | 0.41 / 0.08 (n=278) | 0.23 / 0.23 (n=52) |
| rollforward-vis-pure@claude-sonnet-5 | 0.43 / 0.07 (n=277) | 0.27 / 0.27 (n=52) |
| rollforward-v2@claude-sonnet-5 | 0.46 / 0.07 (n=194) | 0.33 / 0.33 (n=43) |
| blind@gemini-3.6-flash | 0.38 / 0.07 (n=166) | 0.29 / 0.29 (n=41) |
| rollforward-v2@gemini-3.6-flash | 0.43 / 0.07 (n=173) | 0.23 / 0.23 (n=47) |
| blind@claude-sonnet-5 | 0.35 / 0.04 (n=136) | 0.18 / 0.18 (n=28) |
단일/다중
| cell | multi | single |
|---|---|---|
| oracle-loc | 0.77 / 0.26 (n=35) | 0.84 / 0.24 (n=313) |
| frames-uniform@gemini-3.6-flash | 0.24 / 0.09 (n=33) | 0.65 / 0.19 (n=304) |
| rollforward-vis-pure@gemini-3.6-flash | 0.30 / 0.06 (n=33) | 0.66 / 0.18 (n=307) |
| textlog@gemini-3.6-flash | 0.63 / 0.22 (n=27) | 0.66 / 0.17 (n=189) |
| textlog@claude-sonnet-5 | 0.65 / 0.23 (n=26) | 0.63 / 0.16 (n=180) |
| frames-uniform@claude-sonnet-5 | 0.06 / 0.03 (n=34) | 0.42 / 0.11 (n=296) |
| rollforward-vis-pure@claude-sonnet-5 | 0.10 / 0.06 (n=31) | 0.43 / 0.10 (n=298) |
| rollforward-v2@claude-sonnet-5 | 0.19 / 0.12 (n=26) | 0.46 / 0.11 (n=211) |
| blind@gemini-3.6-flash | 0.14 / 0.05 (n=22) | 0.39 / 0.12 (n=185) |
| rollforward-v2@gemini-3.6-flash | 0.25 / 0.11 (n=28) | 0.41 / 0.10 (n=192) |
| blind@claude-sonnet-5 | 0.14 / 0.07 (n=14) | 0.34 / 0.07 (n=150) |
바닥 여부
| cell | floor | not_floor |
|---|---|---|
| oracle-loc | 0.93 / 0.80 (n=30) | 0.82 / 0.19 (n=318) |
| frames-uniform@gemini-3.6-flash | 0.43 / 0.33 (n=30) | 0.63 / 0.17 (n=307) |
| rollforward-vis-pure@gemini-3.6-flash | 0.30 / 0.30 (n=30) | 0.66 / 0.16 (n=310) |
| textlog@gemini-3.6-flash | 0.40 / 0.33 (n=30) | 0.69 / 0.15 (n=186) |
| textlog@claude-sonnet-5 | 0.40 / 0.33 (n=30) | 0.67 / 0.14 (n=176) |
| frames-uniform@claude-sonnet-5 | 0.03 / 0.03 (n=30) | 0.42 / 0.11 (n=300) |
| rollforward-vis-pure@claude-sonnet-5 | 0.07 / 0.03 (n=29) | 0.43 / 0.11 (n=300) |
| rollforward-v2@claude-sonnet-5 | 0.24 / 0.16 (n=25) | 0.46 / 0.11 (n=212) |
| blind@gemini-3.6-flash | 0.10 / 0.10 (n=20) | 0.39 / 0.11 (n=187) |
| rollforward-v2@gemini-3.6-flash | 0.30 / 0.20 (n=30) | 0.40 / 0.09 (n=190) |
| blind@claude-sonnet-5 | 0.00 / 0.00 (n=12) | 0.35 / 0.07 (n=152) |
쌍둥이 인접
| cell | False | True |
|---|---|---|
| oracle-loc | 0.81 / 0.24 (n=304) | 0.98 / 0.25 (n=44) |
| frames-uniform@gemini-3.6-flash | 0.59 / 0.18 (n=295) | 0.76 / 0.19 (n=42) |
| rollforward-vis-pure@gemini-3.6-flash | 0.62 / 0.17 (n=298) | 0.67 / 0.19 (n=42) |
| textlog@gemini-3.6-flash | 0.62 / 0.18 (n=177) | 0.82 / 0.15 (n=39) |
| textlog@claude-sonnet-5 | 0.58 / 0.17 (n=169) | 0.86 / 0.16 (n=37) |
| frames-uniform@claude-sonnet-5 | 0.35 / 0.10 (n=287) | 0.58 / 0.12 (n=43) |
| rollforward-vis-pure@claude-sonnet-5 | 0.38 / 0.09 (n=287) | 0.52 / 0.14 (n=42) |
| rollforward-v2@claude-sonnet-5 | 0.41 / 0.10 (n=203) | 0.59 / 0.18 (n=34) |
| blind@gemini-3.6-flash | 0.34 / 0.11 (n=175) | 0.47 / 0.09 (n=32) |
| rollforward-v2@gemini-3.6-flash | 0.36 / 0.11 (n=180) | 0.53 / 0.10 (n=40) |
| blind@claude-sonnet-5 | 0.30 / 0.06 (n=132) | 0.41 / 0.09 (n=32) |
4 실패 사례 (oracle-loc 기준 — 메모리가 아니라 하네스/정책 한계)
val_11__2 · Pick up the remote control I moved earlier · sources oracle-loc, textlog@gemini-3.6-flash, frames-uniform@gemini-3.6-flash…
FAIL · 476s · sel0_a0(remotecontrol_dbbd): t0:✗600, t1:✗600
FAIL · 476s · sel0_a0(remotecontrol_dbbd): t0:✗600, t1:✗600
첫 pick 키프레임 4장 (bench_exo_L)
val_144__0 · Pick up the apple I moved earlier · sources oracle-loc, blind@gemini-3.6-flash, textlog@gemini-3.6-flash…
FAIL · 494s · sel0_a0(apple_ff93afa9f54b): t0:✗600, t1:✗600
FAIL · 494s · sel0_a0(apple_ff93afa9f54b): t0:✗600, t1:✗600
첫 pick 키프레임 4장 (bench_exo_L)
val_179__0 · Pick up all the ladles · sources oracle-loc
FAIL · 662s · sel0_a0(ladle_efa098c0c2ee): t0:✓300 | sel1_a0(ladle_efa098c0c2ee): t0:✗600 | sel1_a1(ladle_efa098c0c2ee): t0:✗600
FAIL · 662s · sel0_a0(ladle_efa098c0c2ee): t0:✓300 | sel1_a0(ladle_efa098c0c2ee): t0:✗600 | sel1_a1(ladle_efa098c0c2ee): t0:✗600
첫 pick 키프레임 4장 (bench_exo_L)
val_101__1 · Pick up all the bowls I moved earlier · sources oracle-loc
FAIL · 626s · sel0_a0(bowl_46a21212675e4): t0:✓189 | sel1_a0(painting_2fadc8a08): t0:✗600 | sel1_a1(bowl_6befd62f08fd3): t0:✗600
FAIL · 626s · sel0_a0(bowl_46a21212675e4): t0:✓189 | sel1_a0(painting_2fadc8a08): t0:✗600 | sel1_a1(bowl_6befd62f08fd3): t0:✗600
첫 pick 키프레임 4장 (bench_exo_L)
성공 사례
val_101__1 · Pick up the remote control I moved earlier · sources oracle-loc, textlog@gemini-3.6-flash, rollforward-v2@gemini-3.6-flash…
SUCCESS · 318s · sel0_a0(remotecontrol_dbbd): t0:✗600 | sel0_a1(remotecontrol_dbbd): t0:✓118
SUCCESS · 318s · sel0_a0(remotecontrol_dbbd): t0:✗600 | sel0_a1(remotecontrol_dbbd): t0:✓118
마지막 성공 pick 키프레임 4장 (bench_exo_L)
val_125__0 · Pick up the fork · sources oracle-loc, frames-uniform@gemini-3.6-flash, rollforward-vis-pure@gemini-3.6-flash…
SUCCESS · 25s · sel0_a0(fork_a17d780ca6a79): t0:✓55
SUCCESS · 25s · sel0_a0(fork_a17d780ca6a79): t0:✓55
마지막 성공 pick 키프레임 4장 (bench_exo_L)
val_179__0 · Pick up the tissue box · sources oracle-loc, frames-uniform@gemini-3.6-flash, rollforward-vis-pure@gemini-3.6-flash…
SUCCESS · 136s · sel0_a0(tissuepaper_1f2829): t0:✓327
SUCCESS · 136s · sel0_a0(tissuepaper_1f2829): t0:✓327
마지막 성공 pick 키프레임 4장 (bench_exo_L)
5 Takeaway / Next
- 메모리 전략 격차는 Stage B(위치 정확도+접지)에서 갈리고, Stage C는 정책 ceiling이 공통으로 누른다 → 리더보드는 SR_B·SR_C·P(lift|reached)를 나란히 보고.
- 다음: ① online 트랙(매 턴 VLM이 subgoal+메모리 갱신; 설계 `claude/260821/plan-online_exec_track.md`) ② 정책 교체 zero-shot 비교(Cosmos 3-nano-policy-droid) ③ 접지 room-mode 후보 순서(카테고리 인스턴스 우선) v4.