routine 온라인 본 런 최종

1배경/목적

실험 2: 읽기 트랙(VLM) 본표의 답을 초기 계획으로 주고, 로봇이 온라인으로 실행해 목표를 완수하는지 잰다. routine 은 "매일 하던 일 중 남은 것을 제자리에" 과제(682칸 = main 5전략 534 + tagmem 148). 헤드라인은 답 단위 GC(사용자 결정 260921).

2작업 내용

스냅샷 tmp/routine_online/src_r1(find/restore src_w2 와 공유 프롬프트 72개 바이트 동일), SBENCH_PICK_FACE_EXACT=0.
편 단위 샤딩 8(extra) · 잡당 5편 · 200 GB · babysitter 자동 재제출. 클러스터 포화로 extra 선점·풀 대기가 길어 마지막엔 own 으로 옮기고, 한 편에 몰린 잔여 6칸은 칸 단위 6잡으로 분할(submit_tail.sh).
재제출 경계 중복 plan_id 는 첫 행만(집계기 규칙).

3결과 — 전략별

전략	칸	GC(헤드라인)	SR [95% CI]	$/칸
blind	21	0.238	5/21 [0.11, 0.45]	$0.104
textlog	172	0.166	28/172 [0.12, 0.23]	$0.119
frames-uniform	128	0.135	17/128 [0.08, 0.20]	$0.241
rollforward-v3	133	0.160	21/133 [0.11, 0.23]	$0.134
rollforward-vis-pure	80	0.175	14/80 [0.11, 0.27]	$0.241
tagmem-r3	148	0.144	21/148 [0.09, 0.21]	$0.293

자기 셀 수치는 전략마다 셀 집합이 달라 직접 비교 불가 — 아래 LOO(같은 셀의 다른 전략 대비)로 본다.

전략	비교가능 셀	관측 SR	기대 SR	ΔSR ±SE	Δ턴	Δfalse_done	Δ첫nav적중
blind	21	0.238	0.271	-0.033 ±0.088	-0.68 ±2.40	-0.086 ±0.091	-0.013 ±0.068
textlog	162	0.167	0.132	+0.034 ±0.030	+1.07 ±1.13	-0.046 ±0.035	+0.123 ±0.030
frames-uniform	128	0.133	0.131	+0.002 ±0.029	-2.40 ±1.21	+0.114 ±0.045	-0.089 ±0.037
rollforward-v3	132	0.159	0.164	-0.005 ±0.030	+3.33 ±1.35	-0.075 ±0.035	-0.044 ±0.033
rollforward-vis-pure	74	0.189	0.171	+0.018 ±0.043	-2.71 ±1.64	+0.077 ±0.058	-0.061 ±0.038
tagmem-r3	141	0.142	0.183	-0.042 ±0.033	-0.65 ±1.40	-0.008 ±0.036	+0.014 ±0.036

tagmem-r3 Δ −0.042 ±0.033(1.3SE), textlog +0.034 ±0.030(1.1SE) — 모두 2SE 안. 효율 축에서 rf-v3 는 턴 +3.3(2.5SE)로 가장 느리고 frames-uniform·vis-pure 는 false_done 이 +0.11·+0.08 로 높다(성급한 완료 선언).

4읽기 → 온라인 깔때기

전략	읽기 정답 계획	온라인 SR	종단 SR
blind	24/314 = 0.076	5/21 = 0.238	0.018
textlog	235/314 = 0.748	28/172 = 0.163	0.122
frames-uniform	160/314 = 0.510	17/128 = 0.133	0.068
rollforward-v3	169/314 = 0.538	21/133 = 0.158	0.085
rollforward-vis-pure	90/314 = 0.287	14/80 = 0.175	0.050
tagmem-r3	188/314 = 0.599	21/148 = 0.142	0.085

종단 순위(textlog .122 > rf-v3·tagmem .085 > frames .068 > vis-pure .050 > blind .018)는 읽기 정답률이 거의 다 정한다 — 온라인 실행률은 전략 간 비슷(.13~.18)하기 때문.

5실패 분해 · 물체 카테고리

실패 576칸: 정답 근처 미도달 67(12%) · 도달 후 파지 실패 479(83%) · 들었지만 종료 위치·done 미달 36(6%). 바닥 낙하 발생 144칸은 성공 0.

카테고리	GT	도달	들어 올림	들어 올림/도달	VLA 학습 표본(1,900 중)
Spray_Bottle	123	111	42	0.38	24
Hand_Towel	118	107	20	0.19	24
Alarm_Clock	100	89	20	0.22	8
Bread	97	92	1	0.01	0
Tomato	85	80	0	0.00	7
Mug	67	50	27	0.54	151
Fork	47	34	18	0.53	26
Cloth	32	21	0	0.00	1
ButterKnife	27	20	4	0.20	4
Spoon	25	20	12	0.60	24
Plate	23	22	1	0.05	17
Soap_Bottle	9	9	0	0.00	12
Cellphone	6	4	1	0.25	24

find 본 런의 같은 지표는 0.54, 머그·포크·숟가락은 routine 에서도 ~0.55 — VLA 는 학습에 있는 물체는 가족과 무관하게 같은 비율로 든다. routine 의 낮은 GC 는 GT 물체 구성(빨간 행 5종 = GT 44%) 때문이며, 모든 전략이 같은 물체를 보므로 전략 비교는 공정하지만 천장이 낮아 검정력이 준다. 토마토(학습 7)·비누병(12)도 0 이라 표본 수만으로 다 설명되진 않는다(형상·에셋 물리 의심, 미확인).

6사례 — 성공 2 · 실패 3

성공 ① 머그를 찾아 제자리에 — 287624eecad4 · train_1949__7456 · textlog · 턴 4 · 종료 done · GC 1.00
질의: “Put the mug on the dining table away.” · 정답 물체: Mug

nav 1 after (exo)

턴 0 pick — exo 시작·중간·끝 (성공)

턴 0 pick — wrist 시작·중간·끝 (성공)

턴 2 pick_and_place — exo 시작·중간·끝 (성공)

턴 2 pick_and_place — wrist 시작·중간·끝 (성공)

턴	스킬	대상	결과	VLM 판단 원문(앞 220자)
0	pick	Mug	✅	The goal is to put the mug on the dining table away. The mug is right in front of the robot on the dining table. I will pick it up.
1	navigate	stand	✅	The mug has been grasped and lifted. Now I need to navigate to the side table in the bathroom to place the mug.
2	pick_and_place	Mug	✅	I am holding the mug and currently positioned in front of the destination side table. I will place the mug onto the side table.
3	done		—	The mug has been successfully placed onto the side table in the bathroom alongside the dishes, completing the goal.

첫 턴 VLM 프롬프트 원문(텍스트 파트)

Your current view:

Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):

[context] [    0s] The user said: "Put the food from the dining table on the armchair."
[context] [    0s] The user said: "Put the dishes from the dining table on the side table."
[context] [    0s–  146s] I picked up the egg from the dining table in the living room and placed it on the armchair in the living room.
[context] [  146s–  356s] I picked up the bowl from the dining table in the living room and placed it on the side table in the bathroom.
[context] [  356s–  532s] I picked up the apple from the dining table in the living room and placed it on the armchair in the living room.
[context] [  532s–  755s] I picked up the bowl from the dining table in the living room and placed it on the side table in the bathroom.
[plan] 1. [ ] put the mug away on the side table in the bathroom

Object names used in this house:
CD, alarm clock, apple, armchair, baseball bat, bed, book, bowl, box, cell phone, countertop, credit card, cup, desk, desk lamp, dining table, dresser, egg, faucet, floor lamp, fridge, garbage can, laptop, microwave, mug, newspaper, painting, pan, pen, pencil, pillow, plate, plunger, pot, potato, remote control, salt shaker, side table, sink, sofa, spatula, spray bottle, statue, stool, television, toilet, toilet paper, tomato, tv stand, vase, watch

Room names used in this house:
bedroom, bathroom, kitchen, living room

Goal: Put the mug on the dining table away.

Reply with the JSON object only.

판정: 성공(GC 1.0, 4턴).
원인: 계획 원천(읽기 답)이 정답 방·가구를 가리켰고, 학습에 흔한 물체(머그: 학습 표본 151)라 VLA 파지가 한 번에 됐다.
교훈: 성공 칸의 공통점은 '정답 계획 + 흔한 물체' — 메모리가 좋은 계획을 주면 실행은 VLA 가 한다.

성공 ② 드문 물체(손수건)도 되는 경우 — 7c51c0baa9eb · train_3218__6617 · rollforward-v3 · 턴 5 · 종료 done · GC 1.00
질의: “Put the hand towel on the countertop away.” · 정답 물체: Hand_Towel

nav 0 after (exo)

nav 2 after (exo)

턴 1 pick — exo 시작·중간·끝 (성공)

턴 1 pick — wrist 시작·중간·끝 (성공)

턴 3 pick_and_place — exo 시작·중간·끝 (성공)

턴 3 pick_and_place — wrist 시작·중간·끝 (성공)

턴	스킬	대상	결과	VLM 판단 원문(앞 220자)
0	navigate	Hand_Towel	✅	The hand towel is on the countertop in the kitchen, but not close enough to reach. I need to navigate to the hand towel on the kitchen countertop first.
1	pick	Hand_Towel	✅	We are at the kitchen countertop facing the hand towel. Now initiate picking the hand towel.
2	navigate	armchair	✅	Holding hand towel. Proceeding with the plan to navigate to the armchair in the bedroom.
3	pick_and_place	Hand_Towel	✅	We have navigated to the armchair while holding the hand towel. Now we place the hand towel on the armchair using pick_and_place.
4	done		—	The hand towel has been successfully placed onto the armchair, and the gripper is empty. The goal is complete.

첫 턴 VLM 프롬프트 원문(텍스트 파트)

Your current view:

Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):

[context] I was instructed to place kitchen utensils from the countertop onto the side table, gadgets from the countertop into the laundry basket, and clothes from the countertop onto the armchair. In the kitchen, I picked up a spatula from the countertop and placed it onto the side table in the bedroom. Next, I picked up a cell phone from the countertop and placed it into the laundry basket. Then, I picked up a fabric from the countertop and placed it onto the armchair in the bedroom. Afterwards, I picked up a second cell phone from the countertop and placed it into the laundry basket. Most recently, I returned to the kitchen countertop and approached another fabric.
[plan] 1. [ ] put the hand towel away on the armchair in the bedroom

Object names used in this house:
alarm clock, apple, armchair, baseball bat, bed, book, bowl, bread, candle, cell phone, chair, cloth, coffee machine, countertop, desk lamp, dining table, dish sponge, dresser, faucet, floor lamp, fork, fridge, garbage can, hand towel, house plant, knife, ladle, laundry hamper 1, lettuce, mug, painting, pencil, plate, plunger, potato, remote, room decor, side table, sink, sofa, spatula, spoon, teddy bear, television, toilet, toilet paper, tv stand, vase

Room names used in this house:
bedroom, bathroom, kitchen, living room

Goal: Put the hand towel on the countertop away.

Reply with the JSON object only.

판정: 성공(GC 1.0, 5턴).
원인: 손수건은 학습 데이터에 없지만(종이타월만 24) 이 칸은 첫 시도에 들렸다 — 카테고리 커버리지는 확률을 낮출 뿐 0 으로 만들지 않는다.
교훈: 희귀 카테고리 성공률 .06 은 '불가능'이 아니라 '드묾' — 전략 비교에는 같은 분포라 공정하다.

실패 ① 토마토 앞까지 가서 7번 집기 실패 — 4a7c78424913 · train_1102__7832 · rollforward-v3 · 턴 7 · 종료 stalled · GC 0.00
질의: “Put the tomato on the countertop away.” · 정답 물체: Tomato

턴 0 pick — exo 시작·중간·끝 (실패)

턴 0 pick — wrist 시작·중간·끝 (실패)

턴	스킬	대상	결과	VLM 판단 원문(앞 220자)
0	pick	Tomato	❌	I am standing at the kitchen countertop with an empty gripper. The tomato is on the countertop within reach. I will attempt to pick it.
1	pick_and_place	Apple	✅	The gripper picked the apple instead of the tomato. I need to place the apple back on the countertop before I can pick the tomato.
2	pick_and_place	Apple	✅	I am currently holding the apple. I need to place the apple back onto the countertop so my gripper is free to pick the tomato.
3	pick_and_place	Apple	✅	Holding apple; need to place it on the countertop first to free the gripper for the tomato.
4	pick_and_place	Apple	✅	We are currently holding an apple, which prevents us from picking the tomato. First place the apple down onto the countertop using pick_and_place.
5	pick_and_place	Apple	✅	We are currently holding an apple. To free our gripper to pick the tomato, we need to place the apple on the countertop first.
6	pick_and_place	Apple	✅	I am holding an apple while in front of the kitchen countertop. I need to place the apple down onto the countertop first to free the gripper so I can pick up the tomato.

첫 턴 VLM 프롬프트 원문(텍스트 파트)

Your current view:

Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):

[context] I was instructed to "Put the food from the countertop on the side table." In the living room, I saw a bed, a sofa (holding a remote control, cell phone, and newspaper), a tv stand with a bowl, a house plant, a shelf with a vase, and a side table. In the kitchen, the countertop held an alarm clock, an egg, irish potatoes, apples, and tomatoes, while a dining table held a toaster and a house plant. So far, I have placed an egg, an irish potato, a tomato, and an apple onto the living room side table.
[plan] 1. [ ] put the tomato away on the side table in the living room

Object names used in this house:
alarm clock, apple, bed, book, bowl, cell phone, countertop, credit card, dining table, dish sponge, dog bed, dresser, egg, faucet, fork, fridge, garbage bag, garbage can, house plant, laptop, ottoman, painting, pen, pencil, pillow, plunger, potato, remote, remote control, shelving unit, side table, sink, sofa, statue, television, toaster, toilet, tomato, tv stand, vase, watch

Room names used in this house:
bedroom, bathroom, kitchen, living room

Goal: Put the tomato on the countertop away.

Reply with the JSON object only.

판정: 실패(GC 0, stalled).
원인: 정답 위치 도달(최근접 0.4 m 이내)은 성공했지만 pick 7회 모두 들어 올리지 못함. 토마토는 학습 표본 7개, routine 전체 0/76.
교훈: 실패의 83%가 이 유형(도달 후 파지 실패). 메모리 전략 차이가 아니라 VLA 카테고리 커버리지가 천장을 정한다.

실패 ② 토마토를 떨어뜨리고도 '완료' 선언 — 7ceab0a0fe3e · train_1102__7832 · textlog · 턴 16 · 종료 done · GC 0.00
질의: “Put the tomato on the countertop away.” · 정답 물체: Tomato

nav 1 after (exo)

nav 2 after (exo)

턴 0 pick — exo 시작·중간·끝 (실패)

턴 0 pick — wrist 시작·중간·끝 (실패)

턴 3 pick — exo 시작·중간·끝 (실패)

턴 3 pick — wrist 시작·중간·끝 (실패)

턴	스킬	대상	결과	VLM 판단 원문(앞 220자)
0	pick	Tomato	❌	The robot is in front of the countertop in the kitchen with the tomato within reach. I will pick up the tomato.
1	navigate	stand	✅	The gripper is currently holding the apple. I will navigate to the living room side table to place it down before returning for the tomato.
2	navigate	Tomato	✅	The tomato is still on the countertop in the kitchen. I must navigate to the countertop in the kitchen to pick it up.
3	pick	Tomato	❌	I have navigated to the kitchen countertop where the tomato is located. Now I need to pick up the tomato.
4	pick	Tomato	❌	The gripper is aligning with the tomato on the countertop but has not grasped and lifted it yet. Continuing the pick subgoal.
5	navigate	stand	✅	The tomato has been successfully picked up and is held in the gripper. Now navigate to the side table in the living room.
6	navigate	Tomato	✅	The tomato was not successfully picked and is still on the kitchen countertop. I need to navigate back to the kitchen countertop to pick it.
7	pick	Tomato	❌	The tomato is in view on the kitchen countertop. I will pick up the tomato.
8	pick	Tomato	❌	The tomato on the kitchen countertop is in view and within reach, but has not yet been grasped. Issuing pick(tomato).
9	pick	Tomato	❌	The gripper is positioned above the tomato on the countertop but has not grasped it yet. I need to continue the pick action.
10	pick	Tomato	❌	The gripper is aligning with the tomato but has not grasped and lifted it yet. Continuing pick.
11	pick	Tomato	❌	The pick attempt failed and the gripper is empty. The tomato is still on the countertop. Retrying pick.
12	navigate	Tomato	✅	I need to put the tomato on the countertop away onto the side table. I will navigate directly to the tomato on the countertop in the kitchen to ensure I am within grasping range.
13	pick	Tomato	❌	The arm is moving toward the tomato on the countertop but has not finished grasping and lifting it yet. Continue picking.

첫 턴 VLM 프롬프트 원문(텍스트 파트)

Your current view:

Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):

[context] [    0s] The user said: "Put the food from the countertop on the side table."
[context] [    0s–  165s] I picked up the egg from the countertop in the kitchen and placed it on the side table in the living room.
[context] [  165s–  322s] I picked up the potato from the countertop in the kitchen and placed it on the side table in the living room.
[context] [  322s–  470s] I picked up the apple from the countertop in the kitchen and placed it on the side table in the living room.
[plan] 1. [ ] put the tomato away on the side table in the living room

Object names used in this house:
alarm clock, apple, bed, book, bowl, cell phone, countertop, credit card, dining table, dish sponge, dog bed, dresser, egg, faucet, fork, fridge, garbage bag, garbage can, house plant, laptop, ottoman, painting, pen, pencil, pillow, plunger, potato, remote, remote control, shelving unit, side table, sink, sofa, statue, television, toaster, toilet, tomato, tv stand, vase, watch

Room names used in this house:
bedroom, bathroom, kitchen, living room

Goal: Put the tomato on the countertop away.

Reply with the JSON object only.

판정: 실패(GC 0, done — false_done).
원인: 파지 중 토마토가 바닥으로 떨어졌는데(gt_fell_to_floor) VLM 이 이미지로 완료를 판단해 done. 바닥 낙하 칸 144개는 전부 실패.
교훈: done_mode=vlm 에서 '완료 판단'은 이미지 판독 능력에 의존 — false_done 율 .13~.27 이 전략별 차이의 대부분.

실패 ③ 정답 근처에 못 가고 이동만 반복(nav_loop) — 607d1ca68340 · train_1161__7339 · rollforward-v3 · 턴 9 · 종료 nav_loop · GC 0.25
질의: “Clear the dining table. Items on it: alarm clock, fork, cell phone, spatula.” · 정답 물체: Alarm_Clock, Fork, Cellphone, Spatula

nav 1 after (exo)

nav 3 after (exo)

턴 0 pick — exo 시작·중간·끝 (성공)

턴 0 pick — wrist 시작·중간·끝 (성공)

턴 2 pick_and_place — exo 시작·중간·끝 (실패)

턴 2 pick_and_place — wrist 시작·중간·끝 (실패)

턴	스킬	대상	결과	VLM 판단 원문(앞 220자)
0	pick	Alarm_Clock	✅	The robot is in front of the dining table with the alarm clock right under the gripper. Attempting to pick the alarm clock.
1	navigate	sofa	✅	Holding the alarm clock. Now navigating to the sofa in the living room to place it.
2	pick_and_place	Alarm_Clock	❌	We have navigated to the sofa in the living room while holding the alarm clock. Now we need to place the alarm clock onto the sofa using pick_and_place.
3	navigate	diningtable	✅	The alarm clock has been successfully placed on the sofa. The next item to clear from the dining table is the fork, which needs to be moved to the side table in the bedroom. Navigating to the dining table to pick up the
4	navigate	diningtable	✅	The fork is not within reach yet, so we need to navigate specifically to the fork on the dining table in the living room.
5	navigate	diningtable	✅	We need to clear the fork from the dining table. The robot moved near the table but the fork is not yet in view/reach. Navigating to the fork on the dining table in the living room.
6	navigate	diningtable	✅	The fork is not within reach yet, so we must navigate specifically to the fork on the dining table in the living room before picking.
7	navigate	diningtable	✅	The fork is not within reach, so navigate closer to the fork on the dining table in the living room.
8	navigate	diningtable	✅	We arrived near the dining table in the living room. I will navigate specifically to the fork on the dining table to ensure the robot is within reach to grasp it.

첫 턴 VLM 프롬프트 원문(텍스트 파트)

Your current view:

Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):

[context] I was instructed to put gadgets from the dining table onto the sofa, and kitchen utensils from the dining table onto the side table. In the bedroom (green floor, bed, basketball, floor cushion, white side table, brown dresser), the dining table initially held a remote control, two cell phones, a spoon, a spatula, and a knife. In the living room (orange floor, cart, side table with a book, armchair, TV stand, sofa), I placed the remote control and both cell phones onto the sofa. In the bedroom, I placed the knife and the spoon onto the white side table. The spatula remains on the dining table.
[plan] 1. [ ] put the alarm clock away on the sofa in the living room
[plan] 2. [ ] put the fork away on the side table in the bedroom
[plan] 3. [ ] put the cell phone away on the sofa in the living room
[plan] 4. [ ] put the spatula away on the side table in the bedroom

Object names used in this house:
alarm clock, apple, armchair, basketball, bed, book, bottle, bowl, candle, cart, cell phone, countertop, dining table, dog bed, dresser, faucet, floor lamp, fork, fridge, garbage can, knife, ladle, newspaper, painting, pillow, plunger, pot, remote, salt shaker, side table, sink, sofa, spatula, spray bottle, stool, television, toilet, tv stand

Room names used in this house:
bedroom, bathroom, kitchen, living room

Goal: Clear the dining table. Items on it: alarm clock, fork, cell phone, spatula.

Reply with the JSON object only.

판정: 실패(GC 0, nav_loop).
원인: 두 물체 질의에서 navigate 만 6회 연속 — 조작 시도 없이 루프 가드(6)에 걸림. 정답 근처 미도달 67칸(12%)의 전형.
교훈: 계획이 틀리면(읽기 오답·방 착각) 온라인 탐색으로 회복하지 못한다 — 종단 SR 은 읽기 정답률 × 실행률.

7의미 (Takeaway)

routine 온라인에서 메모리 전략 간 성공률 차는 측정되지 않는다(LOO 전부 2SE 안). 차이는 읽기 단계(계획 정확도)와 효율·완료 판단에서 난다.
실행 천장은 VLA 카테고리 커버리지 — 논문에 카테고리 층화 표와 함께 "실행 병목은 파지"를 명시해야 한다.
multi 질의(답 3~4개) 30칸은 GC 0 — 한 칸에 여러 물체를 옮기는 긴 실행은 현재 스택으로 불가.

8보완점/다음

find·restore·routine 온라인 표를 한 표로 합치고 카테고리 층화 열 추가.
resume 온라인: 누름 경로 정비 완료(누름 전용 VLA · 버튼 정면 접근 · datagen 판정 · 무효 누름 제외 · 누름 후 팔 복귀 · 초과 감점) — 파일럿 2b 결과 후 본 런 승인 요청.
토마토·빵 실패 영상으로 "미끄러짐 vs 엉뚱한 조준" 구분(카테고리 문제의 물리 요인 확인).

routine 온라인 본 런 — 최종 (682칸)

TL;DR

1배경/목적

2작업 내용

3결과 — 전략별

4읽기 → 온라인 깔때기

5실패 분해 · 물체 카테고리

6사례 — 성공 2 · 실패 3

7의미 (Takeaway)

8보완점/다음