조건 A: 읽기 트랙의 그 칸 답을 초기 계획으로 주입하고, 계획 갱신은 전략 고유 수단만 허용(rf-v3만 재작성, 나머지는 읽기 전용).
셀 선택: 읽기 답이 방과 가구까지 맞은 칸만 — find 1108 + restore 464 = 1572칸, 예상 API ≈ $150~190(샘플 실측 단가 기준). 근거는 파일럿(성공 7건이 전부 정답 계획 칸에서 나옴) + 샘플(가구 틀린 칸 7/7 실패).
이번에 고친 것: 방 이름 정본을 scene card로 교체(프롬프트 room_N 노출 30/100플랜 → 0), 셀 단위 필터, restore 자명 충족 답 기록, 테스트 fake 6개.
아래 §4에 시스템 프롬프트·계획 블록·턴 입력 전문, §5에 실제 샘플 롤아웃(턴 표·관측 이미지·주행 영상·VLM 원문)을 그대로 실었다.
1572
본 런 셀
~$190
예상 API
5
전략
2
가족(find·restore)
100
편(가족당)
1조건 A — 무엇을 재는 실험인가
읽기 트랙(오프라인 기억 평가)에서 그 전략이 낸 답을 그 칸의 초기 계획으로 주입한다. 그 다음 로봇은 실제 씬에서
턴 루프를 돌며 navigate/pick/pick_and_place/open/close를 고른다.
계획을 고칠 수 있는 수단은 그 전략이 원래 가진 기억 갱신 수단뿐이다 — 이 비대칭이 실험의 핵심이다.
전략
온라인 노트 갱신
온라인 이미지 갱신
계획 갱신 수단
blind
none
none
fixed
textlog
none
none
fixed
frames-uniform
none
uniform
fixed
rollforward-v3
rewrite
none
rewrite
rollforward-vis-pure
none
keep
fixed
왜 이렇게 나눴나. "모든 항목이 [x]면 done" 같은 종료 규칙을 주면 rewrite 전략만 유리해진다.
그래서 종료 조건은 전 전략 공통의 스킬 설명 한 줄(the goal is fully satisfied)뿐이고, 계획 블록은 출처·신뢰도만 알린다.
2세팅 전체
항목
값
기억 모델 (VLM)
gemini-3.7-flash — 읽기 트랙 본표와 같은 모델·같은 답을 초기 계획으로 사용
왜 방 기준(loc_set)을 버렸나(사용자 결정 260916).loc_set은 방만 본다 —
"침대 → 식탁"이 정답인데 "책상 → 책상"이라고 답해도 두 지점이 같은 방이면 1점이다. 그런 칸이 샘플에 섞여 있었고
find 3/3 · restore 4/4 전부 실패했다(파일럿도 같은 방향: loc만 맞은 칸 find 1/3 · restore 0/5).
가구까지 맞은 칸만 남기면 샘플 성공률이 5/7로, 조건 A가 재려던 "기억이 맞을 때 실행이 되는가"에 맞는 집합이 된다.
옛 기준 목록(find 1,444 · restore 804)은 *_cells_loc.json.bak에 보존.
가족
blind
textlog
frames-uniform
rf-v3
rf-vis-pure
합계
find
82
280
235
224
287
1108
restore
2
257
53
64
88
464
총 셀
1572
방 이름 재채점. 옛 방 이름(이벤트 조인)이 틀린 질의 33개(find 13·restore 20)는 scene card 기준으로
다시 채점해서 포함했다. 재채점은 러너와 같은 경로를 쓰고, 영향 없는 행은 저장 점수를 100% 재현했다
(find 65행·restore 100행 검증). 정답 판정은 find +34/−4, restore +25/−1로 뒤집혔다.
4프롬프트 전문
4-1. 계획 블록 — 전략별로 여기만 다르다
[fixed — blind · textlog · frames-uniform · vis-pure]
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
[rewrite — rollforward-v3]
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
You keep this plan up to date. In every reply, return the COMPLETE plan in "plan" as a list of lines, each starting with "[x] " (done) or "[ ] " (not done yet):
1. Mark an item [x] only when the frames clearly show it done — not because you issued the subgoal.
2. If what you see shows an item is wrong, correct it; if something is missing, add an item. Never delete an item that is not done.
4-2. 온라인 시스템 프롬프트 (find · fixed · 텍스트 기억 없음)
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: after you have lifted an object, the robot sets it back down where it is before your next subgoal — that pick already counts, so pick each target once and move on to the next.
Memory — you do not write anything down. Any notes shown above were recorded during the context phase and will not change; apart from those, the frames of your previous subgoal and your current view are all you have. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
4-3. 온라인 시스템 프롬프트 (restore · rewrite · 요약 재작성 기억)
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute, and you update your own notes in the same reply.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your notes so far, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: once you have lifted an object you keep holding it. navigate carries it with you and never places anything; put it down only with pick_and_place(category, surface) at the destination. While you hold an object you cannot pick another one.
Notes — this is the ONLY record you will have of earlier turns; you will not see those frames again:
1. Do NOT claim a subgoal completed unless the last frame clearly shows it. If the frames only show the action in progress, or the final state is unclear, describe only what you actually see — never assume success from what you were trying to do.
2. Prefix anything you are not certain about with "uncertain: ".
3. Record what you attempted, what visibly changed, and what is still left to do.
4. Also record where you see goal-relevant objects (room and surface), even ones you did not act on.
5. Write your COMPLETE updated notes (at most 12 lines). You may compress, correct or drop earlier lines — what you return replaces your notes entirely.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
You keep this plan up to date. In every reply, return the COMPLETE plan in "plan" as a list of lines, each starting with "[x] " (done) or "[ ] " (not done yet):
1. Mark an item [x] only when the frames clearly show it done — not because you issued the subgoal.
2. If what you see shows an item is wrong, correct it; if something is missing, add an item. Never delete an item that is not done.
Output a single JSON object, nothing else:
{"note": "...", "plan": ["[x] ...", "[ ] ..."], "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
4-4. 계획의 출처 — 읽기 트랙 시스템 프롬프트
find 읽기 트랙 프롬프트
You are the memory module of a household robot. The earlier turns of this
conversation are your memory of one episode in this house — what the robot did
and saw. The final image is the robot's CURRENT view at the end of the episode.
The user will ask you to pick up an object, described by its category and by
what happened to it. Answer WHERE that object is now. The object is usually not
in the current view, and nobody will tell you what is in this house: recall it.
Output a single JSON object, nothing else:
{"reasoning": "...",
"selections": [{"category": "...", "room": "...", "surface": "..."}],
"abstain": false}
Rules:
- One entry in "selections" per object the query refers to. If the query says
"all the ...", decide how many there are from your memory. Otherwise give
exactly one.
- "room": one of bathroom | bedroom | kitchen | living room.
- "surface": the kind of thing the object is resting on, from the surface list
below. Use "floor" if it is lying on the floor rather than on a surface —
an object that was dropped ends up there.
- Use the category and surface names exactly as spelled in the lists.
- If your memory genuinely cannot determine the answer, set "abstain": true.
Object categories:
CD | alarm clock | apple | atomizer | bar soap | baseball bat | basketball | boiler
bottle | bowl | box | bread | butter knife | candle | cell phone | coffeemaker
cooking pan | credit card | cup | desktopcomputer | dumbbell | egg | fabric | fork
house plant | irish potato | key chain | knickknack | knife | ladle | laptop | lettuce
microwaveoven | mug | newspaper | paper towel | pen | pencil | pepper shaker | pillow
plate | plunger | pot | remote control | safe | salt shaker | soapdispenser | spatula
sponge | spoon | statue | table lamp | teddy bear | tennisracket | tissue box | toaster
toilet tissue | tomato | vase | watch | wine bottle
Surfaces:
armchair | bed | cart | chair | clothes dryer | coffee table | countertop | desk
dining table | dresser | shelf | sink | sofa | toilet | tv stand | washing machine
floor
restore 읽기 트랙 프롬프트
You are the memory module of a household robot. The earlier turns of this
conversation are your memory of one episode in this house — what the robot did
and saw. The final image is the robot's CURRENT view at the end of the episode.
The user will ask you to pick up an object and put it back where it belongs —
either where it FIRST was, or where it was JUST BEFORE its last move; read the
query. To execute this the robot needs BOTH locations. Answer, for each object:
where it is NOW, and the ORIGINAL place the query asks it returned to. Nobody
will tell you what is in this house: recall it.
Output a single JSON object, nothing else:
{"reasoning": "...",
"selections": [{"category": "...",
"now": {"room": "...", "surface": "..."},
"original": {"room": "...", "surface": "..."}}],
"abstain": false}
Rules:
- One entry in "selections" per object the query refers to. A query joined with
"then ..." lists several objects — one entry each, in any order.
- "room": one of bathroom | bedroom | kitchen | living room.
- "surface": the kind of thing the object rests on, from the surface list below.
Use "floor" if it lies on the floor rather than on a surface.
- Use the category and surface names exactly as spelled in the lists.
- If your memory genuinely cannot determine the answer, set "abstain": true.
Object categories:
CD | alarm clock | apple | atomizer | bar soap | baseball bat | basketball | boiler
bottle | bowl | box | bread | butter knife | candle | cell phone | coffeemaker
cooking pan | credit card | cup | desktopcomputer | dumbbell | egg | fabric | fork
house plant | irish potato | key chain | knickknack | knife | ladle | laptop | lettuce
microwaveoven | mug | newspaper | paper towel | pen | pencil | pepper shaker | pillow
plate | plunger | pot | remote control | safe | salt shaker | soapdispenser | spatula
sponge | spoon | statue | table lamp | teddy bear | tennisracket | tissue box | toaster
toilet tissue | tomato | vase | watch | wine bottle
Surfaces:
armchair | bed | cart | chair | clothes dryer | coffee table | countertop | desk
dining table | dresser | shelf | sink | sofa | toilet | tv stand | washing machine
floor
4-5. 매 턴 VLM에게 들어가는 것
턴 입력은 이미지 4장 + 텍스트 6조각이다: 현재 시야 → 이 집의 물건 이름 목록 →
이 집의 방 이름 목록 → 계획 → 목표 → 출력 지시. 실제 예시는 §5의 각 카드 안 "첫 턴 VLM 입력 전문"에 그대로 있다.
방 이름 목록(3번째 조각)이 이번 수정의 지점이다. 파일럿에서는 여기에
bedroom, room_5, kitchen, living room처럼 원시 라벨이 섞여 나왔고(100플랜 중 30), 지금은 전부 실제 이름이다.
5실제 샘플 롤아웃
아래는 수정본 코드로 돌린 샘플 잡(130053 find · 130054 restore)의 실제 행이다. 턴 표·관측 이미지·주행 영상·VLM 원문 응답을 그대로 실었다.
가족
완료 칸
성공
첫 navigate 적중
평균 턴
평균 시간
API
find
10
5
7
8.5
227s
$1.15
restore
10
3
4
31.0
1010s
$3.96
본 런 기준 성공률 = 8/13 (62%). 샘플 셀은 옛 기준(방만)으로 뽑혀 두 종류가 섞여 있어, 본 런과 같은 기준(방+가구까지 맞은 계획)만 추린 수치다. 파일럿에서 같은 기준으로 보면 find 5/13(38%) · restore 1/9(11%) 였다 — 방 이름 결함(100플랜 중 30플랜의 프롬프트에 room_N 노출)을 고친 것이 가장 큰 차이로 보인다. 다만 n 이 작아 확정 수치로 쓸 수 없다.
가족
완료
본 런 기준 칸
성공
single
multi
방만 맞은 칸(본 런 제외)
find
10
7
5
5/5
0/2
0/3
restore
10
6
3
3/5
0/1
0/4
find 샘플 진단 (10칸)
같은 navigate 3연속 이상(루프): 2/10
예산 초과로 끝난 칸: 2/10
플랜당 API: single 평균 $0.0324(n=7) · multi 평균 $0.3083(n=3)
재질의 사유:
there is no bowl within reach — navigate first — 15회
you already picked up the bowl here — move on — 5회
there is no statue within reach — navigate first — 3회
you already picked up the kettle here — move on — 1회
restore 샘플 진단 (10칸)
같은 navigate 3연속 이상(루프): 3/10
예산 초과로 끝난 칸: 2/10
플랜당 API: single 평균 $0.1594(n=7) · multi 평균 $0.9487(n=3)
재질의 사유:
there is no remote control within reach — navigate first — 24회
there is no box within reach — navigate first — 18회
you are still holding the cell phone — you cannot pick another object; put it do — 4회
you are still holding the remote control — you cannot pick another object; put i — 3회
예산 경계는 한 경로만큼 무르다. 스텝 누계에 들어가는 것은 정책 호라이즌이 아니라
주행(drive) 스텝이다 — 하네스는 서브골 시작 전에만 예산을 확인하므로, 예산 직전에 시작한 navigate 하나가
경로 길이만큼(실측 221스텝) 더 쓸 수 있다. 실측 예: 누계 1465에서 시작한 마지막 navigate가 1686으로 끝나 예산 1500을 186 초과.
파일럿·옛 프로토콜과 같은 동작이라 조건 간 공정성은 유지되지만, "SR@예산"의 경계가 ±1 경로만큼 무르다는 점은 기록해 둔다.
루프의 비용. 실측에서 there is no bowl within reach — navigate first를 15회 받으며
같은 식탁으로 33번 이동한 multi 플랜이 하나 나왔다(턴 상한 34 소진, $0.62 = 평균 single의 25배).
파일럿에서 본 "fixed 계획은 틀린 계획을 못 벗어난다"와 같은 형태이고, 비용의 대부분을 이런 소수 플랜이 만든다.
find — 샘플 10칸 중 4칸
성공 Pick up the kettle
v1_train_1847__1 · blind · single · 계획모드 fixed · 턴 3/22 · 스텝 162/1500 · $0.0177 · 방 이름 3개
읽기 답(= 계획의 출처)
{
"reasoning": "The kettle (categorized as boiler) is located on the dining table in the kitchen.",
"selections": [
{
"category": "boiler",
"room": "kitchen",
"surface": "dining table"
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] pick up the boiler — on the dining table in the kitchen
최종(0회 재작성)
[ ] pick up the boiler — on the dining table in the kitchen
턴
스킬
대상
성공
스텝
VLM 이유
0
navigate
boiler_335af387651d9869b84abf1fb099bf69_1_0_6
✅
48
The goal is to pick up the kettle, which is located on the dining table in the kitchen. First navigate closer to the kettle on the dining table.
1
pick
boiler_335af387651d9869b84abf1fb099bf69_1_0_6
❌
—
The robot is at the dining table in the kitchen next to the kettle. Now execute pick on the kettle.
2
done
—
❌
—
The kettle has already been picked up successfully, satisfying the goal. 재질의: you already picked up the kettle here — move on
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png턴 0 이동 전턴 0 이동 후
첫 턴 VLM 입력 전문 — 텍스트 6조각 + 이미지 1장
Your current view:
Object names used in this house:
alarm clock, apple, armchair, baseball bat, bed, book, bottle, bowl, box, bread, butter knife, candle, cart, cell phone, chair, countertop, credit card, dining table, dresser, egg, floor lamp, fridge, garbage can, house plant, kettle, key chain, ladle, laptop, lettuce, microwave, mug, painting, pan, pen, pencil, pepper shaker, pillow, plate, potato, remote control, salt shaker, soap bottle, sofa, spatula, spoon, spray bottle, statue, television, tomato, tv stand, wine bottle
Room names used in this house:
bedroom, kitchen, living room
Your plan:
1. [ ] pick up the boiler — on the dining table in the kitchen
Goal: Pick up the kettle
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: after you have lifted an object, the robot sets it back down where it is before your next subgoal — that pick already counts, so pick each target once and move on to the next.
Memory — you do not write anything down. Any notes shown above were recorded during the context phase and will not change; apart from those, the frames of your previous subgoal and your current view are all you have. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{"note": null, "reasoning": "The goal is to pick up the kettle, which is located on the dining table in the kitchen. First navigate closer to the kettle on the dining table.", "next": {"skill": "navigate", "category": "kettle", "surface": "dining table", "room": "kitchen"}}
```
성공 Navigate to the butter knife
v1_train_2159__2 · blind · single · 계획모드 fixed · 턴 2/22 · 스텝 296/1500 · $0.0068 · 방 이름 4개
읽기 답(= 계획의 출처)
{
"reasoning": "Butter knives are standardly found in the kitchen on the countertop.",
"selections": [
{
"category": "butter knife",
"room": "kitchen",
"surface": "countertop"
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] go to the butter knife — on the countertop in the kitchen
최종(0회 재작성)
[ ] go to the butter knife — on the countertop in the kitchen
Navigating to the butter knife located on the kitchen countertop as per plan.
1
done
—
❌
—
The robot has arrived in front of the butter knife on the kitchen countertop, satisfying the navigation goal.
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png턴 0 이동 전턴 0 이동 후
첫 턴 VLM 입력 전문 — 텍스트 6조각 + 이미지 1장
Your current view:
Object names used in this house:
armchair, baseball bat, basketball, bed, book, bottle, bowl, box, butter knife, cart, cell phone, chair, countertop, credit card, cup, desk, dining table, dish sponge, dresser, egg, faucet, fridge, garbage can, house plant, key chain, laptop, laundry hamper, lettuce, mug, newspaper, painting, pen, pencil, pepper shaker, pillow, plate, plunger, potato, remote control, salt shaker, sink, soap bar, soap bottle, sofa, spoon, spray bottle, statue, teddy bear, television, tissue box, toaster, toilet, toilet paper, tv stand, vase
Room names used in this house:
bedroom, bathroom, kitchen, living room
Your plan:
1. [ ] go to the butter knife — on the countertop in the kitchen
Goal: Navigate to the butter knife
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: after you have lifted an object, the robot sets it back down where it is before your next subgoal — that pick already counts, so pick each target once and move on to the next.
Memory — you do not write anything down. Any notes shown above were recorded during the context phase and will not change; apart from those, the frames of your previous subgoal and your current view are all you have. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{"note": null, "reasoning": "Navigating to the butter knife located on the kitchen countertop as per plan.", "next": {"skill": "navigate", "category": "butter knife", "surface": "countertop", "room": "kitchen"}}
```
성공 Pick up the pencil
v1_train_899__1 · rollforward-vis-pure · single · 계획모드 fixed · 턴 8/22 · 스텝 853/1500 · $0.0910 · 방 이름 4개
읽기 답(= 계획의 출처)
{
"reasoning": "A yellow pencil is seen resting on the dining table in the kitchen at t=440.0s, and it was not moved afterward.",
"selections": [
{
"category": "pencil",
"room": "kitchen",
"surface": "dining table"
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] pick up the pencil — on the dining table in the kitchen
최종(0회 재작성)
[ ] pick up the pencil — on the dining table in the kitchen
턴
스킬
대상
성공
스텝
VLM 이유
0
navigate
pencil_04d685f59ba31abb2c3114e046a77deb_1_0_6
✅
61
The pencil is located on the dining table in the kitchen (seen in m7). I will navigate to the dining table in the kitchen to pick up the pencil.
1
pick
pencil_04d685f59ba31abb2c3114e046a77deb_1_0_6
❌
—
We are in front of the dining table in the kitchen where the pencil was located. Let's execute pick for pencil.
2
pick
pencil_04d685f59ba31abb2c3114e046a77deb_1_0_6
❌
—
The gripper is reaching for the pencil on the dining table but has not completed the pick yet. Continuing the pick subgoal.
3
pick
pencil_04d685f59ba31abb2c3114e046a77deb_1_0_6
❌
—
The gripper is directly positioned over the pencil on the dining table in the kitchen. Now pick the pencil.
4
pick
pencil_04d685f59ba31abb2c3114e046a77deb_1_0_6
❌
—
The gripper attempted to pick the pencil on the dining table but did not successfully grasp/lift it yet. Retrying pick(pencil).
5
pick
pencil_04d685f59ba31abb2c3114e046a77deb_1_0_6
❌
—
The pencil is located on the dining table right in front of the gripper. We need to execute the pick action to grasp and lift it.
6
pick
pencil_04d685f59ba31abb2c3114e046a77deb_1_0_6
❌
—
The gripper has contacted the pencil; executing pick again to lift it completely.
7
done
—
❌
—
The pencil has been picked up and is held in the gripper as seen in frame F3. The goal is complete.
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png턴 0 이동 전턴 0 이동 후
첫 턴 VLM 입력 전문 — 텍스트 16조각 + 이미지 9장
Your memory frames:
[m1]
[m2]
[m3]
[m4]
[m5]
[m6]
[m7]
[m8]
Your current view:
[F1]
Object names used in this house:
CD, apple, baseball bat, bed, book, bottle, bowl, bread, candle, cell phone, chair, cloth, countertop, credit card, cup, desk lamp, dining table, dog bed, dresser, egg, faucet, fork, fridge, garbage bag, garbage can, house plant, knife, laptop, painting, pan, paper towel roll, pen, pencil, pillow, plate, potato, remote control, salt shaker, side table, sink, sofa, spoon, spray bottle, television, tissue box, toilet, toilet paper, tomato
Room names used in this house:
bedroom, bathroom, kitchen, living room
Your plan:
1. [ ] pick up the pencil — on the dining table in the kitchen
Goal: Pick up the pencil
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute, and you update your visual memory in the same reply.
Each turn you receive: your memory frames, the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: after you have lifted an object, the robot sets it back down where it is before your next subgoal — that pick already counts, so pick each target once and move on to the next.
Visual memory — this is the ONLY record you will have of earlier turns; you will not see any other frame again. You keep a small set of frames (at most 8), labeled m1, m2, ... in the memory block above; the frames from this turn are labeled F1, F2, ...
1. Keep a frame if it shows something you will need later — an object being picked up, an object just placed at its new location, or a clear view of a room you have not memorized yet.
2. Put your decision in "keep" as ONE line: "KEEP: F2", or "KEEP: none", or — if your memory already holds 8 frames and you want to add one — "KEEP: F2 DROP: m5".
3. You keep no written notes. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "keep": "KEEP: ...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{
"note": null,
"keep": "KEEP: none",
"reasoning": "The pencil is located on the dining table in the kitchen (seen in m7). I will navigate to the dining table in the kitchen to pick up the pencil.",
"next": {
"skill": "navigate",
"category": "pencil",
"surface": "dining table",
"room": "kitchen"
}
}
```
성공 Pick up the tissue box I moved earlier
v1_train_2277__2 · textlog · single · 계획모드 fixed · 턴 6/22 · 스텝 960/1500 · $0.0274 · 방 이름 4개
읽기 답(= 계획의 출처)
{
"reasoning": "The log shows that the tissue box was moved from the tv stand in the living room to the dining table in the living room.",
"selections": [
{
"category": "tissue box",
"room": "living room",
"surface": "dining table"
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] pick up the tissue box — on the dining table in the living room
최종(0회 재작성)
[ ] pick up the tissue box — on the dining table in the living room
The robot is in the middle of grasping the tissue box on the dining table. Continuing the pick subgoal.
5
done
—
❌
—
The gripper successfully grasped and lifted the tissue box from the dining table. The goal of picking up the tissue box is satisfied.
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png턴 0 이동 전턴 0 이동 후
첫 턴 VLM 입력 전문 — 텍스트 7조각 + 이미지 1장
Your current view:
Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):
[context] [ 0s– 94s] I picked up remote control #1 from the dresser in the bedroom and placed it on the clothes dryer in the bathroom.
[context] [ 94s– 256s] I tried to move the egg from the countertop in the kitchen to the drawer of the fridge in the kitchen, but the attempt failed.
[context] [ 256s– 387s] I picked up remote control #1 from the clothes dryer in the bathroom and placed it on the dresser in the bedroom.
[context] [ 387s– 491s] I picked up the salt shaker from the countertop in the kitchen and placed it on the dining table in the living room.
[context] [ 491s– 572s] I picked up the tissue box from the tv stand in the living room and placed it on the dining table in the living room.
[context] [ 572s– 690s] I picked up remote control #2 from the tv stand in the living room and placed it on the dresser in the bedroom.
Object names used in this house:
apple, basketball, bed, book, bottle, bowl, box, bread, cell phone, chair, clothes dryer, countertop, credit card, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, kettle, knife, ladle, laptop, mug, newspaper, painting, pan, paper towel roll, pencil, pillow, plunger, remote control, salt shaker, sink, soap bottle, sofa, spatula, spoon, spray bottle, statue, teddy bear, television, tissue box, toilet, toilet paper, tomato, tv stand, vase, watch
Room names used in this house:
bedroom, bathroom, kitchen, living room
Your plan:
1. [ ] pick up the tissue box — on the dining table in the living room
Goal: Pick up the tissue box I moved earlier
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: after you have lifted an object, the robot sets it back down where it is before your next subgoal — that pick already counts, so pick each target once and move on to the next.
Memory — you do not write anything down. Any notes shown above were recorded during the context phase and will not change; apart from those, the frames of your previous subgoal and your current view are all you have. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{"note": null, "reasoning": "The tissue box was placed on the dining table in the living room. I need to navigate there first.", "next": {"skill": "navigate", "category": "tissue box", "surface": "dining table", "room": "living room"}}
```
restore — 샘플 10칸 중 4칸
성공 Pick up the spatula and place it in or on where it first was
v1_train_6728__2 · blind · single · 계획모드 fixed · 턴 4/50 · 스텝 108/5000 · $0.0282 · 방 이름 4개
이 질의의 답 1개는 시작부터 목적지 표면 위에 있다(기록만, 채점 규칙은 불변): spatula_291e31b285474249721c8a7b1bac877a_1_0_6
읽기 답(= 계획의 출처)
{
"reasoning": "The spatula is currently resting on the kitchen countertop, which is also where it was originally located.",
"selections": [
{
"category": "spatula",
"now": {
"room": "kitchen",
"surface": "countertop"
},
"original": {
"room": "kitchen",
"surface": "countertop"
}
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] put the spatula back — it is now on the countertop in the kitchen; return it to the countertop in the kitchen
최종(0회 재작성)
[ ] put the spatula back — it is now on the countertop in the kitchen; return it to the countertop in the kitchen
턴
스킬
대상
성공
스텝
VLM 이유
0
pick_and_place
spatula_291e31b285474249721c8a7b1bac877a_1_0_6
❌
—
The spatula is on the countertop in front of the robot. I will execute pick_and_place to place it on the countertop as required.
1
pick
spatula_291e31b285474249721c8a7b1bac877a_1_0_6
❌
—
The spatula is on the countertop in front of the robot. I will pick it up.
2
pick_and_place
spatula_291e31b285474249721c8a7b1bac877a_1_0_6
❌
—
Since the spatula is already being held, proceed to place it on the countertop in the kitchen. 재질의: you are still holding the spatula — you cannot pick another object; put it down first with pick_and_place(category, surface) at the destination
3
done
—
❌
—
The spatula has been picked up and placed back onto the countertop as required by the goal.
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png
첫 턴 VLM 입력 전문 — 텍스트 6조각 + 이미지 1장
Your current view:
Object names used in this house:
CD, apple, basketball, bed, book, bowl, box, bread, candle, cell phone, chair, coffee machine, countertop, credit card, cup, desk lamp, dining table, dish sponge, dresser, egg, faucet, floor lamp, fork, fridge, garbage bag, garbage can, house plant, microwave, newspaper, painting, pan, paper towel roll, pepper shaker, pillow, plunger, potato, remote control, sink, sofa, spatula, spoon, spray bottle, statue, stool, television, tennisracket, tissue box, toilet, toilet paper, tomato, tv stand, vase, watch
Room names used in this house:
bedroom, bathroom, kitchen, living room
Your plan:
1. [ ] put the spatula back — it is now on the countertop in the kitchen; return it to the countertop in the kitchen
Goal: Pick up the spatula and place it in or on where it first was
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: once you have lifted an object you keep holding it. navigate carries it with you and never places anything; put it down only with pick_and_place(category, surface) at the destination. While you hold an object you cannot pick another one.
Memory — you do not write anything down. Any notes shown above were recorded during the context phase and will not change; apart from those, the frames of your previous subgoal and your current view are all you have. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{"note": null, "reasoning": "The spatula is on the countertop in front of the robot. I will execute pick_and_place to place it on the countertop as required.", "next": {"skill": "pick_and_place", "category": "spatula", "surface": "countertop", "room": null}}
```
성공 Pick up the remote control and place it in or on where it was just before
v1_train_3679__2 · textlog · single · 계획모드 fixed · 턴 29/50 · 스텝 4602/5000 · $0.1752 · 방 이름 4개
읽기 답(= 계획의 출처)
{
"reasoning": "The remote control was most recently moved from the clothes dryer in the bathroom to the bed in the bedroom, so it is currently on the bed in the bedroom and was on the clothes dryer in the bathroom just before.",
"selections": [
{
"category": "remote control",
"now": {
"room": "bedroom",
"surface": "bed"
},
"original": {
"room": "bathroom",
"surface": "clothes dryer"
}
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] put the remote control back — it is now on the bed in the bedroom; return it to the clothes dryer in the bathroom
최종(0회 재작성)
[ ] put the remote control back — it is now on the bed in the bedroom; return it to the clothes dryer in the bathroom
The gripper has reached down to grasp the remote control on the bed, but the pick action is not complete (arm hasn't lifted it up yet). Continuing pick.
The remote control was not picked up and is still on the bed in the bedroom. I need to navigate back to the bed in the bedroom to pick it up. 재질의: there is no remote control within reach — navigate first
I am holding the remote control. I need to navigate to the clothes dryer in the bathroom to place it. 재질의: you are still holding the remote control — you cannot pick another object; put it down first with pick_and_place(category, surface) at the destination
With the remote control picked up, navigate to the clothes dryer in the bathroom to place it back. 재질의: you already picked up the remote control here — move on
28
done
—
❌
—
The remote control has been placed onto the clothes dryer in the bathroom as required. The goal is complete.
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png턴 0 이동 전턴 0 이동 후턴 15 이동 전턴 15 이동 후
첫 턴 VLM 입력 전문 — 텍스트 7조각 + 이미지 1장
Your current view:
Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):
[context] [ 0s– 108s] I picked up the remote control from the bed in the bedroom and placed it on the clothes dryer in the bathroom.
[context] [ 108s– 222s] I picked up the cell phone from the dining table in the bedroom and placed it on the bed in the bedroom.
[context] [ 222s– 365s] I picked up the remote control from the clothes dryer in the bathroom and placed it on the bed in the bedroom.
[context] [ 365s– 481s] I picked up the spatula from the countertop in the kitchen and placed it on the dining table in the living room.
Object names used in this house:
alarm clock, baseball bat, basketball, bed, book, bowl, box, candle, cell phone, chair, clothes dryer, countertop, desk lamp, dining table, dish sponge, dog bed, dresser, egg, faucet, fridge, garbage can, house plant, knife, mug, painting, pan, pen, pepper shaker, pillow, plate, plunger, remote control, safe, shelving unit, sink, soap bottle, spatula, spray bottle, statue, television, toilet, toilet paper, tomato, vase, watch
Room names used in this house:
bedroom, bathroom, kitchen, living room
Your plan:
1. [ ] put the remote control back — it is now on the bed in the bedroom; return it to the clothes dryer in the bathroom
Goal: Pick up the remote control and place it in or on where it was just before
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: once you have lifted an object you keep holding it. navigate carries it with you and never places anything; put it down only with pick_and_place(category, surface) at the destination. While you hold an object you cannot pick another one.
Memory — you do not write anything down. Any notes shown above were recorded during the context phase and will not change; apart from those, the frames of your previous subgoal and your current view are all you have. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{"note": null, "reasoning": "According to the plan, the remote control is currently on the bed in the bedroom. I need to navigate there to pick it up.", "next": {"skill": "navigate", "category": "remote control", "surface": "bed", "room": "bedroom"}}
```
실패 · done Pick up the mug and place it in or on where it first was
v1_train_188__2 · frames-uniform · single · 계획모드 fixed · 턴 2/50 · 스텝 158/5000 · $0.0320 · 방 이름 4개
읽기 답(= 계획의 출처)
{
"reasoning": "The mug is located on the desk in the bedroom, which is also where it was first seen.",
"selections": [
{
"category": "mug",
"now": {
"room": "bedroom",
"surface": "desk"
},
"original": {
"room": "bedroom",
"surface": "desk"
}
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] put the mug back — it is now on the desk in the bedroom; return it to the desk in the bedroom
최종(0회 재작성)
[ ] put the mug back — it is now on the desk in the bedroom; return it to the desk in the bedroom
Navigate to the dining table in the bedroom where the mug is located.
1
done
—
❌
—
The mug is already located on the desk in the bedroom where it was originally. The goal is fully satisfied.
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png턴 0 이동 전턴 0 이동 후
첫 턴 VLM 입력 전문 — 텍스트 7조각 + 이미지 9장
Frames sampled at even intervals over everything you have observed so far:
Your current view:
Object names used in this house:
apple, armchair, basketball, bed, book, bottle, bowl, bread, candle, cell phone, chair, countertop, credit card, cup, desk lamp, dining table, dish sponge, dog bed, dresser, egg, faucet, fridge, garbage can, house plant, key chain, knife, ladle, laptop, lettuce, microwave, mug, newspaper, painting, paper towel roll, pen, pencil, pillow, plate, plunger, potato, remote control, safe, salt shaker, side table, sink, soap bottle, sofa, spoon, spray bottle, statue, stool, teddy bear, television, tennisracket, tissue box, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle
Room names used in this house:
bedroom, bathroom, kitchen, living room
Your plan:
1. [ ] put the mug back — it is now on the desk in the bedroom; return it to the desk in the bedroom
Goal: Pick up the mug and place it in or on where it first was
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute.
Each turn you receive: your memory frames, the frames captured while your previous subgoal was executing, your current view, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: once you have lifted an object you keep holding it. navigate carries it with you and never places anything; put it down only with pick_and_place(category, surface) at the destination. While you hold an object you cannot pick another one.
Memory — the frames above are sampled automatically at even intervals over everything you have observed so far, and they are the ONLY record you will have of earlier turns. You do not choose them, and you keep no written notes. Always return "note": null.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
The plan stays as written — you cannot edit it or tick items off.
Output a single JSON object, nothing else:
{"note": "...", "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{"note": null, "reasoning": "Navigate to the dining table in the bedroom where the mug is located.", "next": {"skill": "navigate", "category": "mug", "surface": "dining table", "room": "bedroom"}}
```
실패 · done Pick up the spatula and place it in or on where it was just before
v1_train_8843__1 · rollforward-v3 · single · 계획모드 rewrite · 턴 4/50 · 스텝 247/5000 · $0.0310 · 방 이름 3개
읽기 답(= 계획의 출처)
{
"reasoning": "The spatula was moved from the desk in the living room to the kitchen countertop. To return it to where it was just before, it should be moved from the kitchen countertop back to the desk in the living room.",
"selections": [
{
"category": "spatula",
"now": {
"room": "kitchen",
"surface": "countertop"
},
"original": {
"room": "living room",
"surface": "desk"
}
}
],
"abstain": false
}
초기 계획 → 최종 계획
[ ] put the spatula back — it is now on the countertop in the kitchen; return it to the desk in the living room
최종(4회 재작성)
[x] navigate to spatula on countertop in kitchen
[x] pick spatula from countertop
[x] place spatula on countertop
턴
스킬
대상
성공
스텝
VLM 이유
0
navigate
spatula_291e31b285474249721c8a7b1bac877a_1_0_6
✅
94
The spatula is on the kitchen countertop. First, I need to navigate to the spatula on the kitchen countertop to reach and grasp it.
1
pick
spatula_291e31b285474249721c8a7b1bac877a_1_0_6
❌
—
We are positioned in front of the kitchen countertop with the spatula in reach. Next step is to pick the spatula.
2
pick_and_place
spatula_291e31b285474249721c8a7b1bac877a_1_0_6
❌
—
The simulator confirmed the spatula is held. The goal is to place it where it was just before (countertop). Placing is done via pick_and_place. 재질의: you are still holding the spatula — you cannot pick another object; put it down first with pick_and_place(category, surface) at the destination
3
done
—
❌
—
The spatula was placed back onto the countertop where it was initially located, completing the goal.
턴 0 관측 — turn_0_obs_0.png턴 0 관측 — turn_0_obs_1.png턴 0 이동 전턴 0 이동 후
첫 턴 VLM 입력 전문 — 텍스트 7조각 + 이미지 1장
Your current view:
Your notes so far:
Here are my running notes from the episode, written by me while acting (subgoal by subgoal):
[context] I picked up a credit card from the kitchen countertop (which held a soapdispenser, egg, paper towel, two loaves of bread, spatula, butter knife, irish potato, sponge, plate, spoon, and a pen) and placed it on the desk in the office room (which had a statue, key chain, spatula, and tissue box, flanked by two chairs; the room also contains a dresser with a bowl, and a sofa with fabric, cell phone, and remote control). I then moved the tissue box from the desk to the coffee table next to a basketball (which rolled onto the floor). Next, I took an irish potato from the kitchen countertop and dropped it on the floor by the refrigerator. Then, I carried the spatula from the desk to the kitchen countertop. Most recently, I picked up the spoon from the kitchen countertop and placed it on the desk next to the statue and key chain.
Object names used in this house:
alarm clock, apple, baseball bat, basketball, bed, book, bowl, box, bread, butter knife, candle, cell phone, chair, countertop, credit card, desk lamp, dining table, dish sponge, dresser, egg, fork, fridge, garbage bag, garbage can, house plant, key chain, painting, pan, paper towel roll, pen, pencil, pepper shaker, pillow, plate, potato, remote control, side table, soap bottle, sofa, spatula, spoon, statue, teddy bear, television, tennisracket, tissue box, tv stand, vase
Room names used in this house:
bedroom, kitchen, living room
Your plan:
1. [ ] put the spatula back — it is now on the countertop in the kitchen; return it to the desk in the living room
Goal: Pick up the spatula and place it in or on where it was just before
Reply with the JSON object only.
이 칸의 시스템 프롬프트 전문
You are the high-level controller of a household robot, and you are also its memory. You are given ONE goal. Each turn you decide the single next subgoal for the low-level policy to execute, and you update your own notes in the same reply.
Each turn you receive: the frames captured while your previous subgoal was executing, your current view, your notes so far, your plan, the goal, and the object and room names used in this house.
Each turn the low-level policy executes your subgoal for a fixed budget of 128 steps (about 4 action chunks); a pick usually needs several turns — if it is not finished yet, issue the same subgoal again; when you can see the object lifted in the gripper, move on.
Skills:
- navigate(room, surface): go to the named surface in the named room and stop in front of it (next to the object you named if it is on that surface). surface may be null = just enter the room; use "floor" for objects lying on the floor. The room is REQUIRED and taken literally — if you name the wrong room you end up in the wrong room, nothing corrects it. ALWAYS also fill category with the object you are ultimately going for. Manipulation only works within reach, so navigate before acting on an object.
- pick(category): grasp and lift an object within reach
- pick_and_place(category, surface): one atomic unit — you cannot split pick and place into separate subgoals
- open(category) / close(category): open or close an articulated object within reach (drawer, cabinet, fridge, door, ...)
- done: the goal is fully satisfied
- abort: impossible to proceed (explain in reasoning)
Rules:
- Refer to objects and rooms by the names in the lists below the images — copy the spelling exactly.
- The low-level policy fails often. Judge from the frames whether your previous subgoal actually worked, and retry it or move on accordingly. Nobody tells you whether it succeeded.
- Holding: once you have lifted an object you keep holding it. navigate carries it with you and never places anything; put it down only with pick_and_place(category, surface) at the destination. While you hold an object you cannot pick another one.
Notes — this is the ONLY record you will have of earlier turns; you will not see those frames again:
1. Do NOT claim a subgoal completed unless the last frame clearly shows it. If the frames only show the action in progress, or the final state is unclear, describe only what you actually see — never assume success from what you were trying to do.
2. Prefix anything you are not certain about with "uncertain: ".
3. Record what you attempted, what visibly changed, and what is still left to do.
4. Also record where you see goal-relevant objects (room and surface), even ones you did not act on.
5. Write your COMPLETE updated notes (at most 12 lines). You may compress, correct or drop earlier lines — what you return replaces your notes entirely.
Plan — before this task started, a plan for it was written from your memory of the earlier part of this episode; it is shown below the images. It can be wrong or incomplete: if what you see contradicts it, trust what you see.
You keep this plan up to date. In every reply, return the COMPLETE plan in "plan" as a list of lines, each starting with "[x] " (done) or "[ ] " (not done yet):
1. Mark an item [x] only when the frames clearly show it done — not because you issued the subgoal.
2. If what you see shows an item is wrong, correct it; if something is missing, add an item. Never delete an item that is not done.
Output a single JSON object, nothing else:
{"note": "...", "plan": ["[x] ...", "[ ] ..."], "reasoning": "...", "next": {"skill": "...", "category": "...", "surface": "...", "room": "..."}}
Use null for fields that do not apply.
첫 턴 VLM 원문 응답
```json
{
"note": "Starting task: pick up the spatula (currently on the kitchen countertop) and place it back on the table/desk where it was previously.",
"plan": [
"[ ] navigate to spatula on countertop in kitchen",
"[ ] pick spatula from countertop",
"[ ] navigate to dining table in living room",
"[ ] place spatula on dining table"
],
"reasoning": "The spatula is on the kitchen countertop. First, I need to navigate to the spatula on the kitchen countertop to reach and grasp it.",
"next": {
"skill": "navigate",
"category": "spatula",
"surface": "countertop",
"room": "kitchen"
}
}
```
6채점 규칙(v2)
주 지표는 SR@예산 — (a) 종료 시점 위치가 정답이고 (b) 모델이 done으로 끝냈고 (c) 스텝 예산 안이어야 성공.
보조로 success_nobudget(예산 무시), first_nav_hit(탐색 전에 말한 방·가구가 정답),
first_action_hit을 함께 기록한다. restore는 "목적지 표면이 그 물체를 지지하는가"로 판정한다.
자명 충족 답. restore 12질의(single 6·multi 6)는 현재 표면과 목적지 표면이 같아 t=0에 이미 충족이다
(물체는 실제로 1.1~3.5 m 떨어져 있지만 채점이 표면 지지만 본다). 채점 규칙은 바꾸지 않고 행에
trivial_answers로 기록만 한다 — 본 런 804칸 중 25칸이 해당하고, 분석에서 빼고 볼 수 있다.
7다음
샘플 20칸 완주 확인 → 본 런 제출(tmp/submit_plan_main.sh, 가족당 4샤드, own QOS) — 사용자 지시 후 제출.
결정됨(260916): 읽기 본표도 scene card 기준으로 재채점한다(API $0) — scripts/rescore_room_names.py, 8모델 × 2가족.
미결(파일럿부터): find 프롬프트 pick each target once 문구, VLM 진행 판단 오류 대응, VLA 교체·혼합, resume·routine 통합.