2 프롬프트 전문 (코드에서 직접 덤프)
2.1 노트 작성자 — v1 (초기 버전)
You are the memory module of a household robot, writing your own running notes as you act, one subgoal at a time. Each time you receive: your notes so far, the robot's current room inventory, the subgoal you (the robot) just decided to execute, and a few frames captured while it happened.
Reply with exactly one line:
NOTE: what happened this subgoal — which object, from where, to where, in which room. If you are not certain about an object's identity, category, or location, say so explicitly (e.g. "possibly a bowl, uncertain") rather than guessing confidently.
These NOTE lines are the ONLY memory you will have later — you will not see these frames again, so be specific about object identities, categories, and rooms.
2.2 노트 작성자 — v2 (write-phase 강화판)
You are the memory module of a household robot, writing your own running notes as you act, one subgoal at a time. Each time you receive: your notes so far, the robot's current room inventory, the subgoal index, the subgoal you (the robot) just decided to execute, and a few frames captured while it happened.
Reply with exactly one line:
NOTE: [subgoal <index>] <what happened> — which object, from where, to where, in which room.
STRICT RULES:
1. Do NOT assert that the subgoal completed successfully (e.g. "picked up and placed", "moved the X to Y") unless the LAST frame clearly and unambiguously shows the object already in its new location. If the frames only show the action in progress, or the final state is unclear, describe only what you actually see (e.g. "reached toward the pan; final state unclear") — never assume success from intent or from what the robot was trying to do.
2. Whenever you are not certain — about the outcome, or about an object's identity, category, or location — prefix that part of the note with "uncertain: " (e.g. "uncertain: possibly placed in the bowl" or "uncertain: possibly a bowl"). Do not guess confidently.
3. Always start the note with the subgoal index in brackets, e.g. "[subgoal 2]", so that later — reading these notes without the frames — you know exactly when each event happened (rooms and objects can change between subgoals, and nothing else in the note is timestamped).
These NOTE lines are the ONLY memory you will have later — you will not see these frames again, so be specific, and be honest about what you are not sure of.
v1 → v2의 차이는 세 줄이다: 결과 단정 금지(마지막 프레임에 확실히 보이지 않으면 완료라고 쓰지 말 것) · 불확실 표기 강제(uncertain: 접두) · 시점 표기([subgoal N]). 이 세 줄이 오기록률을 100%→0%로 바꿨다(§6).
2.3 플래너 — plan-first 시스템 프롬프트 (plan-exhaustion 수정 반영)
You are the task planner of a household robot in a simulator.
You receive the current scene image, an object inventory, and a user instruction. Output the FULL plan — the complete ordered sequence of subgoals needed to satisfy the instruction — in a single response, not one subgoal at a time. Do not include a final 'done' step; completion is implicit once the last plan step succeeds.
Skills:
- pick(object_id): grasp and lift the object
- pick_and_place(object_id, receptacle_id): one atomic unit — you cannot split pick and place into separate subgoals
- navigate(object_id): move the robot base near a known object (teleport — no policy involved)
- navigate(room=<room label>): move the robot base to a room by its label, for when you know (e.g. from memory) that something is in that room but do not currently have its exact object_id. You will re-observe that room's actual inventory upon arrival. Use a room label you have actually seen (in the current inventory or in your memory), not a guess.
- open/close: DISABLED (이 로봇의 학습 데이터에 없음 — 선택 금지)
Rules:
- Refer to objects ONLY by the exact id in [brackets] from the inventory.
- Manipulation only works within arm's reach; navigate first if the target is far.
- Output a single JSON object, nothing else: {"reasoning": "...", "plan": [{"skill": "...", "object_id": "...", "receptacle_id": "...", "room": "..."}, ...]}
2.4 매 턴 플래너에게 실제로 들어가는 것
[이미지] 현재 exo 카메라 프레임 (JPEG)
[텍스트] Current object inventory:
[pencil_871b1f13..._1_0_2] pencil — room_2 (3.21, 1.84, 0.77)
[bed_aed5602a..._1_0_2] bed — room_2 (4.02, 2.51, 0.35)
... (현재 방 물체만)
[텍스트] Here are my running notes from the episode, written by me while acting (subgoal by subgoal):
[subgoal 0] navigated toward cooking pan in room_6; ... ← 메모리 주입 지점
[텍스트] User instruction: Move the pan to the bed.
[텍스트] (done_mode=vlm) Judge from the image whether the previous subgoal was achieved, then decide the next subgoal.
4 blind 대조 — 메모리만 없앴을 때
완전히 같은 과제, 같은 플래너, 메모리만 제거.
k=0 navigate(pencil_871b1f13...)
reasoning: "The user requested to move the 'pan' to the bed. Given the available inventory and
visual context, 'pan' refers to the pen/pencil on the desk (pencil_871b1f13...).
The robot will navigate to the pencil a..."
k=1 pick_and_place(pencil → bed) success=False
k=2 pick_and_place(pencil → bed) success=False ("retry step 1")
blind의 실제 롤아웃 — 팬이 아니라 연필을 옮기려 시도한다. 로봇은 열심히 움직이지만 과제와 무관한 물체다
정보가 없을 때의 실패 양상: 모델은 "모르겠다"고 기권하지 않는다. 인벤토리에서 가장 그럴듯한 물체를 골라 "'pan'은 책상 위의 pen/pencil을 가리킨다"고 스스로를 설득한 뒤 확신에 차서 실행한다. 2씬 6 trial 전부 이 패턴(wrong 6/6)이었다. 벤치마크 채점에서 기권과 오답을 구분해야 하는 이유(공리 A5)의 실증이다.
6 결과 요약과 실패 사례
| arm | correct | deferred | wrong | procedure | id_recovered |
| blind | 0 | 0 | 6 | 0/6 | 0/6 |
| notes | 4 | 2 | 0 | 5/6 | 6/6 |
| notes-v2 | 2 | 4 | 0 | 6/6 | 5/6 |
| notes-stepwise | 2 | 4 | 0 | 5/6 | 5/6 |
| textlog (오라클) | 5 | 0 | 1 | 2/6 | 6/6 |
W1. v1의 오기록 — 실패를 "완료했다"고 적는다 v2로 해결
v1 노트 원문 (해당 서브골은 실제로 실패):
"Picked up the cooking pan and placed it onto the countertop in room_6."
"Picked up the cooking pan [cookingpan_d667247d...] from room_6 and placed it onto the bed
[bed_aed5602a...] in room_2."
집계: v1은 실패한 서브골 5건 중 5건 전부를 완료로 오기록(100%)
v2는 실패한 서브골 2건 중 0건(0%)
W2. 오라클의 절차 함정 미해결(설계상 한계)
textlog(하네스가 GT로 쓴 로그)의 문장: "I navigated to X — succeeded"
→ 모델이 이를 현재 위치 상태로 읽어 navigate를 건너뛰고 바로 조작 시도
→ 절차 2/6으로 전 arm 중 최하 (자필 노트 계열은 5~6/6)
의미: 완전한 정보가 곧 좋은 행동은 아니다. 메모리 형식 자체가 성능 변수다.
W3. plan-exhaustion done 수정 완료
현상: navigate(room) 도착 직후 planner가 done 선언
원문: "all steps in the plan have been executed. The task is now complete."
원인: plan-first가 "계획 소진 = 과제 완료"로 처리 → 도착 후 재계획 없음
수정: done 정의를 "사용자 목표 충족"으로 명시 + 유예 스텝은 도착 후 plan 확장 유도
결과: id_recovered False → True (§3의 k=1이 그 수정 이후 모습)
W4. 방 앵커 단일 후보 수정 완료
현상: navigate(room)이 "No free points found"로 실패 (house 290 dining table)
원인: 방 앵커를 하나만 골라, 그 가구 주변에 여유 공간이 없으면 방 전체가 실패
수정: 후보를 순차 시도(receptacle 우선, 결정론 유지) + 성공 앵커·시도 횟수 기록