← 목록

온라인 1칸 — C: mug → countertop (rollforward-v3, 계획 재작성이 목적지 상실)

2026-09-17 · job 133822 (core-own, 14분 47초) · 본 런과 같은 설정을 셀 하나로
한 줄 요약 — pick 0/5 뒤 에이전트가 "들었다"고 착각했고, rollforward-v3 가 계획을 다시 쓰다 목적지(부엌 조리대)를 잃어 출발지(서랍장)에 놓기로 바꿨습니다. 머그가 이미 서랍장 위라 판정이 통과됐고 done — 실패.
어디를 보시면 되나
① 턴 1~5 pick 영상(exo·손목): 서랍장 위 머그를 5번 시도해 한 번도 못 잡습니다(그리퍼 전환 0·8·18·8·9).
② 턴 6 결정 상자: 초기 계획은 "머그는 침실 서랍장에 있다 → 부엌 조리대로 돌려놔라"였는데, 이 턴에 받은 계획은 1. [x] pick mug from dresser 2. [ ] place mug on dresser in bedroom — 조리대가 사라졌습니다. 각 턴의 "받은 프롬프트 전문"에서 계획 블록이 어떻게 변해 가는지 따라가 보세요.
③ 턴 6·7 pick_and_place(mug → dresser): 빈손인데 머그가 이미 서랍장 위라 엄격 판정(is_placed_on)이 참 → "성공". 턴 7은 0스텝 vacuous.
textlog(salt_v3)와의 대조 — textlog 는 기록을 못 해서 놓고도 과제를 다시 시작했고, rf-v3 는 기록을 다시 쓰다 목적지를 잊었습니다. 둘 다 메모리 전략의 특성이 그대로 드러난 실패입니다.

설정

셀restore · v1_train_2571__1 · rollforward-v3 · gemini-3.7-flash
질의Pick up the mug and place it in or on where it first was
GTmug_ac3a060b0c3963b34d55fe717f30ee52_2_0_6 → place_on
place_modevla (VLA 가 놓는다)
navigate_modedrive
VLApi05_mlspaces_finetune · molmobot
스텝 예산 / 사용5000 / 970
턴 상한 / 사용50 / 9
비용$0.0487 (VLM 호출 9회)

초기 계획 (읽기 답 → 계획)

읽기 트랙 채점: place_set=1 (방·가구 모두 정답). 즉 기억은 맞았고 실행만 남은 칸입니다.
전략 textlog 는 계획 갱신 수단이 없어 plan_mode=fixed — 끝까지 이 계획 그대로입니다(재작성 0회).

결과

최종 판정실패
GT 물체를 들었나아니오
GT 수용체 위에 놓였나아니오
에피소드 종료 시 손에 든 것None
첫 navigate 가 GT 적중예
목표 지점까지 남은 거리11.253 m (xy)
종료 사유done

턴 전수

턴 0 navigate도착  경로 8.914180855723707 m · 393 스텝
에이전트의 이유
The mug is currently on the dresser in the bedroom and needs to be returned to the countertop in the kitchen. First navigate to the bedroom dresser to pick the mug.
skillnavigate
category / surface / roomNone / None / bedroom
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idNone
현재 방room_5
들고 있는 물체None
정책 프롬프트—
VLM 원문 응답
```json
{
  "note": "Starting goal: return mug from bedroom dresser to kitchen countertop where it was initially.",
  "plan": [
    "[ ] pick mug from dresser in bedroom",
    "[ ] place mug on countertop in kitchen"
  ],
  "reasoning": "The mug is currently on the dresser in the bedroom and needs to be returned to the countertop in the kitchen. First navigate to the bedroom dresser to pick the mug.",
  "next": {
    "skill": "navigate",
    "category": "mug",
    "surface": "dresser",
    "room": "bedroom"
  }
}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\n[context] I started at the dining table with a statue, knife, butter knife, pencil, spoon, pepper shaker, and bowl, with a soapdispenser and an irish potato on nearby chairs, and a safe on the floor. I moved the knife to the kitchen countertop (which held a wine bottle, butter knife, statue, atomizer, mug, house plant, toaster, and vase). Next, I moved the mug to the living room desk (alongside a table lamp, cup, and bowl, near a sofa, armchair, an open box on the floor, and a red shelf with a house plant and table lamp). After checking a credit card in the bedroom dresser drawer, I retrieved the mug from the living room desk and placed it on the bedroom dresser (which also had a cup and pencil). Then, I went to the bedroom desk (which had a laptop, statue, alarm clock, watch, and cell phone, near a bed with a pillow, CD, and bottle), picked up the cell phone, carried it to the bathroom (next to a toilet with a cup, bottle, and candle), placed it on the clothes dryer, and then picked the cell phone back up from the clothes dryer.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [ ] put the mug back — it is now on the dresser in the bedroom; return it to the countertop in the kitchen', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 1}
관측 0
관측 1
이동 전
이동(A* 주행)
이동 후
턴 1 pick실패  51 스텝 · 그리퍼 전환 0
에이전트의 이유
We have reached the dresser in the bedroom. The mug is located on top of the dresser within reach. Now we pick the mug.
skillpick
category / surface / roommug / None / None
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idNone
현재 방room_4
들고 있는 물체None
정책 프롬프트Pick up the mug
VLM 원문 응답
```json
{
  "note": "Navigated to bedroom dresser. Mug is visible on top of the dresser.",
  "plan": [
    "[ ] pick mug from dresser in bedroom",
    "[ ] place mug on countertop in kitchen"
  ],
  "reasoning": "We have reached the dresser in the bedroom. The mug is located on top of the dresser within reach. Now we pick the mug.",
  "next": {
    "skill": "pick",
    "category": "mug",
    "surface": "dresser",
    "room": "bedroom"
  }
}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nStarting goal: return mug from bedroom dresser to kitchen countertop where it was initially.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [ ] pick mug from dresser in bedroom\n2. [ ] place mug on countertop in kitchen', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 4}
관측 0
관측 1
정책 실행(exo)
정책 실행(손목)
턴 2 pick실패  128 스텝 · 그리퍼 전환 8
에이전트의 이유
The gripper attempted to pick the mug on the dresser but did not grasp it. I need to retry picking the mug.
skillpick
category / surface / roommug / None / None
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idNone
현재 방room_4
들고 있는 물체None
정책 프롬프트Pick up the mug
VLM 원문 응답
```json
{
  "note": "At dresser in bedroom. Attempted pick mug, but mug was not successfully grasped. Mug remains on dresser.",
  "plan": [
    "[ ] pick mug from dresser in bedroom",
    "[ ] place mug on dresser in bedroom"
  ],
  "reasoning": "The gripper attempted to pick the mug on the dresser but did not grasp it. I need to retry picking the mug.",
  "next": {
    "skill": "pick",
    "category": "mug",
    "surface": "dresser",
    "room": "bedroom"
  }
}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nNavigated to bedroom dresser. Mug is visible on top of the dresser.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [ ] pick mug from dresser in bedroom\n2. [ ] place mug on countertop in kitchen', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 4}
관측 0
관측 1
정책 실행(exo)
정책 실행(손목)
턴 3 pick실패  128 스텝 · 그리퍼 전환 18
에이전트의 이유
The mug is still on the dresser and has not been lifted into the gripper yet. As picks often require multiple turns, issuing pick(mug) again.
skillpick
category / surface / roommug / None / None
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idNone
현재 방room_4
들고 있는 물체None
정책 프롬프트Pick up the mug
VLM 원문 응답
```json
{
  "note": "At dresser in bedroom. Attempted pick mug; mug is still on dresser on top of book. Retrying pick mug.",
  "plan": [
    "[ ] pick mug from dresser in bedroom",
    "[ ] place mug on dresser in bedroom"
  ],
  "reasoning": "The mug is still on the dresser and has not been lifted into the gripper yet. As picks often require multiple turns, issuing pick(mug) again.",
  "next": {
    "skill": "pick",
    "category": "mug",
    "surface": null,
    "room": null
  }
}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nAt dresser in bedroom. Attempted pick mug, but mug was not successfully grasped. Mug remains on dresser.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [ ] pick mug from dresser in bedroom\n2. [ ] place mug on dresser in bedroom', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 4}
관측 0
관측 1
정책 실행(exo)
정책 실행(손목)
턴 4 pick실패  128 스텝 · 그리퍼 전환 8
에이전트의 이유
The mug is still on the dresser and not yet picked up. Low-level pick policy needs another turn to complete the grasp.
skillpick
category / surface / roommug / None / None
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idNone
현재 방room_4
들고 있는 물체None
정책 프롬프트Pick up the mug
VLM 원문 응답
```json
{
  "note": "At dresser in bedroom. Mug is on dresser. Attempted pick mug; mug not yet grasped. Retrying pick mug.",
  "plan": [
    "[ ] pick mug from dresser in bedroom",
    "[ ] place mug on dresser in bedroom"
  ],
  "reasoning": "The mug is still on the dresser and not yet picked up. Low-level pick policy needs another turn to complete the grasp.",
  "next": {
    "skill": "pick",
    "category": "mug",
    "surface": null,
    "room": null
  }
}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nAt dresser in bedroom. Attempted pick mug; mug is still on dresser on top of book. Retrying pick mug.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [ ] pick mug from dresser in bedroom\n2. [ ] place mug on dresser in bedroom', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 4}
관측 0
관측 1
정책 실행(exo)
정책 실행(손목)
턴 5 pick실패  128 스텝 · 그리퍼 전환 9
에이전트의 이유
The arm has moved towards the mug, but the mug has not yet been lifted into the gripper. Retrying pick(mug).
skillpick
category / surface / roommug / None / None
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idNone
현재 방room_4
들고 있는 물체None
정책 프롬프트Pick up the mug
VLM 원문 응답
```json
{
  "note": "At dresser in bedroom. Mug is on dresser. Attempted pick mug; arm positioned over mug but not lifted yet. Retrying pick mug.",
  "plan": [
    "[ ] pick mug from dresser in bedroom",
    "[ ] place mug on dresser in bedroom"
  ],
  "reasoning": "The arm has moved towards the mug, but the mug has not yet been lifted into the gripper. Retrying pick(mug).",
  "next": {
    "skill": "pick",
    "category": "mug",
    "surface": null,
    "room": null
  }
}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nAt dresser in bedroom. Mug is on dresser. Attempted pick mug; mug not yet grasped. Retrying pick mug.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [ ] pick mug from dresser in bedroom\n2. [ ] place mug on dresser in bedroom', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 4}
관측 0
관측 1
정책 실행(exo)
정책 실행(손목)
턴 6 pick_and_place성공  14 스텝 · 그리퍼 전환 1
에이전트의 이유
The mug is now lifted and held securely in the gripper. I need to place it back on the dresser to satisfy the goal.
skillpick_and_place
category / surface / roommug / None / None
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idchestofdrawers_0f1893ca9306d95cbdc6459a01beb2e0_1_0_4
현재 방room_4
들고 있는 물체None
정책 프롬프트Pick up the mug and place it in or on the dresser
VLM 원문 응답
```json
{"note": "At dresser in bedroom. Successfully picked and lifted mug.", "plan": ["[x] pick mug from dresser in bedroom", "[ ] place mug on dresser in bedroom"], "reasoning": "The mug is now lifted and held securely in the gripper. I need to place it back on the dresser to satisfy the goal.", "next": {"skill": "pick_and_place", "category": "mug", "surface": "dresser", "room": null}}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nAt dresser in bedroom. Mug is on dresser. Attempted pick mug; arm positioned over mug but not lifted yet. Retrying pick mug.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [ ] pick mug from dresser in bedroom\n2. [ ] place mug on dresser in bedroom', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 4}
관측 0
관측 1
정책 실행(exo)
정책 실행(손목)
턴 7 pick_and_place성공  0 스텝 · 그리퍼 전환 0 · vacuous(스텝 0)
에이전트의 이유
The mug is held in the gripper. The goal is to return it to where it first was (the dresser in bedroom). I will execute pick_and_place to place the mug on the dresser.
skillpick_and_place
category / surface / roommug / None / None
접지된 object_idmug_ac3a060b0c3963b34d55fe717f30ee52_1_0_4
접지된 receptacle_idchestofdrawers_0f1893ca9306d95cbdc6459a01beb2e0_1_0_4
현재 방room_4
들고 있는 물체None
정책 프롬프트Pick up the mug and place it in or on the dresser
VLM 원문 응답
```json
{
  "note": "At dresser in bedroom. Mug is grasped/held. Now placing mug on dresser.",
  "plan": [
    "[x] pick mug from dresser in bedroom",
    "[ ] place mug on dresser in bedroom"
  ],
  "reasoning": "The mug is held in the gripper. The goal is to return it to where it first was (the dresser in bedroom). I will execute pick_and_place to place the mug on the dresser.",
  "next": {
    "skill": "pick_and_place",
    "category": "mug",
    "surface": "dresser",
    "room": "bedroom"
  }
}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nAt dresser in bedroom. Successfully picked and lifted mug.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [x] pick mug from dresser in bedroom\n2. [ ] place mug on dresser in bedroom', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 4}
관측 0
관측 1
턴 8 donedone
에이전트의 이유
The mug has been successfully placed back onto the dresser where it started.
skilldone
category / surface / roomNone / None / None
접지된 object_idNone
접지된 receptacle_idNone
현재 방room_4
들고 있는 물체None
정책 프롬프트—
VLM 원문 응답
```json
{"note": "Mug picked up and placed back onto the dresser in the bedroom. Task complete.", "plan": ["[x] pick mug from dresser in bedroom", "[x] place mug on dresser in bedroom"], "reasoning": "The mug has been successfully placed back onto the dresser where it started.", "next": {"skill": "done", "category": null, "surface": null, "room": null}}
```
이 턴에 VLM 이 받은 프롬프트 전문
{'texts': ['Frames captured while your previous subgoal was executing:', 'Your current view:', 'Your notes so far:\nHere are my running notes from the episode, written by me while acting (subgoal by subgoal):\n\nAt dresser in bedroom. Mug is grasped/held. Now placing mug on dresser.', 'Object names used in this house:\nalarm clock, apple, armchair, bed, book, bowl, box, butter knife, cell phone, chair, clothes dryer, countertop, cup, desk lamp, dining table, dresser, egg, faucet, fridge, garbage bag, garbage can, house plant, knife, laptop, mug, painting, pen, pencil, pepper shaker, pillow, plunger, pot, potato, remote control, safe, salt shaker, sink, soap bottle, sofa, spoon, spray bottle, statue, television, toaster, toilet, toilet paper, tv stand, vase, watch, wine bottle', 'Room names used in this house:\nbedroom, bathroom, kitchen, living room', 'Your plan:\n1. [x] pick mug from dresser in bedroom\n2. [ ] place mug on dresser in bedroom', 'Goal: Pick up the mug and place it in or on where it first was', 'Reply with the JSON object only.'], 'n_images': 2}
관측 0
관측 1

에이전트가 턴마다 본 것(모아보기)

턴별 관측 프레임 (1 fps)