2026-09-22 · scene_bench · gemini-3.7-flash · resume 161행
resume 실패의 세 갈래 — 영상·프레임·메모리 전문
rf-v3 누적 절제가 왜 전체로는 귀무였는지, 유형별로 뜯어본다
TL;DR
- 한 가지 원인이 아니다. 누름 / 담기·옮기기 / visit_rooms 가 각각 다른 이유로 틀린다.
- 누름 — 메모리에 press 라는 말이 없다. writer 가 세탁기를
black safe·wooden dresser로 부르고 동작을reached toward its top으로 적는다. 시각 접지 문제이지 연산자 문제가 아니다. - 담기 — rf-v3 는 감자 3개를 집은 건 적었는데 목적지를
on the floor behind the hamper로 적어 "아무것도 안 함"이 됐다. 누적은dropped … into the bin을 남겨 맞혔다(+14.8pp). - 옮기기 — 누적이 끝난 심부름을 진행 중처럼 남겨 미완으로 오보고(−29.4pp). 두 효과가 상쇄돼 전체가 귀무가 됐다.
- visit_rooms 는 하네스 결함 — 총 방 수를 안 줘서 오라클조차 .419, 오답 18행 중 17행이 "알 수 없다"며 기권.
0/49
press 라 적은 행
+14.8pp
담기(누적)
−29.4pp
옮기기(누적)
.419
visit 오라클
1 유형별 지도
| 단일답 유형 | n | rf-v3 → append | 메모리가 사건을 언급 | 단위당 샘플 프레임 | 오라클 |
|---|---|---|---|---|---|
| press_button | 25 | .040 → .040 | .16 → .20 | 중앙 2 | 1.000 |
| put_in_container | 27 | .296 → .444 | .81 → .93 | 중앙 11 | 1.000 |
| move_objects | 17 | .588 → .294 | .94 → 1.00 | 중앙 11 | 1.000 |
| visit_rooms | 31 | .097 → .065 | .61 → .48 | 중앙 5 | .419 |
샘플링 밀도는 원인이 아니다 (확인됨). 누름 단위만 샘플 프레임이 중앙 2장으로 적어
밀도를 의심했지만, 260919 관측밀도 30 s→10 s ablation(find, 32B·Gemma4)에서 밀도를 3배로 올려도 접지 점수는 0~+2.8pp 였다.
resume 에서도 팔 접기 재생성(누름마다 흔적이 남게 한 판)으로 rf-v3 누름 행은 .000 → .050(+1/−0)에 그쳤다.
그때 내린 결론 — "병목은 관측 기회가 아니라 writer 의 시각 접지 능력" — 이 아래 사례에서 그대로 확인된다.
2 사례 · 누름 — writer 가 사건 자체를 못 알아본다
resume:train_1843__2000:cut_06 · 컷 frame 4228 · 컷까지 청크 5개
GT: 총 5회 중 3회 완료 → 남은 2
들은 명령
- [t=0s] “Press the button on the washing machine five times.”
- [t=24s] “Hold on — open a drawer of the dresser in the bedroom first.”
- [t=68s] “Okay, continue what you were doing.”
- [t=93s] “Hold on — move the soap bottle to the other end of the sink first.”
실제로 완료된 단위
- 단위 0: frame 0–404 (16 s) — Press the button on the washing machine
- 단위 1: frame 405–601 (8 s) — Press the button on the washing machine
- 단위 2: frame 1702–2334 (25 s) — Press the button on the washing machine
GT 3회 눌렀는데 rf-v3 는 0회(남은 5), append 는 1회(남은 4)로 읽었다. 메모리 어디에도 'press' 가 없다 — 세탁기를 “black safe”·“wooden dresser” 로 부르고 동작은 “reached toward its top” 으로 적었다. 연산자 문제가 아니라 시각 접지 문제다.
영상 — 실제로 일어난 일
단위 0 · 16.2 s · frame 0–404
단위 1 · 7.8 s · frame 405–601
단위 2 · 25.3 s · frame 1702–2334
writer 가 본 프레임 + 그 청크에 쓴 문장
chunk 00 (t=0–30s) — writer 가 받은 4장. 이 청크에 걸친 단위: [0, 1]
writer 가 이 청크에 쓴 것 → I started at a wooden desk where a plunger was visible in the corner. I moved my arm toward a black safe next to the desk, reaching toward its top before turning back.
chunk 02 (t=60–90s) — writer 가 받은 4장. 이 청크에 걸친 단위: [2]
writer 가 이 청크에 쓴 것 → I turned away from the dresser, looking across the room where a dining table with several red cups and a wastebasket stood. I moved over to a black washing machine and reached my gripper toward its upper front edge.
chunk 03 (t=90–120s) — writer 가 받은 4장. 이 청크에 걸친 단위: [2]
writer 가 이 청크에 쓴 것 → I touched the front of the washing machine, then turned and navigated into the bathroom area. I saw a toilet with a roll of toilet tissue, a sink with two faucets and a soapdispenser, and a black bin. I approached the sink and reached my gripper toward the soapdispenser.
메모리 전문
rf-v3 (재작성) — 컷 시점 요약 78 단어
I started at a desk with a plunger and a black safe, reaching toward the safe before leaving through a doorway into another room. I approached a wooden dresser and reached toward its drawers. Turning away, I passed a dining table with cups and reached the upper front of a black washing machine. I then turned around, moved into a bathroom with a toilet, toilet tissue, and a pedestal sink, and picked up a soapdispenser from the sink.
append (누적) — 5 항목 / 177 단어
- [0] I started at a wooden desk where a plunger was visible in the corner. I moved my arm toward a black safe next to the desk, reaching toward its top before turning back.
- [1] I navigated through the open doorway toward a wooden dresser, seeing a white bench with a laptop in the background. I reached my arm toward the lower drawer of the dresser.
- [2] I turned away from the dresser, looking across the room where a dining table with several red cups and a wastebasket stood. I moved over to a black washing machine and reached my gripper toward its upper front edge.
- [3] I touched the front of the washing machine, then turned and navigated into the bathroom area. I saw a toilet with a roll of toilet tissue, a sink with two faucets and a soapdispenser, and a black bin. I approached the sink and reached my gripper toward the soapdispenser.
- [4] I grasped and picked up the soapdispenser from the sink, turned briefly toward the doorway, and then faced back toward the sink while holding it.
세 답
textlog (오라클) resume_ok=1
Press the button on the washing machine five times.→ 남은 2
The user originally commanded the robot to press the washing machine button 5 times. The robot performed 2 presses, completed an interruption to open a dresser drawer, resumed and completed 1 more press (total 3), and then finished a second interruption to move the soap bottle. Thus, 2 button presses remain.
rf-v3 재작성 resume_ok=0
Press the button on the washing machine five times.→ 남은 5Hold on — move the soap bottle to the other end of the sink first.→ 남은 1
The command to press the washing machine button five times was not completed, leaving all 5 presses remaining. The command to move the soap bottle to the other end of the sink was interrupted mid-action after picking up the bottle, leaving 1 move action remaining.
append 누적 resume_ok=0
Press the button on the washing machine five times.→ 남은 4Hold on — move the soap bottle to the other end of the sink first.→ 남은 1
The robot was instructed to press the washing machine button five times. It pressed the button once before being interrupted to move the soap dispenser to the other end of the sink. The robot has picked up the soap dispenser but has not placed it down yet. Therefore, 4 presses remain on the washing machine and 1 object move remains for the soap bottle.
3 사례 · 담기 — 재작성이 목적지를 바꿔 적는다
resume:train_2739__2000:cut_04 · 컷 frame 9188 · 컷까지 청크 12개
GT: 총 4회 중 3회 완료 → 남은 1
들은 명령
- [t=0s] “Pick up four of the potatos from the dining table and place them into the basket.”
- [t=189s] “Hold on — open the fridge door in the kitchen first.”
실제로 완료된 단위
- 단위 0: frame 0–1712 (68 s) — Pick up the potato from the dining table and place it into the basket
- 단위 1: frame 1713–3116 (56 s) — Pick up the potato from the dining table and place it into the basket
- 단위 2: frame 3117–4717 (64 s) — Pick up the potato from the dining table and place it into the basket
rf-v3 는 감자 3개를 집은 것까지 적었지만 목적지를 “on the floor behind the hamper” 로 적었다 — 바구니에 넣은 게 아니게 되어 남은 수를 4(=아무것도 안 함)로 답했다. append 는 클립마다 “dropped … into the bin” 을 남겨 1(정답).
영상 — 실제로 일어난 일
단위 0 · 68.5 s · frame 0–1712
단위 1 · 56.1 s · frame 1713–3116
단위 2 · 64.0 s · frame 3117–4717
writer 가 본 프레임 + 그 청크에 쓴 문장
chunk 00 (t=0–30s) — writer 가 받은 4장. 이 청크에 걸친 단위: [0]
writer 가 이 청크에 쓴 것 → I started in the living room next to a sofa. I turned and approached a green desk where I saw five irish potatoes, a statue, a bowl, and a house plant. I reached out toward one of the irish potatoes on the desk.
chunk 01 (t=30–60s) — writer 가 받은 4장. 이 청크에 걸친 단위: [0]
writer 가 이 청크에 쓴 것 → I picked up an irish potato from the desk, turned around, and carried it towards a cylindrical bin on the floor next to the desk.
chunk 02 (t=60–90s) — writer 가 받은 4장. 이 청크에 걸친 단위: [0, 1]
writer 가 이 청크에 쓴 것 → I dropped the irish potato into the bin on the floor. I then turned back toward the desk and reached out to grasp a second irish potato among the four remaining on the desk.
chunk 03 (t=90–120s) — writer 가 받은 4장. 이 청크에 걸친 단위: [1]
writer 가 이 청크에 쓴 것 → I picked up the second irish potato from the desk, swung my arm around, and dropped it into the bin on the floor, leaving three irish potatoes on the desk.
chunk 04 (t=120–150s) — writer 가 받은 4장. 이 청크에 걸친 단위: [1, 2]
writer 가 이 청크에 쓴 것 → I turned back to face the green desk and reached toward a third irish potato among the three remaining on the desk.
chunk 05 (t=150–180s) — writer 가 받은 4장. 이 청크에 걸친 단위: [2]
writer 가 이 청크에 쓴 것 → I picked up the third irish potato from the desk, swung my arm around toward the bin, and hovered over it to drop the potato inside.
chunk 06 (t=180–210s) — writer 가 받은 4장. 이 청크에 걸친 단위: [2]
writer 가 이 청크에 쓴 것 → I dropped the third irish potato into the bin. I then turned away from the desk toward the living room, where I saw a cart by the window, a tv stand with a television, and a newspaper on the sofa.
메모리 전문
rf-v3 (재작성) — 컷 시점 요약 128 단어
I started in the living room near a red sofa. In front of me was a green desk holding five irish potatoes, a statue, a house plant, and a bowl, with an open hamper beside it. I picked up three irish potatoes one by one from the desk and placed each on the floor behind the hamper. I turned around, seeing a tv stand with a television, a cart by the window, and a newspaper on the red sofa. I then navigated past a dining table with chairs (holding two apples, a pot, and a plate) to an open refrigerator. I explored its empty interior shelves and door storage, manipulated the refrigerator door, looked toward the adjacent room through the doorway, and reached into the refrigerator shelves again.
append (누적) — 12 항목 / 339 단어
- [0] I started in the living room next to a sofa. I turned and approached a green desk where I saw five irish potatoes, a statue, a bowl, and a house plant. I reached out toward one of the irish potatoes on the desk.
- [1] I picked up an irish potato from the desk, turned around, and carried it towards a cylindrical bin on the floor next to the desk.
- [2] I dropped the irish potato into the bin on the floor. I then turned back toward the desk and reached out to grasp a second irish potato among the four remaining on the desk.
- [3] I picked up the second irish potato from the desk, swung my arm around, and dropped it into the bin on the floor, leaving three irish potatoes on the desk.
- [4] I turned back to face the green desk and reached toward a third irish potato among the three remaining on the desk.
- [5] I picked up the third irish potato from the desk, swung my arm around toward the bin, and hovered over it to drop the potato inside.
- [6] I dropped the third irish potato into the bin. I then turned away from the desk toward the living room, where I saw a cart by the window, a tv stand with a television, and a newspaper on the sofa.
- [7] I navigated out of the living room, passed by the dining room containing a dining table with chairs, went past a wall with a framed picture, and arrived in front of an open refrigerator.
- [8] I reached my arm into the shelves of the open refrigerator, glanced back across the dining room toward the kitchen, and then reached back toward the refrigerator shelves.
- [9] I reached further into the open refrigerator, moving my robotic arm across the shelves and down toward the lower compartment.
- [10] I reached for the open refrigerator door and pushed it shut, then retracted my arm.
- [11] I reached my robotic arm back into the open refrigerator, moving across the upper shelf and then down toward the middle shelf.
세 답
textlog (오라클) resume_ok=1
Pick up four of the potatos from the dining table and place them into the basket.→ 남은 1
The user requested four potatoes to be moved to the basket, but only three were completed before opening the fridge door as requested. Thus, one potato remains to be placed in the basket.
rf-v3 재작성 resume_ok=0
Pick up four of the potatos from the dining table and place them into the basket.→ 남은 4
The fridge door was opened as requested. The task to pick up four potatoes and place them into the basket has not been completed, leaving 4 potatoes remaining.
append 누적 resume_ok=1
Pick up four of the potatos from the dining table and place them into the basket.→ 남은 1
The user asked to pick up four potatoes and place them into the basket. The robot moved three potatoes into the bin before being interrupted, leaving 1 potato remaining to be moved. The interruption command to open the fridge door is complete (the fridge is open).
4 사례 · 옮기기 — 누적이 끝난 심부름을 진행 중처럼 남긴다
resume:train_3098__2000:cut_03 · 컷 frame 7639 · 컷까지 청크 10개
GT: 총 4회 중 2회 완료 → 남은 2
들은 명령
- [t=0s] “Pick up four of the apples from the dining table and place them in or on the desk.”
- [t=180s] “Hold on — move the soap bottle to the other end of the countertop first.”
실제로 완료된 단위
- 단위 0: frame 0–2030 (81 s) — Pick up the apple from the dining table and place it in or on the desk
- 단위 1: frame 2031–4495 (99 s) — Pick up the apple from the dining table and place it in or on the desk
사과 개수는 둘 다 맞혔다. 그런데 append 가 끝난 심부름(비누통 옮기기)까지 미완으로 올려 0점이 됐다 — [7][8][9] 세 클립이 비누통을 집고·놓고·다시 뻗는 장면을 각각 남겨 진행 중처럼 읽힌다. rf-v3 는 한 문장 “picked up the soapdispenser and placed it back down” 으로 압축해 끝났음이 분명하다.
영상 — 실제로 일어난 일
단위 0 · 81.2 s · frame 0–2030
단위 1 · 98.6 s · frame 2031–4495
writer 가 본 프레임 + 그 청크에 쓴 문장
chunk 00 (t=0–30s) — writer 가 받은 4장. 이 청크에 걸친 단위: [0]
writer 가 이 청크에 쓴 것 → I approached a white desk containing five apples and a book, situated in front of a dresser with two statues on it. I picked up an apple from the desk and turned around towards the room, where a basketball lay on the floor near a bed.
chunk 01 (t=30–60s) — writer 가 받은 4장. 이 청크에 걸친 단위: [0]
writer 가 이 청크에 쓴 것 → Holding the apple, I turned towards a wooden desk by a door, noticing another apple on it, and moved closer.
chunk 02 (t=60–90s) — writer 가 받은 4장. 이 청크에 걸친 단위: [0, 1]
writer 가 이 청크에 쓴 것 → I placed the apple onto the wooden desk, which now held two apples, and turned back to face the white desk and the dresser.
chunk 03 (t=90–120s) — writer 가 받은 4장. 이 청크에 걸친 단위: [1]
writer 가 이 청크에 쓴 것 → I moved back to the white desk, which held six apples and a book, and picked up one of the apples. Across the room, two apples sat on the wooden desk, and the basketball lay on the floor near the bed.
chunk 04 (t=120–150s) — writer 가 받은 4장. 이 청크에 걸친 단위: [1]
writer 가 이 청크에 쓴 것 → Holding the apple, I turned away from the white desk, past the bed, and approached the wooden desk.
chunk 05 (t=150–180s) — writer 가 받은 4장. 이 청크에 걸친 단위: [1]
writer 가 이 청크에 쓴 것 → I placed the apple onto the wooden desk, bringing the total number of apples on it to three.
메모리 전문
rf-v3 (재작성) — 컷 시점 요약 108 단어
I started in a bedroom with a bed, a basketball on the floor, a white desk holding multiple apples, a dresser with two statues, a tv stand with a TV, and a wooden desk near a door holding one apple. I picked up two apples from the white desk and placed both onto the wooden desk (leaving three apples on it). Next, I navigated through an office (with a desk, chairs, laptop, armchair, pillow, ottoman) into an adjacent room with a dining table, house plant, stool, and a countertop holding an atomizer and a soapdispenser. I picked up the soapdispenser and placed it back down onto the countertop.
append (누적) — 10 항목 / 278 단어
- [0] I approached a white desk containing five apples and a book, situated in front of a dresser with two statues on it. I picked up an apple from the desk and turned around towards the room, where a basketball lay on the floor near a bed.
- [1] Holding the apple, I turned towards a wooden desk by a door, noticing another apple on it, and moved closer.
- [2] I placed the apple onto the wooden desk, which now held two apples, and turned back to face the white desk and the dresser.
- [3] I moved back to the white desk, which held six apples and a book, and picked up one of the apples. Across the room, two apples sat on the wooden desk, and the basketball lay on the floor near the bed.
- [4] Holding the apple, I turned away from the white desk, past the bed, and approached the wooden desk.
- [5] I placed the apple onto the wooden desk, bringing the total number of apples on it to three.
- [6] I turned away from the wooden desk, passed the white desk and a tv stand, and moved through the doorway into an adjacent room where a desk with a laptop, several chairs, and an armchair were located.
- [7] I moved through the office room and entered the kitchen area, where a dining table and a counter were located. I approached the counter and picked up an atomizer sitting on it.
- [8] I placed the soapdispenser onto the countertop, where an atomizer also sat, and turned away from the countertop.
- [9] I turned back to face the countertop and reached toward the soapdispenser lying on it, with an atomizer also resting further along the countertop.
세 답
textlog (오라클) resume_ok=1
Pick up four of the apples from the dining table and place them in or on the desk.→ 남은 2
The robot completed moving two apples to the desk before being interrupted, leaving two of the requested four apples remaining. The interruption task involving the soap bottle was fully completed.
rf-v3 재작성 resume_ok=1
Pick up four of the apples from the dining table and place them in or on the desk.→ 남은 2
The user requested moving four apples to the desk, and two have been moved so far, leaving two remaining. The subsequent interruption to move the soap bottle was fully completed.
append 누적 resume_ok=0
Pick up four of the apples from the dining table and place them in or on the desk.→ 남은 2Hold on — move the soap bottle to the other end of the countertop first.→ 남은 1
For Command 1, the user asked to pick up four apples and place them on the desk. The robot successfully transported 2 apples before being interrupted, leaving 2 apples remaining. For Command 2, the robot has not yet moved the soap bottle to the other end of the countertop, leaving 1 soap bottle to move.
5 visit_rooms — 문항이 답을 결정하지 못한다
오라클(textlog)이 31행 중 18행 오답, 그 중 17행이 기권한다. 사유 원문이 전부 같다:
"the total number of rooms in the house cannot be determined from the available memory"
질의 데이터의 pending 을 보면 n(집의 총 방 수) − m(방문한 방 수) = r 인데,
예시 train_123 은 씬카드상 Bedroom·Bathroom·Kitchen·LivingRoom 4개이고 로봇은 거실·주방 2개만 갔다 → 정답 2.
들어가 본 적 없는 방 2개의 존재를 알 방법이 없다. build_query_parts 는 "후보 리스트 없음(설계 v2)"이고
시스템 프롬프트도 방 목록을 주지 않는다.
| 전략 | 전체 161 | visit 제외 130 | visit 31행 |
|---|---|---|---|
| blind | .062 | .062 | .065 |
| rollforward-v3 | .174 | .192 | .097 |
| append | .186 | .215 | .065 |
| vis-pure | .199 | .215 | .129 |
| tagmem-r3 | .211 | .238 | .097 |
| textlog(오라클) | .789 | .877 | .419 |
고칠 수 있다. 질의 턴에 그 집의 방 목록을 주면 된다 — routine 이 260915 에
multi_items(테이블 위 물건 목록)를 같은 이유로 이미 주입한 전례가 있다. 전 전략에 동일하게 들어가므로 대칭이고,
방 목록은 집 지식이지 에피소드 정보가 아니라 누출도 아니다. 메모리(ingest)는 그대로 재사용하고
읽기 호출만 다시 돌리면 된다 — flash 전 전략 $8.2 · luna $1.7 · 오픈 4종 $0 · sonnet $33.4.