Index
2026-08-19 — Experiment

예산 스윕 — 353질의 × 5전략 × 2모델

scene_bench | 전체 759질의의 47%를 분포 일치 표집 · 총 $72

TL;DR

75
에피소드
353
질의
0.78
textlog: 옮긴 물체
0.58
시각: 본 물체
$72
비용

1 표본 설계

예산 상한 $80에 맞춰 153편 중 75편(353질의, 전체의 47%)을 골랐다. 무작위가 아니라 질의 층화 셀(질의종류 × 정답개수 × 바닥여부 × 쌍둥이 × 동일카테고리 수)의 비율이 전체 759건과 같아지도록 탐욕 선택했다 — 에피소드는 그릇이고 측정 대상은 질의이므로, 질의 분포가 맞으면 대표성이 선다.

분포 L1 거리 0.016 · 축별 (전체 → 표본) find_target 50.3% → 50.4% 쌍둥이 12.9% → 12.5% 여러 개 9.9% → 9.9% 바닥 8.4% → 8.5% 동일카테고리 2개+ 47.7% → 47.3%

2 전략별 총점

gemini-3.6-flash

전략loc_set−blindplacecountf1기권
blind (메모리 없음)0.273—0.1980.5330.3020.405
textlog (이벤트 로그)0.486+0.2130.4640.6180.4930.377
rollforward-v2 (자필 노트)0.299+0.0260.2240.5860.3210.371
vis-pure (자선별 프레임)0.690+0.4170.6030.9350.7220.023
frames-uniform (균등 16장)0.649+0.3760.5540.9180.6870.031

claude-sonnet-5

전략loc_set−blindplacecountf1기권
blind (메모리 없음)0.204—0.1490.4250.2210.535
textlog (이벤트 로그)0.448+0.2440.4140.5920.4550.405
rollforward-v2 (자필 노트)0.351+0.1470.2920.6370.3790.320
vis-pure (자선별 프레임)0.477+0.2730.3850.8610.5200.057
frames-uniform (균등 16장)0.460+0.2560.3530.8580.5060.051

loc_set=위치 집합 일치(개수까지) · −blind=메모리 신호 · place=위치+가구 · count=개수 일치 · 기권은 오답 처리.

총점만 보면 셋이 붙어 있다. 하지만 아래 층화를 보면 같은 점수가 전혀 다른 능력에서 나온다 — 총점 하나로 메모리 방식을 비교하면 안 된다는 뜻이다.

3 층화 — 어디서 갈리는가

질의 종류

gemini-3.6-flash

전략로봇이 옮긴 물체
n=175		지나치며 본 물체
n=173
blind (메모리 없음)0.230.32
textlog (이벤트 로그)0.800.17
rollforward-v2 (자필 노트)0.450.15
vis-pure (자선별 프레임)0.610.77
frames-uniform (균등 16장)0.600.70

claude-sonnet-5

전략로봇이 옮긴 물체
n=175		지나치며 본 물체
n=173
blind (메모리 없음)0.190.21
textlog (이벤트 로그)0.780.12
rollforward-v2 (자필 노트)0.350.35
vis-pure (자선별 프레임)0.370.58
frames-uniform (균등 16장)0.370.55

정답 개수

gemini-3.6-flash

전략1개
n=313		여러 개
n=35
blind (메모리 없음)0.300.00
textlog (이벤트 로그)0.470.60
rollforward-v2 (자필 노트)0.320.14
vis-pure (자선별 프레임)0.730.34
frames-uniform (균등 16장)0.690.29

claude-sonnet-5

전략1개
n=313		여러 개
n=35
blind (메모리 없음)0.230.00
textlog (이벤트 로그)0.430.60
rollforward-v2 (자필 노트)0.370.17
vis-pure (자선별 프레임)0.530.03
frames-uniform (균등 16장)0.510.00

최종 위치

gemini-3.6-flash

전략바닥(떨어뜨림)
n=30		가구 위
n=318
blind (메모리 없음)0.100.29
textlog (이벤트 로그)1.000.44
rollforward-v2 (자필 노트)0.230.31
vis-pure (자선별 프레임)0.370.72
frames-uniform (균등 16장)0.330.68

claude-sonnet-5

전략바닥(떨어뜨림)
n=30		가구 위
n=318
blind (메모리 없음)0.030.22
textlog (이벤트 로그)0.970.40
rollforward-v2 (자필 노트)0.230.36
vis-pure (자선별 프레임)0.200.50
frames-uniform (균등 16장)0.200.48

동일 카테고리

gemini-3.6-flash

전략1개뿐
n=182		2개 이상
n=166
blind (메모리 없음)0.300.24
textlog (이벤트 로그)0.380.60
rollforward-v2 (자필 노트)0.240.37
vis-pure (자선별 프레임)0.790.58
frames-uniform (균등 16장)0.690.61

claude-sonnet-5

전략1개뿐
n=182		2개 이상
n=166
blind (메모리 없음)0.200.20
textlog (이벤트 로그)0.320.58
rollforward-v2 (자필 노트)0.310.39
vis-pure (자선별 프레임)0.540.40
frames-uniform (균등 16장)0.540.37

쌍둥이

gemini-3.6-flash

전략없음
n=304		26cm 내 동일물
n=44
blind (메모리 없음)0.260.39
textlog (이벤트 로그)0.450.75
rollforward-v2 (자필 노트)0.260.55
vis-pure (자선별 프레임)0.690.68
frames-uniform (균등 16장)0.640.73

claude-sonnet-5

전략없음
n=304		26cm 내 동일물
n=44
blind (메모리 없음)0.180.39
textlog (이벤트 로그)0.400.75
rollforward-v2 (자필 노트)0.330.50
vis-pure (자선별 프레임)0.460.61
frames-uniform (균등 16장)0.430.68
텍스트와 시각은 서로의 사각을 덮는다. 이벤트 로그는 로봇이 한 일을 정확히 담아 옮긴 물체·바닥 낙하·개수에 강하고, 안 건드린 물체는 아예 등장하지 않아 무력하다. 프레임은 로봇이 본 것을 담아 그 반대다. S1(n=18)에서 관측한 구조가 n=353에서 그대로 재현됐다.

부록 A — 케이스 4건 (영상 · 메모리 · 질의 · 답)

같은 질문에 전략마다 어떤 메모리를 갖고 어떻게 답했는지를 나란히 본다. 메모리는 실제 실행에서 쌓인 것 그대로다.

1 ① 떨어뜨린 물체 — 사건은 로그에만 남는다

GT는 바닥. 이벤트 로그에는 '옮기다 떨어뜨렸다'가 적히지만, 바닥의 그릇은 키프레임에 거의 잡히지 않는다.

컨텍스트 영상 (에피소드 전체, ×12 타임랩스 — 모델이 기억해야 하는 원본)

val_384__1 · 398초 · 이벤트 4건
질의 시점 관측 (마지막 프레임) — 모든 전략이 이 1장을 받는다

쌓인 메모리 ①: textlog (이벤트 로그, 코드가 결정론적으로 생성)

[ 0s– 71s] I picked up the cell phone from the desk in the bedroom and placed it on the armchair in the bedroom. [ 71s– 149s] I picked up the bowl from the dining table in the bedroom, but I dropped it on the way to the toilet in the bathroom. Afterwards I went over to look at where it had ended up. [ 149s– 318s] I picked up the apple from the countertop in the kitchen and placed it on the side table in the living room. [ 318s– 398s] I picked up the bowl from the toilet in the bathroom and carried it to the dining table in the kitchen, but I could not place it there. Afterwards I went over to look at where it had ended up.

쌓인 메모리 ②: rollforward-v2 (모델이 청크마다 직접 쓴 노트)

[0s–11s] Robot moved from near a desk with dual monitors and cell phones toward a dresser area in what appears to be a bedroom/office; no object manipulation occurred yet. [11s–23s] At the desk in the bedroom/office, the robot picked up one cell phone from the desktop and placed it onto the wooden chair (near a book), while another cell phone remains on the desk. [23s–55s] The robot picked up the second cell phone from the desk and carried it across the room, placing it onto the seat of a gray armchair near a floor lamp. [55s–71s] The robot placed the second cell phone down into the gray armchair seat near the floor lamp, positioning it firmly into the cushion. [71s–87s] The robot moved away from the armchair, passed through the bedroom near a bed and desk, and arrived at a dining table with a bowl, cell phone, and basketball, appearing to inspect the bowl on the table. [87s–99s] At the dining table, the robot picked up the bowl and cell phone from the table, leaving only the basketball on the tabletop. [99s–138s] The robot carried the bowl and cell phone from the dining table, moved through a doorway into a bathroom, and set the cell phone down on the toilet's tank/lid near the toilet tissue roll. [138s–149s] In the bathroom, the robot placed the bowl onto the dark countertop near the sink, then turned and moved away from the toilet toward the sink/doorway area, leaving the cell phone on the toilet tank lid. [149s–228s] The robot left the bathroom, moved through a bedroom past a TV stand, and arrived at a kitchen countertop area with bread, a lettuce/melon-like item, and a knife, appearing to survey the counter. [228s–240s] At the kitchen countertop, the robot picked up a tomato (red round item) from beside the bread and knife, lifting it with its gripper.

쌓인 메모리 ③: vis-pure (모델이 직접 고른 프레임 16장)

KEEP/DROP으로 스스로 남긴 장면 — 질의 시점에 이 프레임들만 다시 본다

질의와 전략별 답

질의: "Pick up the bowl I moved earlier" 정답(GT): [(None, 'floor')] ← (방, 가구) · 바닥이면 surface="floor", 방은 채점 안 함
전략판정답모델의 근거(발췌)
blind기권—No prior memory of a bowl being moved is available in this episode; the current view shows a bathroom with no bowl visible, and I have no record of interacting with a bowl.
textlog정답[('kitchen', 'floor')]The bowl was originally on the dining table in the bedroom, dropped in the bathroom near the toilet, then picked up again and carried to the kitchen dining table, but since it could not be placed there, it ended up dropped on the floor in the kitchen.
rollforward-v2(자필노트)오답[('bathroom', 'countertop')]The robot picked up a bowl from the dining table along with a cell phone, carried it into the bathroom, and placed it on the dark countertop near the sink.
vis-pure(자선별 프레임)오답[('kitchen', 'countertop')]The bowl first appeared on a wooden side table near the bed (t=76.6s), then the robot carried it to the dining table (t=87.4s) and picked it up (t=95.4s). It later reappears resting on a dark countertop next to a toaster and other kitchen items (t=201.8s), whi
frames-uniform오답[('bedroom', 'dining table')]The only bowl seen in the episode appears at t=79.6s resting on a wooden table in the bedroom, next to the bed. No later frame shows it being moved elsewhere, so its last known position is that bedroom table.

2 ② 지나치며 본 물체 — 로그에는 존재하지 않는다

로봇이 건드린 적 없는 vase. 이벤트 로그에 이 물체는 한 줄도 없다 — textlog는 기권할 수밖에 없고, 프레임에는 남아 있다.

컨텍스트 영상 (에피소드 전체, ×12 타임랩스 — 모델이 기억해야 하는 원본)

val_123__0 · 261초 · 이벤트 4건
질의 시점 관측 (마지막 프레임) — 모든 전략이 이 1장을 받는다

쌓인 메모리 ①: textlog (이벤트 로그, 코드가 결정론적으로 생성)

[ 0s– 23s] I picked up the vase from the tv stand in the living room and carried it to the armchair in the bedroom, but I could not place it there. Afterwards I went over to look at where it had ended up. [ 23s– 28s] I picked up the vase from the armchair in the bedroom and carried it to the desk in the bedroom, but I could not place it there. Afterwards I went over to look at where it had ended up. [ 28s– 138s] I picked up the cell phone from the dresser in the living room, but I dropped it on the way to the dining table in the living room. Afterwards I went over to look at where it had ended up. [ 138s– 260s] I picked up the bowl from the desk in the bedroom and placed it on the countertop in the kitchen.

쌓인 메모리 ②: rollforward-v2 (모델이 청크마다 직접 쓴 노트)

[0s–23s] The robot arm moved from the dining table toward a TV stand, approaching a cell phone and cup/bottle resting on the shelf beneath the TV. [23s–28s] The robot arm remained positioned near the TV stand, holding steady above the cell phone and cup on the shelf, not yet grasping either item. [28s–53s] The robot left the TV stand (with cell phone and cup still on shelf) without grasping anything, moved through a doorway past a dresser, and is now approaching another dresser/tv stand in a different room. [53s–65s] The robot arm reached toward a dresser holding a plate/tray, positioning its gripper above the surface but not yet grasping anything. [65s–106s] The robot picked up a cell phone from the dresser near the TV stand and turned toward a dining table/desk area, carrying the phone in its gripper. [106s–119s] The robot carried the cell phone toward the dining table with chairs, positioning its gripper above the table surface where a mug/plate already sits, near the TV stand and dresser with a bowl and cylindrical item. [120s–138s] The robot dropped the cell phone onto the floor near a dresser/tv stand with a bowl on top, and is now positioned above the dresser without holding anything. [138s–158s] The robot left the cell phone on the floor near the dresser with a bowl, moved through a doorway into a new room, and approached a table holding a mug, box/bin, CD, pencil, and cup near a TV and armchair. [159s–171s] The robot arm hovered above a small bowl/cup on the table (with mug, box, atomizer, CD, pencil) in the room with the armchair and painting, not yet grasping anything. [171s–246s] The robot picked up a bowl/cup from the table (with mug, box, atomizer, CD, pencil) and carried it through the room past a desk with laptop and TV stand, then out into a kitchen area with a countertop holding bread and a knife, stools, and a dining table.

쌓인 메모리 ③: vis-pure (모델이 직접 고른 프레임 11장)

KEEP/DROP으로 스스로 남긴 장면 — 질의 시점에 이 프레임들만 다시 본다

질의와 전략별 답

질의: "Pick up the vase" 정답(GT): [('bedroom', 'desk')] ← (방, 가구) · 바닥이면 surface="floor", 방은 채점 안 함
전략판정답모델의 근거(발췌)
blind오답[('kitchen', 'countertop')]The current view shows a kitchen countertop with a vase-like item near the atomizer, but recalling the episode, the vase was on the countertop in the kitchen.
textlog오답[('bedroom', 'floor')]The vase was moved from the tv stand to the armchair (failed placement, fell to floor), then picked up again and carried to the desk in the bedroom, where it again failed to be placed and ended up on the floor near the desk in the bedroom.
rollforward-v2(자필노트)오답[('living room', 'tv stand')]The notes never explicitly name a vase, but a 'cylindrical item' sitting alongside a bowl on a dresser/TV stand in the living room (mentioned at 106s-119s) matches the typical shape of a vase and was never picked up or moved during the episode, so it should st
vis-pure(자선별 프레임)정답[('bedroom', 'desk')]The teal, bulbous-necked object seen on the TV stand at t=22.8s (living room) matches the appearance of a vase. Later, at t=158.5s, the same teal item appears on a small table/desk in a different room with brick walls and a lamp — a bedroom desk. The robot evi
frames-uniform오답[('kitchen', 'countertop')]Scanning the episode, the only vase-shaped object (a slender green bottle with a narrow neck) appears sitting on the kitchen countertop in the later frames (t≈225.8s–260.5s), where the robot is working near the counter with bread and a bucket. No other vase-li

3 ③ 여러 개 세기 — 시각은 같은 물체로 합친다

'내가 옮긴 모든 bowl'. 로그는 인스턴스를 따로 적어 개수를 세지만, 프레임 속 여러 bowl은 한 물체로 뭉뚱그려진다.

컨텍스트 영상 (에피소드 전체, ×12 타임랩스 — 모델이 기억해야 하는 원본)

val_101__1 · 488초 · 이벤트 4건
질의 시점 관측 (마지막 프레임) — 모든 전략이 이 1장을 받는다

쌓인 메모리 ①: textlog (이벤트 로그, 코드가 결정론적으로 생성)

[ 0s– 116s] I picked up the remote control from the dining table in the bedroom and placed it on the toilet in the bathroom. [ 116s– 203s] I picked up bowl #1 from the dining table in the bedroom and placed it on the bed in the bedroom. [ 203s– 375s] I picked up the remote control from the toilet in the bathroom and placed it on the sofa in the living room. [ 375s– 488s] I picked up bowl #2 from the countertop in the kitchen and placed it on the drawer of the dresser in the living room.

쌓인 메모리 ②: rollforward-v2 (모델이 청크마다 직접 쓴 노트)

[0s–16s] Robot is near a dining table with a mug, remote control/pencil, and bowl on it, moving around the room without manipulating any object yet. [16s–28s] Robot picked up a paper towel (roll) from the dining table in the dining room, holding it near its end effector above the table. [28s–100s] Robot carried the paper towel roll from the dining table into the bathroom, placing it on the toilet tank next to an atomizer. [100s–116s] In the bathroom, the robot placed a butter knife on the toilet seat/lid, while the paper towel roll remains on the toilet tank next to the atomizer. [116s–142s] Robot left the bathroom (with paper towel roll and atomizer remaining on toilet tank, butter knife on toilet seat) and traveled through a bedroom back to the dining table, where it picked up a bowl. [142s–154s] Robot is holding the bowl above the dining table in the dining room, with a pencil/remote still on the table, appearing to reposition or lift the bowl before moving off. [154s–187s] Robot carried the bowl from the dining table through the bedroom (passing a desk with teddy bear, tissue box, and knickknack) toward the bed, still holding the bowl. [187s–203s] Robot placed the bowl onto a nightstand/shelf beside the bed in the bedroom, releasing it from its gripper. [203s–232s] Robot left the bowl on the bedside table and traveled from the bedroom through the doorway back into the bathroom, arriving near the toilet where the paper towel roll and atomizer remain on the tank and the butter knife is still on the toilet seat. [232s–244s] In the bathroom, the robot picked up the butter knife from the toilet seat/lid, leaving the atomizer and paper towel roll on the toilet tank.

쌓인 메모리 ③: vis-pure (모델이 직접 고른 프레임 16장)

KEEP/DROP으로 스스로 남긴 장면 — 질의 시점에 이 프레임들만 다시 본다

질의와 전략별 답

질의: "Pick up all the bowls I moved earlier" 정답(GT): [('bedroom', 'bed'), ('living room', None)] ← (방, 가구) · 바닥이면 surface="floor", 방은 채점 안 함
전략판정답모델의 근거(발췌)
blind오답[('kitchen', 'shelf')]In the current view there is a bowl resting on the shelf compartment of the wooden cabinet in the kitchen. This appears to be the bowl that was moved earlier and placed there.
textlog정답[('bedroom', 'bed'), ('living room', 'dresser')]Two bowls were moved: bowl #1 was placed on the bed in the bedroom, and bowl #2 was placed on the dresser (drawer area) in the living room, which matches the current view showing a bowl on the dresser.
rollforward-v2(자필노트)오답[('bedroom', 'shelf')]The robot picked up a bowl from the dining table (142s-154s) and carried it through the bedroom to a nightstand/shelf beside the bed, placing it there (187s-203s). This is the only bowl the robot moved during the episode; the bowl seen later on the kitchen cou
vis-pure(자선별 프레임)오답[('bedroom', 'bed')]The only bowl seen was initially on the dining table (t=15.8s). The robot later picked it up (t=142.3s, shown lifted with the gripper) and carried it into the bedroom, placing it near the bed's edge (t=203.3s, visible as a small white bowl resting on the dark
frames-uniform오답[('kitchen', 'countertop')]Tracking the white bowl: it started on the dining table (t=32.6s), was moved and dropped on the floor near the dining table (t=130.2s), then picked up and carried through the house. In the final views (t=488.3s and current view) a bowl/plate is resting in the

4 ④ 상식으로 풀리는 질의 — blind도 맞힌다

fork는 kitchen 조리대에 있을 법하다. 메모리 없이도 맞는 이런 질의가 blind 바닥선(0.20~0.27)의 정체다.

컨텍스트 영상 (에피소드 전체, ×12 타임랩스 — 모델이 기억해야 하는 원본)

val_372__0 · 223초 · 이벤트 5건
질의 시점 관측 (마지막 프레임) — 모든 전략이 이 1장을 받는다

쌓인 메모리 ①: textlog (이벤트 로그, 코드가 결정론적으로 생성)

[ 0s– 6s] I picked up the bottle from the dining table in the kitchen and carried it to the chair in the living room, but I could not place it there. Afterwards I went over to look at where it had ended up. [ 6s– 12s] I picked up the bottle from the chair in the living room and carried it to the dining table in the kitchen, but I could not place it there. Afterwards I went over to look at where it had ended up. [ 12s– 175s] I picked up the bowl from the dining table in the living room and placed it on the bed in the bedroom. [ 175s– 190s] I went to the desk in the bedroom to get pen #1, but I never found it. Afterwards I went over to look at where it had ended up. [ 190s– 223s] I picked up pen #2 from the chair in the kitchen and carried it to the dining table in the kitchen, but I could not place it there. Afterwards I went over to look at where it had ended up.

쌓인 메모리 ②: rollforward-v2 (모델이 청크마다 직접 쓴 노트)

[0s–6s] Robot arm is positioned near a dining table holding a bar soap in its gripper, with table items (cup, mugs, toaster, fork, cell phone) visible; no placement completed yet in this clip. [6s–12s] Robot arm remains near the dining table with the bar soap held (or already placed) among items like cup, mugs, toaster, and fork; no clear new placement observed in this clip. [12s–38s] Robot placed the bar soap on the dining table (near toaster, cups, fork) then moved empty-handed through the room, passing a house plant, toward another dining table with a bowl, box, and sofa visible. [38s–50s] Robot arm reached toward a second dining table (near sofa, box, laptop) holding a bowl, positioning it above the table with a fabric item also present, not yet fully placed. [50s–159s] Robot placed the bowl on a chair near the second dining table (with laptop/box/sofa), then moved empty-handed through a doorway into a bedroom with a bed, teddy bear, and dresser. [159s–175s] Robot placed a bowl on the bed (near teddy bear) in the bedroom, next to a dresser with a small red item on it. [175s–190s] Robot left the bowl on the bed near the teddy bear and dresser, then moved empty-handed out of the bedroom (passing a toilet) into a room with a desk holding a laptop, table lamp, cell phone, and a baseball bat on the floor. [190s–223s] Robot moved empty-handed from the desk room (laptop, table lamp, baseball bat) through a doorway into a kitchen with a refrigerator, then into a dining room with a table holding a toaster, bottles, apple, cell phone, and knife.

쌓인 메모리 ③: vis-pure (모델이 직접 고른 프레임 10장)

KEEP/DROP으로 스스로 남긴 장면 — 질의 시점에 이 프레임들만 다시 본다

질의와 전략별 답

질의: "Pick up the fork" 정답(GT): [('kitchen', 'dining table')] ← (방, 가구) · 바닥이면 surface="floor", 방은 채점 안 함
전략판정답모델의 근거(발췌)
blind정답[('kitchen', 'dining table')]The current view shows a dining table with a knife, apple, bottles, and a device, but no fork is visible in this room. Based on memory of the episode, no fork was observed being moved elsewhere, so I cannot determine its exact location.
textlog기권—There is no mention of a fork anywhere in the recorded episode, and no fork is visible in the current view. I have no memory of a fork's location.
rollforward-v2(자필노트)정답[('kitchen', 'dining table')]The fork was seen on the first dining table alongside cup, mugs, toaster, and cell phone, and was never picked up or moved during the episode.
vis-pure(자선별 프레임)정답[('kitchen', 'dining table')]At t=0.0s, a fork was visible on the kitchen dining table alongside a toaster, mugs, and other tableware. No later frames show the fork being moved or picked up, so it should still be on the kitchen dining table.
frames-uniform정답[('kitchen', 'dining table')]At t=0.0s the fork was seen resting on the wooden dining table alongside the toaster, mugs, and other kitchen items. The robot never picked it up or moved it in later frames, and subsequent views of the same table (e.g., t=74.4s, t=89.2s, t=223.1s) still show

4 실물 — 모델 출력 원문

① textlog가 강한 곳 — 로봇이 옮긴 물체 정답

claude-sonnet-5 · textlog · val_101__1 질의: "Pick up the remote control I moved earlier" GT : [('living room', 'sofa')]
[모델 출력 원문] {"reasoning": "The remote control was moved from the dining table in the bedroom to the toilet in the bathroom, then later moved again from the toilet to the sofa in the living room.", "selections": [{"category": "remote control", "room": "living room", "surface": "sofa"}], "abstain": false}

② textlog가 무력한 곳 — 지나치며 본 물체 오답

claude-sonnet-5 · textlog · val_123__0 질의: "Pick up the vase" GT : [('bedroom', 'desk')]
[모델 출력 원문] {"reasoning": "The vase was moved from the tv stand to the armchair (failed placement, fell to floor), then picked up again and carried to the desk in the bedroom, where it again failed to be placed and ended up on the floor near the desk in the bedroom.", "selections": [{"category": "vase", "room": "bedroom", "surface": "floor"}]}

③ 시각이 강한 곳 — 프레임에 남은 물체 정답

claude-sonnet-5 · rollforward-vis-pure · val_144__0 질의: "Pick up the cell phone" GT : [('living room', 'tv stand')]
[모델 출력 원문] {"reasoning": "The cell phone last appears at t=334.5-345.2s being picked up near the orange dresser/tv-stand area in the living room, where it was resting on a cart-like black surface next to the tv stand.", "selections": [{"category": "cell phone", "room": "living room", "surface": "cart"}], "abstain": false}

④ 시각이 놓치는 곳 — 바닥에 떨어진 물체 오답

claude-sonnet-5 · rollforward-vis-pure · val_179__1 질의: "Pick up the cell phone I moved earlier" GT : [(None, 'floor')]
[모델 출력 원문] {"reasoning": "The cell phone started on a tv stand, was moved to the bathroom sink, then dropped on the floor near the toilet, and later appears again on the desk in the office/bedroom at t=250.4s and t=262.5s. This desk location is the last place it was seen, so it should still be there.", "selections": [{"category": "cell phone", "room": "bedroom", "surface": "desk"}], "abstain": false}

5 비용 · 시간

모델calls비용wall(샤드 합)
gemini-3.6-flash3,981$33.1411.0h
claude-sonnet-53,989$39.117.6h

에피소드를 3샤드로 나눠 병렬 실행 — 실제 경과는 샤드 합의 약 1/3. 전량(759질의) 환산 시 약 $155.

6 Takeaway

메모리 방식은 총점이 아니라 커버리지로 갈린다

다섯 전략의 총점 차이는 0.03 안팎이지만, 질의 종류별로 보면 최대 0.66(textlog 0.78 vs 0.12) 차이가 난다. 즉 이 벤치마크에서 "어떤 메모리가 더 좋은가"는 질문이 잘못됐고, "무엇을 기억하는 메모리인가"가 옳은 질문이다. 이벤트 로그는 행위 이력을, 프레임은 장면을 기억하며, 둘 중 어느 쪽도 상대의 영역을 대신하지 못한다.

Generated by Claude Code — 2026-08-19 | 원장: tmp/s3_{flash,sonnet}_s*/results.json | 표본: tmp/budget_sample.json