writer 입력 ablation — 시험 실행 결과
2026-10-13
exo 4장 / exo 10장 / exo+wrist 10장, 과제당 녹화 1개. 논문 당시 같은 녹화의 결과와 나란히 비교
요약: 수정된 파이프라인(v2)은 실제 모델로 끝까지 정상 동작했고, reader가 쓴 가구 단어 308개는 전부 프롬프트 목록 안의 단어였습니다(버려진 단어 0). 다만 과제당 녹화 1개(질의 1–8개)라 조건 간 정확도 차이는 이 실행으로 판단할 수 없습니다. 프롬프트·기억·답을 직접 읽어 확인한 개선과 새로 찾은 문제를 아래에 정리했습니다.
$9.22
시험 실행 비용
240
채점된 답
0
어휘 밖이라 버려진 단어
0
형식 오류·재시도
1숫자
맞은 질의 수 / 전체 질의 수입니다. "논문 당시"는 같은 녹화·같은 모델(gemini-3.7-flash)의 보관된 결과입니다. exo 10장 조건은 writer가 있는 세 방법만 돌렸습니다.
| 과제 | 방법 | 논문 당시 | v2 exo 4장 | v2 exo 10장 | v2 exo+wrist 10장 |
|---|---|---|---|---|---|
| finding | No Memory | 1/8 | 2/8 | — | 1/8 |
| finding | GT Action Log | 4/8 | 5/8 | — | 5/8 |
| finding | Uniform Frames | 5/8 | 5/8 | — | 4/8 |
| finding | MEM^R | 5/8 | 3/8 | 1/8 | 5/8 |
| finding | MemER^R | 2/8 | 5/8 | 5/8 | 4/8 |
| finding | R&R Agent | 3/8 | 3/8 | 3/8 | 1/8 |
| restore | No Memory | 0/4 | 0/4 | — | 0/4 |
| restore | GT Action Log | 3/4 | 4/4 | — | 4/4 |
| restore | Uniform Frames | 0/4 | 2/4 | — | 1/4 |
| restore | MEM^R | 1/4 | 0/4 | 1/4 | 1/4 |
| restore | MemER^R | 0/4 | 1/4 | 1/4 | 1/4 |
| restore | R&R Agent | 0/4 | 1/4 | 2/4 | 0/4 |
| routine | No Memory | 0/3 | 0/3 | — | 0/3 |
| routine | GT Action Log | 1/3 | 1/3 | — | 1/3 |
| routine | Uniform Frames | 2/3 | 0/3 | — | 0/3 |
| routine | MEM^R | 1/3 | 1/3 | 1/3 | 1/3 |
| routine | MemER^R | 3/3 | 2/3 | 2/3 | 2/3 |
| routine | R&R Agent | 1/3 | 2/3 | 1/3 | 1/3 |
| resume | No Memory | 0/1 | 0/1 | — | 0/1 |
| resume | GT Action Log | 1/1 | 1/1 | — | 1/1 |
| resume | Uniform Frames | 0/1 | 0/1 | — | 1/1 |
| resume | MEM^R | 1/1 | 1/1 | 0/1 | 1/1 |
| resume | MemER^R | 0/1 | 0/1 | 0/1 | 0/1 |
| resume | R&R Agent | 1/1 | 1/1 | 0/1 | 1/1 |
읽는 법: writer와 무관한 No Memory도 exo 4장과 exo+wrist 10장 사이에 1–2문항이 달라집니다. reader의 실행 간 흔들림이 이 정도이므로, 녹화 1개에서 보이는 1–2문항 차이는 조건의 효과로 볼 수 없습니다.
비용 (괄호는 API 호출 수)
| 조건 | Finding | Restore | Routine | Resume | 합계 |
|---|---|---|---|---|---|
| exo_f4 | $1.05 (102) | $0.76 (57) | $0.64 (54) | $0.38 (42) | $2.82 |
| exo_f10 | $1.01 (78) | $0.65 (45) | $0.62 (45) | $0.54 (39) | $2.82 |
| wrist_f10 | $1.45 (102) | $0.87 (57) | $0.67 (54) | $0.58 (42) | $3.57 |
writer 한 번에 들어가는 이미지: exo 4장 조건은 640×480 4장, exo 10장은 640×480 10장, exo+wrist 10장은 960×480 10장(MemER^R은 보관 프레임 8장이 더 붙어 최대 18장).
2VLM에 들어가는 입력
아래는 API 호출 없이 실제 실행 코드를 그대로 돌려 뽑은 프롬프트 전문입니다(이미지는 크기만 표시).
MEM^R writer가 받는 입력 전문 (v2, 첫 30초 창)
[system]
You are the memory module of a household robot, watching its episode as it
happens, clip by clip. Keep a running account of what the robot does and what
it sees in this house — in your own words, the way you would tell someone
afterwards what happened.
When the episode ends you will be asked about it from memory alone, and you
will not know the question in advance. It could be about where something is
now, where it was earlier, how far the robot got through something it was
told to do, or where things in this house tend to be put away. So an account
that only lists where objects ended up may not be enough — how many times
something happened, and what a place was like before it changed, can matter
just as much.
You will not see these frames again.
In this house the robot moves objects around. Where a thing is now and where
it was before it moved are both worth keeping.
You maintain ONE running language memory: a compact
natural-language summary of everything relevant that has happened so far.
Each time you receive: your current memory summary, and a few frames from
the current clip.
Reply with exactly two blocks:
MEMORY: the UPDATED summary. Rewrite the whole summary so that it now also
covers this clip. Keep it compact by wording it tightly, not by throwing
events away — two descriptions of the same event can become one, two separate
events should not. Write in first person what the robot did, e.g. "I placed
the cup on the kitchen countertop", along with what you saw and where.
NEXT: one line predicting the robot's next action.
Your MEMORY text is the ONLY memory you will have later — you will not see
these frames again.
Names. When you name an object, a piece of furniture or a room, use the name from
the lists below if one of them fits what you see. If none fits, use a short
plain word of your own. Something lying on the floor rather than on a piece of
furniture is on the "floor".
Objects: CD | alarm clock | apple | baseball bat | basketball | bottle | bowl | box | bread | butter knife | candle | cell phone | cloth | coffee machine | credit card | cup | desk lamp | dumbbell | egg | fork | hand towel | house plant | kettle | key chain | knife | ladle | laptop | lettuce | microwave | mug | newspaper | pan | paper towel roll | pen | pencil | pepper shaker | pillow | plate | plunger | pot | potato | remote control | safe | salt shaker | soap bar | soap bottle | spatula | sponge | spoon | spray bottle | statue | teddy bear | tennis racket | tissue box | toaster | toilet paper | tomato | vase | watch | wine bottle
Furniture and fixtures: armchair | bed | cart | chair | clothes dryer | coffee table | countertop | desk | dresser | fridge | laundry basket | shelf | side table | sink | sofa | table | toilet | tv stand | washing machine
Rooms: bathroom | bedroom | kitchen | living room
[user]
Your current memory summary:
(empty)
Current clip [t=0s – 30s]:
[IMAGE 640x480]
[IMAGE 640x480]
[IMAGE 640x480]
[IMAGE 640x480]
Reply with the MEMORY and NEXT blocks only.
같은 입력의 논문 당시(v1) 전문
[system]
You are the memory module of a household robot, watching its episode as it
happens, clip by clip. Keep a running account of what the robot does and what
it sees in this house — in your own words, the way you would tell someone
afterwards what happened.
When the episode ends you will be asked about it from memory alone, and you
will not know the question in advance. It could be about where something is
now, where it was earlier, how far the robot got through something it was
told to do, or where things in this house tend to be put away. So an account
that only lists where objects ended up may not be enough — how many times
something happened, and what a place was like before it changed, can matter
just as much.
You will not see these frames again.
In this house the robot moves objects around. Where a thing is now and where
it was before it moved are both worth keeping.
You maintain ONE running language memory: a compact
natural-language summary of everything relevant that has happened so far.
Each time you receive: your current memory summary, and a few frames from
the current clip.
Reply with exactly two blocks:
MEMORY: the UPDATED summary. Rewrite the whole summary so that it now also
covers this clip. Keep it compact by wording it tightly, not by throwing
events away — two descriptions of the same event can become one, two separate
events should not. Write in first person what the robot did, e.g. "I placed
the cup on the kitchen countertop", along with what you saw and where.
NEXT: one line predicting the robot's next action.
Your MEMORY text is the ONLY memory you will have later — you will not see
these frames again.
Use the canonical names from this scene inventory when writing your memory:
Object categories: CD | alarm clock | apple | atomizer | bar soap | baseball bat | basketball | boiler | bottle | bowl | box | bread | butter knife | candle | cell phone | coffeemaker | cooking pan | credit card | cup | desktopcomputer | dumbbell | egg | fabric | fork | house plant | irish potato | key chain | knickknack | knife | ladle | laptop | lettuce | microwaveoven | mug | newspaper | paper towel | pen | pencil | pepper shaker | pillow | plate | plunger | pot | remote control | safe | salt shaker | soapdispenser | spatula | sponge | spoon | statue | table lamp | teddy bear | tennisracket | tissue box | toaster | toilet tissue | tomato | vase | watch | wine bottle
Surfaces: armchair | bed | cart | chair | clothes dryer | coffee table | countertop | desk | dining table | dresser | shelf | sink | sofa | toilet | tv stand | washing machine | floor
[user]
Your current memory summary:
(empty)
Current clip [t=0s – 30s]:
[IMAGE 640x480]
[IMAGE 640x480]
[IMAGE 640x480]
[IMAGE 640x480]
Reply with the MEMORY and NEXT blocks only.
R&R Agent reader가 받는 입력 전문 (v2, 가짜 기억으로 채운 형식 예시)
[system]
You are the memory module of a household robot. The earlier turns of this
conversation are your memory of one episode in this house — what the robot did
and saw. The final image is the robot's CURRENT view at the end of the episode.
The user will ask you to pick up an object, described by its category and by
what happened to it. Answer WHERE that object is now. The object is usually not
in the current view, and nobody will tell you what is in this house: recall it.
Output a single JSON object, nothing else:
{"reasoning": "...",
"selections": [{"category": "...", "room": "...", "surface": "..."}],
"abstain": false}
Rules:
- One entry in "selections" per object the query refers to. If the query says
"all the ...", decide how many there are from your memory. Otherwise give
exactly one.
- "room": one of bathroom | bedroom | kitchen | living room.
- "surface": the kind of thing the object is resting on, from the surface list
below. Use "floor" if it is lying on the floor rather than on a surface —
an object that was dropped ends up there.
- Use the category and surface names exactly as spelled in the lists.
- If your memory genuinely cannot determine the answer, set "abstain": true.
Object categories:
CD | alarm clock | apple | baseball bat | basketball | bottle | bowl | box | bread
butter knife | candle | cell phone | cloth | coffee machine | credit card | cup
desk lamp | dumbbell | egg | fork | hand towel | house plant | kettle | key chain
knife | ladle | laptop | lettuce | microwave | mug | newspaper | pan | paper towel roll
pen | pencil | pepper shaker | pillow | plate | plunger | pot | potato | remote control
safe | salt shaker | soap bar | soap bottle | spatula | sponge | spoon | spray bottle
statue | teddy bear | tennis racket | tissue box | toaster | toilet paper | tomato
vase | watch | wine bottle
Surfaces:
armchair | bed | cart | chair | clothes dryer | coffee table | countertop | desk
dresser | fridge | laundry basket | shelf | side table | sink | sofa | table | toilet
tv stand | washing machine | floor
[user]
Entries from my memory index relevant to this question (54 stored in total):
[t=180s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[t=210s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[t=240s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[t=270s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[t=300s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 31 · t=300s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
[t=330s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 34 · t=330s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
[t=360s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 37 · t=360s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
[t=390s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 40 · t=390s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
[t=420s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 43 · t=420s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
[t=450s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 46 · t=450s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
[t=480s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 49 · t=480s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
[t=510s] The robot picked up the cup from the dining table. (obj=cup act=pick room=kitchen surf=table n=1)
[IMG 52 · t=510s] image, caption: a cup and a potato on the nightstand (obj=cup,potato room=bedroom surf=nightstand)
[IMAGE 640x480]
RELATED EVENTS (short form):
[0s pick cup → kitchen/table]
[30s pick cup → kitchen/table]
[60s pick cup → kitchen/table]
[90s pick cup → kitchen/table]
[120s pick cup → kitchen/table]
[150s pick cup → kitchen/table]
[480s place egg → kitchen/laundry basket]
[510s place egg → kitchen/laundry basket]
LEDGER (exact counts over my whole index up to now):
place egg → laundry basket [kitchen]: ×18 (n=1,2,3,1,2,3,1,2,3,1,2,3,1,2,3,1,2,3 · t=0,30,60,90,120,150,180,210,240,270,300,330,360,390,420,450,480,510)
rooms seen so far: kitchen (t=0) · bedroom (t=0)
How to read these entries: an object's CURRENT location is given by its LATEST entry; a TIMELINE line reconstructs its moves and a PREV-HOP line names where it was just before the last change — that is what a question about an earlier or original place is asking for. Lines under RELATED EVENTS are the same kind of entries in short form (time, action, object, surface); count them and read them in order like the others. The LEDGER lines are exact counts computed from EVERY note in my index, not only the entries shown: for an instruction that asks for N repetitions, remaining = N minus the ledger count of that action (match by action and destination); if the ledger count already reaches N, that instruction is FINISHED, not pending. An errand whose completing action appears in the ledger is finished. For 'every room', compare the rooms seen so far with the rooms entered after that instruction. Sentences and captions were written on the fly and MAY CONTAIN MISTAKES; when an image is attached, trust what you SEE in the image over any words — the latest image of a container or surface shows how many things are there now. Each image is labelled IMG <id> — cite the one your answer rests on. When the question covers SEVERAL objects, answer every object these entries give you evidence for and leave out only the ones you have no entry for; abstain only if you have evidence for none of them.
[user]
[IMAGE 640x480]
Query: "Pick up the butter knife I moved earlier"
Reply with the JSON object only.
No Memory reader가 받는 입력 전문 (v2)
[system]
You are the memory module of a household robot. The earlier turns of this
conversation are your memory of one episode in this house — what the robot did
and saw. The final image is the robot's CURRENT view at the end of the episode.
The user will ask you to pick up an object, described by its category and by
what happened to it. Answer WHERE that object is now. The object is usually not
in the current view, and nobody will tell you what is in this house: recall it.
Output a single JSON object, nothing else:
{"reasoning": "...",
"selections": [{"category": "...", "room": "...", "surface": "..."}],
"abstain": false}
Rules:
- One entry in "selections" per object the query refers to. If the query says
"all the ...", decide how many there are from your memory. Otherwise give
exactly one.
- "room": one of bathroom | bedroom | kitchen | living room.
- "surface": the kind of thing the object is resting on, from the surface list
below. Use "floor" if it is lying on the floor rather than on a surface —
an object that was dropped ends up there.
- Use the category and surface names exactly as spelled in the lists.
- If your memory genuinely cannot determine the answer, set "abstain": true.
Object categories:
CD | alarm clock | apple | baseball bat | basketball | bottle | bowl | box | bread
butter knife | candle | cell phone | cloth | coffee machine | credit card | cup
desk lamp | dumbbell | egg | fork | hand towel | house plant | kettle | key chain
knife | ladle | laptop | lettuce | microwave | mug | newspaper | pan | paper towel roll
pen | pencil | pepper shaker | pillow | plate | plunger | pot | potato | remote control
safe | salt shaker | soap bar | soap bottle | spatula | sponge | spoon | spray bottle
statue | teddy bear | tennis racket | tissue box | toaster | toilet paper | tomato
vase | watch | wine bottle
Surfaces:
armchair | bed | cart | chair | clothes dryer | coffee table | countertop | desk
dresser | fridge | laundry basket | shelf | side table | sink | sofa | table | toilet
tv stand | washing machine | floor
[user]
[IMAGE 640x480]
Query: "Pick up the butter knife I moved earlier"
Reply with the JSON object only.
논문 당시와 달라진 입력
| 항목 | 논문 당시 (v1) | 지금 (v2) |
|---|---|---|
| 물체 이름 목록 | atomizer, boiler, cooking pan, fabric, irish potato, knickknack, soapdispenser, toilet tissue … | spray bottle, kettle, pan, cloth, potato, soap bottle, toilet paper … (질의문이 쓰는 단어) |
| 가구 목록 | dining table 포함, side table·fridge·laundry basket 없음 | table, side table, fridge, laundry basket 포함 (19종 + floor) |
| writer별 목록 | MEM^R·R&R Agent만 목록을 받고 MemER^R은 받지 않음 | 세 writer가 같은 목록을 받음 |
| 30초 창의 4장 | 0, 10, 20, 30초 | 0, 7.5, 15, 22.5초 |
| Routine 질의 프레임 | 이 녹화에서는 8,071번 프레임(녹화는 8,728프레임) | 실제 녹화 끝(8,727번) |
| Uniform Frames | 8장 중 마지막 장이 질의 프레임과 같은 장 | 질의 프레임과 겹치지 않는 8장 |
exo만 줄 때와 exo+wrist를 줄 때의 한 장
3사례: 입력 → reasoning → 출력 → 채점
사례 A. 같은 답인데 논문 당시에는 오답, 지금은 정답 (Restore)
질의: Pick up the pen and place it in or on where it first was, then pick up the remote control and place it in or on where it first was 방법: GT Action Log
| 정답(채점 기준) | 모델 출력과 reasoning | 채점 | |
|---|---|---|---|
| 논문 당시 | [{"category": "pen", "now": {"room": null, "surface": "sofa"}, "original": {"room": null, "surface": "dining table"}}, {"category": "remote control", "now": {"room": null, "surface": "dining table"}, "original": {"room": null, "surface": "sofa"}}] | {"selections": [{"category": "pen", "now": {"room": "living room", "surface": "sofa"}, "original": {"room": "living room", "surface": "dining table"}}, {"category": "remote control", "now": {"room": "bedroom", "surface": "dining table"}, "original": {"room": "living room", "surface": "sofa"}}], "abstain": false}The pen was moved from the dining table in the living room to the sofa in the living room. Its initial location was the dining table in the living room, and it is currently on the sofa in the living room. The remote control was moved from the sofa in the living room to the dining table in the bedroom. Its initial location was the sofa in the living room, and it is currently on the dining table in the bedroom. | loc 1 · place 0 · abstain 0 · parse_ok 1 · now/orig loc 1/1 · $0.0035 · 3.1s |
| v2 exo_f4 | [{"category": "pen", "now": {"room": "living room", "surface": "sofa", "aliases": ["couch", "loveseat", "settee"]}, "original": {"room": "living room", "surface": "table", "aliases": ["dining table", "kitchen table", "desk"]}, "alts": []}, {"category": "remote control", "now": {"room": "bedroom", "surface": "table", "aliases": ["dining table", "kitchen table"]}, "original": {"room": "living room", "surface": "sofa", "aliases": ["couch", "loveseat", "settee"]}, "alts": []}] | {"selections": [{"category": "pen", "now": {"room": "living room", "surface": "sofa"}, "original": {"room": "living room", "surface": "table"}}, {"category": "remote control", "now": {"room": "bedroom", "surface": "table"}, "original": {"room": "living room", "surface": "sofa"}}], "abstain": false}The pen was moved from the dining table in the living room to the sofa in the living room, so it is currently on the sofa in the living room and was first on the table in the living room. The remote control was moved from the sofa in the living room to the dining table in the bedroom, so it is currently on the table in the bedroom and was first on the sofa in the living room. | place_set 1 |
논문 당시 정답 파일에는 이 녹화의 일부 위치에 방 이름이 비어 있었습니다(
"room": null). 모델은 방과 가구를 모두 맞혔는데 place 점수가 0이었습니다. v2는 방 이름을 채웠고 같은 답이 정답으로 채점됩니다. 정답을 그대로 읽어 주는 GT Action Log가 이 녹화에서 3/4 → 4/4가 된 이유입니다.사례 B. writer가 물체를 다르게 불러서 틀림 (Finding, 감자)
로봇이 실제로 옮긴 것은 감자입니다(조리대 → 식탁). 조리대에는 사과와 토마토도 같이 있었습니다.
질의: Pick up the potato I moved earlier 방법: R&R Agent
| 정답(채점 기준) | 모델 출력과 reasoning | 채점 | |
|---|---|---|---|
| 논문 당시 | [{"category": "irish potato", "room": "kitchen", "surface": "dining table", "hidden": false, "unlabeled": false}] | {"selections": [{"category": "irish potato", "room": "kitchen", "surface": "dining table"}], "abstain": false}The robot picked up the potato (noted as a tomato in some caption logs) from the kitchen countertop at t=450s and placed it on the dining table at t=510s (IMG 60). | loc 1 · place 1 · abstain 0 · parse_ok 1 · $0.0119 · 9.4s |
| v2 exo_f4 | [{"category": "potato", "room": "kitchen", "surface": "table", "aliases": ["dining table", "kitchen table", "desk"], "alts": [], "hidden": false, "unlabeled": false}] | {"selections": [{"category": "potato", "room": "kitchen", "surface": "countertop"}], "abstain": false}The potato was located on the kitchen countertop alongside other items (IMG 69, IMG 71) and remains there. | place_set 0 |
| v2 exo_f10 | [{"category": "potato", "room": "kitchen", "surface": "table", "aliases": ["dining table", "kitchen table", "desk"], "alts": [], "hidden": false, "unlabeled": false}] | {"selections": [{"category": "potato", "room": "kitchen", "surface": "table"}], "abstain": false}At t=510s, the potato was placed on the dining table in the kitchen, where it remains. | place_set 1 |
| v2 wrist_f10 | [{"category": "potato", "room": "kitchen", "surface": "table", "aliases": ["dining table", "kitchen table", "desk"], "alts": [], "hidden": false, "unlabeled": false}] | {"selections": [{"category": "potato", "room": "kitchen", "surface": "countertop"}], "abstain": false}The potato was seen on the kitchen countertop alongside the tomato and apple (IMG 69, IMG 71). While the robot picked up and moved the tomato to the dining table, the potato remained on the countertop. | place_set 0 |
MEM^R writer가 쓴 최종 기억 (같은 녹화, 조건별):
| 조건 | 기억 전문 |
|---|---|
| 논문 당시 (exo 4장) | I started in the kitchen/dining area, observing a wine bottle, tomato, bread, lettuce, bowl, and bottle on the dining table, plus a plate and apple on the countertop. In the living room, I saw a coffee table with a house plant and bowl, a yellow sofa, a floor lamp, a dark armchair, and a tv stand with a television and knickknacks. In the study room, I saw a desk with a basketball and chairs, and a dresser with a table lamp, pen, and remote control. I moved a yellow knickknack from the tv stand into a drawer of the dresser in the study room and closed it. Next, I picked up a butter knife from the armchair in the living room and placed it onto the dining table. Then, I took an irish potato from the kitchen countertop (which also held a cooking pan, sponge, soapdispenser, vase, and spoon) and placed it onto the dining table. |
| v2 exo_f4 | In the kitchen, a table initially held a wine bottle, tomato, bread, lettuce, a knife, a bowl, and a bottle; a countertop held a vase, apple, potato, plate, pan, soap bottle, spoon, sponge, and butter knife. In the living room, a round table held a house plant, bowl, and statue, a yellow sofa held a pencil, and a tv stand held a television and candle. In the bedroom, a table held a basketball and house plant, and a dresser (topped with a desk lamp, pen, and remote control) holds the yellow alarm clock inside its closed drawer. After placing the butter knife from the living room armchair onto the kitchen table, I picked up the apple from the kitchen countertop and placed it on the kitchen table. |
| v2 exo_f10 | In the kitchen, a table initially held a wine bottle, tomato, bread, lettuce, bottle, and a bowl with a spoon, while a countertop held a vase, tomato, potato, spoon, plate, soap bottle, pan, and sponge. Earlier, I moved a knife to a living room armchair, moved an alarm clock from the tv stand into the closed top drawer of a bedroom dresser, and returned the knife to the kitchen table. Most recently, after carrying the tomato from the kitchen countertop toward the doorway, I turned back into the kitchen and placed the tomato onto the kitchen table. |
| v2 wrist_f10 | I picked up a butter knife from the kitchen table (which held a wine bottle, tomato, bread, lettuce, bottle, bowl, another butter knife, and a chair; the countertop held a plate, pan, potato, tomato, apple, sponge, spoon, knife, vase, and soap bottle) and placed it on an armchair in the living room. In the living room (furnished with a sofa, armchair, floor lamp, a table with a house plant and bowl, and a tv stand with a television and candle), I picked up a plunger from the tv stand, carried it into the bedroom (which has a table with a basketball, chairs, a house plant, and a dresser with a desk lamp), placed it into the top drawer of the dresser (which contained a pen), and closed the drawer. I returned to the living room armchair, picked up the butter knife, and brought it back to the kitchen table. Then I picked up a tomato from the kitchen countertop and placed it on the kitchen table. |
정답 라벨은 맞습니다. writer가 옮긴 물체를 조건마다 apple(exo 4장), potato 또는 tomato(exo 10장), tomato(exo+wrist 10장)로 다르게 적었습니다. 채점 문제가 아니라 모델이 작은 물체를 구분하지 못한 것입니다. 한 녹화에서 조건에 따라 이름이 바뀌므로, 본 실행에서 조건 간 차이를 볼 때 이런 흔들림을 감안해야 합니다.
사례 C. 정답이 "vase"인 물체를 어떤 writer도 vase라고 부르지 않음 (Finding)
질의: Pick up the vase I moved earlier 방법: MEM^R
| 정답(채점 기준) | 모델 출력과 reasoning | 채점 | |
|---|---|---|---|
| 논문 당시 | [{"category": "vase", "room": "bedroom", "surface": "dresser", "hidden": true, "unlabeled": false}] | {"selections": [], "abstain": true}The memory summary records moving a knickknack, a butter knife, and an irish potato, but no vase was ever moved during the episode. | loc 0 · place 0 · abstain 1 · parse_ok 1 · $0.0031 · 2.6s |
| v2 exo_f4 | [{"category": "vase", "room": "bedroom", "surface": "dresser", "aliases": ["chest of drawers", "drawers", "chest", "cabinet"], "alts": [], "hidden": true, "unlabeled": false}] | {"selections": [], "abstain": true}The memory summary records moving a butter knife and an apple, but no vase was ever moved during the episode. Therefore, the request refers to an action that did not occur. | place_set 0 |
| v2 exo_f10 | [{"category": "vase", "room": "bedroom", "surface": "dresser", "aliases": ["chest of drawers", "drawers", "chest", "cabinet"], "alts": [], "hidden": true, "unlabeled": false}] | {"selections": [], "abstain": true}The memory summary details movements of a knife, an alarm clock, and a tomato. No vase was moved during the episode, making it impossible to identify 'the vase I moved earlier'. | place_set 0 |
| v2 wrist_f10 | [{"category": "vase", "room": "bedroom", "surface": "dresser", "aliases": ["chest of drawers", "drawers", "chest", "cabinet"], "alts": [], "hidden": true, "unlabeled": false}] | {"selections": [], "abstain": true}The memory summary records moving a butter knife, a plunger, and a tomato, but no vase was ever moved during the episode. | place_set 0 |
새로 확인한 문제: 질의는 "Pick up the vase I moved earlier"이고 정답은 침실 서랍장 안입니다. 그런데 옮겨진 물체는 꽃병처럼 보이지 않습니다. writer들은 이것을 knickknack(논문 당시), alarm clock(exo 4장·10장), plunger(exo+wrist)로 적었습니다. 한편 조리대에는 실제로 꽃병처럼 보이는 흰 물체가 있어, reader는 그쪽을 답하거나 "꽃병은 옮긴 적 없다"고 답합니다. 이 질의는 논문 당시에도 지금도 GT Action Log를 뺀 다섯 방법이 모두 틀립니다. 어휘나 채점을 고쳐서 해결되는 문제가 아니라 이 물체 모델의 생김새 문제입니다.
사례 D. Routine: 논문 당시 오답, 지금 정답
질의: Put the hand towel on the countertop away. 방법: R&R Agent
| 정답(채점 기준) | 모델 출력과 reasoning | 채점 | |
|---|---|---|---|
| 논문 당시 | {"store": [{"category": "hand towel", "room": "living room", "surface": "dining table", "instances": [{"room": "living room", "surface": "dining table", "uid": "diningtable_9c02a064c4b425976de085d4eaaa59cb_1_0_7"}]}], "leave": [], "expect": "store"} | {"selections": [{"category": "hand towel", "room": "bedroom", "surface": "desk"}], "abstain": false}Based on prior actions in memory, items of fabric/towels from countertops are consistently placed on the desk in the bedroom. | loc 0 · place 0 · any 0 · class 0 · abstain 0 · parse_ok 1 · writer v2 · $0.0120 · 23.8s |
| v2 exo_f4 | {"store": [{"category": "hand towel", "room": "living room", "surface": "table", "aliases": ["dining table", "kitchen table"], "alts": [], "instances": [{"room": "living room", "surface": "table", "aliases": ["dining table", "kitchen table"]}]}], "leave": [], "expect": "store"} | {"selections": [{"category": "hand towel", "room": "living room", "surface": "table"}], "abstain": false}The user instructed to put the clothes/cloths from the kitchen countertop onto the dining table, which is in the living room. | place_set 1 |
| v2 exo_f10 | {"store": [{"category": "hand towel", "room": "living room", "surface": "table", "aliases": ["dining table", "kitchen table"], "alts": [], "instances": [{"room": "living room", "surface": "table", "aliases": ["dining table", "kitchen table"]}]}], "leave": [], "expect": "store"} | {"selections": [{"category": "hand towel", "room": "living room", "surface": "table"}], "abstain": false}At t=0s, the user instructed to put clothes/cloths from the countertop onto the dining table. The robot previously moved cloths from the kitchen countertop to the table in the living room (t=90s, 180s, 270s, and seen on the table at t=300s in IMG 48). | place_set 1 |
| v2 wrist_f10 | {"store": [{"category": "hand towel", "room": "living room", "surface": "table", "aliases": ["dining table", "kitchen table"], "alts": [], "instances": [{"room": "living room", "surface": "table", "aliases": ["dining table", "kitchen table"]}]}], "leave": [], "expect": "store"} | {"selections": [{"category": "hand towel", "room": "living room", "surface": "table"}], "abstain": false}At t=0s, the user instructed to put clothes/cloths from the kitchen countertop onto the dining table, which is located in the living room. The robot subsequently placed the cloths onto the table in the living room. | place_set 1 |
논문 당시에는 식탁을 "bedroom의 desk"로 답했고, 지금은 세 조건 모두 "living room의 table"로 답합니다. 녹화 1개라 무엇 때문인지는 단정할 수 없습니다.
사례 E. Resume: 바뀌지 않은 오답
질의: You were interrupted earlier. Go back and finish what you were doing. 방법: MemER^R
| 정답(채점 기준) | 모델 출력과 reasoning | 채점 | |
|---|---|---|---|
| 논문 당시 | {"pending": [{"utterance": "Pick up four of the eggs from the dresser and place them into the basket.", "r": 3}], "finished_texts": ["Pick up three of the mugs from the dining table and place them in or on the tv stand.", "Hold on — go check whether the plate is still on the dining table in the living room first.", "go check whether the plate is still on the dining table in the living room"]} | {"unfinished": [{"task": "Pick up three of the mugs from the dining table and place them in or on the tv stand.", "remaining": 1}, {"task": "Pick up four of the eggs from the dresser and place them into the basket.", "remaining": 2}], "abstain": false}The robot was tasked with moving 3 mugs from the dining table to the TV stand. It moved 1 mug before checking the plate, and moved a 2nd mug after resuming, leaving 1 mug remaining before it was given the egg task. For the egg task, the robot was asked to place 4 eggs into the basket; as seen in the basket at the end of the episode, 2 eggs have been placed inside, leaving 2 eggs remaining. | resume_ok 0 · slot 0.0 · tasks_ok 0 · remaining_ok 0 · named_finished 1 · abstain 0 · parse_ok 1 · $0.0153 · 20.2s |
| v2 exo_f4 | {"pending": [{"utterance": "Pick up four of the eggs from the dresser and place them into the basket.", "r": 3}], "finished_texts": ["Pick up three of the mugs from the dining table and place them in or on the tv stand.", "Hold on — go check whether the plate is still on the dining table in the living room first.", "go check whether the plate is still on the dining table in the living room"]} | {"unfinished": [{"task": "Pick up three of the mugs from the dining table and place them in or on the tv stand.", "remaining": 1}, {"task": "Pick up four of the eggs from the dresser and place them into the basket.", "remaining": 4}], "abstain": false}The robot was tasked with moving three mugs from the dining table to the TV stand. It moved one mug before being interrupted to check the plate, then resumed and moved a second mug by t=255s before receiving a new command at t=260s. Thus, 1 mug remains for the first command. For the egg task, the robot was asked to place four eggs into the basket; it only transported the first egg to the basket and was stopped before successfully finishing and depositing any eggs, leaving all 4 eggs remaining. | resume_ok 0 |
| v2 exo_f10 | {"pending": [{"utterance": "Pick up four of the eggs from the dresser and place them into the basket.", "r": 3}], "finished_texts": ["Pick up three of the mugs from the dining table and place them in or on the tv stand.", "Hold on — go check whether the plate is still on the dining table in the living room first.", "go check whether the plate is still on the dining table in the living room"]} | {"unfinished": [{"task": "Pick up three of the mugs from the dining table and place them in or on the tv stand.", "remaining": 1}, {"task": "Pick up four of the eggs from the dresser and place them into the basket.", "remaining": 3}], "abstain": false}Command 1 required moving three mugs from the dining table to the TV stand. The robot placed the first mug, was interrupted to check the plate (which it did), resumed and placed the second mug, but was interrupted by Command 4 before picking up the third mug. Thus, 1 mug remains for Command 1. Command 4 asked to place four eggs into the basket; the robot successfully moved only 1 egg before being stopped, leaving 3 eggs remaining. | resume_ok 0 |
| v2 wrist_f10 | {"pending": [{"utterance": "Pick up four of the eggs from the dresser and place them into the basket.", "r": 3}], "finished_texts": ["Pick up three of the mugs from the dining table and place them in or on the tv stand.", "Hold on — go check whether the plate is still on the dining table in the living room first.", "go check whether the plate is still on the dining table in the living room"]} | {"unfinished": [{"task": "Pick up three of the mugs from the dining table and place them in or on the tv stand.", "remaining": 1}, {"task": "Pick up four of the eggs from the dresser and place them into the basket.", "remaining": 2}], "abstain": false}Command 1 required placing 3 mugs from the dining table onto the TV stand. The robot placed the first mug (around t=150s) and the second mug (around t=261s), but was interrupted by Command 4 before fetching the third mug, leaving 1 mug remaining. Command 2 was a quick check interruption that was completed. Command 4 required moving 4 eggs from the dresser to the basket; by t=348s, the robot had placed 2 eggs into the basket, leaving 2 eggs remaining. | resume_ok 0 |
MemER^R은 논문 당시와 같은 방식으로 틀립니다(이미 끝난 머그 과제를 남은 일로 셈).
4이전에 비해 확인된 개선과 확인하지 못한 것
| 항목 | 이 실행에서 확인한 것 |
|---|---|
| 어휘 | reader가 출력한 가구 단어 308개가 전부 목록 안의 단어(table 164, countertop 67, sofa 43, dresser 13, desk 10, tv stand 6, side table 3 …). reasoning에서는 "dining table"이라고 쓰면서도 출력은 "table"로 냈습니다. 버려진 단어 0. |
| 정답 라벨 | 방 이름이 비어 있던 Restore 정답이 채워져, 맞은 답이 오답 처리되던 사례가 사라짐(사례 A). |
| 입력 조건 | 세 writer가 같은 이름 목록을 받고, 30초 창의 프레임이 균등 간격이며, Routine 질의 프레임이 실제 녹화 끝임을 프롬프트 전문에서 확인. |
| 정확도 | 판단 불가. 녹화 1개에서는 reader 흔들림(1–2문항)과 구분되지 않습니다. |
| exo 10장·wrist의 효과 | 판단 불가. 이 녹화에서는 장수를 늘리거나 wrist를 넣어도 writer가 감자를 다른 이름으로 적었습니다(사례 B). |
| 남은 문제 | 생김새가 이름과 다른 물체(사례 C의 vase). 물체 모델별로 점검이 필요합니다. |
5다음
- 본 실행(job 190516)은 2시간 돌고 멈춘 상태입니다. 약 $42를 썼고 끝난 녹화는 저장되어 있습니다. 이어서 돌릴지는 결정이 필요합니다.
- vase처럼 이름과 생김새가 다른 물체 모델이 더 있는지 점검합니다(정답 물체를 writer들이 어떤 단어로 적었는지 논문 본 런 기억에서 집계).