Plan — Run RoboMME-trained MemER LoRA on RoboMME (VLM-only inference + viz)

배경/목적 (왜)

Sister repo ~/repos/Robotics/memer/ 에서 공식 Anthropic MemER 의 dusting 체크포인트로 RoboMME 16개 task ep0 sweep 을 돌렸음. 결과는 모두 OOD — 모델이 "place the red block on the * shelf" 같은 dusting vocabulary를 RoboMME 시각 입력에 강제로 매핑함 (sister repo의 eval_outputs/all_tasks_ep0_index.md 참조).

이 repo는 RoboMME 자체에 학습된 MemER LoRA (Yinpei/vlm_subgoal_predictor 의 memer/grounded_subgoal/checkpoint-1300.zip) 를 실제 RoboMME data에 inference로 돌려서 도메인-매칭된 출력을 얻는 게 목표. VLA는 안 붙임.

작업 내용 (어떻게)

순서를 따르세요. 각 단계 끝에 체크박스 채우면 다음 세션이 어디까지 했는지 바로 압니다.

Phase 0 — 코드 이해 (no install yet)

• [x] Read src/mme_vla_suite/dataset_builder/build_vlm_subgoal_dataset_memer.py — single-image front_rgb (256×256), grounded subgoal JSON {"current_subtask": "...<|box_start|>(x,y)<|box_end|>...", "keyframe_positions": [int]}, ms-swift role tags, inline [<image>, ...] placeholders.
• [x] Read examples/robomme/subgoal_prediction/qwenvl/api_memer.py — loads LoRA via swift.llm.PtEngine(model_id_or_path='Qwen/Qwen3-VL-4B-Instruct', adapters=[adapter_path], attn_impl='flash_attention_2'), single call() does inline JSON parse + _parse_subgoal_for_vla (box → scaled coords). recent context = last 8 frames at stride 2.
• [x] Read examples/robomme/subgoal_predictor.py — MemERSubgoalPredictor adds frames per step, calls api.call() per get_subgoal. memory state lives inside Qwen3VLModelMemER.key_frame_paths. Different cluster distance (dist=8) than sister repo (merge_distance=5) but same algo idea. Use this repo's mechanism (Path B).
• [x] Diff vs Anthropic MemER prompt contract — written to notes-prompt_format_diff.md. Multiple axes diverge (camera count, image layout, output box tokens, swift vs HF). Decision: Path B.

Decision gate: ✅ Path B — wrap this repo's Qwen3VLModelMemER directly. Sister repo's MemERDeploymentPolicy cannot be reused because the LoRA was trained against ms-swift templates with grounded <|box_start|> tokens and a single front-view camera, all of which differ from memer_eval/contract.py.

Phase 1 — Env + LoRA download

• [x] Conda env: reused /home/nas_main/taewoongkang/conda_envs/envs/memer/. Existing peft was missing. Installed ms-swift==3.12.6 (downgraded from 4.1.3 because 4.x removed swift.llm/PtEngine and the LoRA was trained on 3.x ms-swift templates). Side effects: transformers 5.6.2→4.57.6, peft→0.18.1. Also added imageio, opencv-python-headless (NOT opencv-python — server has no libGL.so.1).
• [x] Download LoRA to /home/nas_main/taewoongkang/ckpts/robomme-memer-lora-checkpoint-1300/ (note: zip extracts into a nested checkpoint-1300/ subdir, so the actual adapter path passed to driver is /home/nas_main/taewoongkang/ckpts/robomme-memer-lora-checkpoint-1300/checkpoint-1300/).
• [x] Verified artifacts: adapter_config.json (base = Qwen/Qwen3-VL-4B-Instruct, r=16, α=32, target = language model linear modules), adapter_model.safetensors, args.json (template=qwen3_vl, attn_impl=sdpa, max_length=3200, bf16, 2 epochs).
• [x] Bonus — pre-downloaded base Qwen/Qwen3-VL-4B-Instruct (8.3 GB) via hf download to ~/.cache/huggingface/hub. This was necessary because ms-swift defaults to ModelScope for Qwen/... IDs and ModelScope was throttled to ~230 kB/s (90 min timeout hit). With HF cache populated and USE_HF=1, download is instant. Run with USE_HF=1 CUDA_VISIBLE_DEVICES=<N> from now on.

Phase 2 — Adapter for VLM-only inference

Pick one of two paths based on Phase 0's decision:

• Path A — reuse sister repo adapter. If the prompt format and memory mechanism match Anthropic MemER, copy ~/repos/Robotics/memer/scripts/eval_robomme_subtask.py here and modify only the model load path: load base Qwen/Qwen3-VL-4B-Instruct (already cached on HF), then peft.PeftModel.from_pretrained(base, lora_dir) and merge or keep PEFT wrapped. Most of the code stays.

• Path B — wrap this repo's subgoal_predictor.py. If formats differ, write a thin driver that decodes RoboMME mp4s, calls this repo's MemER predictor frame by frame, and dumps a predictions.jsonl in the same schema the sister repo's renderer expects (see ~/repos/Robotics/memer/scripts/render_robomme_subtask_video.py for the required keys). The renderer is reusable verbatim — feeding it a different jsonl just gives a new mp4.

• [x] Wrote driver scripts/eval_robomme_memer_lora.py — wraps Qwen3VLModelMemER (single front-view, ms-swift), emits jsonl in sister repo's renderer schema. Two minor api_memer.py fixes while integrating: drop dead from h5py._hl.dataset import sel import (line 3, sel never used), and add empty-dict guard in merge_key_frame_paths() (latent IndexError when first inference returns keyframe_positions=[]). Also flipped hardcoded attn_impl='flash_attention_2' → 'sdpa' to match training.

• [x] Ran on BinFill ep0 — 550 frames, stride 5 → 108 calls in 178 s (1.65 s/call) on a B200 with sdpa. Output: eval_outputs/robomme_BinFill_ep0/{predictions.jsonl,run_info.json}.
• [x] Sanity-check passes — top subtasks: "pick up the first red cube at \<104, 115>" (12×), "pick up the second red cube at \<58, 161>" (6×), "put it into the bin at \<118, 174>" (32×), "press the button at \<58, 92~94>" (46× across 3 coord variants). Vocabulary is exactly RoboMME-BinFill — no shelf/block dusting contamination. Final memory FIFO has 6 keyframes [87, 105, 130, 174, 349, 430] — clustering/FIFO mechanism works.

Phase 3 — Visualization

The sister repo's renderer takes a predictions.jsonl plus the RoboMME LeRobot path and produces the side-by-side mp4 with the memory thumbnail strip. Reuse it directly:

• [x] Rendered via sister repo's scripts/render_robomme_subtask_video.py (default camera_keys = wrist+ext1 stack works fine — visualization purely cosmetic, model saw ext1 only). Output: eval_outputs/robomme_BinFill_ep0/visualization.mp4 (550 frames @ 10 fps, 1100×840, H.264). 6 memory thumbnails pre-rendered.
• [x] Quantitative comparison (BinFill ep0): | metric | dusting (OOD) | RoboMME-MemER LoRA | |---|---|---| | rows | 110 | 108 | | unique subtasks | 8 | 14 (mostly coord variants of 4 semantics) | | top vocabulary | "place red block on top shelf" 60× | "put it into the bin" 32× | | in-domain match | none (kitchen / shelf vocab) | full (cube / bin / button) | | grounded coords <x,y> | absent (text-only) | present on every prediction |

Phase 4 — All 16 tasks

• [ ] Mirror ~/repos/Robotics/memer/scripts/eval_robomme_all_tasks.py here as scripts/eval_robomme_memer_lora_all_tasks.py. Same structure — load LoRA once, sweep all 16 RoboMME tasks ep0, write per-task predictions.jsonl + run_info.json + predictions_summary.md, render visualization.mp4, then build eval_outputs/all_tasks_ep0_index.md.
• [ ] Symlink-collect to eval_outputs/all_tasks_visualizations/ the same way (<TaskName>_ep0.mp4 symlinks).

Phase 5 — Comparison report

• [ ] Write claude/<YYMMDD>/analysis-dusting_vs_robomme_memer.md contrasting the two checkpoints on the same 16 episodes: subtask vocabulary alignment, keyframe selection density, memory final state size. Quote concrete numbers (parse rate, top subtask ×counts, unique subtask count, etc.).

결과 (수치)

(empty — fill in as work progresses)

의미 (Takeaway)

(empty)

보완점/다음 (Next)

(empty)

Checklist legend

- [ ] open · - [x] done · - [~] partial / blocked (write why on the same line)