ICLR Research Council — Recall & Roll (R&R-Bench)

Summary

The paper introduces R&R-Bench, a simulated MolmoSpaces benchmark with 300 scripted contexts, 228 houses and 1,772 template-generated queries. The queries fall into four families: Finding, Restore, Resume and Routine. The benchmark has three evaluation conditions: memory QA, execution with ground-truth instructions, and execution with recalled instructions. Five memory strategies and a GT action-log reference are compared inside one two-stage VLM+VLA harness. The paper then proposes R&R Agent, an append-only, tag-indexed store of text and frames that also supplies retrieved goal images to a goal-conditioned MolmoBot. The central claims are that text and visual memory are complementary and that R&R Agent gives the best memory-based performance.

Strengths

• The three-condition design separates recall errors from execution errors (Tables 2–4). The honest Multi-Routine result, 60% QA accuracy but 0% execution (p. 10), shows why this separation matters.
• The validation pipeline is careful: reference answers are read from recorded state, there are leak checks, and there are Restore displacement and occupancy constraints (A.3).
• The appendix states what the analyses do not show (C.1, C.5, C.7), and the paper reports its own non-significant p-values (C.6).

Weaknesses

1. The storage budgets are unequal, so the headline comparison is unfair. MEM^R is capped at "at most 12 lines" of rewritten summary (App. D), and MemER^R rebuilds a fixed 8-frame memory. R&R Agent keeps everything and retrieves 8 items at query time. C.8 confirms that the interventions differ by method. The Finding/Restore gains can therefore be explained by capacity rather than design, and the "complementary evidence" insights (§5.1, points 1 and 6) are partly artifacts of the budgets chosen for the baselines.
2. The goal-image contribution is unsupported end-to-end. In Table 6 the Replace@15 result is 19/93 vs 18/93 for the max-area frame, and against no image the paper reports p = 0.0525. Nine runs used reduced sensors. End-to-end Restore success is 29/102 with the selected image vs 31/102 without (p = 0.86, C.6). Yet the abstract and contribution 3 present goal-conditioned execution guided by retrieved evidence as part of the method. In addition, the GT target render (17.2%) scores below the heuristic frame, and a supplied reference image reaches 63.8%@10 cm in E.3 but only about 11.8% here. Neither discrepancy is explained.
3. The execution conditions mostly measure MolmoBot, not memory. The fine-tuned policy opens fixtures 5.6% of the time and places within 10 cm 17.7% of the time (E.2). In Table 3, success rates are within about 6 pp across methods, R&R Agent does not beat No Memory on Finding (36.0 vs 36.7), and Uniform Frames is best. §4.1 promises success conditioned on a correct QA answer, but it is not reported. Insight 4 rests on a 2.6-pp difference (35.3 vs 32.7).
4. The ranking claims have no inferential support. With n = 161 for Resume and about 38–40 queries per Routine Observed cell, several claimed orderings are within binomial noise: Table 4 Finding 24.5 vs 23.0, and Insight 2 resting on roughly 6 questions. The unit of dependence is also ignored, since multiple queries share a context. (score does not reflect this since the experiment may be ongoing)
5. Answerability is not established. Cells such as Restore M-Obs are at most 1.8% for every method, and the GT log scores 0.0 there. The evidence threshold is only ≥50 px. Query-time visibility is labeled but not reported per cell. With no human baseline, a benchmark defect cannot be told apart from genuine difficulty. (score does not reflect this since the experiment may be ongoing)
6. The same model, Gemini Flash, writes memory for all readers, which confounds the reader comparison. Execution uses only Flash. (score does not reflect this since the experiment may be ongoing)
7. Novelty is thin. It amounts to a new cell in Table 1's feature matrix, scored by the authors themselves. The requests come from templates and the contexts come from scripted A*/IK routines.

Questions

1. Do the Table 2 and Table 4 rankings hold at equal storage (baselines given unbounded storage plus the same retrieval, or R&R Agent restricted to the same capacity)? If yes, my rating goes up; if the gap closes, it goes down.
2. What is the success rate conditioned on a correct QA answer, per method (promised in §4.1)?
3. What are the bootstrap CIs, clustered by context, for Tables 2–4?
4. What is human accuracy on the Observed and Multi cells where every method is near zero?
5. Why does the GT render underperform a heuristic frame, and why is there a 5× gap relative to E.3?

Reproducibility & Statements

Reproducibility, Ethics and AI-use statements are all present. Code and data release is promised, not provided. The Reproducibility Statement says §4.1 gives the "exact scoring rules", but those are in App. B. The main text runs to 13 pages because of the retained "PREVIOUS SECTION 5 DRAFT" and the "[TW]" marker. As submitted, it violates the 9-page limit and is a desk-reject risk. This is separate from the scientific assessment below.

Flag For Ethics Review

No ethics review needed.

Scores

• Soundness: 2 / 4 — fair
• Presentation: 1 / 4 — poor
• Contribution: 2 / 4 — fair
• Rating: 4 / 10 — marginally below the acceptance threshold
• Confidence: 4 / 5

What Would Change My Score

Three pieces of evidence would raise the rating to 6: - equal-capacity baseline comparisons, - QA-conditioned execution success, - context-clustered CIs showing that R&R Agent's lead survives.

If the gains vanish at equal storage, or if the goal-image claim is kept despite the p = 0.86 end-to-end result, I would lower the rating to 2.

Summary

The paper introduces R&R-Bench, a simulated household memory benchmark built on MolmoSpaces. It has 300 scripted experience contexts (43 h in total) and 1,772 templated queries in four families: Finding, Restore, Resume and Routine. Queries are tagged by scope (Single/Multi), experience source (Manipulated/Observed) and target visibility (Concealed/Unconcealed). There are three evaluation conditions: memory QA, execution with ground-truth instructions, and execution with recalled instructions. Inside one two-stage VLM+VLA framework, the authors compare reimplementations of MEM and MemER with simple baselines. They then propose R&R Agent, an append-only text+frame memory with tag-based retrieval that passes retrieved frames to a goal-conditioned VLA. The claimed novelty (Table 1) is that no prior benchmark combines long context, navigation plus low-level manipulation, online memory, multiple evaluation modes and rule inference.

Strengths

• The combination of axes is a real gap. Existing manipulation-memory benchmarks (RoboMME, SAM2Act's MemoryBench, RoboMemArena) do not include navigation, and navigation/QA benchmarks (FindingDory, WorldLines) do not include low-level control (§2, p.3–4).
• Separating retrospective memory from execution-state memory is a useful protocol idea (§4.1, p.6). Comparing Tables 2 and 3 shows that QA rankings do not carry over to execution (MemER^R 44.1 vs MEM^R 34.0 on Finding QA, but 32.7 vs 35.3 in execution SR). No single-mode benchmark can show this.
• The Manipulated/Observed axis covers incidental observation, which most manipulation-memory suites leave out.

Weaknesses

1. Table 1 is not fair to prior work. (a) MEMOBench (Sun et al., 2026) is cited twice in the Introduction, including for the paper's own framing that success alone does not show where memory fails, yet it is missing from Table 1. It is plausibly contemporaneous (arXiv 2609.x), but it should be placed in the table. (b) FindingDory's experience videos contain pick-and-place interactions, and it pairs a high-level VLM with a low-level policy. The "×" for navigation plus manipulation needs explanation. (c) The "R" column is claimed only for R&R-Bench. However, RoboMME's procedural/imitation-style suites, and preference-inference work such as Housekeep (Kant et al., 2022) and TidyBot (cited but not in the table), come close to "infer a rule from examples and apply it." (d) The columns seem chosen so that only the proposed benchmark gets every tick. The table omits scale, real vs. simulated data and query count, where competitors may compare better.
2. Missing citations and positioning. The paper should discuss: - EQA and OpenEQA (episodic-memory EQA); only Datta et al. 2022 is cited. - Ego4D Episodic Memory (NLQ, "where did I leave X"). - Long-video memory QA: HourVideo, EgoSchema, EgoLife. - MIKASA-Robo (memory-intensive manipulation). - RoboCerebra (long-horizon System-2 VLM plus System-1 VLA benchmark). This is the closest threat to the "two-stage memory evaluation" framing. - HomeRobot OVMM and BEHAVIOR-1K (navigation plus manipulation). - ProcTHOR/Housekeep-style rearrangement. Restore is essentially Visual Room Rearrangement's walkthrough/unshuffle recast as memory QA, so this lineage needs explicit discussion. - ALFRED/TEACh-style interrupted instruction following, for Resume.
3. The R&R Agent contribution is thin relative to prior work. Append-only, tagged, time-stamped memory with retrieval is standard embodied retrieval-augmented memory: ReMEmbR, Embodied-RAG, KARMA, 3D-Mem, and MemGPT-style agents. None of these is cited or compared against. Conditioning a VLA on retrieved goal images is close to goal-image-conditioned policies such as GCBC and SuSIE. Both "isn't this just X" questions go unanswered.
4. Reimplementation fidelity. MEM^R and MemER^R are adapted, not the original systems, and MEM is only a technical report. Claims about "textual vs. visual memory" (§5.1, Insights 1–4) risk being claims about these particular reimplementations.
5. One-stage memory VLAs (MemoryVLA, HAMLET, MemoAct) are cited but not evaluated (score does not reflect this since the experiment may be ongoing).
6. Single runs with no confidence intervals, and one memory writer (score does not reflect this since the experiment may be ongoing).
7. The main text runs to page 13 and still contains the "PREVIOUS SECTION 5 DRAFT" block and the "[TW]" marker, which puts the submission at desk-reject risk.

I cannot fully verify concurrent work past my knowledge cutoff in this fast-moving subfield. Under ICLR policy, work published within about 4 months of the deadline counts as contemporaneous, so MEMOBench, RoboMemArena and LongAct (all 2026 arXiv) should be positioned but not held against novelty.

Questions

1. Why is MEMOBench not in Table 1? If it covers process-level diagnosis over navigation plus manipulation, the "multiple evaluation modes" claim weakens (lowers Contribution).
2. Please justify FindingDory's "×" for navigation plus manipulation, and the R-exclusivity claim against RoboMME and Housekeep. A corrected table that still leaves R&R-Bench as the only benchmark covering every column would raise my score.
3. How does R&R Agent differ from ReMEmbR and Embodied-RAG? A head-to-head comparison would raise my assessment of the agent contribution.
4. Did the original MEM and MemER authors' code or prompts inform the reimplementations, and how faithful are they?

Reproducibility & Statements

There is a Reproducibility Statement that promises release of contexts, queries and the evaluation harness; no code is available yet. There are also an Ethics Statement and an AI-use statement. The main text is over the 9-page limit.

Flag For Ethics Review

No ethics review needed.

Scores

• Soundness: 3 / 4 — good
• Presentation: 2 / 4 — fair
• Contribution: 3 / 4 — good
• Rating: 4 / 10 — marginally below the acceptance threshold
• Confidence: 4 / 5

What Would Change My Score

I would move to a 6 with three changes: - a corrected Table 1 that includes MEMOBench, RoboCerebra, MIKASA-Robo and Housekeep, with justified entries, in which R&R-Bench still covers every column; - a comparison of R&R Agent against a standard embodied retrieval-memory baseline; - a manuscript that fits the page limit.

I would lower my score if a pre-deadline benchmark already covers navigation plus low-level manipulation with rule inference and decoupled evaluation modes.

Summary

The paper introduces R&R-Bench: 300 scripted MolmoSpaces contexts totalling 43 h, with 1,772 template-generated queries in four families (Finding, Restore, Resume, Routine). It is evaluated under three conditions: memory QA, execution with ground-truth instructions, and execution with recalled instructions. Five memory strategies are compared inside one two-stage VLM+VLA harness, all using a single Gemini 3.7 Flash writer and three readers. The authors then propose R&R Agent, which stores tagged, append-only text and frames and uses retrieved frames as goal images. The benchmark design is careful, but the statistics do not support many of the comparative claims.

Strengths

• The validation pipeline is concrete and auditable (App. A.3). It includes answer-on-camera ≥50 px, same-name distractor checks, planner-verified executability, a temporal-leak check, and reference answers read from recorded state rather than from the plan.
• The appendix is unusually frank about its limits. C.1 and C.7 disclaim causal attribution, Tables 11–12 say their groups are not comparable, and C.6 reports p=0.0525 and a null end-to-end result.
• The benchmark separates methods clearly at the coarse level. The MEM^R/MemER^R ordering by task average holds for all 3 readers (Table 2), and the GT Action Log's Manipulated/Observed split (85.5 vs 14.0) confirms that the Observed axis tests something action logs do not contain.

Weaknesses

1. The main text contradicts the appendix on goal images. §5.3.4 (retained draft) says the selected images "achieve the highest placement accuracy." Table 6 has n=93: R&R gets 19/93 vs Max-area 18/93, a one-trial gap, and 19/93 vs 9/93 without an image gives p=0.0525. End-to-end, selected images are slightly worse (29/102 vs 31/102, p=0.86; C.6). Nine R&R runs also used reduced sensor inputs. Insight 5 is not supported.
2. Eligible subsets are undisclosed and not constant. - Back-computing from Table 2 gives Finding n=717 (not 722) and Routine n=310 (not 314). - Restore S-Man is 215 for Flash but 212 for Qwen and Gemma. Finding S-Conc denominators switch between 48 and 49 across strategies for the same reader. - Table 3 SRs cannot be reproduced from 717, 574 or 310. They fit only much smaller denominators (Finding ≈150, Restore ≈148, Routine ≈123), yet the caption says "all runs." - Table 4 Finding cells use n=50 and 122, against 49 and 119 in QA.
3. The claims are not tested against noise. - With Finding at n≈150, the Table 3 SR spread (32.7–38.7) is within about 1.5 SE. Insight 4 ("text wins execution") rests on gaps of 2.6 and 3.4 pp. - Insight 2 comes down to Routine S-Obs 13/38 vs 7/38. - "Best on average" in Table 4 includes margins of 1.5 pp (Finding) and 4 queries of 161 (Resume). - There are no paired tests (e.g. McNemar), no bootstrap CIs, and no repeated decoding runs. (score does not reflect this since the experiment may be ongoing)
4. The baseline comparison confounds budget with design. MEM^R is capped at 12 lines and MemER^R at 8 frames. R&R stores an unbounded append-only log and retrieves from it once the query is known. Table 5's "text-only ≥ MEM^R" therefore mixes storage capacity with design, and Table 13 changes different quantities for each method. Tokens, API calls and wall-clock per method are not reported, and nothing checks the reimplementations against published MEM/MemER behaviour.
5. Single writer. All readers consume memory written by Flash, so the "three VLMs" results show reader robustness to Flash-written memory, not memory construction by those models. (score does not reflect this since the experiment may be ongoing)
6. Leakage and priors. No Memory scores 17–28% on Finding S-Obs across readers. A.3 labels query-time visibility but says it is "reported but not used for exclusion," and I cannot find that breakdown anywhere. Separately, when a scripted grasp fails, scoring is downgraded to "navigation-only" (A.1), so part of the benchmark uses a more lenient contract that is not stated in the tables.
7. Text metrics have no table. §5.3.2 cites repetition (38.9→8.7%, 9.0 vs 12.1%) and overrun, but Table 3 reports only Ins./ED/SR. The retained block refers to Rep./Ovr. columns that no longer exist. The Routine twin-pair metric (A.2) is defined but never reported.
8. The GT Action Log reference is ill-defined for Observed queries (e.g. Routine S-Obs 2.6%). It cannot serve as an upper bound, and no human or oracle-memory ceiling is given. (score does not reflect this since the experiment may be ongoing)

Questions

1. What exact n does each cell of Tables 3 and 4 use, and why do denominators differ across readers and strategies? If this is explained by harmless run failures, Soundness would rise.
2. Do paired tests show significant differences between R&R and MemER^R/MEM^R on the Table 2 and Table 4 averages? If the differences are not significant, the "complementary representations" narrative weakens.
3. What are the per-query token and image budgets for each method, and what happens when R&R is matched to MEM^R's budget?
4. How does QA accuracy change after excluding queries whose answer is visible at query time?
5. What is the decoding temperature, and how much do results vary across repeated runs?

Reproducibility & Statements

Reproducibility, Ethics and AI-use statements are all present. The Reproducibility Statement says §4.1 gives "exact scoring rules," but §4.1 only summarizes them. The headline numbers depend on proprietary Gemini 3.7 Flash as the only writer, with no model snapshot or decoding parameters given. Release is promised, not provided. The paper is overlength (13 pages) and still contains the retained draft block and a "[TW]" marker, which is a desk-reject risk.

Flag For Ethics Review

No ethics review needed.

Scores

• Soundness: 2 / 4 — fair
• Presentation: 2 / 4 — fair
• Contribution: 3 / 4 — good
• Rating: 4 / 10 — marginally below the acceptance threshold
• Confidence: 4 / 5

What Would Change My Score

To move to 6, I would need: - per-cell denominators with fixed eligible sets across methods and readers; - paired significance tests on the Table 2 and Table 4 averages; - a budget-matched MEM^R/MemER^R comparison; - rewriting Insight 5 and §5.3.4 to match C.6.

If the confirmatory analysis shows the R&R vs MemER^R/MEM^R gaps are within noise, I would lower my score.

Summary

The paper introduces R&R-Bench, a simulated household benchmark built on MolmoSpaces. It has 300 recorded contexts, 43 h of experience and 1,772 queries across four task families: Finding, Restore, Resume and Routine. Queries are further split by scope (Single/Multi), experience source (Manipulated/Observed) and target visibility (Concealed/Unconcealed). Each query is evaluated under three conditions: memory QA, execution with ground-truth instructions, and execution with recalled instructions. Five memory strategies are compared inside one two-stage VLM+VLA framework. The paper then presents R&R Agent, which uses append-only, tagged text and frame memory plus retrieved goal images. The central finding is that text and visual memory have complementary strengths (Table 2) and that correct recall does not guarantee task success (Tables 3–4).

Strengths

• The four-family structure is easy to remember and easy to pitch. Figure 2 has one row per memory type with a one-line request each ("Put this away as usual."), and a reader grasps the benchmark in 10 seconds.
• The three evaluation conditions are a clean diagnostic idea. They separate recall from execution, which Table 1 shows no prior benchmark does together with navigation and manipulation.
• The paper is honest about scope. Caveats appear in the text itself: tag contribution is not isolated (p. 10), correctly recalled subsets differ across methods (p. 10), and GT render < max-area frame (Table 6).
• Figure 3 is an effective mechanism figure. It shows how three queries over one context need different evidence.

Weaknesses

1. The paper is overlength and still in draft state, which risks desk rejection. The block "PREVIOUS SECTION 5 DRAFT (RETAINED FOR REVISION)" (pp. 10–13) duplicates §5.3 almost paragraph for paragraph, and the conclusion opens with "[TW]". The main text ends on p. 13 against a 9-page limit. The whole block must be deleted.
2. Metric names are inconsistent between the table and the text. Table 3 reports Ins./ED/SR, but §5.3.2 (p. 9) still discusses "repetition (9.0% vs 12.1%)" and "mean overrun", which Table 3 no longer contains. In addition, Table 3's caption is an 11-line scoring algorithm; move it to Appendix B.
3. The paper cannot decide whether it is a benchmark paper or a method paper. The title and primary area say benchmark, but the abstract's last third, contribution 3 and Figure 4 sell R&R Agent. Its gains are modest: it does not beat No Memory on Finding SR (36.0 vs 36.7, Table 3), and it is +1.0 over the max-area frame in Table 6. Recast R&R Agent as a "strong reference baseline motivated by the diagnostics" and make the benchmark findings the headline.
4. The §5.1 insights are weakly sourced and in the wrong order. Insight 5 cites Table 6, which appears two pages later. Insight 6 cites no table at all. Insight 3 claims drops "across all four families" but points to Table 4, not Table 2. Cut the list to three evidence-backed claims, each with its number inline.
5. Figure 1 is spent on dataset statistics. Swap Figures 1 and 2. The page-1 teaser should show one context timeline, the three evaluation conditions, and a headline result such as "QA 60.6% → execution 0% on Multi-Manipulated Routine". That gap is the paper's most striking finding, and it is currently buried on p. 10.
6. Terminology drifts. The text alternates "execution-state memory" and "execution state memory", and "ground truth" and "ground-truth". "System 1/2" is never defined before p. 2. The Table 4 header has a typo ("M-/Obs."). The name "MemoryBench (Fang et al., 2025)" does not match its bibliography entry, which is SAM2Act. MEMOBench is cited in the introduction but missing from Table 1.
7. No seeds or confidence intervals, and execution results use only the Flash reader (score does not reflect this since the experiment may be ongoing).

Questions

1. Can the non-duplicated main text fit in 9 pages with Tables 2–4 kept in the body? If so, my Presentation score rises.
2. Is R&R Agent a contribution or a baseline? If the latter, would you reframe contribution 3 accordingly? A clear yes raises my Rating.
3. Which single sentence is the elevator pitch? I suggest: "Memory QA accuracy does not predict embodied success; text and vision are complementary."
4. Why does the GT target render underperform the max-area frame (Table 6)? An unexplained result like this weakens the goal-image story.
5. Does "update memory" in the title have evidence beyond Resume Ins./ED in Table 3?

Reproducibility & Statements

The Reproducibility, Ethics and AI-use statements are all present. The Reproducibility statement says Section 4.1 has "exact scoring rules", but those rules are actually in Appendix B. The code and benchmark are promised but not released.

Flag For Ethics Review

No ethics review needed.

Scores

• Soundness: 3 / 4 — good
• Presentation: 2 / 4 — fair
• Contribution: 3 / 4 — good
• Rating: 4 / 10 — marginally below the acceptance threshold
• Confidence: 4 / 5

What Would Change My Score

A clean 9-page version would move me to 6. It needs the retained draft and "[TW]" removed, metric names unified, Figure 1 rebuilt around the recall-versus-execution gap, and R&R Agent reframed as a reference baseline. If the submitted version keeps the retained block, I would drop to 2 on compliance grounds.

Summary

The paper introduces R&R-Bench: 300 simulated household contexts in MolmoSpaces (about 43 hours of experience) and 1,772 templated queries in four families (Finding, Restore, Resume, Routine). Each query is labelled Single/Multi, Manipulated/Observed and Concealed/Unconcealed. Evaluation has three conditions: memory QA, execution given ground-truth (GT) instructions, and execution given recalled instructions. All memory strategies run inside one two-stage VLA stack, with Gemini 3.7 Flash writing the memory, three different VLM readers, and MolmoBot as the low-level policy. From the baseline comparisons the paper concludes that visual memory is better for spatial recall and text memory is better for tracking progress and rules. It then proposes R&R Agent, an append-only text-plus-frame memory with tag-based retrieval, which leads task-average QA and end-to-end success.

Strengths

• Fills a real gap (Table 1). It combines long context, navigation, low-level manipulation and rule inference. The Routine family (inferring a storage rule from examples) is new and interesting for the agent and long-context communities.
• The QA track is separate from execution. In principle, VLM and retrieval researchers could use the memory-QA track without a robot stack. That is the most likely path to wide adoption.
• Findings that matter beyond robotics. The GT Action Log control shows that a perfect action log still fails on passively observed evidence: Finding S-Obs is 8.7–14.0% (Table 2). This bears on anyone building agent memory from action traces.
• Careful validation (App. A.3): temporal-leak checks, query-time visibility labels, and randomized rules to block shortcuts from common priors.

Weaknesses

1. The execution tracks show a floor effect, not headroom. In the full pipeline (Table 4), Restore averages 5.7% and every Multi Restore/Routine cell is about 0. Even GT Action Log gets only 8.2% on Restore and 17.4% on Routine. With GT instructions (Table 3), Finding success is 32.7–38.7% for every method, and R&R Agent (36.0) is level with No Memory (36.7). These tracks mostly measure MolmoBot, not memory. As a result they cannot rank memory methods today.
2. The headline modality finding may be an artifact of the setup. Frame-based methods get 8 images for contexts of about 9 minutes. Halving that budget drops Uniform Frames on Finding from 35.7 to 21.6 (App. C.8), so "visual is better for X" depends on the budget. There is also no native long-context video baseline (the whole subsampled context fed to Gemini). That is the first baseline the ICLR long-context multimodal community will ask for. (score does not reflect this since the experiment may be ongoing)
3. Results depend on specific closed models. One closed writer (Gemini 3.7 Flash) serves every reader, and the rankings move a lot by reader. For example, Qwen with MemER^R gets 6.1 on Routine and Uniform Frames gets 3.5. The benchmark's long-term value depends on results that do not rest on one model version that will be deprecated. (score does not reflect this since the experiment may be ongoing)
4. Saturation risk and limited language diversity. Requests come from "a few string templates per family" (A.2). A frontier model given the full video could plausibly saturate Finding QA quickly, and there is no human reference point to judge by. (score does not reflect this since the experiment may be ongoing)
5. R&R Agent adds little as a method. Tagged, append-only retrieval is standard RAG practice. The goal-image gain is 20.4 vs 19.4 for the max-area-frame baseline (Table 6). Against no image it is p=0.0525, and online success is 29/102 vs 31/102 (C.6).
6. No seeds or confidence intervals. Many subtype cells come from small subsets. (score does not reflect this since the experiment may be ongoing)
7. The paper is still in draft form. It includes the "PREVIOUS SECTION 5 DRAFT" block (pp. 10–13), a "[TW]" marker, and a 13-page main text. As submitted, it risks desk rejection.

Questions

1. Can the QA track run entirely from the released videos and annotations, with no simulator and no GPU policy? If yes, my view of adoption goes up.
2. What does Gemini score when given the full context video (no memory module) on Finding/Restore QA? If it matches or beats R&R Agent, the architecture findings lose force.
3. What is MolmoBot's execution ceiling with oracle navigation and oracle goals? This would separate the memory signal from the floor effect in Tables 3–4.
4. Do the modality rankings hold with an open-weight writer, and with paraphrased requests?
5. What is human QA accuracy on a subset?

Reproducibility & Statements

The Reproducibility, Ethics and AI-use statements are all present. The paper promises to release data and the evaluation harness, but no code is available at review time. The execution tracks need the MolmoSpaces simulator and a fine-tuned MolmoBot trained on 8 B200 GPUs (App. E), which is a high bar for the wider community.

Flag For Ethics Review

No ethics review needed.

Scores

• Soundness: 2 / 4 — fair
• Presentation: 2 / 4 — fair
• Contribution: 3 / 4 — good
• Rating: 4 / 10 — marginally below the acceptance threshold
• Confidence: 4 / 5

What Would Change My Score

Three things would raise my rating to 6: a full-video long-context baseline, an oracle-execution ceiling that shows the execution tracks can separate methods, and a lightweight QA-only release path. The rating would drop if a plain long-context VLM matches R&R Agent on QA, since that would mean the memory-architecture findings don't carry over.