VGGRPO Experiments

Job ID	Duration	State	Experiment
#28142	13m	COMPLETED	Exp 1
#28148	23m	COMPLETED	Exp 1
#28160	3h 9m	COMPLETED	Exp 2
#29807	3m	CANCELLED	Exp 3
#29808	1h 56m	CANCELLED	Exp 3
#29843	1h 50m	PREEMPTED	Exp 3
#30037	15m	COMPLETED	Exp 4
#30561	4h 20m	COMPLETED	Exp 4
#31093	12h 26m	PREEMPTED	Exp 4
#30176	24m	CANCELLED	Exp 5
#30745	45m	CANCELLED	Exp 5
#30758	7m	CANCELLED	Exp 5
#30759	9h 43m	CANCELLED	Exp 5
#31770	18h 30m	CANCELLED	Exp 5
#36031	6h 46m	PREEMPTED	Exp 5
#36457	10h 4m	PREEMPTED	Exp 5
#36535	7h 9m	PREEMPTED	Exp 5
#37393	25h+	RUNNING	Exp 6

Experiments

Exp 1: Phase 1 Validation Apr 1

Hypothesis
Wan2.1-1.3B + Any4D can produce valid outputs on DROID data

#28142 13m COMPLETED #28148 23m COMPLETED

Result
Wan generation 33f 480x832 OK. Any4D depth/pose/flow OK on DROID wrist video.

Insight
Pipeline works E2E. DROID pose is Euler XYZ, not axis-angle.

Exp 2: LGM Smoke Test Apr 1

Hypothesis
Conv3d connector can bridge VAE latent to Any4D

#28160 3h9m COMPLETED

Result
Pseudo-label loss 3.54 → 1.81, GT loss 1.23 → 1.06 (5 steps each)

Insight
Gradient flows through. But 5 steps insufficient for meaningful geometry.

Exp 3: LGM Short Training Apr 3

Hypothesis
190 steps produces usable geometry

#29807 3m CANCELLED #29808 1h56m CANCELLED #29843 1h50m PREEMPTED

Result
Loss decreases but geometry is noisy point cloud, not useful.

Insight
Minimum 25K steps needed (paper uses 50K). Submitted longer jobs.

Exp 4: LGM Medium Training Apr 3–4

Hypothesis
Longer training converges to usable geometry quality

#30037 15m COMPLETED #30561 4h20m COMPLETED #31093 12h26m PREEMPTED

Result
Gradual improvement but still not converged.

Insight
Preemption is major obstacle for long training. Auto-checkpoint every 1000 steps needed.

Exp 5: Extended Training Apr 4–6

Hypothesis
25K+ steps with checkpointing can survive preemptions

#31770 18h30m CANCELLED #36031 6h46m PREEMPTED #36457 10h4m PREEMPTED #36535 7h9m PREEMPTED

Result
Frequent preemptions prevent convergence.

Insight
Need own-QOS for critical long jobs, or implement robust auto-resume.

Exp 6: Current Long Run Apr 7

Hypothesis
25K+ step LGM training with auto-checkpoint reaches convergence

#37393 25h+ RUNNING

Result
Ongoing — longest uninterrupted run so far.

Insight
Since Apr 7 04:47. Monitoring for preemption.

Training Configuration

Model LGM: Conv3d connector + Any4D decoder (frozen) Optimizer AdamW Data DROID processed (156 episodes dev, 3162 full) GPU B200 x 1 (LGM is lightweight)

GPU Usage

~60h+ Total GPU

~35h Effective

~40% Preemption Rate

~25h Time Lost

Preemption rate higher than other projects. Primary bottleneck for convergence. Mitigation: own-QOS reservation or auto-resume with checkpoint-on-SIGTERM.

SLURM Summary

Experiments

Training Configuration

GPU Usage