Experiments

DreamData — SLURM Job Details, Training Config & Hypotheses
Apr 3 – 8, 2026
~30 Total Jobs
12 Completed
6 Running
8 Cancelled
3 Preempted

SLURM Job Summary

GPU Time Distribution (~80h+ total) 6 concurrent jobs running
COMPLETED CANCELLED RUNNING PREEMPTED
Experiment Job IDs Duration State Purpose
Exp 1: Initial Training #30770-30773 ~9h each CANCELLED Finetrainers default config
Exp 2: Config Tuning #31091-31092 14min each COMPLETED Pipeline validation
Exp 2 (cont.) #31132-31138 mixed MIXED Preprocessing v1 tests
Exp 3: Long Training v1 #31146 13h COMPLETED Full v1 training run
Exp 3 (cont.) #31147 10h COMPLETED v1 training (alt config)
Exp 4: Long Training v2 #35493 1d 3h COMPLETED v2 preprocessing training
Exp 4 (cont.) #35494 1d 4h COMPLETED v2 task-balanced sampling
Exp 5: Inference/Eval #36497-36765 2min-3h COMPLETED Generation quality check
Exp 6: Current Training #37651 21h+ RUNNING Extended v2 training
Exp 6 (cont.) #37988-37990 3h+ RUNNING Multiple configs
Exp 6 (cont.) #38001-38002 2h+ RUNNING Extended configs
Exp 6 (cont.) #38019 1h+ RUNNING Latest config variant

Training Configuration

Wan2.1-T2V-14B Base Model
LoRA Fine-tuning Method
Finetrainers v0.2.0 Framework
480p / 33 frames Resolution
left|right|wrist Multi-View Concat
B200 (core) GPU / Partition
Data RoboCasa multi-view concatenation — 3 camera views (left, right, wrist) stitched horizontally per frame
GPU Usage 80h+ total compute time, peak 6 concurrent B200 jobs

Experiment Details

Exp 1: Initial Training Attempts Apr 3
CANCELLED #30770 #30771 #30772 #30773
Hypothesis Finetrainers default config works out-of-the-box for RoboCasa data
Duration ~9h each before cancellation
Result Config issues discovered, data loading failures with multi-view concat format
Insight Need custom data loader for multi-view concat format (left|right|wrist)
Exp 2: Config Tuning Apr 4
#31091 #31092 #31132-31138
Hypothesis Shorter validation runs can verify pipeline correctness before committing to long training
Duration 14min each (validation), mixed for follow-up runs
Result Pipeline validated successfully. Preprocessing v1 issues identified — robot motion compressed too much
Insight Robot motion compressed too much in v1 preprocessing — spatial information loss in stitched views
Exp 3: Long Training v1 Apr 4
#31146 #31147
Hypothesis v1 preprocessing with full training convergence produces usable video generation
Duration 13h (#31146) + 10h (#31147)
Result Training converges but generated videos lack motion diversity. Loss curves look healthy but output quality insufficient.
Insight Confirmed preprocessing v1 problem. Led directly to v2 redesign of data pipeline.
Exp 4: Long Training v2 Apr 5–6
#35493 #35494
Hypothesis v2 preprocessing with task-balanced sampling fixes motion diversity problem
Duration 1d 3h (#35493) + 1d 4h (#35494)
Result Improved motion diversity in training data. v2 pipeline addresses spatial compression issues.
Insight Task-balanced sampling critical for multi-task robotic video generation. Prevents mode collapse to dominant tasks.
Exp 5: Inference / Eval Runs Apr 6
#36497-36765
Hypothesis v2-trained model produces qualitatively better samples than v1
Duration 2min – 3h each (multiple eval jobs)
Result Qualitative improvement over v1 confirmed. Motion diversity visibly better in generated videos.
Insight v2 preprocessing validated end-to-end. Ready for extended training runs.
Exp 6: Current Training Apr 7–8
#37651 (21h+) #37988-37990 (3h+) #38001-38002 (2h+) #38019 (1h+)
Hypothesis Extended training with v2 data + multiple config variants finds optimal hyperparameters
Duration 6 jobs running — longest at 21h+
Status Ongoing. Multiple config variants running in parallel for hyperparameter search.

Experiment Timeline

Apr 3 (Thu)
Initial training attempts with default config. 4 jobs cancelled after ~9h each due to data loading issues.
Apr 4 (Fri)
Config tuning: quick validation runs (14min) confirmed pipeline. Long training v1: 13h + 10h completed. Discovered preprocessing v1 motion compression issue.
Apr 5 (Sat)
v2 preprocessing redesign begins. Long training v2 started with task-balanced sampling.
Apr 6 (Sun)
v2 training completes (1d 3h + 1d 4h). Inference/eval runs confirm qualitative improvement over v1.
Apr 7 (Mon)
Extended training launched. Job #37651 started (now at 21h+). Additional config variants queued.
Apr 8 (Tue) — Today
6 jobs running concurrently. Hyperparameter search across multiple configurations with v2 data.

Key Findings So Far

1 Default Finetrainers config does not handle multi-view concatenated video — custom data loader required
2 Preprocessing v1 over-compressed robot motion in spatially stitched views, leading to poor motion diversity in generation
3 v2 preprocessing with task-balanced sampling addresses both spatial compression and task distribution imbalance
4 Short validation runs (14min) effective for fast pipeline verification before committing GPU hours
5 ~80h+ GPU compute spent across ~30 jobs; current phase is hyperparameter search with 6 concurrent runs