260406-260412 · ~20 SLURM jobs, ~5 GPU·d wall time
이번 주 SLURM 작업은 두 가지 목적이 섞여 있다: (1) Resume / NaN guard / viz 인프라 검증, (2) LGM Phase A/B 학습 누적. 그러나 04-09에 design bug가 발견되면서 후반 작업의 결과는 baseline 무효 처리. 인프라 검증 결과는 유효.
| Param | Value |
|---|---|
| Connector | Conv3d, 2.05M params |
| LoRA rank | 32 |
| Dataset | DROID + EgoExo4D 옵션 추가 |
| GPU | 4 × B200 |
| Grad clip | 0.5 (강화됨, 1.0→0.5) |
| Loss skip threshold | 5.0 |
| Depth clamp | 100m |
| Job ID | State | Elapsed | Period | Note |
|---|---|---|---|---|
| 36031 | PREEMPTED | 06:45 | 04-05 17:35 → 04-06 00:21 | 4GPU LGM |
| 36457 | PREEMPTED | 10:04 | 04-06 13:59 → 04-07 00:03 | — |
| 36535 | PREEMPTED | 07:09 | 04-06 17:47 → 04-07 00:57 | — |
| 37393 | PREEMPTED | 1d 03:44 | 04-07 13:47 → 04-08 17:32 | 가장 긴 단일 run |
| 38366 | CANCELLED | 15:03 | 04-09 01:47 → 16:51 | Resume LGM |
| 38638 | COMPLETED | 04:20 | 04-09 18:37 → 22:58 | — |
| 38796 | PREEMPTED | 13:55 | 04-10 00:01 → 13:56 | — |
| 38815 | PREEMPTED | 22:21 | 04-10 00:36 → 22:58 | — |
| 40365 | PREEMPTED | 12:30 | 04-11 01:47 → 14:18 | — |
| 41053 | PREEMPTED | 06:26 | 04-11 16:44 → 23:10 | — |
| 41671 | PREEMPTED | 06:56 | 04-12 03:46 → 10:43 | — |
| 42101 | PREEMPTED | 06:31 | 04-12 10:43 → 17:14 | — |
| 42511 | RUNNING | ~01:52+ | 04-13 12:59 → ... | 현재 진행 |
실행 명령: scripts/train_lgm.sh · scripts/resume_lgm.sh · scripts/train_lgm_egoexo4d.sh