lwam_fb_rec_ft (recovery 1:1) 08:52 완료(19h10m, 4 GPU). 대조군 193354 lwam_fb_ctrl_ft는 RUNNING(19h18m+). 둘 다 끝나면 p768 평가 경로(768 frame history 유지) 확정 후 삽입 평가 → 회복 데이터가 조건부 성공률을 올리는지 확인. fb_insert_vid 193358–193360 RUNNING(own, 1 GPU × 3).--qos=own으로 올리는 것을 검토.ll_egox_long, 6h47m) 완료. 체크포인트 2000/4000 wild sweep(relrope, sink 0) 결과 확인 후 이후 ckpt 평가 결정. 현재 RUNNING job 없음.lift_pot 0/10 재확인 필요.--dependency 잔존(Job dependency problem) + extra 선점 시 첫 저장(500 step, 3.5 h) 이전 손실(약 2.5 h × 8 GPU).exp-recovery_finetune.md: 193004/193005(10/14 18:09 시작)가 첫 checkpoint(500 step) 전 선점 → 자동 재제출도 Job dependency problem으로 실패(20:12, 20:38). 교훈: sbmr에는 --dependency 금지.--qos=own, dependency 없음, SAVE_EVERY 500→250(≈1.7 h마다), --time 1-08:00:00으로 재제출. 설정: INIT step_013690.pt, p768, 4 GPU × batch 2 × accum 8 = 64, 3000 step, lr 5e-5.ab_pi05), RoboCasa365 진단 oracle/foreign/static(rc365_*) 완료, fb_insert_lwam(step 13690) 완료. 영상 녹화 eval ab_pi05_vid, ab_lwam_vid 실행.lwam_ab_train(4 GPU, extra) 선점 반복: 192506 3h04m, 193361 3h08m, 193495 43m, 193798 1h07m, 193864 2h50m, 194092 13m, 194139 2h21m.fb_insert_vid)이 QOSMaxMemoryPerUser로 대기했었다.lwam_ab_train own 승격 또는 체크포인트 간격 점검.ll_egox_long: configs/egox_ar_long.yaml, 4 GPU extra, 6h47m COMPLETED (01:24→08:12 UTC).ll_sweep: logs/egox_ar_long/checkpoint_002000 / checkpoint_004000, use_relative_rope=true inference.sink_size=0 wild sweep, 각 약 16m, COMPLETED.lwam_sweep0–6, 192942/192943 lwam_domino: imagine_vs_sim_b200.sh 9건 모두 COMPLETED(각 43m–1h13m, 1 GPU, isaac-lab 컨테이너).train_b200.sh 자동 resume, step_005700) 여부 결정. 약 2,700 GPU-h로 Lingbot-VA/Zero-WAM(약 150 GPU-h)의 약 18배 — 곡선 보고 단축 스케줄 판단.lift_pot 0/10(공개 5/5) 원인 확인.| Job ID | 프로젝트(WorkDir) | JobName / 실행 명령 | GPU | 상태 | 실행 시간 | 노드 |
|---|---|---|---|---|---|---|
| 192079 | RoboTTA | lwam_fb_train bash scripts/furniture/train_longwam_b200.sh | 4 | COMPLETED | 09:17:14 | n72 |
| 192505 | EgoX_LongLive | ll_egox_long [egox-longlive] LOGDIR=logs/egox_ar_long CONFIG=configs/egox_ar_long.yaml bash scripts/egox/train_egox.sh | 4 | COMPLETED | 06:47:22 | n87 |
| 192506 | RoboTTA | lwam_ab_train bash scripts/assemblebench/train_longwam_b200.sh | 4 | PREEMPTED | 03:03:59 | n111 |
| 192507 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_4mm | 1 | CANCELLED | 02:50:33 | n64 |
| 192508 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_8mm | 1 | CANCELLED | 02:50:33 | n67 |
| 192509 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_12mm | 1 | CANCELLED | 02:49:33 | n61 |
| 192510 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_16mm | 1 | CANCELLED | 02:49:33 | n70 |
| 192741 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_4mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 6 | 1 | COMPLETED | 00:43:22 | n52 |
| 192742 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_8mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 6 | 1 | COMPLETED | 00:43:52 | n61 |
| 192743 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_12mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 6 | 1 | PREEMPTED | 00:10:30 | n64 |
| 192744 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_16mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 6 | 1 | COMPLETED | 00:44:22 | n100 |
| 192745 | RoboTTA | fb_insert_lwam bash scripts/furniture/eval_insert_longwam.sh outputs/longwam/furniture_one_leg/libero-init outputs/longwam/furniture_one_leg/libero-init/checkpoin… | 1 | PREEMPTED | 00:56:00 | n103 |
| 192813 | RoboTTA | rc365_oracle bash scripts/robocasa365/diag_eval.sh | 1 | COMPLETED | 00:35:55 | n78 |
| 192814 | RoboTTA | rc365_foreign bash scripts/robocasa365/diag_eval.sh | 1 | COMPLETED | 00:39:45 | n80 |
| 192815 | RoboTTA | rc365_static bash scripts/robocasa365/diag_eval.sh | 1 | COMPLETED | 02:04:10 | n81 |
| 192818 | RoboTTA | fb_rec_scale1000 [furniture-bench] bash scripts/furniture/gen_recovery_scale.sh scale1 1000 1103 8 | 1 | COMPLETED | 01:20:44 | n2 |
| 192819 | RoboTTA | fb_rec_scale1104 [furniture-bench] bash scripts/furniture/gen_recovery_scale.sh scale1 1104 1207 8 | 1 | COMPLETED | 01:24:58 | n46 |
| 192820 | RoboTTA | fb_rec_scale1208 [furniture-bench] bash scripts/furniture/gen_recovery_scale.sh scale1 1208 1311 8 | 1 | COMPLETED | 01:24:01 | n56 |
| 192821 | RoboTTA | fb_rec_scale1312 [furniture-bench] bash scripts/furniture/gen_recovery_scale.sh scale1 1312 1415 8 | 1 | COMPLETED | 01:21:15 | n60 |
| 192853 | RoboTTA | ab_pi05 bash scripts/assemblebench/eval_pi05.sh peg_round_12mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 6 | 1 | COMPLETED | 00:44:22 | n83 |
| 192934 | LongLive (Long-WAM) | lwam_sweep0 bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:43:06 | n104 |
| 192935 | LongLive (Long-WAM) | lwam_sweep1 bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:44:14 | n71 |
| 192936 | LongLive (Long-WAM) | lwam_sweep2 bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:44:19 | n78 |
| 192937 | LongLive (Long-WAM) | lwam_sweep3 bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:56:09 | n80 |
| 192938 | LongLive (Long-WAM) | lwam_sweep4 bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:47:27 | n68 |
| 192939 | LongLive (Long-WAM) | lwam_sweep5 bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:59:00 | n70 |
| 192940 | LongLive (Long-WAM) | lwam_sweep6 bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:54:58 | n60 |
| 192942 | LongLive (Long-WAM) | lwam_domino bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 01:13:03 | n51 |
| 192943 | LongLive (Long-WAM) | lwam_domino bash scripts/robotwin2/imagine_vs_sim_b200.sh | 1 | COMPLETED | 00:55:14 | n112 |
| 192999 | RoboTTA | fb_rec_convert bash scripts/furniture/convert_recovery.sh one_leg_recovery outputs/furniture/recovery/scale1 | 0 | CANCELLED | 00:00:00 | None assigned |
| 193004 | RoboTTA | lwam_fb_rec_ft [furniture-bench] TASK=furniture_one_leg_recovery INIT=RoboTTA/outputs/longwam/furniture_one_leg/libero-init/checkpoints/weights/step_013690.pt BAT… | 4 | PREEMPTED | 02:30:00 | n99 |
| 193005 | RoboTTA | lwam_fb_ctrl_ft [furniture-bench] TASK=furniture_one_leg_recovery INIT=RoboTTA/outputs/longwam/furniture_one_leg/libero-init/checkpoints/weights/step_013690.pt BAT… | 4 | PREEMPTED | 02:03:00 | n105 |
| 193006 | RoboTTA | fb_rec_convert bash scripts/furniture/convert_recovery.sh one_leg_recovery outputs/furniture/recovery/scale1 | 0 | COMPLETED | 00:04:39 | n102 |
| 193034 | RoboTTA | fb_insert_lwam bash scripts/furniture/eval_insert_longwam.sh outputs/longwam/furniture_one_leg/libero-init outputs/longwam/furniture_one_leg/libero-init/checkpoin… | 1 | COMPLETED | 00:57:04 | n47 |
| 193126 | EgoX_LongLive | ll_sweep [egox-longlive] bash scripts/egox/eval_wild_sweep.sh logs/egox_ar_long/checkpoint_002000 long_s2000_relrope_nosink use_relative_rope=true inference… | 1 | COMPLETED | 00:16:25 | n41 |
| 193212 | EgoX_LongLive | ll_sweep [egox-longlive] bash scripts/egox/eval_wild_sweep.sh logs/egox_ar_long/checkpoint_004000 long_s4000_relrope_nosink use_relative_rope=true inference… | 1 | COMPLETED | 00:16:14 | n43 |
| 193353 | RoboTTA | lwam_fb_rec_ft [furniture-bench] TASK=furniture_one_leg_recovery INIT=RoboTTA/outputs/longwam/furniture_one_leg/libero-init/checkpoints/weights/step_013690.pt BAT… | 4 | COMPLETED | 19:10:41 | n13 |
| 193354 | RoboTTA | lwam_fb_ctrl_ft [furniture-bench] TASK=furniture_one_leg_recovery INIT=RoboTTA/outputs/longwam/furniture_one_leg/libero-init/checkpoints/weights/step_013690.pt BAT… | 4 | RUNNING | 19:18:40 | n19 |
| 193355 | RoboTTA | ab_pi05_vid bash scripts/assemblebench/eval_pi05.sh peg_round_12mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 6 | 1 | COMPLETED | 00:45:57 | n31 |
| 193356 | RoboTTA | ab_pi05_vid bash scripts/assemblebench/eval_pi05.sh peg_round_16mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 4 | 1 | PREEMPTED | 00:34:32 | n34 |
| 193357 | RoboTTA | ab_pi05_vid bash scripts/assemblebench/eval_pi05.sh peg_round_4mm assemble-bench/checkpoints/pi05-AssembleBench-cgdagger-r3 5 2 | 1 | COMPLETED | 00:18:51 | n42 |
| 193358 | RoboTTA | fb_insert_vid bash scripts/furniture/eval_insert_longwam.sh outputs/longwam/furniture_one_leg/libero-init outputs/longwam/furniture_one_leg/libero-init/checkpoin… | 1 | RUNNING | 00:06:40 | n80 |
| 193359 | RoboTTA | fb_insert_vid bash scripts/furniture/eval_insert_longwam.sh outputs/longwam/furniture_one_leg/libero-init outputs/longwam/furniture_one_leg/libero-init/checkpoin… | 1 | RUNNING | 00:06:10 | n116 |
| 193360 | RoboTTA | fb_insert_vid bash scripts/furniture/eval_insert_pi05.sh pi05_furniture_all_low_b128 scenemem/openpi/checkpoints/pi05_furniture_all_low_b128/all_low_b128_v1/3999… | 1 | RUNNING | 00:06:40 | n85 |
| 193361 | RoboTTA | lwam_ab_train [flexpi-eval] bash scripts/assemblebench/train_longwam_b200.sh | 4 | PREEMPTED | 03:08:30 | n47 |
| 193363 | RoboTTA | ab_lwam_vid bash scripts/assemblebench/eval_longwam.sh outputs/longwam/assemblebench_round_pegs/libero-init outputs/longwam/assemblebench_round_pegs/libero-ini… | 1 | COMPLETED | 00:21:11 | n53 |
| 193495 | RoboTTA | lwam_ab_train [flexpi-eval] bash scripts/assemblebench/train_longwam_b200.sh | 4 | PREEMPTED | 00:43:00 | n124 |
| 193798 | RoboTTA | lwam_ab_train [flexpi-eval] bash scripts/assemblebench/train_longwam_b200.sh | 4 | PREEMPTED | 01:07:29 | n98 |
| 193864 | RoboTTA | lwam_ab_train [flexpi-eval] bash scripts/assemblebench/train_longwam_b200.sh | 4 | PREEMPTED | 02:50:00 | n63 |
| 194092 | RoboTTA | lwam_ab_train [flexpi-eval] bash scripts/assemblebench/train_longwam_b200.sh | 4 | PREEMPTED | 00:13:30 | n127 |
| 194139 | RoboTTA | lwam_ab_train [flexpi-eval] bash scripts/assemblebench/train_longwam_b200.sh | 4 | PREEMPTED | 02:21:30 | n15 |