← index
2026-10-22

Human hand data survey — EgoDex · EgoVerse · VITRA

Demo-conditioned dexterous WAM/VLA 후보 데이터: 손 라벨 품질을 영상 위 오버레이로 직접 확인

TL;DR

1데이터셋 비교

데이터규모손 라벨 (획득 방식)영상언어에피소드 길이(중앙)wrist jitter (mm/f², 중앙)라이선스품질
EgoDex (Apple)829 h · 338k ep (test 3,243 ep / 111 task)Vision Pro 직접 트래킹 (68 joint SE(3) + confidence) + 카메라 6-DoF1920×1080 30fpsLLM 설명 (정/역방향 태스크)10.0 s0.48CC-BY-NC-ND · Apple CDN깨끗함 (센서 GT)
EgoVerse · MicroAGI (mono)2,191 h · 186,188 ep21 MANO keypoint + 카메라 6-DoF + 에피소드별 K · 라벨 30 Hz640×360 30fps태스크 17,855종20 s1.24에피소드별 (CC BY-SA 4.0 / Apache-2.0 / 미표기)추정 (벤더별 파이프라인)
EgoVerse · MicroAGI1,823 h · 328,051 ep21 MANO keypoint + 카메라 6-DoF + 에피소드별 K · 라벨 30 Hz640×360 30fps태스크 21,924종11 s2.03에피소드별 (CC BY-SA 4.0 / Apache-2.0 / 미표기)추정 (벤더별 파이프라인)
EgoVerse · Mecka828 h · 41,617 ep21 MANO keypoint + 카메라 6-DoF + 에피소드별 K · 라벨 10 Hz640×360 30fps태스크 4,859종81 s– (선형 보간이라 0)에피소드별 (CC BY-SA 4.0 / Apache-2.0 / 미표기)추정 (벤더별 파이프라인)
EgoVerse · Scale AI217 h · 17,090 ep21 MANO keypoint + 카메라 6-DoF + 에피소드별 K · 라벨 30 Hz640×480 30fps태스크 71종19 s2.42에피소드별 (CC BY-SA 4.0 / Apache-2.0 / 미표기)추정 (벤더별 파이프라인)
EgoVerse · Aria gen1 (GaTech·ETH 등 4개 랩)110 h · 2,570 ep21 MANO keypoint + 카메라 6-DoF + 에피소드별 K · 라벨 30 Hz640×480 30fps태스크 26종118 s11.73에피소드별 (CC BY-SA 4.0 / Apache-2.0 / 미표기)추정 (벤더별 파이프라인)
EgoVerse · Trace102 h · 7,700 ep21 MANO keypoint + 카메라 6-DoF + 에피소드별 K · 라벨 30 Hz1280×720 30fps태스크 461종33 s1.53에피소드별 (CC BY-SA 4.0 / Apache-2.0 / 미표기)추정 (벤더별 파이프라인)
VITRA-1M1.22M ep (egoexo4d 67k)영상 → MANO 재구성 (HaWoR 계열), 저자 추정 정확도 ~90%원본 영상 직접 준비 (Ego4D/Epic/EgoExo4D/SSv2, undistort)손별 atomic 지시문 + GPT rephrase0.8 s3.31MIT (라벨) · 영상은 원본 라이선스대규모·노이즈
VITRA-TeleData (로봇)2,611 epRealman 7-DoF + XHand 12-DoF 텔레옵 (state/action GT)1280×720 30fps (RealSense D455)짧은 지시문6.2 s0.45MIT로봇 GT (덱스 핸드)

wrist jitter = 연속 3프레임 wrist 위치의 2차 차분 크기 중앙값(작을수록 매끈). 프레임레이트·카메라 움직임이 달라 절대 비교는 참고용.

VITRA-1M egoexo4d 라벨 2,000개 샘플의 에피소드 길이 분포 (s): <0.5: 0 · 0.5–1: 1402 · 1–1.5: 509 · 1.5–2: 80 · 2–3: 9 · 3–5: 0 · >5: 0 — 손 유효 프레임 비율 중앙값 100%.

2EgoDex — Vision Pro 센서 트래킹 (공식 test split, 태스크별 1개 · 16개)

원본 hdf5 + 1920×1080 mp4. 21 keypoint(손목+각 손가락 4관절)를 68 joint 중에서 골라 inv(camera)로 카메라 좌표 변환 후 파일의 intrinsic으로 투영. 빈 원 = confidence < 0.25. 에피소드 중앙 30 s 구간, 2프레임마다.

arrange_topple_dominoes · 19s · 1920×1080
Arrange eight dominoes upright in a pattern on a green table.
conf L/R 0.64/0.62 · jitter 0.61 mm
assemble_disassemble_furniture_bench_drawer · 2s · 1920×1080
Assemble the drawer from furniture bench on a blue tablecloth with a pink background while sitting.
conf L/R 0.90/0.80 · jitter 1.53 mm
assemble_disassemble_furniture_bench_stool · 8s · 1920×1080
Assemble the leg onto the stool while sitting at a table with a green tablecloth and blue background.
conf L/R 0.55/0.61 · jitter 0.84 mm
charge_uncharge_airpods · 4s · 1920×1080
Charge the lightning-sized airpods on the green tablecloth.
conf L/R 0.64/0.68 · jitter 0.48 mm
deal_gather_cards · 10s · 1920×1080
Deal cards onto a wooden table for a game of pyramid solitaire while sitting.
conf L/R 0.43/0.56 · jitter 0.43 mm
flip_coin · 10s · 1920×1080
Place the coin on top of the thumb, flip the coin, catch it with the other hand, and put it back in position.
conf L/R 0.75/0.74 · jitter 0.33 mm
insert_remove_utensils · 6s · 1920×1080
Insert utensils into a container on a green tablecloth while sitting against a white background.
conf L/R 0.64/0.00 · jitter – mm
make_sandwich · 20s · 1920×1080
Make a sandwich using breads, cheese, petty, tomato, onion, lettuce, pickles, and egg while sitting at a table with a blue tablecloth and pi
conf L/R 0.64/0.74 · jitter 0.70 mm
pick_place_food · 7s · 1920×1080
Pick a food item from the food basket and place it onto the plate next to it.
conf L/R 0.00/0.45 · jitter 1.89 mm
Point and click remote · 10s · 1920×1080
Point at the monitor and click the gray remote while sitting at a wooden table with a red background.
conf L/R 0.59/0.02 · jitter 2.99 mm
push_pop_toy · 19s · 1920×1080
Press the bubbles on the circle fidget Pop It toy one by one using your fingers while sitting at a blue tablecloth with a pink background.
conf L/R 0.55/0.54 · jitter 0.35 mm
sleeve_unsleeve_cards · 37s · 1920×1080
Pick up a vanguard card from the metal table and insert it into a card sleeve.
conf L/R 0.58/0.68 · jitter 1.13 mm
slot_batteries · 7s · 1920×1080
Insert batteries into a remote while sitting at a table with a green tablecloth and brown background.
conf L/R 0.59/0.63 · jitter 0.16 mm
stack_unstack_bowls · 7s · 1920×1080
Stack three bowls on a green tablecloth while sitting against a blue background.
conf L/R 0.68/0.65 · jitter 0.61 mm
type_keyboard · 10s · 1920×1080
Use fingers of one or both hands to type on a keyboard while sitting against a blue background without a tablecloth.
conf L/R 0.83/0.83 · jitter 0.20 mm
wrapping · 10s · 1920×1080
Wrap a box with wrapping paper on a pink tablecloth with a lavender background while sitting.
conf L/R 0.66/0.67 · jitter 0.73 mm

3EgoVerse — 공식 zarr (출처별 무작위 태스크 4개 · 내부 "debug" 태스크 제외 · 23개)

world 좌표 21 MANO keypoint를 obs_head_pose(xyz+quat wxyz)의 역변환으로 카메라 좌표로 옮기고, zarr.json의 front_1 intrinsics로 투영 (egomimic의 keypoints_headframe 변환과 동일). 영상은 zarr 안 JPEG 프레임. "label Hz" = 카메라 기준 wrist 궤적이 꺾이는 프레임 비율 × fps (30 = 매 프레임 라벨, 10 = 3프레임마다 키프레임 + 선형 보간).

Aria gen1 (GaTech·ETH 등 4개 랩) (rl2) · 207s · 640×480 · f=267px
In this task, the collector will sit or stand in front of a table which has two objects, a container, and a small object to put inside the c
label 30 Hz · 오른손만 · jitter 18.28 mm
Aria gen1 (GaTech·ETH 등 4개 랩) (rl2) · 116s · 640×480 · f=267px
pick up the bag and make it upright, place the objects into the bag
label 30 Hz · jitter 20.67 mm
Aria gen1 (GaTech·ETH 등 4개 랩) (rl2) · 108s · 640×480 · f=267px
sorting markers and pens
label 30 Hz · jitter 8.91 mm
Mecka (mecka) · 102s · 640×360 · f=251px
Fill small plastic cups with a chopped vegetable mixture and arrange them on the preparation surface.
label 10 Hz
Mecka (mecka) · 75s · 640×360 · f=251px
Sorting small colorful garment accessories from a large container into individual plastic zip-lock bags.
label 10 Hz · head pose 없음(상수)
Mecka (mecka) · 90s · 640×360 · f=254px
Manually assembling metal eyelets by joining two components together and sorting them into plastic cups.
label 10 Hz · head pose 없음(상수)
Mecka (mecka) · 80s · 640×360 · f=254px
Organizing various items on a desk, including notebooks, pens, and papers, by repositioning them to tidy the workspace.
label 10 Hz
MicroAGI (microagi) · 14s · 640×360 · f=240px
Pick up a small metal container from the counter, rinse it under the kitchen faucet, and place it back on the counter.
label 29 Hz · jitter 3.55 mm
MicroAGI (microagi) · 14s · 640×360 · f=263px
Fold a pair of khaki pants in half lengthwise on a white table, then fold them again into a compact folded shape.
label 29 Hz · jitter 3.31 mm
MicroAGI (microagi) · 23s · 640×360 · f=437px
Place a blue keychain into a small cardboard gift box and tie twine around the box to secure it.
label 30 Hz · jitter 0.97 mm
MicroAGI (microagi) · 84s · 640×360 · f=250px
Wipe and clean a pair of headphones with a cloth over a basin.
label 30 Hz · jitter 1.98 mm
MicroAGI (mono) (microagi) · 19s · 640×360 · f=244px
Transfer cooked food from a frying pan into a green bowl using a wooden spoon.
label 30 Hz · jitter 2.44 mm
MicroAGI (mono) (microagi) · 73s · 640×360 · f=244px
Stir the mixture inside the pot on the stove using a utensil to prepare the sauce.
label 30 Hz · jitter 1.20 mm
MicroAGI (mono) (microagi) · 37s · 640×360 · f=258px
Transfer batter from a mixing bowl to a pan using a spatula.
label 30 Hz · jitter 2.40 mm
MicroAGI (mono) (microagi) · 11s · 640×360 · f=457px
Place a garment onto a hanger and hang it in the closet.
label 30 Hz · jitter 1.63 mm
Scale AI (scale) · 30s · 640×480 · f=171px
Portion out snacks by scooping
label 30 Hz · jitter 3.39 mm
Scale AI (scale) · 46s · 640×480 · f=175px
Placing utensils into a drawer
label 30 Hz · jitter 2.02 mm
Scale AI (scale) · 101s · 640×480 · f=174px
Hang shirt on hanger and place on rack
label 30 Hz · jitter 2.37 mm
Scale AI (scale) · 12s · 640×480 · f=175px
Do dishes
label 30 Hz · jitter 1.79 mm
Trace (trace) · 26s · 1280×720 · f=476px
pick up lens, grind on wheel, wipe hands, and fit lens into frame
label 30 Hz · jitter 1.17 mm
Trace (trace) · 23s · 1280×720 · f=478px
sealed puffed grains
label 30 Hz · jitter 1.30 mm
Trace (trace) · 20s · 1280×720 · f=489px
fold cloth by the use of both hand
label 30 Hz · jitter 3.71 mm
Trace (trace) · 29s · 1280×720 · f=478px
fishing net repair
label 30 Hz · jitter 1.82 mm

4VITRA-1M — 영상 기반 MANO 재구성

EgoExo4D: 로컬 원본(448 downscaled Aria)에 VITRA와 같은 Aria→pinhole undistort를 448 해상도로 적용 후 라벨 투영. 한 take의 연속 에피소드를 이어붙임(각 0.5–1.5 s).

Ego4D (repo 예시 1개 에피소드)
EgoExo4D fair_cooking_07_2 · 연속 에피소드 10개 이어붙임
R: | L: Hold the edge of the metal tray containing vegetables.
R: | L: Touch the red bell pepper.
R: | L: Grasp the bunch of leafy greens.
EgoExo4D georgiatech_cooking_01_01_2 · 연속 에피소드 10개 이어붙임
R: | L: Hold the recipe sheet.
R: | L: Hold and adjust the left edge of the recipe sheet.
R: | L: Hold the recipe sheet.
EgoExo4D uniandes_cooking_008_8 · 연속 에피소드 10개 이어붙임
R: Grasp the frying pan handle. | L:
R: Remove the frying pan from the stove. | L:
R: Pick up the wooden spatula. | L:

5VITRA-TeleData — XHand 로봇 텔레옵

hand_mount 6-DoF 축(카메라 좌표) + 하단 막대 = XHand 12 관절 (에피소드 내 min–max 정규화).

Place mango into white box. · 6.0s
wrist jitter 0.63 mm
Pick up the cube. · 5.2s
wrist jitter 0.45 mm
Place the blackboard eraser into the white container. · 2.2s
wrist jitter 1.26 mm
Place the pot onto the table. · 6.3s
wrist jitter 0.43 mm
Sweep the paper balls into the dustpan. · 16.0s
wrist jitter 0.45 mm
Pick up the power screwdriver. · 12.8s
wrist jitter 0.26 mm

6EgoDex — 공식 센서 라벨 vs EgoSteer 추정 라벨 (같은 영상, 8개)

EgoSteer-Egocentric의 egodex 폴더(147,588 ep · 논문 표 370 h; HF 데이터 카드는 손을 "EgoSmith pipeline" 추정이라 표기, 논문 표는 원본 annotation)에서 에피소드를 고르고, source_media로 같은 공식 hdf5/mp4를 Apple CDN에서 받아 비교. 둘 다 EgoSteer 형식(손목 + 손끝 5개)으로 줄여 각자의 head camera(공식 inv(camera), EgoSteer head_world2cam)로 옮기고 공식 1080p intrinsic으로 투영. 왼쪽 = 공식 Vision Pro 라벨 (손목 confidence > 0.5일 때만, 빈 원 = 손끝 confidence < 0.5), 오른쪽 = EgoSteer (흰 얇은 선 = 공식; 완전히 겹쳐서 안 보임).

에피소드프레임L 최대 차이 (mm)L 존재 일치L 손끝 conf<0.5R 최대 차이 (mm)R 존재 일치R 손끝 conf<0.5head camera 차이 (중앙)
assemble_disassemble_jigsaw_puzzle
extra/assemble_disassemble_jigsaw_puzzle/500		8040.000100%61%0.000100%63%0.001° / 0.000 mm
basic_pick_place
extra/basic_pick_place/1012		7700.000100%23%0.000100%57%0.000° / 0.000 mm
basic_pick_place
extra/basic_pick_place/970		2360.000100%47%0.000100%64%0.000° / 0.000 mm
blowdry_hair
extra/blowdry_hair/251		6600.00092%92%0.00098%65%0.000° / 0.000 mm
build_unstack_lego
extra/build_unstack_lego/892		1500.000100%39%0.000100%41%0.000° / 0.000 mm
make_sandwich
extra/make_sandwich/540		4380.000100%82%0.000100%73%0.000° / 0.000 mm
makeup
extra/makeup/764		431–100%–%0.000100%67%0.000° / 0.000 mm
play_mancala
extra/play_mancala/426		2110.000100%94%0.000100%66%0.000° / 0.000 mm
EgoSteer의 EgoDex 손 라벨 = 공식 Vision Pro 라벨 그대로: 8개 에피소드 3,700 프레임 전체에서 손목·손끝 위치 최대 차이 0.0004 mm, head camera도 동일. 논문 Fig. 8(a) 표에서도 EgoDex는 Hand·Camera 칸이 비어 있음(= 원본 annotation 사용, Depth·Language만 파이프라인 생성) — HF 데이터 카드의 "EgoSmith pipeline" 표기가 틀린 것. 손 존재 플래그는 공식 손목 confidence > 0.5와 중앙 100% 일치.
잃는 정보 = confidence: EgoSteer에는 관절별 confidence가 없음. 공식 라벨에서 손끝 confidence < 0.5인 비율이 중앙 64% (가려지거나 물건을 쥔 손가락). EgoSteer로 학습하면 이 손끝 값을 확신 있는 값과 구분 없이 쓰게 됨 → EgoDex는 공식 hdf5로 confidence 마스크를 쓰고, EgoSteer에서는 지시문(최대 5단계)만 빌리는 게 낫다.
assemble_disassemble_jigsaw_puzzle · 27s · 최대 차이 L/R 0.000/0.000 mm · 손끝 confidence<0.5 L/R 61/63%
EgoSteer 지시문: Dismantling and reassembling a puzzle. ⟶ Dismantling a completed shark puzzle and then attempting to reassemble the pieces on the table surface.
basic_pick_place · 26s · 최대 차이 L/R 0.000/0.000 mm · 손끝 confidence<0.5 L/R 23/57%
EgoSteer 지시문: Assembling a puzzle. ⟶ Selecting puzzle pieces from a pile and connecting them one by one to form a section on the table.
basic_pick_place · 8s · 최대 차이 L/R 0.000/0.000 mm · 손끝 confidence<0.5 L/R 47/64%
EgoSteer 지시문: Assembling a jigsaw puzzle. ⟶ Picking up individual puzzle pieces and connecting them to a larger section on the table.
blowdry_hair · 22s · 최대 차이 L/R 0.000/0.000 mm · 손끝 confidence<0.5 L/R 92/65%
EgoSteer 지시문: Drying and styling a wig. ⟶ Picking up a pink hair dryer to blow-dry a blonde wig while using the other hand to ruffle the hair and adjust a clip.
build_unstack_lego · 5s · 최대 차이 L/R 0.000/0.000 mm · 손끝 confidence<0.5 L/R 39/41%
EgoSteer 지시문: Stacking toy blocks. ⟶ Picking up various colored toy blocks from the surface and stacking them vertically on top of a white base block.
make_sandwich · 15s · 최대 차이 L/R 0.000/0.000 mm · 손끝 confidence<0.5 L/R 82/73%
EgoSteer 지시문: Assembling a toy taco. ⟶ Picking up a yellow shell and filling it with various colorful ingredients like cheese, lettuce, and tomato before setting it down.
makeup · 14s · 최대 차이 L/R –/0.000 mm · 손끝 confidence<0.5 L/R –/67%
EgoSteer 지시문: Applying lipstick to a doll head. ⟶ Picking up a lipstick tube, removing the cap, applying the color to a doll's lips, and placing the tube back on the table.
play_mancala · 7s · 최대 차이 L/R 0.000/0.000 mm · 손끝 confidence<0.5 L/R 94/66%
EgoSteer 지시문: Playing a board game. ⟶ Picking up pieces from a yellow game board and dropping them into adjacent slots.

7Takeaway / Next

"Clean pretraining" 축은 EgoDex(센서 GT, 손가락 관절까지)가 가장 명확. EgoVerse 공식 데이터는 21 keypoint 전체를 주고 규모가 가장 크지만(출처 6곳), 벤더별 추정 파이프라인이라 라벨 주기·정밀도가 출처마다 달라 "large & mixed" 쪽에 가까움. VITRA-1M은 규모는 최대이나 조각이 짧고 라벨이 추정치라 "large & dirty" 비교군으로 적합.
generated 2026-10-22T04:52:07 · tmp/hand_data/build_gallery.py