# Format

The EgoDex dataset consists of 800+ hours of 30 Hz, 1080p egocentric video and paired 30 Hz 3D pose annotations for the upper body, hands, and camera extrinsics. The data consists entirely of tabletop manipulation tasks across ~200 diverse tasks. 

The files are divided into a training set with 99% of the data and test set (`test.zip`) with 1% of the data. Due to size, the training set is separated into 5 zip files each with size <350GB: `part1.zip`, `part2.zip`, and so on. (Note - for users interested in the `basic_pick_place` or `furniture_bench` tasks, these are in `part2.zip`.) There is also an `extra.zip` with additional samples collected after freezing the initial train and test set. 

Within each zip file are folders named by task, and within each task folder is a set of *paired* HDF5 files and MP4 files. **Corresponding files have the same index** (e.g., `0.hdf5` and `0.mp4`). The pose annotations at each frame of the MP4 file are contained in the corresponding HDF5 file. The files are structured as follows: 

```
part1
└──task1
    └──0.hdf5
    └──0.mp4
    └──1.hdf5
    └──1.mp4
    ...
└──task2
    └──0.hdf5
    └──0.mp4
    ...
...

test
└──task1
    └──0.hdf5
    └──0.mp4
└──task2
... 
```

Each HDF5 file has the structure below, where `N` is the number of frames. 

```
camera
└──intrinsic            # 3 x 3 camera intrinsics

transforms              # all joint transforms, all below have shape N x 4 x 4
└──camera               
└──leftHand             
└──rightHand            
└──leftIndexFingerTip
└──leftIndexFingerKnuckle
└──(64 more joints...)

confidences             # scalar joint confidences, all below have shape N
└──leftHand
└──rightHand
└──(66 more joints...)
```

If the corresponding MP4 file is `T` seconds long, then `N = 30 * T`. The first transform of each joint corresponds to the first frame of the video. The file contains skeletal SE(3) pose data for all joints, estimated using ARKit on visionOS. Note that all transforms (including the camera extrinsics, `transforms/camera`) are expressed in the *ARKit origin frame*: a stationary frame on the ground set at the beginning of a recording session. Since this depends on device initialization, this world frame is not necessarily consistent across episodes (though it is stationary during an episode).

Most (*but not all!*) HDF5 files also contain *confidences*, a scalar value between 0 and 1 indicating how confident the ARKit model is in its prediction. A confidence of zero indicates that the joint is fully occluded or otherwise not detectable. The leftHand and rightHand confidences indicate confidence for the hands as a whole; if these are low or zero, the finger joint confidences may not be reliable.

Lastly, language metadata annotations can be accessed under the HDF5 file attributes. In Python, if `f` is the hdf5 file, you can access this with `f.attrs['llm_description']`. Reversible tasks also have a `f.attrs['llm_description2']`. In this case, you can determine which description applies to that particular episode with `f.attrs['which_llm_description']`, which will be either `1` for `llm_description` or `2` for `llm_description2`. Note that there may be errors in `llm_description` and `which_llm_description` as they are auto-generated by a LLM and VLM respectively (specifically, GPT-4).

# Collection Process

This data was collected with Apple Vision Pro using ARKit for 3D pose tracking. Operators wore the headset while performing various manipulation tasks with their hands.

# References 

Data can be loaded for model training (e.g., in a PyTorch dataset) with [torchcodec](https://github.com/pytorch/torchcodec) for MP4 files and [h5py](https://www.h5py.org) for HDF5 files. If you use the data, please cite the paper: https://arxiv.org/abs/2505.11709. The dataset is licensed under CC-by-NC-ND terms.