Recent findings suggest that TL;DR Today we release LeRobotDataset:v3! In our previous LeRobotDataset:v2 release, we stored one episode per file, hitting file-system limitations when scaling datasets to millions of episodes. LeRobotDataset:v3 packs multiple episodes in a single file, using relational metadata to retrieve information at the individual episode level from multi-episode files. The fresh format also natively supports accessing datasets in streaming mode, allowing to process large datasets on the fly.We provide a one-liner util to convert all datasets in the LeRobotDataset format to the new format, and are very excited to share this milestone with the community ahead of our next stable release!

LeRobotDataset is a standardized dataset format designed to address the specific needs of robot learning, and it provides unified and convenient access to robotics data across modalities, including sensorimotor readings, multiple camera feeds and teleoperation status. Our dataset format also stores general information regarding the way the data is being collected (metadata), including a textual description of the task being performed, the kind of robot used and measurement details like the frames per second at which both image and robot state streams are sampled. Metadata are useful to index and search across robotics datasets on the Hugging Face Hub!

Within lerobot, the robotics library we are developing at Hugging Face, LeRobotDataset provides a unified interface for working with multi-modal, time-series data, and it seamlessly integrates both with the Hugging Face and Pytorch ecosystems. The dataset format is designed to be easily extensible and customizable, and already supports openly available datasets from a wide range of embodimentsβ€”including manipulator platforms such as the SO-100 arms and ALOHA-2 setup, real-world humanoid data, simulation datasets, and even self-driving car data! You can explore the current datasets contributed by the community using the dataset visualizer! πŸ”—.

According to recent reports, besides scale, this latest release of LeRobotDataset also enables support for a streaming functionality, allowing to process batches of data from large datasets on the fly, without having to download prohibitively large collections of data onto disk. You can access and use any dataset in v3.0 in streaming mode by using the dedicated StreamingLeRobotDataset interface! Streaming datasets is a key milestone towards more accessible robot learning, and we are excited about sharing it with the community πŸ€—.

lerobot is the end-to-end robotics library developed at Hugging Face, supporting real-world robotics as well as state of the art robot learning algorithms. The library allows to record datasets locally directly on real-world robots, and to store datasets on the Hugging Face Hub. You can read more about the robots we at the moment support here!

LeRobotDataset:v3 is going to be a part of the lerobot library starting from lerobot-v0.4.0, and we are very excited about sharing it early with the community. You can install the latest lerobot-v0.3.x supporting this newly introduced dataset format directly from PyPI using:.

Follow the community's progress towards a stable release of the library here πŸ€—.

Once you have installed a version of lerobot which supports the new dataset format, you can record a dataset with our signature robot arm, the SO-101, by using teleoperation alongside the following instructions:.

Head to the official documentation to see how to record a dataset for your use case.

According to recent reports, A core design choice behind LeRobotDataset is separating the underlying data storage from the user-facing API. This allows for efficient serialization and storage while presenting the data in an intuitive, ready-to-use format. Datasets are organized into three main components:.

It has been revealed that to support datasets with potentially millions of episodes (resulting in hundreds of millions/billions of individual frames), we merge data from different episodes into the same high-level structure. Concretely, this means that any given tabular collection and video will not contain information about one episode only, but a concatenation of the information available in multiple episodes. This keeps the pressure on the file system manageable, both locally and on remote storage providers like Hugging Face. We can then leverage metadata to gather episode-specific information, e.g. the timestamp a given episode starts or ends in a certain video.

Datasets are organized as repositories containing:.

LeRobotDataset:v3.0 will be released with lerobot-v0.4.0, together with the possibility to easily convert any dataset currently hosted on the Hugging Face Hub to the new v3.0 using:.

We are very excited about sharing this fresh format early with the community! While we develop lerobot-v0.4.0, you can still convert your dataset to the newly updated version by using the latest lerobot-v0.3.x supporting this new dataset format directly from PyPI using:.

According to recent reports, note that this is a pre-release, and generally unstable version. You can follow the status of the development of our next stable release here!

It has been revealed that the conversion script convert_dataset_v21_to_v30.py aggregates the multiple episodes episode-0000.mp4, episode-0001.mp4, episode-0002.mp4, .../episode-0000.parquet, episode-0001.parquet, episode-0002.parquet, episode-0003.parquet, ... into single files file-0000.mp4/file-0000.parquet, and updates the metadata accordingly, to be able to retrieve episode-specific information from higher-level files.

Every dataset on the Hugging Face Hub containing the three main pillars presented above (Tabular and Visual Data, as well as relational Metadata), and can be accessed with a single line.

Sources indicate that most robot learning algorithms, based on reinforcement learning (RL) or behavioral cloning (BC), tend to operate on a stack of observations and actions. For instance, RL algorithms typically use a history of previous observations o_{t-H_o:t}, and BC algorithms are instead typically trained to regress chunks of multiple actions. To accommodate for the specifics of robot learning training, LeRobotDataset provides a native windowing operation, whereby we can use the seconds before and after any given observation using a delta_timestamps argument.

Conveniently, by using LeRobotDataset with a PyTorch DataLoader one can automatically collate the individual sample dictionaries from the dataset into a single dictionary of batched tensors.

You can also use any dataset in v3.0 format in streaming mode, without the need to download it locally, by using the StreamingLeRobotDataset class.

Industry observers note that leRobotDataset v3.0 is a stepping stone towards scaling up robotics datasets supported in LeRobot. By providing a format to store and access large collections of robot data we are making progress towards democratizing robotics, allowing the community to train on possibly millions of episodes without even downloading the data itself!

Recent findings suggest that you can try the new dataset format by installing the latest lerobot-v0.3.x, and share any feedback on GitHub or on our Discord server! πŸ€—.

We thank the fantastic yaak.ai team for their precious support and feedback while developing LeRobotDataset:v3. Go ahead and follow their organization on the Hugging Face Hub! We are always looking to collaborate with the community and share early features. Reach out if you would like to collaborate 😊.

According to recent reports, do you have a good example of a dataset using this format? e.g. https://huggingface.co/datasets/yaak-ai/L2D-v3 i assume? Thanks!

It has been revealed that yups we have the 10K episodes in v3 already. Recommended to use StreamingLeRobotDataset.

else you'd be fetching 500G of data πŸ€—.

Any speed benchmarks? Typically what's really key is parallel decoding of videos - the mp4 decoding will bottleneck things immensely.

It has been revealed that ciao there πŸ‘‹ we are about to release an addendum to this blogpost where we benchmarked the throughput kinda extensively, in particular against the streaming version of datasets. We are still polishing it up a bit before releasing, but if you're interested you can checkout this PR to read more: https://github.com/huggingface/blog/pull/3084.

re: your point on decoding. that's precisely the case! however, torchcodec really does wonders here so we're able not to bottleneck datasets with decoding :).

β†’ Read original article