Digital Event Horizon
Hugging Face and SkyPilot have joined forces to provide zero-egress storage for AI models and datasets. This partnership enables users to store and run their AI workloads on any cloud without incurring egress fees, thanks to SkyPilot's advanced compute capabilities.
Hugging Face collaborates with SkyPilot for running AI workloads on any cloud. Hugging Face Storage becomes a first-class SkyPilot backend, enabling zero egress fees. No cross-cloud transfer taxes or extra fees for reading data onto GPUs. Use of Xet-backed storage for Hugging Face Buckets leads to cost savings and improved performance. Integration of hf-mount FUSE backend with SkyPilot's storage solution.
Hugging Face, a leading provider of natural language processing (NLP) tools and models, has recently announced a major collaboration with SkyPilot, a pioneering platform for running AI workloads on any cloud. The partnership brings together the best of both worlds, enabling users to store their models and datasets in Hugging Face's storage solution while leveraging SkyPilot's advanced compute capabilities.
At the heart of this collaboration is Hugging Face Storage, which has been upgraded to become a first-class SkyPilot backend. This means that users can now run their AI workloads on any cloud, with zero egress fees, thanks to SkyPilot's ability to automatically determine where to run the workload based on GPU availability. No longer do users need to worry about cross-cloud transfer taxes or paying extra for reading their own data onto GPUs.
One of the key benefits of this partnership is the ability to use Hugging Face Storage without incurring egress fees, regardless of which cloud the compute resources are running on. This is particularly significant for AI workloads that require frequent reads from storage, such as fine-tuning models or serving predictions. By streaming data directly from the storage solution, users can avoid paying for egress fees, which can add up quickly.
Another important feature of this collaboration is the use of Xet-backed storage for Hugging Face Buckets. This technology allows for deduplication of checkpoints and model variants, meaning that only the changed chunks need to be transferred when updating a checkpoint or running a new variant. This can lead to significant cost savings and improved performance.
The partnership also involves the integration of Hugging Face's hf-mount FUSE backend with SkyPilot's storage solution. This allows users to mount their Hugging Face Buckets directly into SkyPilot tasks, making it easy to access and manage their data without having to jump through hoops.
To get started with this new capability, users simply need to install the "skypilot[huggingface]" package, log in to their Hugging Face account, and add an hf:// mount to their SkyPilot task. This enables them to use Hugging Face Storage with zero egress fees, while still benefiting from SkyPilot's advanced compute capabilities.
In conclusion, the partnership between Hugging Face and SkyPilot marks a significant milestone in the field of AI storage and computing. By providing users with a seamless and cost-effective way to run their AI workloads on any cloud, this collaboration has the potential to revolutionize the way businesses and organizations approach AI development and deployment.
Related Information:
https://www.digitaleventhorizon.com/articles/Hugging-Face-and-SkyPilot-Join-Forces-to-Revolutionize-AI-Storage-with-Zero-Egress-Capability-deh.shtml
https://huggingface.co/blog/skypilot-hf-storage
Published: Tue Jul 7 13:53:52 2026 by llama3.2 3B Q4_K_M
