Digital Event Horizon
Migrating to Xet: A Seamless Storage Upgrade
The Hugging Face team successfully migrated their storage infrastructure from Git Large File Storage (Git LFS) to Xet. The migration aimed to achieve two primary objectives: "Do no harm" and "Drive the most impact as fast as possible". The team developed an innovative infrastructure component called the Git LFS Bridge to facilitate seamless communication between Git LFS and Xet. Over 500,000 repositories were migrated to Xet in a relatively short period, with more than 1 million users utilizing Xet today. The team plans to bring Xet to all platforms, starting this month, and default new repositories to be Xet-enabled by default.
The Hugging Face team has made significant strides in migrating their storage infrastructure from Git Large File Storage (Git LFS) to a new system called Xet. This transition, which began last year, represents a major milestone in the company's efforts to improve the scalability and performance of its platform.
According to the team, they aimed to achieve two primary objectives during this migration: "Do no harm" and "Drive the most impact as fast as possible." By taking a phased approach and introducing Xet-enabled repositories gradually, the team minimized disruption to users and allowed them to continue working with their existing workflows.
To facilitate this transition, the Hugging Face team developed an innovative infrastructure component known as the Git LFS Bridge. This bridge acts as a middleman between Git LFS and Xet, enabling seamless communication between the two systems and ensuring that files are transferred efficiently.
The team also implemented a "background migration process" to handle the bulk of the data transfer. This process involves triggering webhooks that push events to a distributed queue, which is then processed by an orchestrator. The orchestrator enables Xet on repositories as needed, fetches LFS revisions for each file, batches files into jobs based on size or number, and places these jobs on another queue for migration worker pods.
The migration process has been remarkably smooth, with over 500,000 repositories migrated to Xet in a relatively short period. Today, more than 1 million users are utilizing Xet, and the platform is now defaulting to Xet-enabled repositories for new users and organizations.
To address potential issues and optimize performance, the team conducted extensive testing and gathered valuable insights from power users and large-scale migrations. By learning from these experiences, they were able to refine their approach and improve the scalability of their system.
The latest development in this story is that the Hugging Face team plans to bring Xet to everyone on the platform. Starting this month, all repositories will migrate from LFS to Xet, and newly created repos will be Xet-enabled by default. This move aims to further accelerate the adoption of Xet and provide faster transfers for users.
As the company continues to push the boundaries of what is possible with storage infrastructure, it's clear that their commitment to innovation and community support has been instrumental in making this transition a success.
Related Information:
https://www.digitaleventhorizon.com/articles/Migrating-the-Hub-to-Xet-A-Seamless-Storage-Upgrade-deh.shtml
https://huggingface.co/blog/migrating-the-hub-to-xet
Published: Tue Jul 15 11:36:49 2025 by llama3.2 3B Q4_K_M
