Digital Event Horizon
The world of AI infrastructure is rapidly evolving, and Hugging Face's AI Infrastructure team has been at the forefront of this effort. In a recent article, the team shares their experiences and lessons learned in implementing a cutting-edge scheduling system that prioritizes high-impact research while maintaining full occupancy of their GPU clusters. Read on to learn more about their approach to fair-share scheduling and the results of their simulation and real-world testing.
The AI Infrastructure team at Hugging Face implemented a cutting-edge scheduling system to prioritize high-impact research while maintaining full occupancy of their GPU clusters. The team's approach is based on the concept of "fair-share," where managers allocate a portion of GPU time to researchers and incentivize them to manage their budgets. The system includes a hierarchical fair-share scheduler and a "scheduling contract" to ensure workloads are protected from preemption during their minimum runtime. The simulation results showed that the new scheduler has a larger mix of colors on each GPU, illustrating occupancy rotation across teams. The rollout of the new system was met with enthusiasm, with researchers consistently receiving their allocated GPU time and teams delivering 98% of the GPU hours they were owed. The team encountered challenges, including a steeper learning curve and capacity fragmentation, but implemented new visualizations and roadmap projects to address these issues.
The world of artificial intelligence (AI) is rapidly evolving, and with it, the need for more efficient and effective GPU cluster scheduling. Hugging Face, a leading provider of AI infrastructure, has been at the forefront of this effort. In a recent article, the AI Infrastructure team at Ai2 shares their experiences and lessons learned in implementing a cutting-edge scheduling system that prioritizes high-impact research while maintaining full occupancy of their GPU clusters.
At Ai2, the team is responsible for providing the institute's GPU compute capacity, specifically targeting large, distributed training workloads. They have identified four key metrics that build on each other: availability, occupancy, impact, and utilization. These metrics form the foundation of their scheduling system, which aims to optimize the allocation of GPU resources to maximize the impact of research projects.
The team's approach is built around the concept of "fair-share," where managers allocate a portion of GPU time to researchers and researchers are incentivized to manage their budgets. This system is designed to prevent "squatting," where researchers hold onto GPUs for extended periods, and to promote a culture of transparency and collaboration.
The team's new scheduling system is based on a hierarchical fair-share scheduler, which tracks occupancy over a sliding lookback window and sorts workloads from under-utilized allocations above those from over-utilized allocations. The system also includes a "scheduling contract," which ensures that workloads are protected from preemption during their minimum runtime, allowing researchers to make progress on their projects.
To test the effectiveness of their new system, the team built a small simulation environment that takes a set of workloads and their submission schedule as input and allows the scheduler to make preemption and GPU assignment decisions. The simulator was run against both historical submission data and constructed scenarios to predict queue wait times and distribution of GPU time across projects.
The results of the simulation were promising, with smaller-scale simulator visualization showing that the new scheduler has a larger mix of colors on each GPU, illustrating occupancy rotation across teams. The team's rollout of the new system was met with enthusiasm, with researchers consistently receiving their allocated GPU time and teams delivering 98% of the GPU hours they were owed.
However, the team also encountered challenges, including a steeper learning curve than anticipated and the need to address issues related to capacity fragmentation. To address these challenges, the team introduced new visualizations and roadmap projects to improve the user experience and address emerging issues.
In conclusion, Hugging Face's AI Infrastructure team has made significant strides in optimizing GPU cluster scheduling, prioritizing high-impact research while maintaining full occupancy. Their approach to fair-share scheduling and the use of a hierarchical fair-share scheduler has shown promising results, with smaller-scale simulator visualization and real-world results demonstrating improved queue wait times and distribution of GPU time across projects.
Related Information:
https://www.digitaleventhorizon.com/articles/Revolutionizing-GPU-Cluster-Scheduling-How-Hugging-Faces-AI-Infrastructure-Team-Tackled-the-Tragedy-of-the-Commons-deh.shtml
https://huggingface.co/blog/allenai/impactful-scheduling
Published: Fri Oct 9 12:45:13 2026 by llama3.2 3B Q4_K_M
