The Allen Institute for AI rebuilt its GPU cluster scheduler after a priority-based system let every job claim top priority, the lab said in a blog post Oct. 9.
Jobs marked 100% high priority and idle workloads parked on hardware had made the old system unworkable, according to the post.
The new system allocates GPU time in hierarchical budgets that managers set based on research impact, tracks usage with a fair-share algorithm averaged over a rolling seven-day window, and lets jobs declare a minimum runtime upfront in exchange for protection against being preempted, according to Ai2.
Over a 30-day test, 13 of 15 team allocations hit at least 95% of their assigned GPU hours while the cluster stayed 98% occupied, Ai2 said. Median queue wait time on its largest cluster fell from five minutes to 24 seconds, debug job wait time dropped from two hours to 30 seconds, and on-call interventions needed to manually resolve scheduling conflicts fell 74%.
Ai2 runs thousands of Nvidia H100, B200 and B300 GPUs for roughly 150 researchers across distributed training, robotics reinforcement learning and other scientific work, according to the post.
Every lab running shared GPU capacity hits the same failure mode once everyone learns to mark their own job high priority. Ai2's fix, trading guaranteed capacity for an upfront commitment to a minimum runtime, is a trade any team fighting the same scheduler queue could borrow directly.