Digital Event Horizon
Together AI's Road to Revolution: How the Company Ditched Megakernels for a New Era of Continual Compilation
Together AI shifted from megakernels to Polymorphic Inference in their Together Compiler. The company removed an infrastructure to help agents reason about a model's forward pass to focus on high-fidelity measurements and agent experimentation throughput. Polymorphic Inference achieves comparable performance to megakernels without the complexity and rigidity of a single persistent kernel. The new approach allows agents to optimize forward passes without explicit guidance, paving the way for a new era of production open-source inference systems.
In the ever-evolving landscape of artificial intelligence, Together AI has taken a significant step forward in their pursuit of optimizing AI inference. Last year, the company built megakernels, a type of optimized execution system that can significantly improve performance. However, this year, they took a drastic turn by deleting megakernels and embracing a new approach: Polymorphic Inference, which is being implemented in their Together Compiler.
To understand the context behind this shift, it's essential to delve into the history of megakernels and the limitations they posed. Megakernels were designed to optimize performance by eliminating intermediate kernel launches, overlapping work across operations, and dynamically managing GPU resources. However, as the company delved deeper into their research, they discovered that megakernels were not the ideal target for optimization. In fact, they found that the benefits of megakernels were limited and that alternative solutions could achieve similar performance gains without the complexity and rigidity of a single persistent kernel.
One of the key findings was that the benefits of megakernels, such as eliminating kernel launch overhead and overlapping operations, could be achieved through simpler primitives like CUDA graphs and Programmatic Dependent Launch (PDL). By leveraging these primitives, Together AI was able to achieve performance levels comparable to their megakernel baseline without the need for a monolithic, single-kernel approach.
However, the company's journey didn't end there. As they continued to explore the optimization space, they realized that their agents needed a more structured framework to optimize forward passes. They built an elaborate infrastructure to help agents reason about a model's forward pass, but ultimately found that this scaffolding could constrain their agents.
In a significant shift, Together AI decided to remove this infrastructure entirely, letting their agents roam free. This approach allowed them to focus on providing high-fidelity measurements and ensuring high agent experimentation throughput. The result was a system that not only achieved performance levels comparable to their megakernel baseline but also demonstrated the potential for agents to optimize forward passes without the need for explicit guidance.
TheTogether Compiler, now under development, represents a major milestone in the company's pursuit of continual compilation for AI inference. By embracing Polymorphic Inference and removing the need for explicit guidance, Together AI is poised to revolutionize the field of AI inference. With their new approach, the company aims to provide a scalable, flexible, and efficient solution for optimizing AI models, paving the way for a new era of production open-source inference systems.
Related Information:
https://www.digitaleventhorizon.com/articles/The-Road-to-Together-Compiler-A-Journey-from-Megakernels-to-Polymorphic-Inference-deh.shtml
https://www.together.ai/blog/road-to-together-compiler
Published: Mon Nov 16 20:09:55 2026 by llama3.2 3B Q4_K_M