Digital Event Horizon
Researchers at Hugging Face have developed custom kernels for AMD MI300 GPUs, achieving significant speedups over traditional kernel implementations. By employing a range of techniques, including fused residual connections, optimized memory management, and sparsity trick optimization, these custom kernels have transformed performance on this device.
Hugging Face collaborated with AMD to deliver state-of-the-art performance on AMD platforms and benefit the open-source community. The team employed techniques such as fused residual connections, optimized memory management, warp specialization, and sparsity trick optimization to achieve significant speedups. The MI300X GPU features multiple threads, warps, compute units, and XCDs that can be optimized for matrix multiplications. The sparsity trick optimization technique involves padding inputs with fewer rows than the maximum supported by the tensor core instruction to take advantage of sparse nature of certain matrices. Warps are groups of threads that execute concurrently, and optimizing these warps can result in additional speedups. The custom kernels developed for AMD MI300 GPUs have achieved notable speedups over traditional kernel implementations, ranging from 98.80% to 127.84%.
Creating a custom kernel is an arduous task that requires a deep understanding of the inner workings of the GPU. Kernels are low-level, highly-optimized algorithms that are tailored to run on specific hardware devices, and they play a crucial role in executing operations within neural networks. However, kernel developers often overlook AMD GPUs, despite their comparable or superior specifications. To address this issue, Hugging Face collaborated with AMD to deliver state-of-the-art performance on AMD platforms and make it benefit the open-source community.
As part of this partnership, Hugging Face focused on delivering optimized kernels to improve the performance of serving Llama 3.1 405B in FP8 on a node powered by MI300X GPUs using VLLM. The goal was to achieve significant speedups when running VLLM on these devices. To accomplish this, the team employed a range of techniques, including the use of fused residual connections, optimized memory management, and warp specialization.
The MI300X GPU features multiple threads, warps, compute units, and XCDs, which are specialized instructions for performing matrix multiplications. The tensor cores on the MI300X handle these multiplications at high speeds. However, not all instructions are equally beneficial, and some can be optimized to take advantage of the sparse nature of certain matrices.
For instance, when dealing with inputs that have fewer rows than the maximum supported by the tensor core instruction, padding is required. This results in wasted compute resources. By employing a sparsity trick, where half of the matrix is described using a dense 8-row matrix and the other half using a sparse 16-row matrix, significant speedups can be achieved.
In addition to this optimization technique, warp specialization was also employed. Warps are groups of threads that execute concurrently, and by optimizing these warps, additional speedups can be gained. Asynchronous execution further enhances performance by allowing multiple warps to operate simultaneously.
The results of these optimizations were impressive, with speedups ranging from 98.80% for inputs with 32 rows to an astonishing 127.84% for low-input sizes. These gains are substantial and demonstrate the potential for custom-tailored kernels to transform the performance landscape for neural networks on AMD GPUs.
Furthermore, the team also explored how matrix multiplication is handled by the tensor cores on the MI300X GPU. The format of tensor core instructions can be optimized for specific use cases, such as using sparse instructions for matrices with a 4:2 sparsity pattern. By leveraging these optimization techniques, Llama 3.1 405B in FP8 can achieve notable speedups over traditional kernel implementations.
In conclusion, the partnership between Hugging Face and AMD has yielded significant gains in performance for neural networks on the MI300X GPU. Through a combination of fused residual connections, optimized memory management, warp specialization, and sparsity trick optimization techniques, custom kernels have been developed to tackle specific challenges associated with matrix multiplication on this device.
These findings demonstrate the potential for kernel customization to drive performance improvements across various hardware platforms. As researchers continue to push the boundaries of what is possible in AI development, the importance of optimizing kernels cannot be overstated.
The hf-rocm-kernels repository now offers a range of optimized kernels that can be used by developers and researchers seeking to explore these techniques further. With the emergence of new optimizations and advancements in GPU technology, it is essential for the community to continue exploring the vast potential offered by customized kernels.
Researchers at Hugging Face have developed custom kernels for AMD MI300 GPUs, achieving significant speedups over traditional kernel implementations. By employing a range of techniques, including fused residual connections, optimized memory management, and sparsity trick optimization, these custom kernels have transformed performance on this device.
Related Information:
https://www.digitaleventhorizon.com/articles/Unlocking-the-Potential-of-Customized-Kernels-for-AMD-MI300-A-Game-Changing-Partnership-deh.shtml
https://huggingface.co/blog/mi300kernels
Published: Wed Jul 9 11:43:54 2025 by llama3.2 3B Q4_K_M
