Introduction to AutoKernel
AutoKernel is a revolutionary tool that leverages autonomous AI agents to optimize GPU kernels for any given PyTorch model. Inspired by the concept of autoresearch, AutoKernel applies the same philosophy to GPU kernel optimization, allowing users to simply provide a PyTorch model and let the agent work its magic.
How AutoKernel Works
AutoKernel operates by first profiling the provided PyTorch model to identify bottleneck kernels. It then extracts these kernels as standalone Triton or CUDA C++ files and optimizes them autonomously through an edit-benchmark-keep/revert loop. This process ensures that each kernel is optimized to its fullest potential without compromising the overall model’s correctness.
Key Components of AutoKernel
- Profile.py: Profiles any PyTorch model and ranks kernels by GPU time.
- Extract.py: Extracts top-N bottleneck kernels into standalone Triton or CUDA C++ kernel files.
- Orchestrate.py: Decides which kernel to optimize next using Amdahl’s law and tracks aggregate progress.
- Bench.py: A fixed benchmark that checks correctness and performance.
- Verify.py: Plugs optimized kernels back into the model and checks end-to-end correctness, reporting total speedup.
Supported Kernels and Example Models
AutoKernel supports 9 kernel types covering core deep learning operations and comes with self-contained model definitions for GPT-2, LLaMA, and BERT. Users can easily integrate their custom models and optimize them using AutoKernel.
Kernel Types
- Matmul: Dense matrix multiplication.
- Softmax: Row-parallel numerically stable softmax.
- LayerNorm: Layer normalization with affine transform.
- RMSNorm: RMS normalization (LLaMA-style).
- Flash Attention: Scaled dot-product attention with causal masking.
- Fused MLP: SwiGLU-style fused MLP.
- Cross Entropy: Fused cross entropy loss.
- Rotary Embedding: Rotary position embeddings (RoPE).
- Reduce: Parallel reduction (sum).
Getting Started with AutoKernel
To start using AutoKernel, users need to install the required dependencies, clone the AutoKernel repository, and set up their environment. The quick start guide provides step-by-step instructions on how to profile a model, extract bottleneck kernels, and verify the benchmark.
KernelBench Integration
AutoKernel integrates seamlessly with KernelBench, allowing users to evaluate and optimize kernels using the standard benchmark for AI-generated GPU kernels. This integration enables systematic exploration of the optimization space instead of relying on one-shot LLM generation.
Conclusion and Call to Action
AutoKernel is a powerful tool that simplifies the process of optimizing GPU kernels for deep learning models. By leveraging autonomous AI agents, users can achieve significant speedups without requiring extensive expertise in kernel optimization. To get started with AutoKernel, follow the quick start guide and explore the project’s documentation for more information.
Frequently Asked Questions
- What is AutoKernel, and how does it work?
- How do I get started with AutoKernel?
- What types of kernels does AutoKernel support?
- Can I use AutoKernel with my custom model?
- How does AutoKernel integrate with KernelBench?
