How It Works
To understand why daisy-vllm exists, you have to look at what GPUs actually spend their time doing during large language model inference.
The Bottleneck: Why MLP Layers Dominate Execution Time
When profiling transformer inference on modern GPUs, there is a common misconception that attention is the primary compute bottleneck. In reality, across modern decoder models (such as LLaMA and Qwen), the vast majority of GPU kernel time is consumed by non-attention dense matrix multiplications especially during prefill stage.
Here is real profiling data from an AMD Instinct MI210 (gfx90a) running Qwen 2.5-1.5B in stock vLLM (VLLM_COMPILE mode, 128 input tokens):
Profiled on AMD Instinct MI210 (gfx90a), Qwen 2.5-1.5B, VLLM_COMPILE mode, 128 input tokens. Raw data from profile_ops.py.
| Kernel Category | GPU Kernel Time | Share of Total |
|---|---|---|
| GEMM (MLP Projections: Gate, Up, Down) | 37.0 ms | 82.9% |
| Attention (PagedAttention + FlashAttention) | 5.2 ms | 11.7% |
| LayerNorm + Elementwise Activations + Other | 2.4 ms | 5.4% |
Over 82.9% of total GPU time is spent in matrix multiplications inside the MLP layers.
How docc Integrates with vLLM
daisy-vllm does not replace vLLM. vLLM is an industry standard for high-throughput serving, request scheduling, continuous batching, and paged memory management. Instead, daisy-vllm surgically accelerates the bottleneck by embedding docc (docc.pytorch) directly into vLLM's execution pipeline.
Compilation Roadmap: MLP is Target #1
daisy-vllm targets MLP layers first because that is where 83% of the execution time is spent. Accelerating MLP provides the highest performance return per line of code.
However, the piecewise architecture is designed to expand across the entire transformer block:
| Compilation Target | Splitting Op | Status | Technical Focus |
|---|---|---|---|
| SwiGLU MLP | docc_mlp | Validated | Fused Gate/Up GEMM + SiLU epilogue + Down GEMM i.e. the bottleneck |
| Multi-Head Attention | docc_attention | Planned | Custom flash-attention kernels tailored to AMD CDNA/RDNA matrix cores |
| KV Cache Management | docc_kv_caching | Planned | Shape-specialized paging and cache reshape kernels |
As additional compilation targets mature, docc coverage expands across the transformer layer, progressively minimizing dependence on generic library kernels while maintaining full vLLM compatibility.
Next Steps
- Quickstart: Set up the container and run your first benchmark.
- Workload Benchmarking: Compare latency and throughput against stock vLLM.
- Correctness Verification: Verify numerical accuracy against eager PyTorch.
- Supported Models & Precisions: Check validated model families and configuration flags.