Supported Models & Precision

Today, daisy-vllm surgically targets the compute-heavy MLP projections, the layers that consume over 80% of your GPU time. This is the first compilation target; attention and KV cache management are planned as the next layers to bring under docc compilation.

---

1. Model Support Matrix

daisy-vllm accelerates inference by replacing compute-heavy SwiGLU MLP feed-forward layers with specialized, fused docc ROCm kernels while keeping vLLM's native PagedAttention:

Model Family	Specific Checkpoints	Accelerated Submodules	Status
Qwen 2.5 (incl. Coder)	Qwen2.5-1.5B, Qwen2.5-7B	Fused Gate/Up & Down MLP Projections (docc_mlp)	Validated
Qwen 3.8	Qwen3.8-27B	Fused Gate/Up & Down MLP Projections (docc_mlp)	Validated
Qwen 3.8	Larger variants	Multiple layers: docc_mlp, docc_attention, docc_kv_caching	Planned

Roadmap Note: MLP acceleration (docc_mlp) is validated and available today. The next planned targets are attention (docc_attention) and KV cache management (docc_kv_caching), which will extend docc coverage across the full transformer block. See the Qwen 3.8 "Planned" row above for the first model family targeting all three.

---

2. Precision & Data Types

You can configure model execution precision via the -dt or --dtype CLI flag, or via the dtype parameter in Python:

Dtype	PyTorch Equivalent	Bytes / Param	Status	Description
bfloat16	torch.bfloat16	2 bytes	Validated	Recommended standard for AMD Instinct (MI200/MI300) and RDNA 3/4 hardware.
float16	torch.float16	2 bytes	Validated	Standard half-precision floating-point inference.
float32	torch.float32	4 bytes	Validated	Full single-precision baseline and numerical debugging.
fp8	torch.float8_e4m3fn	1 byte	Planned	Next-gen 8-bit floating point high-throughput inference on MI300/RDNA4.
int8	torch.int8	1 byte	Planned	8-bit integer quantized weight and activation GEMM acceleration.

---

3. DaisyLLM Configuration Parameters

When using DaisyLLM or passing arguments to the engine harnesses, you can customize the engine's memory footprint, batching limits, context window, and parallelism.

Code Example

from daisy_vllm.entrypoints.llm import DaisyLLM

llm = DaisyLLM(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    dtype="bfloat16",
    max_model_len=4096,
    max_num_seqs=1,
    gpu_memory_utilization=0.70,
    tensor_parallel_size=1,
    seed=1320,
    trust_remote_code=True,
    enforce_eager=False,
)

Complete Parameter Reference

The following table details all configuration parameters accepted by DaisyLLM with their tested defaults:

Parameter	Type	Default	Description
model	str	(Required)	Hugging Face model repository ID (e.g. Qwen/Qwen2.5-1.5B-Instruct) or absolute path to a local weights directory.
dtype	str	"bfloat16"	Model parameter and activation precision. Recommended: bfloat16 for modern ROCm hardware.
max_model_len	int	4096	Maximum total context window size (prompt tokens + generated tokens). Default 4096 guarantees stability without excessive KV cache pre-allocation.
max_num_seqs	int	1	Maximum concurrent sequences batched in parallel. Kept at 1 for reproducible, lowest-latency single-stream benchmarking.
gpu_memory_utilization	float	0.70	Fraction of GPU VRAM reserved for weights and KV cache. Default 0.70 provides a safe buffer that prevents ROCm OOM crashes.
tensor_parallel_size	int	1	Number of AMD GPUs to shard model layers across via ROCm NCCL (default 1 for single-GPU).
seed	int	1320	Deterministic random seed used for sampling, ensuring repeatable benchmark outputs across runs.
trust_remote_code	bool	True	Whether to allow execution of custom Python modeling code from the Hugging Face repository.
enforce_eager	bool	False	Disables vLLM compilation and CUDA graphs when True, but docc-compiled MLP custom ops still execute. Use stock vllm.LLM with enforce_eager=True for a non-docc eager baseline.
quantization	str | None	None	Quantization algorithm to apply when loading weights (e.g., None, fp8, awq, gptq).
compilation_config	CompilationConfig	(Auto)	Automatically configured by daisy-vllm to use CUDAGraphMode.PIECEWISE with daisy_vllm::docc_mlp in splitting_ops.
tokenizer	str | None	None	Override Hugging Face tokenizer name or local path if different from model.
chat_template	str | Path | None	None	Path to a custom Jinja2 template file for formatting conversational messages into token strings.
hf_token	str | None	None	Hugging Face authentication token for accessing gated repositories. Loaded automatically from .env.