Correctness Verification
Speed means nothing without accuracy. When daisy-vllm fuses and recompiles MLP kernels through docc, it must produce mathematically identical results to uncompiled PyTorch eager execution. Here's how we prove it.
All four mathematical invariants passing for Qwen 2.5-1.5B-Instruct.
1. Running the Correctness Suite
Inside your container environment, execute the pytest test runner:
pytest -v -s tests/test_inference_correctness.py
This test suite executes across diverse test prompts (covering factual Q&A, math reasoning, system descriptions, and algorithmic code generation) and evaluates the generated logits against four mathematical invariants.
2. Evaluated Mathematical Invariants
The test suite validates four numerical invariants against the uncompiled eager baseline:
| Invariant | Metric | Passing Threshold | Purpose |
|---|---|---|---|
| Token Alignment | Greedy token overlap | >= 95% | Verifies the model outputs the same token sequence without divergence. |
| Candidate Overlap | Top-5 Jaccard index | >= 80% | Confirms the top 5 predicted token candidates match eager PyTorch. |
| Logprob Tolerance | Max Absolute Error (MAE) | <= 0.05 | Ensures floating-point differences on token logprobs remain strictly bounded. |
| Distribution Drift | Top-5 KL-Divergence | <= 0.01 | Confirms the candidate probability distribution is statistically faithful. |
3. Interpreting Test Results
When running pytest -v -s tests/test_inference_correctness.py, the test suite prints per-prompt metrics and verification status:
Test #1: [PASS]
Prompt : Explain the difference between latency and throughput in computer systems in two sentences.
Generated Tokens : Eager=32 | Compiled=32
Token Alignment : 100.00% (Exact Match: True)
Max Logprob Delta (L∞) : 0.014231 (Threshold <= 0.05)
Avg Logprob Delta : 0.004120
Max Top-K KL-Divergence: 0.001824 (Threshold <= 0.01)
Top-5 Candidate Overlap: 92.50% (Threshold >= 80%)
============================================================
RESULT: ALL MATHEMATICAL INVARIANT TESTS PASSED.
============================================================
Each invariant provides a different guarantee. Together, they ensure that daisy-vllm's compiled kernels are not just fast, they are mathematically faithful.