AI Trending
All 全部 Deep Analysis 深度解析 Foresight 前瞻 AI News AI资讯 Open Source 开源项目 AI Products AI产品 Research Papers 论文研究 AI Security AI安全 AI Practices AI实践 AI Skills AI技能 AI Overseas AI出海
AI Skills AI技能 8h ago • Updated 5h ago 更新于 5小时前 50

How To Build Your Own LLM Runtime From Scratch 如何从零开始构建自己的LLM运行时

The article details the development of a custom, from-scratch CUDA inference runtime for Qwen2.5-Coder-7B-Instruct on NVIDIA H100 GPUs, emphasizing deep ownership of the decoding stack over using black-box solutions. CUDA Graphs are identified as a critical optimization rather than an optional feature, yielding a 7x performance improvement by reducing driver overhead (dropping latency from ~119 ms/token to ~17 ms/token). Significant engineering challenges were encountered, including undefined be 从零构建针对 NVIDIA H100 (sm_90) 的 Qwen2.5-Coder-7B-Instruct CUDA 推理运行时，旨在深入理解底层机制而非替代生产级工具。 CUDA Graph 是性能优化的核心杠杆，将 eager decode 从 ~119 ms/token 降低至 ~17 ms/token，实现约 7 倍的性能提升。 当前实现稳态解码吞吐量约为 60 tok/s，首 token 延迟为 128 ms，虽低于 llama.cpp 参考值，但验证了自定义栈的可行性。 开发过程中揭示了三个关键陷阱：warp-specialized 分支内使用 `__syncthreads()`

65
Hot 热度
85
Quality 质量
70
Impact 影响力

Analysis 深度分析

TL;DR

  • The article details the development of a custom, from-scratch CUDA inference runtime for Qwen2.5-Coder-7B-Instruct on NVIDIA H100 GPUs, emphasizing deep ownership of the decoding stack over using black-box solutions.
  • CUDA Graphs are identified as a critical optimization rather than an optional feature, yielding a 7x performance improvement by reducing driver overhead (dropping latency from ~119 ms/token to ~17 ms/token).
  • Significant engineering challenges were encountered, including undefined behavior from __syncthreads() in warp-specialized branches and hardware-specific instruction choices where prmt.b32 outperformed __dp4a for specific GEMV shapes.
  • While the custom runtime achieves ~60 tok/s, it serves primarily as an educational reference against llama.cpp (~200 tok/s) to illustrate the complexity of high-performance LLM inference rather than to compete in raw speed.

Why It Matters

This resource provides invaluable insight for AI infrastructure engineers and researchers aiming to optimize LLM inference beyond standard libraries, highlighting the low-level realities of GPU programming such as memory barriers, warp specialization, and kernel scheduling. It demonstrates that achieving production-grade performance requires not just algorithmic efficiency but also meticulous management of hardware-specific constraints and driver interactions. By documenting common pitfalls and architectural decisions, it helps practitioners avoid costly debugging sessions when building custom inference engines.

Technical Details

  • Architecture & Hardware: The runtime targets NVIDIA H100 (sm_90) and implements a custom CUDA/C++ stack for Qwen2.5-Coder-7B-Instruct, handling weight packing, INT4 dequantization, paged KV cache, and attention mechanisms.
  • Performance Metrics: The custom runtime achieves a steady-state decode latency of ~16.7 ms/token (~60 tok/s) and a first-token latency of ~128 ms for a 512-token prompt, compared to llama.cpp's ~4.95 ms/token (~200 tok/s) under similar conditions.
  • CUDA Graph Optimization: The most significant performance lever was wrapping the decode loop in a captured CUDA Graph. Eager execution resulted in ~119 ms/token, while graph replay reduced this to ~17 ms/token, eliminating substantial CPU-side driver overhead.
  • Low-Level Pitfalls: The author highlights three critical lessons: using __syncthreads() inside warp-specialized branches causes undefined behavior and corrupts softmax outputs; CUDA graphs are mandatory for performance; and instruction selection matters, with prmt.b32 beating __dp4a for certain GEMV shapes, leading to the removal of an attempted INT8-activation path.

Industry Insight

  • Invest in Observability and Customization: Relying solely on existing frameworks like llama.cpp may limit the ability to implement novel quantization schemes, custom samplers, or architecture-specific optimizations. Building or deeply understanding a custom runtime provides the flexibility needed for cutting-edge research and specialized deployment scenarios.
  • Driver Overhead is a Major Bottleneck: The drastic performance gain from CUDA Graphs underscores the importance of minimizing CPU-GPU synchronization and driver API calls in high-throughput inference loops. Engineers should prioritize graph capture techniques for steady-state decoding phases.
  • Hardware-Specific Tuning is Non-Negotiable: Generic implementations often fail to exploit the full potential of modern GPUs like Hopper. Success requires deep knowledge of specific hardware instructions (e.g., prmt.b32 vs __dp4a) and careful management of warp-level synchronization to avoid subtle, hard-to-debug errors.

TL;DR

  • 从零构建针对 NVIDIA H100 (sm_90) 的 Qwen2.5-Coder-7B-Instruct CUDA 推理运行时，旨在深入理解底层机制而非替代生产级工具。
  • CUDA Graph 是性能优化的核心杠杆，将 eager decode 从 ~119 ms/token 降低至 ~17 ms/token，实现约 7 倍的性能提升。
  • 当前实现稳态解码吞吐量约为 60 tok/s，首 token 延迟为 128 ms，虽低于 llama.cpp 参考值，但验证了自定义栈的可行性。
  • 开发过程中揭示了三个关键陷阱：warp-specialized 分支内使用 __syncthreads() 导致未定义行为、CUDA Graph 的必要性以及 Hopper 架构上 prmt.b32 优于 __dp4a。
  • 项目通过精简代码库（约二十多个源文件）和详尽注释，展示了权重打包、INT4 反量化、Paged KV 及 TMA 等高级特性的实现细节。

为什么值得看

对于希望深入掌握 LLM 推理底层原理、摆脱黑盒依赖的 AI 工程师而言，本文提供了在 H100 架构上从零构建高性能解码栈的实战指南。它揭示了通过细粒度控制内核启动和同步原语来优化性能的具体路径，特别是 CUDA Graph 带来的巨大收益。

技术解析

  • 硬件与模型配置：运行环境为 NVIDIA H100 (sm_90)，模型为 Qwen2.5-Coder-7B-Instruct。模型参数包括 28 层解码器、隐藏层大小 3584、MLP 宽度 18944 及词表大小 152064。
  • 性能基准对比：在 batch=1, pp512/tg128 协议下，该运行时 TTFT 为 128ms，解码 ITL 为 16.7 ms/token (60 tok/s)。作为参考，llama.cpp (Q4_K_M) 在同一硬件上 TTFT 为 43ms，解码 ITL 为 4.95 ms/token (200 tok/s)。
  • CUDA Graph 优化：最大的性能突破来自将解码过程封装在 CUDA Graph 中。Eager 模式下的解码耗时约 119 ms/token，而捕获并回放 Graph 后降至 ~17 ms/token，证明了减少驱动开销的重要性。
  • 关键工程教训：
    1. 在 warp-specialized 分支内部使用 __syncthreads() 是未定义行为，会导致 KV 页面尾部的 Softmax 数据损坏。
    2. 在 Hopper 架构上，对于特定的 GEMV 形状（gate/up/down），prmt.b32 指令比 __dp4a 更快，因此放弃了 INT8 激活路径尝试。
  • 代码结构：项目名为 annotated-llm-runtime，包含约二十多个 CUDA/C++ 源文件，重点在于对每个热路径进行“为什么”而非仅仅“是什么”的注释，涵盖权重打包、INT4 反量化、Paged KV 管理等。

行业启示

  • 掌控力优于单纯的速度：虽然 llama.cpp 等成熟框架在绝对性能上领先，但构建自定义运行时赋予开发者对量化格式、采样器和注意力变体的完全控制权，这对于适配新硬件或特定业务需求至关重要。
  • CUDA Graph 是必选项：在现代 GPU 推理优化中，消除 CPU-GPU 调度开销是提升吞吐量的关键。任何追求极致性能的 LLM 推理引擎都必须将 CUDA Graph 作为标准组件，而非可选优化。
  • 架构特异性优化：不同 GPU 架构（如 Hopper）的指令集特性差异显著（如 prmt.b32 vs __dp4a）。通用优化策略往往失效，必须针对具体硬件架构进行细粒度的内核调优和指令选择。

Disclaimer: The above content is generated by AI and is for reference only. 免责声明：以上内容由 AI 生成，仅供参考。

LLM 大模型 Inference 推理 GPU GPU Code Generation 代码生成 Deployment 部署
Read Original → 阅读原文 →

Related Articles 相关文章

[GitHub] keras-team/keras Keras 3 多后端深度学习框架
Anthropic Surpasses OpenAI: The 'Code is King' Logic Behind $965 Billion Valuation Anthropic超越OpenAI：9650亿估值背后的「代码为王」逻辑
The Second Half of the AI War: No Longer About Who Has the Strongest Model, But Who Can Use It AI 战争下半场：不再比谁的模型强，比谁用得上
Loop Engineering for RAG Generation: iterate top-k one at a time RAG生成的循环工程：逐个迭代top-k
AI Teammates: how monday.com runs production AI agents on Amazon Bedrock AI队友：monday.com如何在Amazon Bedrock上运行生产级AI智能体