Sylvia Xiao | AI Engineer
文章
14
分类
2
标签
16
Read Blog
🏋🏼
Memory and Computation in LLM Inference: Transformers vs. RWKV
发布于: 2025-2-23
最后更新: 2026-5-28
次查看
llm
RWKV
Tansformer
目录
0%
Memory and Computation in LLM Inference: Transformers vs. RWKV
1. Memory Usage and GPU Utilization
1.1 Memory Usage (VRAM)
1.2 GPU Utilization
2. Data Flow in Transformer-Based LLMs
2.1 Two-Stage LLM Inference with KV-Cache
2.1.1 Prompt Stage
2.1.2 Decoding Stage
2.2 Self-Attention and KV-Cache in Standard Transformers
2.2.1 Computational Complexity Analysis
2.2.2 Space Complexity
3. The RWKV Architecture
3.1 Data Flow in RWKV
3.2 Time Mixing and Channel Mixing Details
3.3 RWKV Inference Process
3.3.1 Processing the Prompt
3.3.2 Generating New Tokens
3.4 Memory Usage in RWKV
3.5 A Pitfall in the MLC Framework’s RWKV5 Implementation
4. Comparison of RAM Usage: Transformers vs. RWKV
SylviaXiao
AI Engineer
文章
14
分类
2
标签
16
最新发布
Qwen-Image Text-to-Image Pipeline Note
2026-6-22
Whisper’s Encoder-Decoder Pipeline, Special Tokens, and a Comparison with Qwen3-ASR
2026-6-17
Financial / Universe Robot Dialogue AI Interface
2026-6-2
Real-time Logging in Model Training Using Python
2026-5-28
Building MLC-LLM: A Step-by-Step Guide
2026-5-28
Memory and Computation in LLM Inference: Transformers vs. RWKV
2026-5-28
目录
0%
Memory and Computation in LLM Inference: Transformers vs. RWKV
1. Memory Usage and GPU Utilization
1.1 Memory Usage (VRAM)
1.2 GPU Utilization
2. Data Flow in Transformer-Based LLMs
2.1 Two-Stage LLM Inference with KV-Cache
2.1.1 Prompt Stage
2.1.2 Decoding Stage
2.2 Self-Attention and KV-Cache in Standard Transformers
2.2.1 Computational Complexity Analysis
2.2.2 Space Complexity
3. The RWKV Architecture
3.1 Data Flow in RWKV
3.2 Time Mixing and Channel Mixing Details
3.3 RWKV Inference Process
3.3.1 Processing the Prompt
3.3.2 Generating New Tokens
3.4 Memory Usage in RWKV
3.5 A Pitfall in the MLC Framework’s RWKV5 Implementation
4. Comparison of RAM Usage: Transformers vs. RWKV
