LLM Inference Scheduling: A Survey of Techniques, Frameworks, and Trade-offs
Morgan Heisler, Zahra Yousefijamarani, Xinglu Wang, Qian Wang, Ge Shi, Hanieh Sadri, Timothy Yu, Yifei Li 等 26 位
Huawei Technologies (Canada)
阅读操作
确认中在文库中上传 PDF 后可生成中文音频讲解。
摘要与影响
Large Language Models (LLMs) require substantial computational resources, making efficient inference a critical challenge. As the demand for LLMs grows in serverless and large-scale deployments, scheduling techniques play a pivotal role in optimizing inference performance, balancing latency, cost, and throughput. This survey provides a comprehensive overview of scheduling strategies for LLM serving across both the control and data planes. We explore key challenges such as cold start latency, HOL blocking, resource utilization, and dynamic workload variations. We analyze scheduling techniques, including reinforcement learning-based workload management, adaptive batching, speculative execution, and multi-tenant scheduling. Additionally, we compare scheduling frameworks across latency reduction, cost efficiency, and scalability. Finally, we identify open research directions, including ML-driven adaptive scheduling, energy-aware inference, and heterogeneous hardware utilization. This survey aims to serve as a foundational reference for researchers and practitioners optimizing LLM inference at scale and is maintained as a live document, updated monthly to reflect ongoing developments.
逐年被引趋势
关键指标
同类平均 = 1
同领域 · 同年份 · 同类型
Google Scholar 与 OpenAlex 的被引统计范围不同,数值存在差异属正常。
AI 辅助阅读
依据:摘要
可就本文提问;依据不足时会说明。
学术脉络
学科主题
计算机 / AICloud Computing and Resource Management
Parallel Computing and Optimization Techniques · Distributed and Parallel Computing Systems