Academic Insight Talks

Mingxing Zhang

Associate Professor, Tsinghua University.

Talk Title:

How Can We Reduce Token Costs in the Agent Era?

Abstract: Large language model applications are evolving from single-turn question answering to complex multi-turn agentic reasoning. A single task is no longer just one model generation, but instead involves multiple model invocations, extensive context reads and writes, tool interactions, and continuous state maintenance. This trend is driving rapid growth in token consumption and making inference cost a central challenge for scaling agent applications. This talk focuses on "how to reduce token costs" and presents a whole-system heterogeneous collaboration approach. In the cloud, the Mooncake architecture enables KVCache-centric disaggregated inference. At the edge, the KTransformers framework maps different parts of a model onto the most suitable heterogeneous hardware. During agent execution, environment reuse reduces the overhead of repeated initialization and state management. Together, these techniques improve compute utilization, lower inference costs, and help AI infrastructure evolve toward more efficient and broadly accessible intelligent services.

Speaker Bio: The speaker is an Associate Professor at Tsinghua University whose research focuses on memory systems. He is the initiator of the open-source projects Mooncake and KTransformers. His work has been published in leading international conferences and journals, including OSDI, SOSP, ASPLOS, HPCA, and EuroSys, with more than 40 papers in total. His research has received several distinctions, including a FAST Best Paper Award, a SIGSOFT Distinguished Paper Award, and the first OSDI paper from a university in mainland China. He has also received the ChinaSys Rising Star Award, the ChinaSys Distinguished Doctoral Dissertation Award, and the IEEE TCSC Outstanding Ph.D. Dissertation Award. He has been selected for the Ministry of Education's U40 Program and the China Association for Science and Technology's Young Talent Support Program, and has served as a project lead for a National Key R&D Program of China. Previously, he served as Chief Algorithm Technology Expert and Director of the Innovation Research Institute at Sangfor Technologies, where the incubated products were deployed by tens of thousands of customers.