Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Serving

Can you explain how inflation affects interest rates and the broader economy? ... According to the Taylor Rule, monetary policy should adjust the federal funds rate in response to deviations of actual inflation from the target rate and output from potential GDP. (User pauses to think and digest, while the LLM continues generating content) This increase in rates raises the cost of borrowing, reduces consumer spending and business investment, and can lead to slower economic growth ... Waiting User Normal User (Unaware of Change) Satisfied User How do I make olive oil garlic pasta? Sure! I'd be more than happy to help you with that. Olive oil garlic pasta is a wonderfully simple yet flavorful dish that's perfect for just about any occasion. (Streaming paused due to limited computational resource, leaving the user idle and disengaged) ... Can you explain how inflation affects interest rates and the broader economy? How do I make olive oil garlic pasta? ... According to the Taylor Rule, monetary policy should adjust the federal funds rate in response to deviations of actual inflation from the target rate and output from potential GDP. (Streaming is slowed, allowing the user time to digest the content and freeing up computational resources) ... Sure! I'd be more than happy to help you with that. Olive oil garlic pasta is a wonderfully simple yet flavorful dish that's perfect for just about any occasion. To get started, you'll need a few basic ingredients: spaghetti, garlic, extra virgin olive oil, red pepper flakes, salt, and parsley... (Streaming continues smoothly thanks to sufficient computational resources, and proceeds at a faster pace since the content is low in complexity) Normal User Figure 1: (a) In regular LLM streaming, the streaming speed is not associated with the content being delivered. Complex content may be streamed faster than users can comfortably read, resulting in wasted resource. Conversely, simple content may be streamed too slowly, causing users to wait unnecessarily. (b) In our proposed approach, the streaming speed is adapted based on the estimated content complexity. Complex content is slowed down to support user comprehension and optimize resource usage, while simpler content is streamed more quickly to better align with the user's natural reading speed.