LazyLLM is a novel method designed to accelerate the inference of transformer-based large language models (LLMs). This technique focuses on improving the time-to-first-token (TTFT) during the inference process, which can be a bottleneck for long prompts.

Key Features of LazyLLM:

• Dynamic Pruning: Selectively computes the key-value (KV) cache for important tokens, reducing unnecessary computations.
• Seamless Integration: This can be integrated with existing language models without requiring fine-tuning.
• Performance Boost: Accelerates the pre-filling stage by 2.34x in models like LLama 2 7B while maintaining accuracy.

Understanding LLM Inference Stages

LLM inference consists of two sequential stages:

1. Prefilling Stage: Computes and saves the KV cache of each token from the prompt and predicts the first token.
2. Decoding Stage: Uses the cached KVs to iteratively generate subsequent tokens.

For long prompts, computing the KV cache during the prefilling stage can be time-consuming, significantly increasing TTFT.

The Concept of LazyLLM

LazyLLM dynamically selects important tokens for the KV computation in both the prefilling and decoding stages. Unlike static pruning methods that reduce the prompt at once, LazyLLM allows the model to select different subsets of tokens at different generation steps, even if they were pruned in previous steps. This dynamic approach helps maintain accuracy while reducing computational load.

Implementation and Results

LazyLLM has been tested extensively on standard datasets, showing significant improvements across various tasks. For instance, in the multi-document question-answering task, LazyLLM accelerates the prefilling stage of the LLama 2 7B model by 2.34x while maintaining accuracy.

Progressive Token Pruning

LazyLLM prunes tokens progressively, based on their importance determined by attention scores. Tokens with lower attention scores are pruned in successive layers of the transformer, reducing computational overhead. An auxiliary cache (Aux Cache) is used to store the hidden states of pruned tokens, ensuring they can be recomputed efficiently if needed later.

Comparative Analysis

LazyLLM has been compared with standard LLM inference and other baselines, such as random token dropping, static token pruning, and prompt compression. The results consistently show that LazyLLM achieves better TTFT speedup with negligible accuracy drop across multiple tasks.

Detailed Evaluation

• Single-Document QA: LazyLLM achieved a TTFT speedup of 1.36x with minimal accuracy loss.
• Multi-Document QA: Showed a TTFT speedup of 2.34x, maintaining accuracy.
• Summarization: Improved TTFT speedup by 1.46x with negligible performance impact.
• Few-Shot Learning: Achieved a TTFT speedup of 2.19x while retaining high accuracy.
• Synthetic Tasks: Showed a significant TTFT speedup of 2.89x with negligible accuracy drop.
• Code Completion: Improved TTFT speedup by 1.94x with minimal impact on performance.

LazyLLM presents a practical solution for improving the efficiency of LLM inference, particularly under long context scenarios. By dynamically pruning tokens and leveraging an auxiliary cache, LazyLLM significantly reduces TTFT and overall computational load while maintaining accuracy. This method can be seamlessly integrated into existing transformer-based LLMs without requiring any fine-tuning, making it a valuable tool for optimizing LLM performance.