The Fundamentals of Context Management and Compaction in LLMs
Introduction
I’ve been thinking about context management in LLMs and agents lately and decided to start writing a few posts on them and share what I read and learn. In the first post, I try to explain the fundamentals. We are getting to an era that agents are more capable and can solve more complex jobs and can run longer. This means a context window of 200k or 400k is not enough and we need to find a solution to manage what we put there and what we remove from it. Even if the model can use the whole context window, there are usually several factors that we cannot consider:
- computational costs skyrocket
- inference slows
- and models suffer from “context rot”
In order to prevent this, we need to do context management and compaction using techniques like pruning, summarization, and optimization of information dynamically.
What is Context Management
Context management is usually considered as managing and controlling what input is fed into the LLM, like instructions, history, tools, and external data. Ensuring relevant data to the task is fed to the model while not overloaded. Poor context management leads to hallucinations or stalled tasks; effective management enables agents to maintain state over hours or days.
One of the main pillars of effective context management is compaction, which tries to reduce the size of a long and inefficient context intelligently. This not only preserves performance but also unlocks the potential for handling vastly larger effective contexts. Let’s dive deeper into its role and key methods.
The Role of Compaction
Compaction compresses context without losing its essence, often extending effective windows to millions of tokens by distilling information into more efficient forms. This is super important for agentic systems where interactions with the environment generates more data/tokens, which needs to be used by the agent, which means an overload context. How can we then decrease the number of tokens in the context window and be sure that we have not lost or thrown away the required info for the agent to complete the task? There are several ways like training the model to do it intelligently itself, which we will explore in the next posts, or using some external strategies like the followings:
Semantic Compression
This technique introduced in this paper and this paper asks a simple question: instead of using classical compression (like ZIP) or embeddings, can an LLM itself compress long text (or code descriptions) into a shorter prompt, and later “decompress” it well enough to still answer questions or regenerate code? The core idea is “approximate / semantic compression”. In many real tasks, you don’t actually need to reconstruct every original character perfectly. You mainly need the meaning (intent) to survive, so you can keep working despite context-window limits.
Loss-Aware Pruning
The main idea is to drop the parts of the prompt that least increase the model’s loss (or perplexity), under a token budget. Basically, you define a token budget, and you keep the subset of the context that best preserves the model’s ability to do the task.
This paper tries to make long inputs cheaper by pruning redundant parts of the context while keeping downstream quality roughly the same. The key idea is to score pieces of the input by how “important” they are for prediction (they use perplexity/LM confidence style scoring), then keep the best parts and drop the rest. They report large context reduction with small quality drop on summarization, QA, and long conversations.
Another major family is prompt compression methods like LLMLingua. LLMLingua does token-level pruning/compression guided by model signals (again, often perplexity-like “how predictable / low-information is this token here?”), plus budget controllers so you don’t over-compress and break meaning. That’s “loss-aware” in spirit: keep tokens that the model “needs,” drop the ones that don’t move the needle much.
There are also methods like Provence, which trains a small “pruner” model to remove irrelevant sentences from retrieved passages given the question.
Dynamic Summarization
The other category is dynamic summarization. The idea is: instead of deleting old messages (pruning) or squeezing them token-by-token (compression), you keep a living summary that you update over time. So when the chat gets long, you don’t carry the entire transcript in the prompt. You carry (1) the last N turns in full, plus (2) a compact summary of everything older. That summary is rewritten repeatedly as the conversation evolves. Langchain and LlamaIndex both have solutions for this.
The term dynamic in practice means two things:
- First, it’s triggered by state. For example: “when we hit 70% of the token budget, summarize the oldest part.” That’s why frameworks expose token limits and do automatic roll-ups. We see this a lot in coding agents like Claude Code or Codex.
- Second, it’s incremental. You don’t re-summarize the whole conversation every time (too expensive and it drifts). You usually do: “existing summary + new chunk of recent messages → updated summary.”
A paper called “Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models” is directly related to this category. The idea is exactly a running summary that gets updated again and again: old summary + new dialogue chunk → new summary, so you can keep coherence over very long conversations without keeping all turns.
Another interesting and related paper is DTCRS. The main idea is to build a tree of summaries only when it helps: given a question, the system first judges the question type, then decides whether recursive summarization is needed at all. If needed, it dynamically groups document chunks and creates higher-level summaries that match both the document’s structure and the query’s needs. This reduces wasted summaries and keeps useful evidence available for multi-step, abstractive questions in RAG-style settings, and speeds answers under token limits.
That it for this first post on this series. If you liked it and find it valuable, I appreciate if you repost and distribute, so more interested people can see it.
