LlamaIndex re-retrieves your chunks — and re-sends up to 75% of your context — on every chat turn

You wire up a LlamaIndex agent, give it a couple of tools and a retriever, and it just works — reason, retrieve, call a tool, reason again, answer. The part the quickstart never puts a number on is what the agent re-sends to the model on every pass of that loop. On run-llama/llama_index@main, it's two independent re-sends, stacking, every single turn.

1. The default memory hands the model up to 75% of your context window#

LlamaIndex agents remember through ChatMemoryBuffer. Building the next call, it asks memory for the history via get_all() — the entire accumulated buffer: every user turn, every assistant reasoning step, every fat tool observation. Nothing is trimmed until the buffer crosses a ceiling. And the ceiling is not small:

DEFAULT_TOKEN_LIMIT_RATIO = 0.75
token_limit = token_limit or int(context_window * DEFAULT_TOKEN_LIMIT_RATIO)

(memory/chat_memory_buffer.py.) On a 128k-context model that's a ~96,000-token memory budget — allowed to grow to ~96k of accumulated history and re-sent on each agent step. It isn't a runaway that grows forever; it's a very high floor you pay over and over. Ten steps of a tool-heavy agent re-send a large, near-constant buffer ten times. Cost ≈ buffer size × number of steps.

2. In a RAG chat, retrieval re-runs every turn#

index.as_chat_engine() gives you ContextChatEngine. Its .chat() doesn't retrieve once and reuse — it re-runs the retriever against every new message (chat_engine/context.py):

nodes = self._get_nodes(message)   # -> self._retriever.retrieve(message)

Those fresh chunks get re-stuffed into the system prompt each turn via "{context_str}", with no dedup against what you already sent. In a follow-up-heavy chat the retriever frequently pulls the same top-k chunks again — and you pay to upload them again, on top of the growing history buffer. Two re-sends, stacking.

The knobs are real — set them on purpose#

This isn't a bug. LlamaIndex hands you the levers: cap ChatMemoryBuffer(token_limit=...) explicitly instead of inheriting the 0.75-of-context default, and reuse retrieved context across turns instead of re-retrieving blind. But you can't tune what you can't see.

Measure it first. npx @wartzar-bee/tokenscope breaks down exactly what your agent re-sends per step — buffer vs. re-retrieved context — so you cap the right thing instead of guessing. Then put a ceiling in CI so it never regresses: wartzar-bee/ci-guardrail fails the build past a max-usd you set.

Full walkthrough with every source anchor: https://dev.to/wartzarbee/llamaindex-re-retrieves-your-chunks-and-re-sends-up-to-75-of-your-context-on-every-chat-turn-10ka

#llamaindex #rag #ai #python #llm