Guide · 7 min
How to Set a Prompt Token Budget for RAG, Agents, and Chat
A simple framework to allocate tokens across system prompt, retrieval, history, and output — then validate with Fluxkit AI Lab.
Split the window into jobs
A context window is a budget. If retrieval eats 80% and history eats the rest, the model has no room for a useful answer. Decide up front: system instructions, tools/schemas, retrieved chunks, conversation history, and reserved output.
Write those slices as percentages or hard caps (for example: 10% system, 50% RAG, 20% history, 20% output reserve). Then measure real text — do not trust word-count guesses.
Validate with one draft
Assemble a worst-case prompt (long system + top-k chunks + multi-turn history) in AI Lab. Compare GPT/Claude/Gemini estimates. If you are over budget on the family you ship, shrink retrieval first, then history, then instructions.
- Prefer fewer high-signal chunks over dumping an entire corpus.
- Summarize old turns instead of replaying full transcripts.
- Rebuild the prompt with the Prompt Builder template, then re-measure.
Revisit when models change
New model tiers change both price and practical context. Keep a golden fixture prompt and re-run it in AI Lab whenever you switch vendors or raise top-k. Share the hash link in the PR so reviewers see the budget impact.
FAQ
How large should the system prompt be?
As small as it can be while staying reliable. Many products waste hundreds of tokens on unused rules. Measure, delete, and re-test.
Do RAG chunks count as input tokens?
Yes. Retrieved text is input. It often dominates cost more than the user question itself.