A Token Budget is an Architectural Constraint

In the early stages of building a production AI system, developers often treat Large Language Model (LLM) tokens like an infinite resource. During the prototyping phase, a few thousand extra tokens in a prompt seem negligible. But as I transitioned systems from proof-of-concept to production environments—specifically within the rigorous demands of HealthTech—I realized that tokens are not just a billing line item. They are a hard architectural constraint, as definitive as memory limits in an embedded system or bandwidth caps in a mobile app. Throughout my 8+ years in professional software engineering, I have shipped 18+ production applications across iOS, Android, web, and desktop. In my role as a founding engineer, I owned the architecture for a HealthTech AI platform from 0 to 1, building out the React Native, Next.js, and NestJS stacks alongside HIPAA-aligned RAG and LLM pipelines. When you are serving clinical AI, "out of memory" or "context window exceeded" errors aren't just bugs; they are failures in care delivery. To solve this, I moved away from reactive error handling and toward a proactive "Estimate, Reserve, Settle" framework. This approach treats the LLM context window as a finite buffer that must be managed with the same discipline as a database transaction. The Problem: The Elasticity Illusion The primary challenge with LLM-integrated systems is that the input size is often non-deterministic. In the RAG pipelines I managed for clinical data, a single query could pull in three paragraphs of physician notes or thirty pages of FHIR-formatted lab results. If you treat the context window as elastic, you encounter three failure modes: Truncation Loss: The model ignores the most recent (and often most relevant) data because the prompt was too long. Latency Spikes: Time-to-first-token (TTFT) scales with input length. In a clinical setting, a five-second delay can break a provider's workflow. Cost Cascades: Without a budget, a recursive loop or a large document retrieval can spike costs by 10x in a single afternoon. When I scaled our engineering team from 0 to 21 engineers in 13 months, one of the first architectural standards I implemented was the enforcement of a token budget. We had to stop thinking about "sending a prompt" and start thinking about "allocating context." Architecture: Estimate, Reserve, Settle To maintain 99.9% uptime for our clinical AI, I built a middleware layer that treats every LLM call as a three-stage lifecycle. 1. Estimate Before a single byte is sent to the provider (OpenAI, Anthropic, or a self-hosted Llama instance), the system must calculate the weight of the request. This involves more than just a string.length check. We use tiktoken or similar libraries to get an exact count. We categorize tokens into three buckets: System Instructions (Static): The base prompt that defines the model's persona and constraints. Contextual Data (Variable): The RAG fragments, FHIR resources, or user history. Response Buffer (Reserved): The minimum number of tokens we need the model to output to be useful. 2. Reserve Once we have an estimate, the system "reserves" space. If the total of System + Context + Buffer exceeds the model’s hard limit (e.g., 128k tokens), the request is rejected at the edge—not by the LLM provider. This prevents unnecessary latency and billing for a request that is destined to fail or be truncated. In our NestJS architecture, this was implemented as a guard. If the RAG retrieval returned 150k tokens of data, the "Reserve" phase triggered a summarization strategy or a "Rank and Prune" logic to bring the payload back within the architectural budget. 3. Settle After the LLM returns a response, the "Settle" phase calculates the actual usage. This data is fed back into our telemetry to adjust future estimates. If we consistently reserve 4,000 tokens for a clinical summary but the model only uses 500, we are over-provisioning and potentially starving other parts of the prompt of necessary context. Implementing the Token Budget Guard In the RAG pipelines I managed, we integrated FHIR/HL7 data and wearables telemetry. This data is notoriously verbose. Below is a simplified representation of how we enforced a budget within our NestJS services to ensure we stayed within the constraints of our HIPAA-aligned pipeline. async function executeClinicalQuery(patientId: string, userQuery: string) { const BUDGET_LIMIT = 8192; // Hard ceiling for this specific model const RESPONSE_RESERVATION = 1000; // 1. Estimate Static & User Input const basePromptTokens = tokenizer.encode(SYSTEM_PROMPT).length; const queryTokens = tokenizer.encode(userQuery).length; // 2. Calculate remaining budget for RAG const availableForContext = BUDGET_LIMIT - basePromptTokens - queryTokens - RESPONSE_RESERVATION; // 3. Retrieve and Prune const rawContext = await clinicalDataService.getPatientHistory(patientId); const prunedContext = tokenBudgeter.fit(rawContext, availableForContext, { strategy: 'priority-rank' // Prioritize recent vitals over old notes }); // 4. Final Verification const finalPayload = assemble(SYSTEM_PROMPT, prunedContext, userQuery); if (tokenizer.encode(finalPayload).length > BUDGET_LIMIT - RESPONSE_RESERVATION) { throw new TokenBudgetExceededException(); } return llmProvider.generate(finalPayload); } This logic ensures that the model always has at least 1,000 tokens available to formulate its answer. Without this reservation, the model might start a high-quality clinical analysis only to be cut off mid-sentence because the input data consumed 99% of the context window. Trade-offs and Failure Modes Enforcing a hard token budget requires making difficult choices about data priority. In my experience building for Synapsis Medical Technologies, we faced a recurring trade-off: Recency vs. Breadth. If a patient has ten years of medical history, a token budget forces you to choose between a shallow overview of those ten years or a deep, high-fidelity look at the last six months. We handled this by implementing a multi-stage retrieval process. We would use a smaller, cheaper model to summarize older records into a "condensed history" (consuming fewer tokens) while keeping recent lab results in their raw, high-precision format. Another failure mode is "Token Drift." Different tokenizers (GPT-4 vs. Claude vs. Llama 3) count tokens differently. When I overhauled our CI/CD cycles—cutting release times from 2 days to 4 hours—we included automated "Token Sensitivity" tests. These tests ensured that changes to our system prompts didn't unexpectedly balloon our token usage and break the budget for our mobile users on React Native. What it Cost to Learn Maintaining 99.9% uptime for clinical AI taught me that reliability is a byproduct of constraint. Early on, we attempted to be "clever" by dynamically switching to larger context models (like moving from an 8k to a 32k model) on the fly when the budget was exceeded. This was a mistake. It introduced non-deterministic latency and made cost forecasting impossible. I learned that it is better to have a system that predictably prunes data than one that unpredictably increases its operational footprint. By treating the token budget as a fixed architectural constraint, we were able to stabilize our NestJS and Next.js services, ensuring that the mobile applications I shipped remained responsive even when the underlying clinical data was massive. Practical Recommendations For architects building similar LLM-integrated systems, I recommend the following: Define a "Minimum Viable Response" (MVR): Determine the smallest number of tokens your model needs to provide a useful answer. Subtract this from your model's maximum context window. That is your actual operating budget. Implement Token-Aware RAG: Do not just retrieve top-k documents. Retrieve top-k, then use a "fit" function to fill your token budget starting from the highest relevance score downward. Middleware Validation: Don't wait for the LLM API to return a 400 error. Validate your token counts in your backend (NestJS, Go, etc.) before the request is dispatched. Monitor "Budget Headroom": Track how close your requests are coming to the limit. If 90% of your requests are using 99% of the budget, you have no room for prompt iterations or model updates. Conclusion Tokens are the new RAM. In the same way that we wouldn't ship a React Native app that consumes 4GB of memory on a low-end device, we shouldn't ship AI features that treat context windows as infinite. By implementing an Estimate, Reserve, and Settle framework, you move from a reactive posture to a proactive one. This discipline is what allowed us to scale our engineering operations and maintain clinical-grade reliability. A hard ceiling on tokens isn't a limitation; it is the foundation of a stable, production-ready AI architecture. Amit Chakraborty is a founding engineer and senior architect — React Native, AI/RAG systems and production architecture. Portfolio: www.amitchakraborty.dev · LinkedIn · GitHub. Open to senior and founding engineering roles, remote worldwide.