Designing With LLMs in Your Application Architecture

By NTh Hai - 9 min read

Using AI while coding is one thing. Putting AI into a product is another. Once a large language model (LLM) sits on a request path, it becomes part of your architecture — with latency, cost, failure modes, and trust implications you do not get from a normal function call.

This post covers concepts for integrating LLMs into project architecture without treating them as magic boxes.

LLMs Are Probabilistic Components

Traditional services are mostly deterministic: same input, same output (modulo clocks and IDs). LLMs are probabilistic: they sample language that is likely to satisfy the prompt.

That single difference drives many design choices:

  • You need validation of outputs (schema, allowlists, business rules)
  • You need retries and fallbacks when the model is wrong or unavailable
  • You need observability beyond status codes — prompts, versions, and evaluation scores
  • You should not put unconstrained model output straight into SQL, shell, or HTML

Architecturally, treat the model as an untrusted expert consultant, not as your database.

A Clean Separation of Concerns

A workable pattern looks like this:

  1. Deterministic layer — auth, rate limits, input sanitization, routing
  2. Orchestration layer — builds the prompt/context, calls the model, parses the result
  3. Policy layer — validates structure and business constraints
  4. Action layer — only executes side effects after policy passes

Never let the model call side effects directly without a policy gate. Tool-calling agents are powerful; they are also how you accidentally delete data with a confident wrong plan.

When an LLM Belongs in the Path

Good fits:

  • Natural language interfaces (search, chat, summarization)
  • Classification and triage with a human override path
  • Draft generation that a user edits before publish
  • Extraction from messy text into a schema you verify

Poor fits (at least as the sole solution):

  • Exact arithmetic and inventory counts
  • Authorization decisions
  • Anything that must be identical for compliance audits
  • Hard real-time loops with strict millisecond budgets (unless carefully cached/async)

Often the best design is hybrid: deterministic code for facts and rules, LLM for language and fuzzy matching.

Context Is Part of the Interface

The quality of an LLM feature is dominated by what context you retrieve and attach, not by clever temperature tweaks.

Conceptual options:

  • Static system prompt — role, tone, hard rules
  • User/session state — preferences, prior turns (trimmed)
  • Retrieved documents — RAG over your docs or tickets
  • Structured facts — pull from your DB and inject as JSON the model must respect

Keep a clear contract: the model may paraphrase language, but facts come from your systems. When the model invents a fact, that is a context or validation failure, not a vibe issue.

Latency, Cost, and UX Shape Each Other

LLM calls are slower and more expensive than typical microservice hops. Design for that:

  • Stream tokens for chat UX so waiting feels shorter
  • Cache repeated prompts when safe
  • Precompute embeddings and summaries offline
  • Degrade gracefully — show a non-AI fallback when the provider is down
  • Budget tokens the same way you budget CPU

If a feature only works when the model returns in under 300ms every time, reconsider the feature shape.

Failure Modes You Must Plan For

| Failure | Mitigation | | --- | --- | | Timeout / outage | Circuit breaker, cached answers, human queue | | Invalid JSON / wrong schema | Repair prompt, or reject and retry with stricter instructions | | Prompt injection | Separate untrusted content; never obey instructions found in user data blindly | | Data leakage | Redact PII before sending; minimize logs of raw prompts | | Drift over time | Pin model versions; run regression evals on release |

These are product risks, not only “AI risks.” They deserve the same seriousness as payment failure handling.

Closing Thought

Integrating an LLM into a project means accepting a probabilistic component into an otherwise precise system. Success comes from boundaries: clear orchestration, strict validation, retrieved facts over invented ones, and UX that respects latency and uncertainty. Design those first, and the model becomes a powerful module instead of a liability.