The problem: paying a frontier model to do nothing
When you wire an AI assistant into your daily workflow, the bill creeps up in a way
that's easy to miss. The surprise isn't the hard reasoning; it's how much of the spend
goes to trivial work: routing a message, formatting a summary, deciding whether anything
even needs a model at all. You end up paying frontier prices for tasks a plain rule could
have answered for free.
The fix isn't a cheaper model. It's deciding, per task, **whether a model is needed at
all, and if so, how capable it has to be**.
Deterministic before AI
The first rule of the system is that a model is the last resort, not the first. If a
deterministic rule can solve the case (a lookup, a template, a conditional), no model
runs. Reasoning models are reserved for what genuinely requires reasoning.
This single principle removes a large share of calls before any routing happens. It also
makes the system faster and more predictable, because the cheap path is the common path.
Tiers and a daily budget guard
What survives the deterministic filter goes through a small router that classifies the
work into tiers:
- fast: short, low-stakes tasks (classification, light summaries).
- balanced: everyday reasoning that needs decent quality but not the top model.
- quality: the hard, high-stakes work that justifies a frontier model.
- local: anything a self-hosted model can handle for free.
Each tier has a daily budget. When a tier approaches its limit, the system warns and
falls back to a cheaper tier instead of silently overspending. The budget isn't a
hard wall that breaks the assistant; it's a guardrail that keeps cost bounded while the
work still gets done.
A self-hosted cheap tier, with a fallback chain
The biggest lever is moving background work (research drafts, long syntheses, routine
organization) onto self-hosted models. They aren't as sharp as a frontier model,
but for bulk work that runs unattended, "good enough and free" beats "excellent and
metered."
The catch with self-hosted infrastructure is that it isn't always up. So the router runs
a fallback chain: if the self-hosted tier is unavailable, the request degrades to the
next option that *is* available rather than failing. The assistant stays responsive; the
only thing that changes is which engine answered.
The result is a simple cost hierarchy: rules first (free), self-hosted next (free),
cheaper hosted tiers for everyday work, and the expensive model only for the small slice
that truly earns it.
What changed
Two things, both qualitative:
1. Background work stopped touching the expensive tier. Long-running, unattended
tasks now route to self-hosted or fast tiers by default: exactly the work that used
to quietly dominate the bill.
2. Spend became predictable. With per-tier budgets and a fallback chain, a busy day
no longer means a runaway invoice. The system trades a little quality on low-stakes
tasks for a cost ceiling you can reason about.
Takeaway
You don't cut LLM costs by hunting for a cheaper model. You cut them by **not calling a
model when you don't need one**, routing what's left to the *least* capable engine that
can still do the job, and keeping a frontier model in reserve for the work that's
actually worth it. The architecture (deterministic-first, tiered routing, a budget
guard, a self-hosted fallback) is what turns "AI is expensive" into a line item you
control.
This routing layer is part of a larger
self-governing personal assistant, the
same system that powers this site's PublicBrain pipeline.