BV-SALA: inside the LLM cost layer

The bill of an LLM-powered system doesn't grow with the value it produces: it grows with the tokens it moves. And most of those tokens never needed to move. Context repeated on every turn, declared tools that are never called, questions whose answer was already computed a minute ago. BV-SALA is the layer that sits in front of your provider and, on every request, answers a single question: what is the cheapest execution that still meets your quality contract?

This article explains how we think about that question: the formalization, the three ledgers every saving falls into, and the measurement methodology — which is, deliberately, the strictest part of the system.

The decision, formalized

For a request $x$ (prompt, context, tools), the provider charges a function that is essentially linear in tokens:

$$C(x) \;=\; c_{\text{in}} \cdot T_{\text{in}}(x) \;+\; c_{\text{out}} \cdot T_{\text{out}}(x)$$

where $T$ counts tokens with the provider's own tokenizer — not with an approximation of ours — and $c_{\text{in}}, c_{\text{out}}$ are their current prices. BV-SALA has a catalog $\Pi$ of request transformations: removing redundancy, pruning tools, compressing the encoding, resolving without calling. Choosing well is a constrained optimization problem:

$$\pi^{*} \;=\; \operatorname*{arg\,min}_{\pi \,\in\, \Pi} \; \mathbb{E}\big[\, C(\pi(x)) \,\big] \qquad \text{s.t.} \qquad Q(\pi(x)) \,\ge\, \tau$$

The constraint $Q \ge \tau$ is the quality contract: the cheapest transformation is worthless if it degrades the answer. And here lies the intrinsic difficulty of the problem: $Q$ is not observable before executing. That is why every transformation in the catalog carries its own guard — it is applied only when it pays for itself, with the cost of deciding accounted inside the saving, not hidden outside it.

The three ledgers

Every optimized request is attributed to exactly one ledger, by dominant mechanism. This partition is not cosmetic: it is what makes the numbers mean something.

01 — Avoid the call entirely. Deterministic resolvers for what doesn't need a model, an exact cache for what was already answered, and singleflight to collapse identical concurrent requests into a single execution. Measured ledger saving: 17.9%.

02 — Remove what didn't need to be sent. Tools pruned from the declared catalog, selective retrieval instead of full context, and elimination of the redundant context that travels turn after turn. Measured saving: 38.6%.

03 — Compress what remains. Lighter encoding of what does have to travel — and only when the compression itself pays off. Measured saving: 45.1%.

The saving of a ledger $\ell$ over the set $R_\ell$ of requests attributed to it is defined against the naive arm:

$$S_\ell \;=\; 1 \;-\; \frac{\sum_{x \in R_\ell} C\big(\pi^{*}(x)\big)}{\sum_{x \in R_\ell} C(x)}$$

Two direct consequences of the partition. First: these figures do not add up — each speaks over its own denominator, and presenting them aggregated would count the same euro twice. Second: a step whose ledger doesn't clear zero is not published. If a transformation doesn't demonstrably save, its figure doesn't exist.

How we measure: the floor of the interval, not the flattering estimate

The methodology is the part of the system we trust most, because it is the one that leaves the least room for interpretation:

1. Same model, two arms. Every request is evaluated as your application would send it (naive arm) and as BV-SALA sends it. We never compare different models or different time periods.
2. Cost quoted with the provider's own meter. No modeled baseline: tokens are counted by the official tokenizer and prices are the provider's. The number we publish is the one you would see on your bill.
3. We publish the floor of the 95% confidence interval. With $n$ requests and sample saving $\hat S$, we publish the one-sided lower bound

$$S_{\text{pub}} \;=\; \hat S \;-\; z_{0.95}\,\widehat{\operatorname{se}}\big(\hat S\big)$$

that is, the value the true saving exceeds with 95% confidence — never the point estimate, which is always more photogenic.

Held to that yardstick, the results published today are:

The $\ge$ sign on the first is not rhetoric: it is the lower bound of the interval. And the second figure shows ledger 01 on its best terrain — when the context prefix repeats, the provider's cache turns expensive tokens into cheap ones, and the provider's own meter certifies it.

Why the problem is harder than it looks

Three sources of intrinsic complexity that shape the whole design:

The constraint is stochastic. $Q(\pi(x)) \ge \tau$ is decided before seeing the response. Aggressive transformations (rewriting, compression) only activate where accumulated evidence indicates they preserve intent; in doubt, the request travels untouched. We would rather save less than degrade silently.

The cost of deciding counts. Analyzing every request also consumes. BV-SALA's entire accounting is net: the published saving discounts the cost of the layer itself. An optimizer that pays itself in the fine print is not an optimizer.

Cache correctness is semantic, not syntactic. An exact cache and a singleflight are only a saving if the reused answer is still valid for the new request. Keys incorporate everything that conditions the response — model, version, visible tools, effective context — and on any difference, we call.

Where this is today

BV-SALA is productized as an independent product at savings.bivelio.com: a licensed SDK on npm, a five-line install and a measure-only mode to start — first we tell you what you would save on your real traffic, without changing any of your code. Inside BiVelio it is the Optimize layer: the same engine deciding, request by request, how much of your AI bill never needed to be paid.