BV-SALA Savings Layer: Verifiable Measurement of Cost Savings in Large Language Model Applications

Abstract

Cost claims for large language model (LLM) optimizations require explicit workloads, uncertainty, and a documented cost basis. This paper presents BV-SALA Savings Layer, an in-process SDK and local gateway that combine provider-reported usage with versioned list prices and separate four savings ledgers. Live SDK double-runs used a percentile cluster bootstrap with 10,000 resamples. On GSM8K-derived traffic, gpt-4o-mini showed a 31.5% cost reduction (95% CI: 25.7–36.7; 100 families, 200 requests per arm) and claude-sonnet-5 showed 75.1% (95% CI: 69.3–79.6; 31 families, 62 requests per arm). The Anthropic arm included 41 percentage points from a prefix-cache effect enabled by cache_control in the instrumented condition, not a pre-existing provider discount. A separate counterfactual replay of 19,779 gateway requests estimated 2.49% (95% CI: 1.50–3.57); it was not a live double-run or invoice reconciliation. We distinguish measured, reference-valued, and estimated amounts, state quality and latency observations without claiming non-inferiority, and present only provisional economic extrapolations.

Keywords: LLM; inference cost; provider-reported usage; prefix caching; cluster bootstrap; prompt optimization.

Results

Table 2 and Figure 2 separate two live SDK double-runs from a gateway counterfactual replay. Amounts use provider-reported usage and versioned list prices; none is presented as an invoice-reconciled result.

The gateway’s 2.49% does not come from an invoiced A/B or a live double-run: it is a counterfactual estimate with a partially reconstructed baseline and session-level resampling. In the Sonnet flow, observed accuracy was 56/56 in each arm and median provider-call latency was 1,969 ms in A and 2,225 ms in B; these observations establish neither equivalence nor non-inferiority. Gemini contributes operational non-breakage checks only, with no publishable savings interval.

Method and cost basis

In the live SDK experiments, base condition A without the layer and instrumented condition B use the same model and constructed content. Cost is calculated from provider-reported usage and the versioned list prices in force for the analysis. This provides documentary traceability to usage and price records, not strict metrological traceability or proof of an invoice amount. Negotiated prices, credits, taxes, rounding and later provider adjustments remain outside the evidence.

Live intervals use a non-parametric percentile cluster bootstrap with 10,000 resamples. The family, not the individual request, is the resampling unit: requests within a family stay together, and the 95% interval uses the 2.5th and 97.5th empirical percentiles. The gateway instead resamples whole sessions. These intervals summarize variability under the observed design, not all temporal drift, model or price changes, or counterfactual reconstruction error.

Four ledgers and three evidence bases

The per-request savings report (SavingsReport) keeps four ledgers separate to reduce double counting and make each mechanism’s dependence on traffic and provider behavior explicit. Amounts should not be summed without a defined estimand.

An amount is measured when directly supported by provider-reported usage for that execution and a documented price version; reference-valued when a counterfactual amount uses an earlier equivalent or identical request under explicit matching conditions; and estimated when reconstructed from logs, list-price calculations or other assumptions without such a direct reference. A stream without final usage is unmeasured. These labels describe evidentiary basis; they do not turn reference values or estimates into invoice-verified savings.

Limitations and future work

• The samples are small and use constructed traffic: 100 OpenAI families and 31 Anthropic families. The gateway covers 51 sessions, 13 days and one organization; temporal stability and generalization to other models, providers or production workloads are not established.
• Anthropic attribution requires a factorial deduplication-by-cache_control design. Further work includes a repeat-rate sweep, multi-day hierarchical experiments and adequately powered preregistered non-inferiority tests for answer quality, freshness and latency.
• Exact caching can reduce freshness and distributional diversity. Results cannot be generalized to creative, time-sensitive, multi-tenant or semantically similar traffic without new experiments. The published package includes no general prompt compressor, and TOON is not evaluated as a shipped optimization.
• The reference semantic store is in-memory and linear per lookup; transactional multi-replica coordination, consistency and production-scale latency need further work. Current routing lacks evidence of semantic correctness.
• List-price calculations are not invoice reconciliation. Private gateway logs, no independent replication and the author’s developer role limit verification. External replication, invoice reconciliation including negotiated prices and credits, and immutable resampling artifacts are needed.

In BiVelio Factory

This paper is product 01 on the output dock of BiVelio Factory, where the applications, papers and companies that leave the factory appear. The algorithm whose savings it measures is explained in BV-SALA · BiVelio Savings Layer.

Download and cite

Download the paper as a PDF (9 pages, 444 KB). The public bench with the trials is at savings.bivelio.com.

How to cite it:

Vidal, Adrián (2026). BV-SALA Savings Layer: Verifiable Measurement of Cost Savings in Large Language Model Applications. BiVelio Inc - research.bivelio.com, August 2026. https://bivelio.com/en/research/papers/bv-sala-savings-layer

For more context on the algorithm: BiVelio Savings Layer and the article BV-SALA: inside the LLM cost layer.