Zero markup does not determine LLM gateway TCO

Zero token markup answers only one question: how much an intermediary added to the model's public rate. It says nothing about a discount under a direct enterprise agreement, the price of reserved capacity, foreign exchange spreads, support, or engineering hours. A zero-markup gateway can therefore cost more than a direct contract, while a discounted contract can lose to a gateway after the first quarter of operation.

You need to compare the full cost of two working architectures at the same volume, quality, and service level. I call an option cheaper only when the difference holds across several load scenarios and after accounting for the team's work. Without that calculation, a price list gives an exact number for a much too narrow part of the expense.

Zero markup does not mean the lowest cost

A zero-markup gateway rate normally matches the provider's base price for input and output tokens. If a direct agreement gives the company a discount d, the gateway is already more expensive by the value of that discount at the consumption level: with a monthly provider bill of KZT 10 million and a 12% discount, the difference is KZT 1.2 million before any other cost items. This is not a hidden gateway fee. It is a direct-contract term that the company gives up.

The discount does not operate in a vacuum. A provider may tie it to annual volume, prepayment, a selected region, a specific model family, or total cloud spend. The finance team should calculate the effective rate on the actual invoice, not the promised percentage: divide every payment to the provider during the period by the input, cached input, and output tokens actually processed, accounting for their different rates.

One distinction often gets erased in a single Excel column. A consumption discount lowers the price of a completed request. A capacity reservation buys available throughput even when the application does not use it. In the first case, traffic growth creates the risk. In the second, the company pays for a forecasting error. Combining both mechanisms under the word discount is dangerous because an underused reservation can easily make an attractive contract rate the most expensive line in the budget.

Check four numbers in the commercial proposal: the discount on variable consumption, the minimum payment, the commitment term, and the overage billing rules. If the seller shows only a percentage, you cannot compare the proposal with a gateway yet.

Compare two architectures that deliver the same result

A fair comparison starts with the same business outcome, not the same model name. Both architectures must handle the required request peak, deliver acceptable latency, store data in an approved jurisdiction, retain the necessary audit trail, and provide a working escalation path. A cheaper design that fails one mandatory condition is not an alternative.

Define the baseline monthly load profile. You need input tokens, cached input tokens as a separate value, output tokens, request count, peak tokens per minute, batch-processing share, and distribution across models. Average volume without the peak is insufficient because the peak determines reserved capacity, quotas, and failover headroom.

Then describe both setups. For the direct option, list each provider, separate agreement, region, billing arrangement, SDK or adapter, backup route, and integration owner. For the gateway option, record its model rates, local controls, invoice currency, per-key limits, and behavior during a provider failure. Do not transfer a capability from one setup to the other by default. For example, a second model in a catalog does not prove automatic failover, and an enterprise agreement does not guarantee help with application code.

Choose the comparison unit in advance. For a stable service, the cost per million successful, useful answers is more informative than the cost per million sent requests. Retries after 429 responses, timeouts, empty answers, and outputs rejected by validation consume tokens and labor but produce no useful answer. If model quality differs, add the cost of reaching the defined threshold on your own evaluation set. A cheaper model with more repeat calls can raise the final cost even though its token price is lower.

The calculation horizon must match the longest commitment. A monthly comparison misleads when the contract runs for a year because a seasonal dip, a delayed launch, and a model replacement appear only in the annual table. I normally calculate month by month for 12 months and show expected, low, and high traffic separately.

An enterprise discount has entry and exit costs

A direct agreement wins when the company actually consumes the promised volume and can apply the discount to the required models. It loses when the minimum commitment turns an unused balance into an expense. The formula is simple: the effective discount equals one minus all mandatory payments divided by the public-rate cost of the same actual consumption. If mandatory payments exceed the public cost of actual traffic, the discount becomes negative.

Provider documentation illustrates the mechanics well, although specific rates change. OpenAI Scale Tier sells predetermined units of input and output throughput for a specific model snapshot with a minimum purchase term. Amazon Bedrock Provisioned Throughput charges hourly for Model Units and lowers the rate for a longer commitment; billing continues until the resource is deleted. Microsoft Foundry charges for deployed PTUs regardless of the number of processed tokens and separately warns customers to create the deployment before buying the reservation. Google Cloud describes Provisioned Throughput as a commitment that cannot be canceled in the middle of its term. The shared conclusion is dull but useful: evaluate a reservation by utilization, not discount size.

A direct contract also has an exit cost. The application may replace the model before the commitment ends, the product may miss its traffic plan, or security may prohibit the selected region. Put the expected cost of an unused commitment into TCO instead of treating it as zero until a problem occurs. For scenario planning, multiply the remaining mandatory payments by the probability of early exit agreed by the team for budgeting.

Do not add the token discount and the reservation discount as two independent percentages. Confirm which requests the reservation covers, how overage is billed, whether general cloud discounts apply, and whether the commitment moves between models. If you cannot express the terms as a formula from the contract, ask the provider for sample invoices at 60%, 90%, and 120% utilization.

Currency and settlement change the effective rate

Foreign exchange cost includes more than the rate on the payment date. The company pays the bank's spread, international transfer fees, possible card or acquiring fees, conversion cost, and labor to reconcile closing documents. On a large bill, even a modest spread matters more than several engineering optimizations to token use.

Calculate in the currency used for management budgets, usually tenge for a company in Kazakhstan. For every month, use the actual debit rate with fees rather than an official reference rate. If the direct provider invoices in dollars while the gateway issues a B2B invoice in tenge, evaluate the difference using bank statements and accounting time. A predictable tenge invoice does not guarantee a lower total, but it removes part of the variance and administrative work.

A yearly model benefits from three currency scenarios. The base case uses the budget rate, the stress case reflects a weaker tenge, and the reverse case shows a stronger tenge. You do not need to pretend you can forecast the exact rate. The purpose is to see the rate at which the winner changes. Calculate the currency threshold as the alternative's full cost in tenge divided by the direct setup's dollar payments after adding fixed local expenses.

Settle taxes and their accounting treatment with a finance specialist instead of copying another company's template. VAT, withholding, input-tax credits, and the recognition of imported services depend on the agreement and the company's status. TCO needs the amount that remains an expense for this specific company after available credits. Otherwise, two teams can honestly reach different answers because their tax positions differ.

Finally, estimate the cost of payment timing. An invoice with deferred payment preserves working capital, prepayment ties it up, and a mandatory deposit creates a permanent balance. This item is smaller than tokens and labor for most teams, but it should not hide in a footnote when the annual commitment is large.

Separate integrations quietly consume the discount

Team labor often decides the argument at moderate API volume. A direct contract with one provider may require little work. A portfolio of three providers adds different authentication schemes, error formats, rate limits, streaming behavior, tool-calling details, model versions, usage metrics, and access procedures. An OpenAI-compatible interface reduces the number of code changes, but endpoint compatibility does not make quotas, error semantics, and operations identical.

Do not estimate integration as one task called connect the API. Separate initial development from monthly operations. The first category includes contract and security review, network access, secret management, an adapter, tests, observability, load testing, documentation, and launch. The second includes SDK updates, model migrations, invoice investigation, quota requests, incident response, key reviews, and backup-route maintenance.

The hourly cost should be the fully loaded internal rate, not salary divided by working hours. Finance usually adds employer taxes, benefits, equipment, management, and overhead. If that rate does not exist, use an agreed team-hour cost and show it as a separate assumption. A disagreement about the rate will then not destroy the whole model.

The most common failure looks like this. An architect budgets 80 hours for direct integrations, the team spends about that much on launch, and everyone declares the estimate accurate. Three months later, the provider releases a new model version, structured output behaves differently, the quota cannot handle the morning peak, and finance cannot reconcile project tags with the invoice. Those hours enter the platform team's general queues and disappear from TCO. On paper, the direct contract kept its discount. In the company budget, operations consumed it.

Count only the difference in labor between the options. Both setups still need evaluation, prompt controls, and application observability. You cannot credit the gateway with saving the whole LLM team, and you cannot give the direct option free functions that the gateway already includes in the agreed setup.

Support costs as much as downtime costs

Compare support packages by what happens during a failure, not by plan name. Confirm the contact channel, operating hours, first-response target, responsibility boundaries, escalation procedure, and who can see the problem between your application and the model. An account manager without technical escalation rarely shortens an incident.

First calculate the cost of one hour of degradation. For an external product, that includes lost margin, customer compensation, and support labor. For an internal assistant, use the number of affected employees, lost time, and the price of the manual process. Then estimate how many hours of downtime each option can prevent or remove. This is a range rather than a promised point, so the model needs at least a best and worst case.

A gateway creates economic value if a single observation point and access to multiple models actually shorten recovery. It adds risk if it becomes another component without transparent diagnostics or contractual accountability. A direct contract helps when the provider owns most of the stack and the team has an agreed urgent escalation path. It helps less when a failure requires a manual move to another API that nobody has tested recently.

You also cannot monetize an SLA at the full value of possible damage. Service credits are usually limited by the agreement and do not restore lost revenue. Include expected operational loss and the cost of risk controls in the main calculation. Show a credit separately only when the company can realistically claim it.

The test is practical: run a failover exercise, measure detection, decision, and recovery time, then multiply the difference between the setups by the cost per hour. Without that exercise, the support discussion remains an opinion held by people who have not yet had to restore the service at night.

A working TCO sheet reveals the break-even point

The model must be simple enough for a finance partner to check every row and complete enough that an engineer cannot hide work in comments. Create one row per month and one column per cost item. You can paste this header into a CSV file or spreadsheet:

month,direct_list_usage,direct_discount,direct_commitment_floor,direct_unused_commitment,direct_support,direct_fx_fees,direct_labor_hours,gateway_usage,gateway_fixed_fees,gateway_support,gateway_fx_fees,gateway_labor_hours,internal_hour_rate,direct_total,gateway_total

For the direct setup, use direct_usage_net = max(direct_list_usage * (1 - direct_discount), direct_commitment_floor). Do not add direct_unused_commitment a second time if the minimum is already handled through max; that column exists to expose the share of payment with no useful consumption. The direct total equals net consumption plus support, foreign exchange, differential labor hours, and expected downtime cost. Apply the same logic to the gateway without inventing zero labor or risk.

There is no single universal formula for the volume break-even point because commitments create steps. Calculate volumes from 50% to 150% of forecast in 10% increments and find the first interval where the sign of gateway_total - direct_total changes. Then refine it with a smaller increment. Display reservation utilization, the number of active providers, and operating hours next to the result so a reader can see why the lines crossed.

Add validation equations. Monthly totals must sum to the annual total; token volumes on invoices must reconcile with telemetry within an accepted tolerance; a discount cannot apply to expenses excluded by the contract; hours cannot be negative. Mark inputs and actual data separately. A sheet that makes a commercial proposal, a forecast, and an actual result look identical will inevitably trigger an argument about where the numbers came from.

Take the decision from a range. If a direct contract costs less only when the optimistic plan lands exactly, there is no winner yet. If it wins under low, base, and high traffic even after differential hours double, the economic signal is strong.

Normalize caching as well. One setup may charge a separate rate for cached input tokens, while another may fail to pass the required cache identifier or may limit its lifetime. Compare the actual hit rate with identical prompts. You cannot take the best theoretical cache price from a price list and apply it to all input. If the application regularly changes the system prompt, tools, or message order, a large share of requests will miss the cache.

Keep unsuccessful traffic on its own line. Providers may account differently for tokens when a stream breaks, a request is canceled, or a call is retried after an error, while client libraries may retry automatically. Use telemetry to find the number of attempts per answer accepted by the application and apply it to each route. Even a difference of a few percent matters at scale, but you cannot invent it in advance. Until a load test exists, use the same assumption and mark it as unknown.

Record the model mix as separate rows instead of using an average token price. Input and output have different prices, the output share varies by task, and a discount may not cover the full catalog. Classification, document generation, and a voice use case need different profiles. When the team replaces a model, move only the traffic that passed the set quality threshold. Otherwise the savings come from a worse product and the comparison loses its meaning.

Finally, assign owners to source data. Finance confirms exchange rates, fees, and commitments; the platform team confirms hours and incidents; product owns the forecast and degradation cost; the ML team confirms model mix and evaluation results. A shared sheet works because every assumption receives an owner and a review date.

The example shows where the savings disappear

Consider a fictional team to test the mechanics without relying on current rates. The public cost of its monthly consumption is KZT 40 million. A direct provider offers a 15% discount with a KZT 34 million monthly minimum and a one-year commitment. The gateway charges the same public rates without markup. On the first comparison line, the direct option costs KZT 34 million and the gateway KZT 40 million, so the contract saves KZT 6 million.

Now add differential costs. The direct setup needs 140 launch hours and the gateway needs 40. At an internal rate of KZT 25,000, the one-time difference is KZT 2.5 million. In operation, the direct option needs 32 more hours per month for separate quotas, invoices, and adapters, or KZT 800,000. Assume the direct provider's support costs another KZT 700,000 per month, while exchange fees and spread add KZT 450,000. In the first month, the direct agreement's advantage falls to KZT 1.55 million: 6 - 2.5 - 0.8 - 0.7 - 0.45.

The one-time development cost disappears from the second month, and the direct contract keeps KZT 4.05 million at the same traffic. It clearly wins at a stable load. But if the public-rate cost of consumption falls to KZT 26 million, the minimum payment keeps the direct token cost at KZT 34 million while the gateway drops to KZT 26 million. The direct option loses KZT 8 million for that month even before labor and support.

There is a reverse stress case too. If usage rises to KZT 60 million at public rates, the direct payment after discount is KZT 51 million, provided the contract applies the discount to all volume and does not set a separate overage rate. Foreign exchange and extra hours will rarely consume the KZT 9 million difference. But the conclusion changes if part of the traffic moves to a model outside the discount or if reserved capacity limits throughput and overage is billed differently.

The example teaches two lessons. The minimum commitment determines the loss when demand falls, while the discount percentage determines the gain above the minimum. Procurement should therefore ask for a monthly distribution and sensitivity to model changes, not one forecast point.

Direct contracts win with concentration and predictability

A direct agreement is usually cheaper when a large share of traffic remains consistently with one provider, the company consumes the minimum commitment, and the enterprise discount exceeds the additional operating cost. The higher the volume and the fewer the integrations, the easier it is to recover the contract and engineering work. A strong existing cloud team also shifts the calculation toward the direct path because identity, networking, billing, and support already fit its processes.

Some nonfinancial reasons must become monetary values or constraints. Direct procurement may provide the required capacity reservation, a specific region, agreed data-processing terms, or access to the provider's technical team. If the product cannot meet its requirements without one of these, the gateway can remain available for other models but cannot replace the main contract.

I would not sign a large commitment for a discount alone under three conditions: the product is still searching for a repeatable use case, the team often changes the leading model after evaluation, and the forecast uses average traffic without seasonality. The popular advice to lock in the price sounds sensible for a mature load. For a young LLM product, it often locks in a mistaken assumption about model and volume instead.

Before signing, check whether the discount moves between models, the overage price, what happens to the commitment when a model is retired, the right to change region, and termination terms. Then run TCO with a one-quarter launch delay. If that single delay breaks the economics, the deal has too little margin for error.

Gateways win on flexibility, and a hybrid often beats both extremes

A gateway wins more often when traffic is fragmented, models change quickly, each provider receives modest volume, and platform-team time is expensive. Zero markup helps, but the main savings come from consolidated integrations, settlement, and controls. Confirm those savings with hours and invoices instead of accepting them from a product description.

For teams in Kazakhstan, AI Router brings routes to 500+ models behind one OpenAI-compatible endpoint, issues monthly B2B invoices in tenge, and charges provider API rates without markup. In TCO, those properties belong under integration labor, foreign-exchange and accounting cost, and consumption cost respectively. Local storage, PII masking, audit logs, and per-key limits should receive value only when they replace separate work in your setup.

The choices do not need to be absolute. A company can sign a direct agreement for one primary model with a predictable baseline load and use a gateway for occasional models, experiments, locally hosted open-weight models, and backup. This hybrid keeps the discount on concentrated volume without multiplying direct integrations for a small tail.

Calculate the hybrid honestly too. It adds a second agreement, route validation, data-classification rules, and recurring failover tests. It saves money only when the team explicitly defines which traffic consumes the commitment, which remains variable, and who monitors underused capacity.

Put a review threshold into the financial model itself. For example, recalculate when actual volume leaves the agreed range for two months, the primary model's share changes materially, or operating work exceeds its budget. Direct contracts and gateways then stop being an ideological choice. They are two procurement forms for different parts of the load, and each must prove its price again with actual data.