GPUs for model fine-tuning or a managed endpoint

The choice between dedicated GPUs and a managed endpoint cannot be reduced to accelerator hourly rates. A team is buying more than compute. It is choosing who owns the queue, training environment, checkpoints, encryption keys, logs, and the procedure for moving weights out.

For infrequent, predictable runs, a managed service usually costs less once the total effort is counted. For a continuous series of experiments, long-term on-site state retention, and strict isolation, a dedicated pool often gives more control. Between these poles is a broad area where five quantities decide the answer: run frequency, the share of occupied GPU time, artifact lifetime, acceptable queue delay, and the actual trust boundary.

I have watched teams compare two attractive GPU-hour prices, then spend engineer time on manual environment recovery and wait for a slot in a shared queue a month later. I have seen the opposite too: a rack of accelerators sat idle while one specialist maintained drivers for four training runs per quarter. The right choice starts with the workload profile, not a GPU catalog.

Run frequency determines what you own

Frequency does not matter on its own. What matters is how consistently the team can fill accelerators with useful work and how quickly the experiment configuration changes. One large run every month and twenty short trials every week create very different loads, even when their total GPU-hours match.

A managed endpoint fits when jobs arrive as separate batches, long pauses sit between them, and the team does not need to keep a warm environment. You submit data and configuration, the service provisions resources, performs the work, and releases them. The convenience premium is justified when an in-house cluster would spend most of the calendar waiting for the next job.

Dedicated GPUs start to win when experiments form a continuous pipeline. A researcher changes a dataset, runs a short check, fixes parameters, resumes from a checkpoint, and immediately queues another run. A permanent pool cuts the delay between an idea and its result. It also lets the team preinstall libraries, warm dataset caches, and keep intermediate state close to compute.

Do not count only successful full runs. In the log for the last eight weeks, mark preparatory runs, crash restarts, quality evaluations, adapter merges, quantization, and exports. These operations use a GPU or block the same work path. If a team counted three training runs but executed forty-seven related jobs, the frequency decision must use forty-seven.

There is a simple sanity threshold. If an engineer cannot predict what the pool will run tomorrow, buying permanent capacity is premature. If the experiment queue is already booked for weeks, renting every job as an exception creates needless coordination.

Count occupied time, not the run price

The comparison starts with each GPU's calendar. A dedicated pool bills continuously, while useful time includes only training and the operations needed around it. For a managed service, the bill is usually closer to job duration, but environment preparation, data loading, storage, and sometimes endpoint time before explicit deletion are added. The exact billing rules must come from the provider contract, not a marketing calculator.

Use one cost model for both options:

месячная_стоимость = вычисления + хранение + передача_данных + операции_платформы + труд_инженеров + стоимость_ожидания
полезная_загрузка = полезные_GPU_часы / доступные_GPU_часы
стоимость_полезного_GPU_часа = месячная_стоимость / полезные_GPU_часы

Engineering labor cannot be set to zero. In a dedicated environment, someone updates drivers, checks CUDA compatibility, recovers nodes, configures the scheduler, watches disk capacity, and investigates failures. That labor does not disappear with a managed option: the team prepares the image, access permissions, dataset, job parameters, and cleanup rules. Its volume and nature change.

Calculate the cost of waiting separately. If a model release slips by two days because of a queue, that may cost more than the difference in rates. Do not turn the delay into a made-up exact amount. Record a consequence you can verify: a missed release window, two days of team work without a result, or a production fix that remained unavailable.

Run the calculation in three modes. A normal month shows the average picture. A peak month tests whether a dedicated pool can handle parallel jobs and what a managed-service spike will cost. An empty month reveals the price of idling and storage. A decision that pays off only at 100 percent utilization rarely survives real research work, where failed hypotheses and pauses are unavoidable.

Depreciation of purchased hardware is not the same as the price of useful compute either. Add servers, networking, disks, power, redundancy, warranty replacements, and time until retirement. For a rented dedicated pool, add the minimum commitment and the ability to reduce capacity. A single line labeled GPU hides too many expenses to support a decision.

Weight retention determines the architecture

A checkpoint needed for three hours and a private model that must be reproducible three years later need different systems. Before buying compute, the team should define artifact classes and the lifetime of each: source weights, adapters, full checkpoints, optimizer state, tokenizer, training configuration, environment snapshot, metrics, and final inference weights.

The PyTorch guide to saving models distinguishes model weights from a general checkpoint used to resume training. Resumption needs at least the model and optimizer states, the epoch number, and related parameters. Such a file can be substantially larger than the weights alone. This is more than a format detail: if a service saves only the final state_dict, you can run inference but may not be able to continue interrupted training with the same optimizer state.

Retention length affects space, encryption, backups, and recovery testing. An artifact is not saved until the team has loaded it in a clean environment and obtained the expected result. A hash confirms file integrity, but it does not confirm compatibility among the code, tokenizer, and library versions.

It helps to create a small manifest next to the weights for every run:

run_id: ft-2026-07-014
base_model: internal-registry/model-a@sha256:...
dataset_snapshot: s3://restricted/train/v17/
code_commit: 3f92c1a
container_digest: sha256:...
precision: bf16
method: lora
checkpoint_format: safetensors
optimizer_state_exported: true
created_at_utc: 2026-07-27T14:10:00Z
retention_class: regulated-3y

Immutable identifiers matter here, not attractive names. A latest reference does not prove what the model was trained on. A dataset path without a version cannot reproduce a run. An image tag rather than a digest can change unnoticed.

Hugging Face describes safetensors as a simple tensor format that does not execute pickle code during loading. It is a sensible format for final weights, but it does not replace a manifest, optimizer state, or an environment snapshot. Teams often blur weight portability and run reproducibility. The first asks whether tensors can be opened somewhere else. The second asks whether the work can be repeated or continued.

With a managed service, check what happens after a job or endpoint is deleted. Are checkpoints retained separately, who assigns the retention period, can a legal hold be applied, how are copies deleted, and who can access the keys? The same questions belong to your team in a dedicated environment. Owning the disk does not create a retention policy automatically.

Artifact export must be proven before the contract

The promise that weights can be downloaded is too vague. You need a test performed before private data moves: train a small model, export the complete artifact set, delete the source environment, deploy a clean environment elsewhere, and continue training for at least a few steps. If a provider does not allow that test, you still do not know the exit cost.

Check export at four layers. At the data layer, you need final weights and intermediate checkpoints. At the training layer, you need optimizer state, scheduler, random seeds, and the exact configuration. At the execution layer, you need the container digest, driver version, and library versions. At the provenance layer, you need the base-model identifier, dataset snapshot, code commit, and evaluation metrics.

The test itself can be written as an acceptance scenario:

1. Run short training on a synthetic dataset with no private data and save a checkpoint halfway through.
2. Export every promised artifact and calculate SHA-256 for each file.
3. Delete the job and its work volume, then request confirmation of the state through an API or log.
4. In another project or local environment, load the artifacts, continue training, and compare the metric structure.
5. Record manual actions, proprietary formats, and fields that had to be reconstructed from memory.

Do not demand bitwise-identical metrics after moving between arbitrary GPUs and library versions. Nondeterministic operations can produce differences. Demand loading without undocumented conversions, continuation from the expected step, and quality within a predefined tolerance. The tolerance must be in the validation plan before the test.

A dangerous lock-in pattern rarely looks like a ban on downloads. More often, a service exports an adapter but does not reveal the exact base-model revision, preserves weights but not optimizer state, returns a proprietary package that only its SDK can load, or throttles egress until the migration window stretches out. Cover each case with a separate acceptance requirement.

For a private model, establish the rights to derivative weights too. The technical ability to export a file does not replace the contractual right to use it elsewhere. The base-model license, provider terms, and dataset rights must permit the intended route. This is a legal question with a technical consequence: prohibited or ambiguous export destroys the benefit of a portable format.

The queue is part of the model release time

A managed endpoint does not guarantee an instant GPU. Between job submission and the first training step may sit a quota check, a search for the right accelerator type, image loading, data mounting, and a wait for shared capacity. A dedicated pool also has a queue when several teams share limited nodes. The difference is who manages priorities and who can explain the delay.

Request measurable job states, not abstract availability. You need acceptance time, queue time, the reason for waiting, node assignment time, preparation start, first training step, last checkpoint, and completion. Without those timestamps, a ten-minute training failure and a six-hour wait both appear as job pending.

The Kubernetes scheduler assigns a Pod to a suitable node based on constraints and available resources. ResourceQuota limits aggregate consumption in a namespace. These mechanisms are useful in an in-house cluster, but a quota does not reserve a physical GPU. A team may have permission to request four accelerators and still wait if suitable nodes are occupied or the memory, topology, and device-type requirements do not match.

Define two SLOs. The first limits the time from submission to start for a normal job. The second covers an urgent run, such as a model fix after a failed security check. For each one, specify job size, GPU class, time of day, region or site, and the permitted share of violations. A promise of a priority queue cannot be tested without these parameters.

Check the preemption policy. Can a higher-priority job stop yours? Does the platform save a checkpoint before stopping? Who pays for lost minutes? Set the same rules among teams in your own pool. If every job has the highest priority, the scheduler gets no useful signal.

The queue is especially risky for chains where training, evaluation, merging, and quantization use different resource types. The first stage can finish while the whole release waits on the second. Measure the duration of the complete chain to a ready artifact, not only the speed of the main training loop.

A managed service does not manage the experiment

A managed service removes some infrastructure work, but it does not own experiment quality. It can provision nodes, restart a container, and collect system logs. The team remains responsible for leakage between train and validation sets, labeling quality, data licenses, hyperparameters, stopping criteria, and final-model evaluation.

This distinction matters when staffing the work. A platform engineer and an ML engineer cover different risks. Buying a managed endpoint can reduce on-call work for GPU nodes, but it does not reduce the number of data and evaluation decisions. Buying dedicated GPUs does not turn researchers into cluster administrators without reducing their research speed.

Write down the responsibility boundary as a RACI table or a simple owner list. Who updates the base image? Who approves a new library? Who investigates a stuck NCCL operation? Who restores a checkpoint? Who deletes a temporary dataset copy? Who proves to an auditor which user started the training run? If a row names two vendors and no accountable person on the team, an incident will bounce between support queues.

A managed endpoint helps when its standard path matches yours. A normal container, supported data format, permitted network route, and standard export method let the provider automate repeatable work. If every training run needs a custom kernel, modified driver, direct device access, or special inter-node networking, the team will fight the service limits. A dedicated environment then becomes the way to own the environment, not a luxury.

Do not buy dedicated GPUs just to keep familiar SSH access. Interactive access is convenient, so teams often mistake it for control. Actual control appears in a reproducible image, access policy, observable queue, backup recovery, and automatic temporary-data deletion. A terminal without those properties merely makes manual errors faster.

Isolation starts with a threat model

The phrase dedicated GPU does not describe an isolation boundary. An accelerator may be assigned to a single job while the control plane, network, image registry, logging, and support staff remain shared. Conversely, a managed service may run a job on a separate node with your encryption key and private network. A service-tier name proves nothing.

Start with the data and the adversary. The training set, source weights, adapter, evaluation prompts, final weights, and even metrics can be private. For each object, state who it must be protected from: another team in your organization, another provider customer, a platform administrator, a contractor, a compromised account, or an external intruder.

Then test concrete boundaries:

• the physical node and the GPU-sharing mode;
• the project, namespace, service accounts, and support permissions;
• inbound and outbound network traffic, DNS, and registry access;
• work volumes, object storage, backups, and keys;
• command, parameter, path, and error logs that may contain secrets.

Isolation also has a time dimension. What happens to video memory, local NVMe, and the image cache after completion? When is the work volume deleted? Does a diagnostic bundle remain? Can support staff take a memory snapshot? The answer that data is encrypted does not settle these questions because the system must decrypt data during training.

NIST SP 800-53 treats access control, system protection, and logging as separate control families. This is a useful correction to a procurement checklist: one mechanism does not replace the others. A separate node reduces part of the sharing risk, but without restricted roles and auditing it does not explain who read an artifact.

For a regulated workload, ask for evidence rather than a declaration: a data-flow diagram, the list of support roles, audit events, deletion procedure, key management, recovery results, and incident response procedure. For Kazakhstan, separately compare actual storage and processing locations with your organization's requirements and applicable law. Legal counsel defines the rule, while the architecture must show where the data physically travels.

A failure reveals the hidden cost of the choice

Imagine a team that runs fine-tuning every two weeks through a managed service. On Friday evening, training reaches 82 percent, the node is lost, and the latest checkpoint is six hours old. The service automatically repeats the job from the beginning because the user container wrote state to local disk. The bill grew, the release moved, and the platform technically performed the promised restart.

The cause is not that managed services are unreliable. The team did not agree on recovery semantics. It tested startup but did not test node loss. Proper acceptance should have forcibly stopped a short test job, confirmed that the checkpoint lived in external storage, and measured the step from which work resumed.

Now take the opposite case. A bank rents a dedicated pool for isolation. One GPU starts reporting correctable memory errors. The scheduler keeps assigning jobs to the node because the metric is not connected to automatic cordoning. Three experiments produce unstable results before an engineer manually excludes the node. A managed provider might include this automation by default, while the private environment required the team to design it.

Both failures expose the same boundary: responsibility without observability is useless. If the team owns the pool, it needs hardware telemetry, node health checks, retirement rules, and spare capacity. If the provider owns the nodes, the customer needs job lifecycle events, a retry policy, an external checkpoint, and a clear escalation route.

Run game-day tests before a private workload. Stop a node, fill a disk, revoke storage access, delay GPU assignment, and corrupt a checkpoint copy. The goal is not to break the platform in every possible way. You are checking whether the team detects the failure, preserves data, and can explain the final status without guessing.

The decision fits in one matrix

Choose dedicated GPUs when a persistent work queue keeps them occupied, startup delay affects releases, the environment needs low-level customization, artifacts must remain under your control for years, or the threat model requires control of the node and control plane. This choice only makes sense with operations owners and a redundancy budget.

Choose a managed endpoint when runs are infrequent or bursty, a standard container covers the job, the team accepts a tested export mechanism, and the provider offers a measurable start time and the required isolation boundary. Do not call that choice a compromise. For a small ML team, avoiding cluster maintenance often leaves more time for data and evaluation.

Score the options with weights from your system, not a consultant.s template:

• For frequency, check how many related GPU jobs arrive each week. Dedicated GPUs pay off with a steady flow, while a managed endpoint pays off with pauses and bursts.
• For artifacts, test export of a full checkpoint and manifest. In your pool you choose the format and storage, while in a service the answer depends on the API and contract.
• For the queue, measure time to the first step. In your pool you manage priorities, while in a service the provider manages capacity.
• For isolation, find out who shares the node, network, and control plane. A dedicated environment gives more control with the right architecture, while strong service boundaries need evidence.
• For operations, name who fixes drivers and nodes. That is your team or contractor in your pool and the provider within the SLA in a managed service.
• For scale, estimate how quickly a rare large run is needed. Your pool needs spare capacity or expansion, while service elasticity depends on available quota.

Give each criterion a weight from 1 to 5, then score each option from 1 to 5. Attach evidence to every score: a contract excerpt, test result, queue metric, or utilization calculation. A number without evidence only disguises an assumption.

Do not let the total hide a stopping condition. If private weights cannot be exported in the required format, a high convenience score does not rescue the service. If the organization cannot provide on-call coverage and node replacement, paper savings do not make a dedicated pool workable.

A hybrid design is often more honest than a binary choice

A team does not have to train every model in one environment. Short research runs on de-identified or synthetic data can go to a managed environment, while final training on a private dataset stays on a dedicated pool. A small permanent pool can handle normal work, with managed capacity covering rare peaks. This design works only when the container, checkpoint format, and manifest match on both sides.

A hybrid adds data transfer and two control surfaces. Define in advance which artifacts may cross the boundary, who approves movement, and where evaluation runs. If a researcher manually rewrites the configuration for the second environment every time, environment drift will quickly destroy the promised flexibility.

The two environments also need one evaluation contract. The same test set, prompt versions, quality thresholds, and release rules must apply regardless of the training location. Otherwise, procedural differences will distort the comparison, and the team may wrongly attribute a quality gap to the GPUs or service.

AI Router hosts open-weight models on its own GPU infrastructure for teams that need data storage within Kazakhstan, low latency, or fine-tuned variants. It can be one of the verifiable environments in the matrix, but export, retention, and isolation requirements still need to be recorded for the specific project.

Apply one final reversibility test. For a managed endpoint, name the time and procedure needed to export one complete run. For a dedicated pool, name the time needed to reduce capacity and move jobs when utilization drops. A choice the team knows how to leave will survive changing models, rates, and requirements better than the most precise table made today.