Why Long-Running Tool Calls Need Leases and Heartbeats?

A long-running tool call cannot be considered alive just because the database contains a running status. That status quickly turns into garbage: the worker may have crashed, stalled on the network, entered a runtime pause, or received a response from an external API without having time to save the result. You need a design in which the worker receives temporary permission to execute the task and regularly confirms that permission.

Leases and heartbeats solve two different problems. A lease keeps a second worker from claiming the work too early. A heartbeat lets a short lease survive a normal long-running operation. Neither one makes the system exactly-once by itself. Safe side effects also require idempotency and a fencing token.

This is especially visible in agent systems. A model may call search, a browser, an ERP system, OCR, report generation, or an internal API. One tool call finishes in seconds, while another waits several minutes for an external service. A fixed timeout in such a queue is almost always wrong: one that is too short creates parallel duplicates, while one that is too long makes the team wait until an obviously dead task becomes available again.

A lease grants temporary permission, not a promise of success

A lease is a record showing that a particular worker may execute a task until a specific point in time. After that point, the task becomes available again. The worker does not own the task forever and does not gain exclusivity through owner_id alone.

It is useful to store at least these fields:

create table tool_jobs (
  id uuid primary key,
  state text not null check (state in ('queued', 'running', 'succeeded', 'failed')),
  owner_id text,
  lease_until timestamptz,
  fencing_token bigint not null default 0,
  heartbeat_at timestamptz,
  attempt integer not null default 0,
  idempotency_key text not null unique,
  payload jsonb not null,
  result jsonb,
  error jsonb
);

owner_id answers who currently holds the lease. lease_until says when that right stops being valid. fencing_token separates the old owner from the new one if the old process wakes up too late. heartbeat_at is useful for observability and investigations, but it must not be the only criterion for reclaiming a task.

Do not conflate the lease with the status. running describes the intention and current state of the work. The lease describes the right to own it. A task may remain running even though its previous owner has lost the right and the new owner has not yet started useful work. This is a normal brief state during a handoff.

The etcd documentation states the idea directly: a lease expires if the cluster does not receive keepalives within the TTL. In etcd, this is a storage primitive, not a ready-made protocol for processing your business task. You still need to decide what a worker may do after its right expires and how the receiver of a side effect will distinguish the old owner from the new one.

A running status cannot tell a live call from a stuck one

A typical failure is unremarkable. Worker W1 claims a task: "obtain a statement, extract the fields, and send them to the approval system." It sets state = 'running', calls an external service, and gets stuck in the HTTP library: the connection has not formally closed, the client timeout was never set, or a DNS request is waiting longer than expected.

Ten minutes later, the database still says running. The scheduler does not know whether the work is still running, completed on the provider's side, or whether process W1 was killed by the orchestrator long ago. If the team forbids retries until manual intervention, the queue stops being a queue. If the team retries based on the status age, it starts a second call and gets a double charge, a duplicate email, or two updates to the same record.

A heartbeat does not measure useful progress. It answers a narrower question: is the worker still alive, connected to the coordinator, and continuing to hold the right to the task? That distinction matters.

For example, a browser tool may be alive, sending heartbeats, and waiting forever for a modal window. According to the heartbeat, the operation has not stalled, but it is no longer useful to the product. That is why you need two separate controls:

• a lease and heartbeat for ownership safety;
• a stage-duration limit and a tool timeout for execution quality.

If you combine these signals, you get poor automation. It will retry slow but working operations or keep an endless lease on a task that stopped producing results long ago.

Claiming a task must be atomic

A worker should not first read a free task and then update it separately. Another worker can see the same state between those requests. Grant the lease in one transaction and check the time on the database side, not on the container's clock.

Here is an example claim operation for PostgreSQL. It selects one waiting or expired task, locks it for the duration of the transaction, assigns an owner, moves the lease deadline, and increments the fencing token.

with candidate as (
  select id
  from tool_jobs
  where state = 'queued'
     or (state = 'running' and lease_until < now())
  order by id
  for update skip locked
  limit 1
)
update tool_jobs j
set state = 'running',
    owner_id = :worker_id,
    lease_until = now() + interval '90 seconds',
    heartbeat_at = now(),
    fencing_token = fencing_token + 1,
    attempt = attempt + 1
from candidate
where j.id = candidate.id
returning j.id, j.payload, j.fencing_token, j.lease_until;

A typical result looks like this:

id: 8f1b7c3d-...
fencing_token: 42
lease_until: 2026-07-23T21:14:30Z

for update skip locked is not magic or a universal recipe. It is useful when several workers read one table and you do not need a separate broker. It does not replace per-user concurrency limits, a limit for a specific tool, or priorities. But it closes the race in which ten workers select the same queued row before the first update.

Do not use application time in a condition such as lease_until < :client_now. Container clocks drift, virtual machines can make abrupt time corrections, and a worker may continue using stale time after a pause. Database now() does not make a distributed system perfect, but it gives one arbiter a shared decision point.

A heartbeat should renew the lease only for the current owner

A heartbeat is not an instruction to "make the task alive." It is a conditional renewal: the database must verify that the request came from the current owner and that its lease has not expired. Without this check, an old worker could revive a task that was handed over long ago.

The renewal request should look like this:

update tool_jobs
set lease_until = now() + interval '90 seconds',
    heartbeat_at = now()
where id = :job_id
  and state = 'running'
  and owner_id = :worker_id
  and fencing_token = :fencing_token
  and lease_until > now()
returning lease_until;

A zero-row response does not mean "try the request again." It means the worker is no longer the owner. There are several possible reasons: the lease expired, another worker already took over the task, an operator canceled it, or a failure policy changed its state. After a zero-row result, the worker must stop actions that can change the outside world.

The loop can look like this:

job = claim(worker_id)
start heartbeat every 30 seconds

run tool with a bounded client timeout

on each heartbeat:
  if extend(job.id, worker_id, job.fencing_token) returns no row:
    cancel tool if the client supports cancellation
    mark local execution as lease_lost
    do not commit a success result

on tool completion:
  stop heartbeat
  commit result only if the same token still owns the job

Do not run the heartbeat in the same thread that performs the tool call. The call may block the event loop, occupy the only thread, or hang in a native library. A separate runtime task, thread, or control process is usually more reliable. But this controller must not blindly renew the lease if the main executor has already finished or is no longer locally observable.

The Amazon SQS documentation recommends this principle for unpredictable durations: start with a short visibility timeout and renew it periodically while the consumer works. It also states directly that a short visibility timeout can cause a duplicate before the first consumer finishes, while a timeout that is too long delays retry after a failure.

Choose intervals by their safety margin, not by a nice-looking number

The pattern "90-second lease, 30-second heartbeat" is often suitable for a first deployment, but it is not a rule. Choose a duration that lets the worker survive one failed heartbeat delivery, a brief database outage, and a normal runtime pause.

A practical model is:

• L is the lease duration;
• H is the heartbeat period;
• J is the margin for network jitter, process pauses, and database overload;
• R is the number of consecutive missed heartbeats you are willing to tolerate.

You need L > R × H + J. If the heartbeat runs every 30 seconds, you want to survive one missed delivery, and you estimate the margin at 20 seconds, a 90-second lease gives you a reasonable buffer. A 35-second lease in the same setup means ordinary latency can take ownership away from a worker that is still working.

Do not wait until one second before expiration to renew the lease. This popular way to reduce requests breaks at the first spike in latency. Renew in advance, when at least one full heartbeat interval and enough reserve for a failed request remain.

Separate work classes. A classifier call expected to take a few seconds should not hold the same lease as an accounting-system file export. You can store lease_duration in the task type or pass it to the claim operation, but do not let the model choose the TTL directly. A model can make a mistake, and it may receive a user instruction to perform pointlessly long work.

SQS has another unpleasant limit for visibility-timeout renewal: the maximum of 12 hours is counted from the first message receipt, and later renewals do not reset it. This is a limitation of that service, but the lesson applies to any architecture: a task renewed forever should usually become a sequence of persistable stages.

Losing the lease requires stopping, even if the tool returned success

The most dangerous moment comes not when a worker crashes, but when an old worker wakes up. W1 received lease token 41 and sent a request to the provider. Then the network between W1 and the database failed. The heartbeat does not get through, and the lease expires. W2 claims the task, receives token 42, and runs it again.

W1 then receives a delayed successful response. If it runs a simple query:

update tool_jobs
set state = 'succeeded', result = :result
where id = :job_id;

it will overwrite the new owner's result. Worse, W1 may send a second payment or approve a request in an external system. The status in your table cannot stop a request that has already been sent.

The final write must be conditional:

update tool_jobs
set state = 'succeeded',
    result = :result,
    lease_until = null
where id = :job_id
  and state = 'running'
  and owner_id = :worker_id
  and fencing_token = :fencing_token
  and lease_until > now();

If the update returns no rows, the result must not be declared accepted. Save it in a technical log for investigation, but do not publish it to the user or use it as the state of a business process.

This is where the fencing token comes in. owner_id is not enough as the only protection: after a restart, the same worker may receive the same logical name, and an old network connection may continue to live. A monotonically increasing token ties each lease grant to the order of ownership. The receiver of a critical side effect must remember the last accepted token and reject lower values.

For example, a payment service accepts the header X-Execution-Fence: 42. If a command with token 42 has already been accepted, a command with token 41 has no right to change the state, even if its authentication is valid. This does not make an external API idempotent automatically, but it removes the class of errors in which "the old owner writes after the new one."

Idempotency covers what a lease cannot

A lease manages concurrency before and during execution. It cannot revoke an HTTP request that has already reached an external service. The network can drop the response after the recipient has performed the action. The worker sees a timeout and does not know whether it is safe to retry.

That is why every operation with a side effect needs an idempotency key. Do not generate a new UUID for every attempt. Use a stable key tied to the meaning of the operation, such as payment:{invoice_id}:capture or ticket:{job_id}:create.

A request to an external tool might look like this:

{
  "operation": "create_case",
  "idempotency_key": "tool-job:8f1b7c3d:create_case",
  "execution_fence": 42,
  "input": {
    "customer_id": "c-1048",
    "summary": "Проверить расхождение в счете"
  }
}

The recipient should return the previous result when it sees the same idempotency_key a second time. If it supports a fencing token, it should also reject a request with a value lower than the last accepted value for the same entity. These mechanisms work together:

• an idempotency key protects against the same command being sent again;
• a fencing token protects against an old command arriving after a new one;
• a lease reduces the chance that two attempts start at the same time at all.

The popular advice to "add retries" is almost always incomplete. Retries without idempotency increase the chance of an expensive duplicate. Retries without a task deadline create an endless load. Retries without distinguishing errors repeat validation failures as persistently as temporary network failures.

The risk is lower for read-only tools, but duplicates still cause harm. They consume API limits, may return data from different points in time, and force the model to build an answer from an inconsistent picture. Inside an agent loop, this looks like "the model is hallucinating," although the real cause is often two unsynchronized executions of the same task.

Do not treat heartbeat errors as ordinary network errors

If one heartbeat fails, the worker may still be the owner. If it does not know whether the lease was renewed, the situation is already ambiguous. The request may have reached the database while the response was lost. The safe reaction depends on the time remaining and the type of operation.

I use a simple rule: a worker stops starting new external actions when it cannot confirm the lease before a conservative deadline. It may wait for a heartbeat retry while it still has a safety margin, but it should not start the next step in a chain during the last second of its ownership.

It helps to divide errors into three groups:

• The database replied that no row was updated. The lease is lost, and the work must stop.
• The database is unavailable or no response arrived. Ownership is unknown, so no new irreversible step is allowed.
• The external tool did not respond. Ownership may still be valid, but the result of the external action is unknown, so an idempotent retry or an operation-status query is needed.

The state "ownership unknown" is often hidden behind an automatic retry. Do not hide it. Log it as a separate event with job_id, the token, the time of the last confirmed lease, and the external request ID. These are the fields you need at 2:00 a.m., when an operator sees two requests from one user and tries to understand who created them.

An expired task does not have to be retried immediately. If it has exhausted its attempts, has an unrecoverable error, or belongs to a canceled session, the scheduler should move it to a clear terminal state. A queue without an attempt policy simply moves manual work from one list to another.

Observability should show ownership, not just task counts

A running jobs graph is almost useless. It shows how many rows have that status, but not which of them truly owns the work or how close the lease is to expiring.

The minimum set of metrics includes the age of the last successful heartbeat, the remaining lease time, the number of expired leases, the number of failed renewal attempts, and the number of tasks claimed again. Break them down by tool type, model, and external provider. Otherwise, a spike in browser-tool timeouts will blend into normal short search calls.

A log for one execution should contain at least these events:

job_claimed job=8f1b... token=42 lease_until=21:14:30Z
heartbeat_ok job=8f1b... token=42 lease_until=21:15:00Z
tool_request_started job=8f1b... request=ext-791
heartbeat_lost job=8f1b... token=42 reason=zero_rows
result_rejected job=8f1b... token=42

Do not write complete prompts, documents, and tool responses to these events by default. For LLM applications, a log can easily become another personal-data store. Technical identifiers, a payload hash, the tool class, duration, and error code are enough. Store the full content only where there is a clear need and a retention policy.

AI Router can be a convenient single OpenAI-compatible layer for model calls, but the lease must remain in your orchestration layer, next to the queue and tool-call state. The model gateway should not decide whether a worker has the right to retry a payment or change a request.

Long-running agent work is better divided into persistable stages

One huge tool call with a lease renewed for hours looks simpler until the first failure. Afterward, you do not know what the tool has already done, where the intermediate file is, or whether it is safe to continue.

Divide the work where a verifiable result appears. For example, a report-preparation process can be represented as: obtain the source data, save a normalized set, build a draft, send it for review, and publish it. Each stage gets its own idempotency key, result, and new lease. The system can stop between stages without losing track of what has been done.

Do not split things up for the sake of splitting them up. If a stage takes 200 milliseconds and has no separate side effect, a new task adds more complexity than reliability. A boundary is useful where an operation waits for a long time, interacts with the outside world, is expensive to repeat, or needs human review.

For an LLM agent, it is especially useful to separate planning from execution. The model may suggest a sequence of actions, but the dispatcher should create separate tasks with an allowed tool type, time limit, idempotency key, and retry policy. Then losing the lease for one call does not turn the entire agent run into an unmanageable state.

Run one unpleasant test before production. Let W1 claim a task and send an external request. Then cut off its access to the database, wait for the lease to expire, let W2 claim the same work, and only then restore W1's network. Verify that W1 cannot renew the lease, cannot write the result, and cannot create a second irreversible side effect. If this test fails, your heartbeat is only producing reassuring logs.