How to Configure TCP Keepalive for an LLM API Behind NAT

A long-running LLM call does not by itself mean that the connection has failed. The model may be thinking normally, a queue may be waiting for a GPU, or a streaming response may arrive in occasional fragments. But a connection that silently died in a NAT device, firewall, or load balancer looks almost the same: the client waits, the number of busy sockets grows, and the error appears only after a random write or an overall timeout.

TCP keepalive for an LLM API is not meant to make generation faster. It helps distinguish a live wait from a socket that the kernel still considers established even though a device in the middle has already forgotten its state. You need to address this in four places at once: the client pool, the TCP stack, the proxy, and the load balancer. A single sysctl setting is not enough.

A stalled request and a dead socket are different failures

A stalled request remains on a working TCP connection, while a dead socket exists only in the memory of one side. If you mix these cases together, the team will either start terminating normal long generations or leave users waiting on a connection that will never respond.

Consider a typical chain: an application service holds an HTTPS connection to an LLM gateway through an outbound NAT. After a period of inactivity, the NAT removes the translation entry. The client process does not know this: the local kernel received neither FIN nor RST, so the socket remains in the ESTABLISHED state. The next request is sent by the library through the old connection from the pool. The packet disappears on the way, and the client waits for a response until the overall deadline.

Now consider a different case. The request reached the upstream, the model is busy, and the response has not started yet. The socket is healthy, but the user is waiting for the first byte. TCP keepalive will not provide a useful diagnosis because the stack may have unacknowledged data or an active transfer. This is where you need a first-byte timeout and, if the product supports streaming output, a limit on the pause between events.

Separate at least four times in your observability data:

• when the client took the connection from the pool;
• when the TCP and TLS connection completed, if a new connection was needed;
• the first byte of the HTTP response;
• the last response byte or the reason for closing.

If you see a long interval before the first byte on a new connection, look at the queue, DNS, connection establishment, or the upstream. If the delay occurs almost exclusively on connections with a high idle age, investigate the pool, NAT, and idle rules. This distinction can save days of useless prompt tuning.

TCP keepalive checks the path, not the model's work

TCP keepalive sends probe TCP segments on an idle connection and waits for a response from the peer. It helps the kernel close a locally open socket when the remote side or the network path has disappeared. It does not measure inference speed, validate an HTTP session, or make the server send the next token.

RFC 9293 describes keepalive as an optional mechanism. The standard requires that an application be able to enable and disable it for a specific connection, and that it be disabled by default. It also preserves the historical default minimum idle period of two hours. For an API behind NAT, this is almost always useless: many intermediate devices clear inactive state much sooner.

An important qualification from RFC 9293 is often lost in summaries: failure to receive a response to one particular probe does not prove that the connection is dead. TCP does not guarantee delivery of pure ACKs. One lost probe must not therefore turn into an immediate disconnect. You need several attempts and a finite detection budget.

Do not confuse these three similar terms:

• an HTTP persistent connection allows one TCP socket to be used for multiple requests;
• TCP keepalive checks an idle transport socket;
• an application heartbeat sends a protocol-meaningful message, such as an SSE comment or a WebSocket ping.

For streaming APIs, an application heartbeat is usually a more reliable signal that the response is alive. For a silent connection in a pool, TCP keepalive is more appropriate. To limit how long the user waits, use a request deadline. They complement one another rather than replace one another.

Calculate the detection time before configuring anything

On Linux, the time before a dead connection is closed is approximately keepalive_time + keepalive_intvl × keepalive_probes. This is a simplified planning model, not a promise of millisecond-level accuracy. Scheduling, packet loss, proxy implementation, and the network will add variation.

Suppose the client starts checking after 30 seconds of idle time and sends three probes at 10-second intervals. The expected detection budget is about 60 seconds. If the NAT removed its entry at the 45-second mark, the next request may enter the window before the first check. That is normal: keepalive does not predict when an entry will be removed. It reduces the time it takes the local side to find out.

Choose values based on path constraints, not personal preference. You need to know:

1. the shortest idle timeout on the outbound NAT, firewall, ingress, egress, and load balancer;
2. how quickly the service must recognize that a connection is unusable;
3. the number of connections that may be idle at peak time;
4. whether losses on mobile or interregional networks are acceptable;
5. whether an application layer already sends data regularly.

If the shortest known timeout is 60 seconds, a first probe after 50 seconds may be too late. If you probe every 5 seconds on hundreds of thousands of sockets, you turn liveness checking into constant network noise. RFC 9643 on TCP management notes that frequent keepalive traffic loads the network and endpoints, and that the idle interval should not be reduced below 15 seconds without a particular reason. For a server-side LLM API, it is more sensible to limit connection age and idle time in the pool first, and leave TCP keepalive as a safety net.

The client pool should discard old connections by itself

The connection pool creates most problems with "eternal" calls. It saves TLS handshakes and reduces latency, but keeps sockets longer than the devices along the route remember them.

The wrong response to an incident sounds like this: "Let's turn off keep-alive." Sometimes this temporarily hides the failure because every request creates a new TCP connection. The cost of hiding it soon appears as more handshakes, higher proxy load, and worse tail latency. Disabling the pool makes sense only as a diagnostic experiment, not as a permanent architecture.

It is better to give the pool two separate limits. idle timeout says how long a connection may remain unused. max lifetime limits its total age even when it is used periodically. The first limit should be shorter than the minimum timeout on the network path. The second protects against long-lived connections that have accumulated rare failures, a route change, or state left over after an infrastructure update.

You also need a limit on connection acquisition. If the entire pool is occupied by long streaming requests, a new ordinary request must not wait forever for a slot to become available. The pool_acquire_duration metric often explains a "hang" better than a TCP graph.

Set separate budgets for every outbound call:

• connection timeout, including DNS and TCP/TLS if the library combines them;
• timeout until the first response byte;
• timeout for the pause between bytes during streaming;
• the complete operation deadline;
• a bounded wait for an available connection in the pool.

Do not make all the numbers identical. For example, an overall deadline of five minutes and a five-minute inter-byte pause mean that the user will not learn about a lost stream until the entire budget has elapsed. For streaming, set the pause limit based on the model's real behavior and the response format, not on the average generation duration.

Socket settings matter more than global sysctl

net.ipv4.tcp_keepalive_time, tcp_keepalive_intvl, and tcp_keepalive_probes set system-wide defaults on Linux. Linux kernel documentation explains that the probe interval multiplied by the number of probes determines the timing of repeated checks. However, these sysctl values do not enable keepalive on every socket by themselves.

The application first enables SO_KEEPALIVE, then, when needed, sets values for the individual socket with TCP_KEEPIDLE, TCP_KEEPINTVL, and TCP_KEEPCNT. The tcp(7) man page documents these socket options for Linux. If the HTTP library does not expose the socket, changing sysctl may not solve the problem or may affect processes you did not intend to change.

Check the current system values on the host:

sysctl net.ipv4.tcp_keepalive_time \\
      net.ipv4.tcp_keepalive_intvl \\
      net.ipv4.tcp_keepalive_probes

# Example output:
# net.ipv4.tcp_keepalive_time = 7200
# net.ipv4.tcp_keepalive_intvl = 75
# net.ipv4.tcp_keepalive_probes = 9

This configuration means that a socket with keepalive enabled may remain silent for two hours before the first probe. That is not protection against a NAT with a short-lived entry. But do not rush to change parameters across the whole machine. First find out whether your HTTP client applies SO_KEEPALIVE and whether it lets you set parameters on the outbound connections used by the LLM client.

In Go, this is usually done with a custom dialer and a socket control function. In Java, look for transport settings in the specific HTTP client, not only JVM parameters. In Node.js, check whether the agent in use calls socket.setKeepAlive() and which values it passes. In Python, the decisive factor is the transport used by the selected library and how it creates the connection pool. Method names vary, but the principle is the same: verify the setting on a real socket, not merely in the process configuration.

The proxy must distinguish silence from a long response

A proxy can easily create a false diagnosis. Nginx uses proxy_read_timeout 60s by default. The official Nginx documentation clarifies an important detail: this timeout applies between two successful read operations from the upstream, not to the entire response transfer. If the upstream sends nothing during that interval, Nginx closes the connection.

For an LLM, this means the following. A non-streaming request may legitimately remain silent until the final JSON response is ready. A streaming response may run for an hour as long as no pause between fragments exceeds proxy_read_timeout. You cannot use one value for both behaviors and expect clear failures.

A minimal Nginx configuration fragment for a streaming route might look like this:

location /v1/chat/completions {
    proxy_http_version 1.1;
    proxy_set_header Connection "";

    proxy_connect_timeout 5s;
    proxy_send_timeout 30s;
    proxy_read_timeout 90s;

    proxy_buffering off;
    proxy_pass http://llm_upstream;
}

These are not universal values. proxy_connect_timeout limits the time to establish a connection to the upstream. proxy_send_timeout applies to pauses while sending the request to the upstream. proxy_read_timeout limits the pause while reading the response. For SSE, proxy_buffering off is needed so that the proxy does not accumulate fragments and turn the stream into one late, complete response.

Also check timeouts on the incoming side: the client to your ingress, ingress to the application, application to the gateway, and gateway to the provider. The shortest timeout wins. If the client waits 120 seconds but the ingress closes the response after 60 seconds of silence, increasing the SDK timeout will change nothing.

A load balancer and NAT can lose state without notifying you

A stateful device is not required to tell both sides that it has forgotten a TCP session. It may simply remove the entry after a period of inactivity. The local machine therefore sees ESTABLISHED until it tries to send data or keepalive receives several failures.

The most common investigation mistake is asking the network owner only, "What is your TCP timeout?" There may be several answers in the chain: container NAT on the host, a corporate firewall, cloud egress, WAF, ingress, and the provider's load balancer. You need the minimum timeout for the traffic path, not one attractive value from a document.

Create a table with four columns: segment, idle timeout, owner, and how you confirmed it. In the last column, do not write "the team said so." Include a configuration, administrative document, test, or trace. If a particular device is inaccessible, set a conservative pool limit and verify it with a controlled disconnect.

Passive TCP keepalive does not always preserve a NAT entry as you expect. Some devices count probe segments, some have their own rules, and some problematic connections fail not because of idle cleanup but because of a route change, a peer restart, or an overloaded state table. The goal is not to keep a NAT entry alive forever at any cost. The goal is to stop handing a dead socket out of the pool quickly.

Test with a controlled disconnect, not an error graph

An error graph will not prove that keepalive works. It shows only the consequences, mixed with DNS errors, upstream overload, and API limits. You need a short, reproducible test in which you know when the path disappears.

In a test environment, create a persistent connection, let it remain idle beyond the selected threshold, and then block traffic to the upstream on one segment. On Linux, you can temporarily add a rule that drops outgoing packets to the test address and port. Do this only in an isolated environment or on a dedicated test route.

sudo iptables -I OUTPUT -p tcp -d 203.0.113.20 --dport 443 -j DROP

# Remove exactly the rule you added after the test.
sudo iptables -D OUTPUT -p tcp -d 203.0.113.20 --dport 443 -j DROP

The 203.0.113.0/24 range is reserved for documentation. Replace it with the address of your test upstream, but do not run this experiment against a shared production endpoint.

During the test, capture packets on both sides of the failure point:

sudo tcpdump -ni any 'host 203.0.113.20 and tcp port 443'

Look for this sequence: the last useful application message, then keepalive probes or a new request after the socket is taken from the pool, no responses, socket closure, and a clear application error. At the same time, record the connection age, request ID, retry attempt, and reason for closure. Without these fields, tcpdump confirms the network fact but does not explain why the application continued waiting.

After DROP, repeat the test with RST if your lab allows it. DROP imitates a silent loss of state and usually exposes the worst case. RST checks whether the client can quickly remove a connection from the pool after an explicit failure.

A request retry is not a cure for the network

When an old socket dies, the library often suggests a retry. This is useful for idempotent reads or an operation with a reliable deduplication key. For a POST to an LLM API, an unconditional automatic retry is dangerous: the upstream may have received the body, started generation, called a tool, or stored a result, while the response was lost on the way back.

A network failure before the first byte does not prove that the upstream did nothing. The client's TCP stack knows only that it received no response. It does not know whether the server processed the request. Therefore, separate retries by operation semantics.

A safe approach looks like this: the client creates a request ID, the server or application layer stores the operation result for that ID for a limited time, and a retry returns the same result instead of starting the work again. If no such contract exists, retry only errors that occurred before the body was sent, or clearly show the user that the outcome is unknown.

For calls that send a simple prompt without tools or side effects, the duplication risk may be acceptable, but this must be a product decision. An engineer should not quietly accept it through an HTTP client setting.

Checklist before changing production timeouts

Start not with global tcp_keepalive_time, but with one route and one specific symptom. After every change, repeat the controlled disconnect and compare not only the error rate but also detection time, the number of new connections, and pool queueing.

Check the following:

• The client has limits for connection establishment, the first byte, the pause between bytes, the complete request, and pool acquisition.
• The pool limits idle age and the total age of sockets instead of relying on eternal HTTP connections.
• SO_KEEPALIVE is confirmed for outbound sockets, and per-socket intervals are shorter than the shortest known timeout on the path.
• Nginx and load balancers have separate rules for ordinary and streaming calls, and the team knows their shortest timeout.
• Logs connect the request ID, connection ID, socket age, first-byte time, and closure reason.

If you use a single OpenAI-compatible gateway, do not shift responsibility for these limits to the gateway. For example, AI Router can simplify routing to different models through one endpoint, but the client pool still determines whether an outdated connection will be reused.

A good result from this work looks boring: after a silent disconnect, the client quickly receives a network error, removes the socket from the pool, creates a new connection, and applies only an allowed retry. The user does not wait for the overall timeout, and the on-call engineer does not search for a "stuck model" when NAT removed the entry long ago.