Telemetry shutdown cannot wait for the last signal

A span lost during process shutdown rarely points to an error in the span itself. More often, the application called end() correctly, the SDK placed the result in a queue, and the orchestrator killed the process a second later along with that queue. In a backend, this looks like a random gap in a trace. In reality, it is a predictable race between the application lifecycle and the exporter lifecycle.

Telemetry shutdown should be designed as part of the normal shutdown sequence, not added to a SIGTERM handler on the eve of a release. This applies to HTTP services, consumer processes, cron jobs, migrations, and short-lived LLM tasks. Their shutdown patterns differ, but they share one unpleasant trait: the process can disappear before telemetry leaves memory.

end() completes the measurement, not the delivery

Calling span.end() records the end time and passes the span into the processor chain. It does not mean the backend has received the data. With batch processing, a memory queue, an export timer, serialization, a network request, the receiver's response, and sometimes another collector queue stand between these two events.

Even experienced teams often blur this distinction. They say, "We close our spans," when they mean, "We hope an agent sends them at some point." For a long-running process, that hope sometimes works: another batch arrives a few seconds later. For a task that runs for 400 milliseconds and then exits, there is nobody left to wait.

The OpenTelemetry specification separates these stages explicitly. The processor calls OnEnd synchronously during Span.End, but BatchSpanProcessor then accumulates completed spans. The standard values described in the specification and SDK environment variables can easily create a trap: the default delay between exports is 5 seconds, the maximum queue size is 2048, the batch size is 512, and the export timeout is 30 seconds.

If your container receives SIGTERM and has a 10-second grace period, a span completed at the start of shutdown may wait up to 5 seconds for the scheduled flush. The exporter may then wait on the network longer than the process remains alive. You do not have telemetry that "sometimes disappears." You have conflicting timers built into the design.

Check unfinished spans separately. If the code receives a signal and exits immediately, active HTTP requests, database operations, and model calls may never reach their end(). A flush cannot save something the processor never received. First stop new work from entering the system and let current work finish or cancel it deliberately.

BatchSpanProcessor is convenient while it has time

A batch processor is almost always the right choice in production. It reduces the number of network requests and protects application threads from exporter latency. Switching the whole service to the synchronous SimpleSpanProcessor so that "nothing gets lost" may seem like an easy solution, but it usually moves the problem into request latency and creates a wave of errors when the receiver is slow.

BatchSpanProcessor works only under one important condition: the process remains alive long enough for the queue to drain. It sends a batch when the periodic timer fires, the queue reaches the batch size, or the calling code requests a flush. When the queue is full, it starts dropping newly completed spans. Once shutdown has started, late spans may also be dropped.

The OpenTelemetry semantic conventions documentation for self-observability contains a useful operational detail. For a full queue, queue_full is the recommended value. For spans created after the processor has stopped, use already_shutdown in the error.type attribute of the span processing metric. Support for this metric depends on the SDK version and language, but the check itself should be part of your list of observable signals.

Do not confuse these different kinds of loss:

• queue_full means the application completed spans faster than the pipeline could process them.
• An exporter timeout or error means the processor took the data but did not finish exporting it within the available time.
• already_shutdown means your code continued running and completing spans after you had already closed the SDK.
• Missing span.end() means the application operation never reached normal completion.

Different teams own these failures. Increasing the queue is pointless for already_shutdown. Raising the exporter timeout is also pointless when the queue is overflowing. Name the type of loss first, then change the setting.

Shutdown must come after work stops

The correct shutdown sequence is simple on paper and regularly breaks in code. The process should stop taking new work, finish or cancel work it has already accepted, close its application resources, and only then close telemetry. After the provider shuts down, you cannot expect it to accept new spans.

For an HTTP service, the order is usually:

1. Receive SIGTERM or another shutdown signal and mark the process as draining.
2. Remove it from readiness and stop accepting new connections or jobs from the load balancer.
3. Wait for active requests until a chosen deadline, then cancel the remaining contexts.
4. Close consumers, pools, and background workers that could still create spans.
5. Call the telemetry provider's shutdown with a separate timeout and check the result.

The third step cannot be replaced with "wait a little." Keep a counter of active units of work. In HTTP, these are requests. In a queue, they are messages. In a batch pipeline, they are the items already picked up by a worker. A shutdown signal puts this counter into draining mode. Once it reaches zero or the deadline expires, telemetry can shut down.

In Go, a particularly harmful pattern is putting defer tp.Shutdown(ctx) in main while calling os.Exit(1) from an error branch. os.Exit does not run deferred functions. In Node.js, the equivalent mistake is calling process.exit(1) immediately after handling an exception. In Java, you may see Runtime.getRuntime().halt(), which bypasses shutdown hooks. All three can be justified when the process is corrupted, but they should not be normal control-flow branches.

Here is a Go skeleton where the order is explicit. It is not tied to a particular exporter, but it shows where the deadline belongs:

rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)\ndefer stop()\n\n\u003c-rootCtx.Done()\n\nserver.SetReady(false)\nserver.StopAccepting()\n\nworkCtx, cancelWork := context.WithTimeout(context.Background(), 20*time.Second)\nerr := workers.Drain(workCtx)\ncancelWork()\nif err != nil {\n    logger.Error(\"work drain failed\", \"error\", err)\n}\n\ntelemetryCtx, cancelTelemetry := context.WithTimeout(context.Background(), 5*time.Second)\nerr = tracerProvider.Shutdown(telemetryCtx)\ncancelTelemetry()\nif err != nil {\n    logger.Error(\"telemetry shutdown failed\", \"error\", err)\n}\n```

This example does not promise delivery during a total network failure. It does something else: the process does not destroy its own queue before trying to send it, and the log records failure explicitly.

## Timeouts must fit within the grace period

Do not choose the shutdown timeout by intuition. It is part of the total time budget provided by the platform. Kubernetes, systemd, a container runtime, a supervisor, and a serverless environment all have different rules, but none of them care whether you managed to export the last trace. When the external deadline expires, the process stops existing.

Build the budget backward. Suppose the orchestrator gives the process 30 seconds. You may need 2 seconds for the load balancer to stop sending traffic, 18 seconds for active requests, 6 seconds for consumers, and 4 seconds for telemetry. These are not universal values, just an example of how the total is allocated. If an active operation can legitimately last a minute, a 30-second grace period already conflicts with your contract, regardless of OpenTelemetry.

The exporter should have less time than the complete shutdown. Otherwise, the external deadline will cut it off in the middle of a network call. Do not set `OTEL_BSP_EXPORT_TIMEOUT` equal to `terminationGracePeriodSeconds` or the systemd timeout. Leave room for transitions between stages, code execution, and thread scheduling.

A configuration might look like this:

```bash
OTEL_BSP_SCHEDULE_DELAY=1000\nOTEL_BSP_EXPORT_TIMEOUT=4000\nOTEL_BSP_MAX_QUEUE_SIZE=4096\nOTEL_BSP_MAX_EXPORT_BATCH_SIZE=512\nAPP_DRAIN_TIMEOUT=20s\nAPP_TELEMETRY_SHUTDOWN_TIMEOUT=5s\n```

Reducing `OTEL_BSP_SCHEDULE_DELAY` shortens the average wait during normal operation, but increases the frequency of exports. This is a cost and load setting, not a replacement for a proper shutdown. A larger queue provides room for a short burst, but consumes memory and does not create network capacity. If the exporter consistently falls behind, the queue only hides the delay for longer.

Check one more limitation: the context timeout you pass to shutdown does not always cancel all internal background threads in exactly the same way across SDKs. Read the documentation for the implementation you chose and test it with a real exporter. The specification requires `Shutdown` and `ForceFlush` to report success, failure, or timeout, but the specific API differs by language.

## `ForceFlush` is useful for short-lived processes and risky environments

OpenTelemetry explicitly says that `ForceFlush` should be called only where it is truly necessary, and gives FaaS as an example: the environment may suspend the process after an invocation before the scheduled batch export. This is a useful guideline, but not a reason to add a flush after every request.

In a long-running API service, flushing after every HTTP response turns batch telemetry into expensive synchronous delivery. You get more network work and more tail latency, without a 100 percent guarantee when the receiver fails. A normal service relies on batching during operation and on shutdown when it stops.

The picture is different for a short-lived task. A script that reads a CSV, calls an external API, and exits after 200 milliseconds will never wait for a five-minute or even five-second scheduler. It must own the SDK lifecycle. After all work is complete, it should call `Shutdown`, wait for the result, and only then return its exit code.

Separate `ForceFlush` and `Shutdown` by meaning:

- `ForceFlush` asks the processor to send data it has already received without destroying the provider.
- `Shutdown` stops the pipeline, includes the effect of a flush, and releases resources.
- Work started after `Shutdown` must not rely on the old provider.
- A flush error means the attempt did not complete. It does not mean the data is safely stored somewhere outside the process.

In a serverless function, calling `ForceFlush` before returning from the handler may be appropriate if the instance can handle the next invocation and the provider must not be closed after every call. Measure the cost first. At high traffic, this adds network work to every invocation. If the environment creates a new process for each task, shutting down the provider is usually clearer and safer.

## Background tasks are most likely to shut down the SDK too early

A task queue and an HTTP server have different traps. A server usually keeps the process alive through its event loop and active connections. A worker may receive a message, start several goroutines or promises, acknowledge the message to the broker, and consider the task complete while background operations are still running.

The most troublesome version looks like this. A worker creates a parent span for a job, starts three parallel model calls, calls span.end() in the main flow, and moves on to shutdown. Two child operations are still running. Their spans either finish after the processor has closed or never finish when the context is canceled. In the observability interface, you see a short job without its most expensive calls and draw the wrong conclusion about latency and cost.

Do not close the telemetry provider after acknowledging one message if the worker is meant to keep running. Close the provider when the entire worker process stops. For the task itself, use a counter or structured concurrency: the parent should end its span only after it has waited for child operations or recorded that they were canceled.

Manual process.exit() is particularly dangerous in Node.js. The event loop might wait for an open exporter network request, but a forced exit gives it no chance. Set process.exitCode instead, complete a controlled shutdown with await, log the error, and let the process finish naturally. If a library or runtime keeps the process alive too long, find that descriptor separately rather than treating it with an immediate exit.

For scheduled jobs, add an explicit finalization stage to the task contract. Its result should appear in the normal process log: run ID, number of processed items, telemetry shutdown result, and duration. When the next incident reveals a gap in a trace, this log will help distinguish exporter loss from an emergency process kill.

Check the entire chain, not just the message "flush completed"

A "flush completed" log is useful, but it does not prove that the event is available in the final storage system. The exporter may have successfully sent the batch to a collector, while the collector queued it, dropped it because of a limit, or failed to deliver it further. A trace crosses several independent failure boundaries.

Test the exact chain you run in production: application, local SDK, network, collector or gateway, and backend. Do not stop at a unit test that replaces the exporter with an in-memory object. That test checks call order but hides DNS, TLS, proxies, queue limits, and container shutdown timing.

The following scenario belongs in CI for every type of executable process.

1. Generate a run_id, such as a UUID, and add it as an attribute to the root span of the test run.
2. Create several child spans, finish some immediately, and finish one after a short delay.
3. Run the same shutdown path the application uses on SIGTERM, not a separate test function.
4. Wait for run_id to appear in the final backend within a chosen time limit.
5. Repeat the run with an artificial receiver delay and with a timer close to the external grace period.

Check more than the presence of one trace. Compare the expected number of completed spans, the root span status, and the duration of the child operation. One root span arriving while child spans are missing creates a plausible but false picture.

Add a separate overflow test. Temporarily slow down the exporter, set a small queue limit, and complete more spans than it can hold. The team should see a controlled loss in a metric or log. If you cannot detect an artificially created queue_full, you will learn about it from an analyst's complaint during a real incident.

Monitor the pipeline itself

Telemetry that nobody monitors fails silently. The receiver may be available while the exporter uses the wrong endpoint. The collector may accept the request but hit its own queue limit. The process may shut down on time even though the SDK queue was already full before shutdown began.

For each service, collect a small set of signals: completed application requests, exported spans, export errors, queue drops, shutdown duration, and the number of active tasks when SIGTERM arrives. Not every SDK publishes the same metrics, so some of these can come from logs or an exporter wrapper. The specific dashboard is less important than being able to compare the flow of work with the flow of telemetry.

Do not alert only on the absolute number of exporter errors. A night service with three requests and one error needs attention, while a streaming consumer processing millions of spans may survive one temporary failure. Look at the share of unsuccessful processing, repeated timeouts, and queue growth. One of the most useful comparisons is business operations versus root spans with the same service name and operation type.

Shutdown logs should be structured. Useful fields include the shutdown reason, the number of active tasks when draining began, drain time, the SDK shutdown result, and the exporter error. Do not put request payloads or entire prompts in them. Telemetry often passes through more systems than application logs, so unnecessary data creates a separate access problem.

This is especially visible in LLM services: a short model request can create a long chain of retries, tool calls, and background checks. If the process ends immediately after returning a response to the client, you may lose the very spans that explain the expensive request. AI Router can route LLM calls through one OpenAI-compatible endpoint, but correctly completing telemetry remains the responsibility of the application and its runtime.

Checklist before enabling production

Treat this list as a check of the shutdown contract, not as a list of OpenTelemetry settings. If you cannot give a precise answer to any item, the next restart may leave gaps in the data.

• The process has one shutdown owner, and it does not force an exit before cleanup finishes.
• On SIGTERM, the service stops accepting new work before closing the telemetry provider.
• Active HTTP requests, messages, and background operations have their own drain deadline.
• The exporter timeout is shorter than the time remaining before the external kill after all application stages are complete.
• Logs and metrics show queue overflows, send errors, timeouts, and late spans after shutdown.

Also check the deployment configuration. A preStop hook does not replace a SIGTERM handler because its order and available time depend on the platform. A readiness probe does not help if your consumer continues to take messages. A longer grace period does not help if the code calls os.Exit or process.exit 20 milliseconds after receiving the signal.

The final check is simple and unpleasant: send SIGTERM to the process under load, repeat it dozens of times, and compare the number of started, completed, and delivered root operations. Do this in an environment with a real collector and the same network rules as production. If you cannot explain every missing span after this test, the configuration is not ready for the next deployment.