API Image Normalization Should Be a Separate Layer

Image formats cannot be passed to a model exactly as the client sent them. A URL, a base64 data URL, and a file with an identifier may all look like three ways to say «here is an image», but they create three different obligations for the gateway: download an object, parse binary data, or find a stored resource.

Image normalization should happen before provider selection and before serializing an OpenAI-compatible request. This lets the application preserve message-part order, apply security checks consistently, and work with one contract instead of a growing set of branches that diverge with every new endpoint.

The most unpleasant bug here rarely looks like an image-processing error. A user writes: «The package is in the first photo, and the defect is in the second. Compare them». The service groups images separately from text, sorts them again by filename, and sends the model a different sequence. The model responds convincingly, but analyzes the wrong objects. The logs still show two valid images and a successful response. That is why content order is part of the request data, not an interface detail.

One internal contract is better than three formats

The internal contract should describe an image as a separate message part with a position, a source, and already validated metadata. It should not copy the image_url shape from one API, because another endpoint may expect input_image, file_id, or a separate upload altogether.

A practical contract looks like this:

type NormalizedImage = {
  kind: "image";
  position: number;
  source: "remote_url" | "inline_bytes" | "managed_file";
  mimeType: "image/jpeg" | "image/png" | "image/webp" | "image/gif";
  bytes?: Uint8Array;
  remoteUrl?: string;
  fileId?: string;
  sha256: string;
  originalName?: string;
  detail?: "low" | "high" | "auto";
};

type NormalizedPart =
  | { kind: "text"; position: number; text: string }
  | NormalizedImage;

type NormalizedMessage = {
  role: "system" | "developer" | "user" | "assistant";
  parts: NormalizedPart[];
};

Here, position is not duplicating the array index for appearance's sake. It is needed when parts go through different asynchronous operations. A URL needs to be downloaded, base64 needs to be decoded, and file_id needs metadata to be read. The results will be ready in an arbitrary order. After that, you cannot simply append completed elements with push().

The contract also separates origin from representation. Two inputs may contain identical bytes even though one came from a URL and the other from base64. Their further processing should depend on routing and storage rules, not on how the frontend packaged the image.

Do not make «always a URL» your internal format. Inline data would have to be placed somewhere temporarily. A private file URL may be useless to the selected model. An external URL raises the question of who is allowed to download it. A universal string quickly becomes a universal problem.

Message-part order cannot be reconstructed by guesswork

The order should be established while parsing the original message and preserved through the outgoing payload. Do not sort images by filename, hash, upload time, or type. Do not move all text before all images just because that makes request construction easier.

Consider this Chat Completions user content:

[
  {"type":"text","text":"The price tag is in the first photo."},
  {"type":"image_url","image_url":{"url":"https://cdn.example.org/price.jpg"}},
  {"type":"text","text":"The shelf is in the second photo. Find the discrepancy."},
  {"type":"image_url","image_url":{"url":"data:image/png;base64,iVBORw0KGgo..."}}
]

The normalizer should immediately create four parts with positions 0, 1, 2, and 3. Downloading price.jpg and decoding the PNG can run in parallel, but the final result must be assembled by position.

const settled = await Promise.all(
  rawParts.map((part, position) => normalizePart(part, position))
);

const parts = settled
  .flat()
  .sort((a, b) => a.position - b.position);

This example is intentionally simple. In real code, do not use flat() without clear rules: each input part should produce exactly one normalized part or return an error for the entire message. If a function sometimes expands one element into several, it must assign stable positions such as 2.0 and 2.1 to child elements, or keep a separate array inside the original part. Otherwise, order will be «usually correct», which is a poor property for data.

An especially dangerous optimization collects textParts and imageParts in separate collections and then builds [...textParts, ...imageParts]. It looks harmless in tests with one image. In multistep visual instructions, it changes the meaning of the request.

Treat URLs as untrusted network input

An external URL is convenient for the client, but it turns your service into an HTTP client whose destination is chosen by the user. This is a classic path to SSRF: requests to local services, internal networks, and cloud metadata endpoints.

Checking url.startsWith("https://") does not solve the problem. The address may point to a public domain that redirects into an internal network. A DNS name may resolve to a private IP. The server may return a huge file or redirect indefinitely.

A loader's minimum policy should include the following:

• Accept only https:, and, if required for legacy infrastructure, a separately allowlisted http:.
• Reject credentials in URLs, nonstandard schemes, an empty host, and excessively long addresses.
• After DNS resolution, block loopback, private, link-local, multicast, and service address ranges.
• Check every redirect again and set a low redirect limit.
• Limit connection time, read time, body size, and allowed MIME types.

Do not check only the Content-Type header. A server may label an HTML document image/jpeg, while a proxy may return an error page with status 200. After downloading, read the binary file signature. JPEG starts with FF D8 FF, PNG has a fixed eight-byte signature, and WebP uses a RIFF container with the WEBP marker. A byte-based MIME detection library avoids a homemade table, but its result still has to be matched against the allowlist.

Downloading a URL does not require storing the image permanently. For a single request, validated bytes can live in a temporary object with a short lifetime. If the application wants to reuse the resource, create a managed file and record its owner, hash, size, and deletion time.

Base64 must be decoded before validation

Base64 is not an image format. It is a text encoding of bytes, and it cannot be treated as an image before decoding. The string may be plain base64, a data URL, or simply corrupted text that passes a superficial check.

A data URL parser should split the header and payload strictly at the first comma. For an image, expect the form data:<mime>;base64,<payload>. Do not silently accept data:text/html;base64,... just because the field is called image_url.

function parseDataUrl(value: string) {
  const comma = value.indexOf(",");
  if (comma < 0) throw new InputError("invalid_image_reference");

  const header = value.slice(0, comma).toLowerCase();
  const payload = value.slice(comma + 1);
  const match = /^data:(image\/(jpeg|png|webp|gif));base64$/.exec(header);

  if (!match || payload.length === 0) {
    throw new InputError("invalid_image_reference");
  }

  if (!/^[a-z0-9+/=\r\n]+$/i.test(payload)) {
    throw new InputError("invalid_image_reference");
  }

  return { declaredMimeType: match[1], payload };
}

Then decode the bytes with a size limit. The limit should not be checked only against string length. Base64 increases size by roughly one third, but spaces and line breaks change the text length, while the attack targets memory after decoding. A streaming decoder or a preliminary estimate of the expected size provides better protection than calling Buffer.from() on a string of unknown length.

Next, determine the actual MIME type from the signature. If the client declares PNG but the bytes look like JPEG, there are two reasonable options: reject the request as inconsistent or replace the declared type with the actual one and record an audit event. For systems handling documents and medical images, I prefer rejection. A mismatch is rarely accidental, and the cost of later diagnosis is higher.

Do not re-encode an image without a reason. Re-saving JPEG degrades it, may remove a useful color profile, and consumes CPU. Re-encoding makes sense when you intentionally remove metadata, convert an unsupported type to an allowed one, or limit pixel dimensions.

A file is a resource with a lifecycle, not just another URL

file_id gives the client a short request and saves it from sending the same bytes again. But an identifier does not mean the file can be attached to every model without conditions. An endpoint may accept file_id, support only URLs, or require a separate content form.

In the OpenAI Responses API documentation, an image can be passed through image_url or file_id in an input_image element. Files are described there as a separate input_file type. The older Chat Completions style uses a different content-part structure. The schemas are similar in meaning, but not in fields. That is why endpoint adapters should remain separate instead of spreading conditionals throughout the application.

File storage should answer at least four questions:

1. Who owns the object and has the right to provide its ID?
2. Which bytes and MIME type were confirmed during upload?
3. Until what time is the object available?
4. Which route can retrieve its original bytes or a temporary link?

Never accept file_id and insert it into a request without authorization. Otherwise, a user from one tenant may guess or obtain identifiers for objects belonging to another. UUIDs reduce the chance of guessing, but they do not replace an ownership check.

It is useful to separate two states. «Uploaded» means the bytes reached storage. «Ready for the model» means that you checked the MIME type, size, decodability, antivirus rules where applicable, and the owner's policy. The model should see only the second state.

OpenAI's public Uploads API documentation describes multipart upload as an intermediate object that becomes a File after completion. This is a useful model for your own state machine: do not let clients use a resource that is still being assembled in parts or has not passed validation.

The endpoint adapter should build the provider payload last

Do not expose the normalized object as-is. At the final boundary, the adapter chooses the capabilities of the specific route and builds the required form. This is where a model capability table is useful: does it accept external URLs, data URLs, or managed files, which detail level does it understand, and what maximum size does your route allow?

For a Responses-like endpoint, outgoing content might look like this:

{
  "role": "user",
  "content": [
    {"type":"input_text","text":"Compare the labeling on the two packages."},
    {
      "type":"input_image",
      "image_url":"https://media.example.net/tmp/2f7c.jpg",
      "detail":"high"
    },
    {
      "type":"input_image",
      "file_id":"file_01HXYZ...",
      "detail":"high"
    }
  ]
}

For a Chat Completions-like endpoint, the same semantics often require a different shape:

{
  "role": "user",
  "content": [
    {"type":"text","text":"Compare the labeling on the two packages."},
    {
      "type":"image_url",
      "image_url":{"url":"https://media.example.net/tmp/2f7c.jpg","detail":"high"}
    },
    {
      "type":"image_url",
      "image_url":{"url":"data:image/jpeg;base64,/9j/4AAQ...","detail":"high"}
    }
  ]
}

The second example does not mean that every compatible server accepts data URLs. It shows why calling one JSON document «the OpenAI format» does not settle the issue. Compatibility usually applies to a route and a set of fields, not to identical behavior across all providers.

If a model accepts only URLs, the adapter can create a signed temporary link to the validated bytes. If a route accepts only inline data, the adapter can encode the bytes as base64. If neither option meets the route's rules, return an error before calling the model. Hoping that «the provider will probably understand» creates expensive failures that are difficult to explain.

AI Router makes sense precisely at this boundary: the application keeps one OpenAI-compatible call, while route selection rules stay outside business code. This does not remove the need for normalization on your side, because only the application knows the original order, file owner, and meaning of the user's operation.

Image validation must account for bytes, pixels, and cost

A file can be small in storage but enormous after pixel decompression. An image with extreme dimensions can consume a lot of memory during decoding, even if the compressed PNG looks harmless. Before sending it to the model, read the width, height, frame count for animated formats, and orientation.

Validation should separate acceptability from usefulness. A JPEG may be technically valid but unsuitable for the task if the photo has too few pixels to read small text. The gateway does not need to guess the user's goal, but it can pass detail when the selected API supports it and record normalized dimensions in the audit log.

Calculate SHA-256 over the confirmed bytes. This hash helps to:

• eliminate repeated uploads of the same image;
• associate requests with an object without storing the original URL in analytics;
• reproduce an investigation using the same resource;
• detect that two different file_ids contain identical data.

Do not use the hash as the only access right. A hash is predictable for a known file and is not a secret by itself. It identifies content; it does not authorize access.

Handle EXIF separately. Camera metadata may contain coordinates, capture time, and device model. If the image comes from an app for property inspections, insurance claims, or medical work, passing EXIF onward is often unnecessary. Remove it in a controlled transformation step, but do not do so silently if the client relies on image orientation. Apply the orientation to the pixels first, then remove the tag.

Errors should tell the client what to fix

Provider errors rarely provide a good contract for your client. One server may say unsupported image, another may return HTML through a proxy, and a third may reject file_id without explaining why. The normalizer should catch problems earlier and return a limited set of predictable codes.

I would start with these codes:

{
  "error": {
    "code": "unsupported_media_type",
    "message": "JPEG, PNG, WebP, and GIF are supported.",
    "param": "messages[0].content[3]"
  }
}

invalid_image_reference means an invalid data URL, an empty file_id, or a disallowed URL. remote_fetch_denied means the link violated network policy. image_too_large means your byte or pixel limit was exceeded. file_not_found and file_access_denied should not be combined if the client is allowed to know that the object existed. For an external client, returning the same response is safer, while the exact reason stays in the audit log.

Store the request ID, tenant ID, element position, source type, declared and actual MIME types, size, hash, and selected route in the log. Do not log base64. Do not leave complete temporary URLs with tokens in the logs. Logs should help reconstruct the processing chain, not become a second unprotected image store.

Test transitions between formats, not just a successful JPEG

One test with a public JPEG checks almost nothing. You need a transition matrix: URL to data URL, file_id to a temporary URL, base64 to a managed file, and rejection at every stage.

The minimum test set should include a message with text between two images, a URL redirecting to a blocked network, a data URL with a false MIME type, a file belonging to another tenant, and an image whose pixel dimensions violate policy. For each case, check not only the response code but also that the provider is not contacted when the error is local.

Add a concurrency test. Make the first image download slowly while the second decodes instantly. The final array must remain in the original order. This catches the temptation to assemble results by operation completion time.

A good transformation layer does not make the model smarter. It ensures that the model sees the same objects, in the same order, with the same access boundaries that the user specified. For a multimodal application, this is part of response correctness, not auxiliary plumbing.