← Back to the notebook

ENGINEERING NOTES

Why a streaming retry needs a boundary

When an AI provider times out or runs out of capacity, trying another provider can keep an application useful. But a streaming response changes the decision: some of the answer may already be on the screen.

The first visible output changes the contract

Before the first output chunk, a failed attempt has not yet become part of the user’s answer. The router can move to the next model and return one coherent response.

After output starts, switching silently can splice together two independently generated answers. The second model does not automatically know what the first model promised, explained or assumed. A connection may recover while the answer becomes less trustworthy.

A small rule with a useful consequence

In my LLM Gateway project, streaming failover is allowed only before the first output chunk. After that point, a failure needs to stay visible to the calling application. The application can explain the interruption or offer an explicit retry.

This does not make the network more reliable. It makes the response boundary easier to reason about. Availability matters, but so does knowing which answer the user actually received.

What I would inspect when integrating a provider

  • Which errors justify retrying, and which indicate invalid credentials or a bad request?
  • Has any response content already reached the caller?
  • Can cancellation stop the current attempt cleanly?
  • Does the application surface an interrupted response honestly?

These are product decisions expressed through error handling. The useful result is not simply fewer error messages. It is behaviour a user and the next developer can understand.

Implementation context: LLM Gateway routing policy and public source.

Prepared for this portfolio with AI assistance, based on the public project descriptions.