← Back to the notebook

ENGINEERING NOTES

Choose the task before you choose the model

A summary, a structured classification and a difficult technical analysis have different needs. Putting the same model name into every caller makes those differences harder to manage.

Keep the caller’s intent visible

In my LLM Gateway project, a caller selects a task. A routing policy maps that task to an ordered set of models. The catalogue can refresh as model availability changes, while the caller continues to say what kind of work it needs.

The policy can put cost and speed first for some jobs and reasoning quality first for others. This is a design choice, not a universal ranking of model quality. The right policy still needs evaluation against the actual task.

Discovery should not erase the last working state

A failed catalogue refresh should not replace a useful configuration with an empty one. The project retains the previous catalogue if discovery fails or returns no text models. Clients also keep a last-successful configuration cache.

The broader idea is to separate a changing dependency from a stable expression of intent. That leaves room to change providers without scattering routing decisions through every application.

Implementation context: LLM Gateway source and design notes.

Prepared for this portfolio with AI assistance, based on the public project descriptions.