Your AI Vendor Is a Single Point of Failure. Plan for It.

This month, OpenAI's first developer conference handed teams a long list of reasons to build more on its platform. Within days, the same platform reminded everyone what it means to depend on a single provider.

At DevDay on November 6, OpenAI announced GPT-4 Turbo with a 128K context window, priced 3x cheaper for input tokens and 2x cheaper for output tokens than GPT-4, alongside a new Assistants API in beta and custom versions of ChatGPT called GPTs. Two days later, ChatGPT and the API went down in a major outage, and OpenAI confirmed it was dealing with periodic outages caused by an abnormal traffic pattern reflective of a DDoS attack. Then on November 17, the board announced that Sam Altman would depart as CEO, with CTO Mira Murati named interim CEO effective immediately.

None of this is a reason to stop using hosted models. It's a reason to architect for them the way you would any critical external dependency: assume it will be slow, down, changed or repriced at an inconvenient moment.

Three kinds of vendor risk

It helps to separate the failure modes, because each one needs a different control.

• Availability risk. The API is down, rate-limited or degraded. Your feature either degrades gracefully or takes your product down with it.
• Change risk. Models are versioned, updated and retired, and new versions behave differently on the same prompts. Pricing moves too. A change you didn't schedule can still break your output contract.
• Business and governance risk. Leadership, strategy, terms of service and regional availability can shift quickly. You can't control any of it, but you can control how hard it is to move.

Many teams that wired a model directly into their application this year have planned for none of these. The model call is an HTTP request scattered across the codebase, with the vendor's SDK, API key and prompt format hard-coded at every call site.

Put a gateway between your code and the model

The single most useful structural change is an internal model gateway: one service or library that every feature goes through to reach a model. It doesn't need to be elaborate. It needs to own the things that should never be duplicated:

• One interface, many providers. Application code asks for a capability such as summarize, classify or chat, and the gateway maps it to a provider and model version.
• Timeouts, retries and circuit breakers. Fail fast when the provider is struggling instead of piling up requests.
• Fallback routing. A secondary provider, a smaller self-hosted model, or a cached or non-AI response when the primary is unavailable.
• Pinned model versions. Upgrades happen deliberately, behind a flag, after evals have run.
• Central credentials and quotas. Keys live in your secrets manager, scoped per environment, with per-feature budgets and rate limits.
• Observability. Latency, error rates, token usage and cost per feature, with alerts, so you hear about degradation before your customers tell you.

A gateway also makes fallbacks testable. If switching providers is a configuration change, you can exercise it in staging on a schedule, rather than discovering mid-outage that the secondary path never worked.

Design for degraded mode

Fallbacks aren't only about swapping models. Ask what each AI feature should do when no model is available at all. A support assistant can hand off to a human queue. A summarizer can show the original text. A classification step can route to a default bucket for later review. Write these behaviours down, build them, and test them by switching the model off in a non-production environment.

Keep your prompts, eval sets and output contracts in your own repository, in as provider-neutral a form as you can. They're the real asset. If they live only inside one vendor's hosted features, moving becomes a rebuild rather than a reconfiguration. Weigh that carefully before building on any vendor-specific layer, including the newly announced hosted assistants and GPTs.

Governance is catching up

Canada's Voluntary Code of Conduct on the Responsible Development and Management of Advanced Generative AI Systems, published in September, distinguishes between developers who build these systems and managers who operate them, and sets out outcomes including accountability, safety, human oversight and monitoring, and validity and robustness. If you run a generative AI feature in production, you're closer to a manager than you might think. Knowing your dependencies, and how you'll respond when one fails, is part of operating responsibly.

What we recommend

However this week's leadership news plays out, do a short dependency review of every AI feature you run before the next outage:

• List each feature, the provider and model version it uses, and who owns it.
• Decide its degraded-mode behaviour and confirm that it's implemented.
• Route all model calls through one gateway with timeouts, fallbacks and monitoring.
• Pin model versions and gate upgrades on eval results.
• Write a one-page exit plan: what it would take to move each feature to another provider, and how long it would take.

A hosted model is a dependency like a payment processor or a cloud region. Treat it with the same discipline, and a bad week for your vendor stays a bad week for your vendor.

If you want help designing a model gateway or a fallback strategy that holds up under real outages, talk to us.