Build Multi-LLM Apps Easier with Google Cloud's AI Model Routing

Google Cloud has announced a new model routing feature for API Gateway, now available in Public Preview. The update gives developers a single API endpoint that can route requests to different AI models without changing application code every time they switch providers or model versions. As AI applications continue to rely on more than one large language model, managing separate APIs, payload formats, authentication methods and routing logic has become an everyday challenge. Google’s latest update aims to reduce that extra work by moving model selection into the API Gateway configuration instead of the application itself.

One endpoint for multiple models

With the new feature, developers can define routing rules inside an OpenAPI specification. Instead of hardcoding individual endpoints for Gemini, Claude, or OpenAI-compatible models, applications send requests to one API Gateway endpoint. The gateway then decides which backend model should handle the request based on the configured routing rules.

For example, a team can set Gemini as the default model while automatically forwarding specific requests to Claude when the request includes a matching model identifier. This keeps the client application unchanged while the routing logic stays inside the gateway configuration. Google says the gateway also converts OpenAI-style requests into the native request format expected by supported Vertex AI models.

Why developers may care

Many production AI applications no longer depend on a single model. One model might perform better for coding tasks, another for long-form writing, while a third offers lower costs for high-volume requests. Without a routing layer, developers usually have to maintain separate SDKs, authentication flows, and request formats for each provider. That complexity grows as new models are added to an application. Google’s approach keeps the application talking to a single endpoint while routing decisions happen inside API Gateway. This also makes it easier to change routing policies without modifying application code.

OpenAI-compatible requests

Another notable part of the announcement is OpenAI compatibility. Applications already built around the familiar Chat Completions format do not necessarily need to rewrite every request when moving workloads onto supported Google-hosted models. API Gateway accepts OpenAI-compatible requests and translates them into the backend format required by Vertex AI services before forwarding the request. This reduces migration work for existing applications that already depend on OpenAI-style APIs.

Current limitations

The feature is still in Public Preview so there are a few restrictions. According to Google, every backend used by a single router must share the same host. In practice, routing happens between supported models hosted through Vertex AI rather than directly switching between unrelated external endpoints. Developers planning multi-provider deployments should review the current documentation before moving production workloads to ensure a smooth transition.

What this means

This update reflects a broader change in how AI infrastructure is evolving. A year ago, most applications focused on choosing one language model. Today, many teams combine several models, depending on factors such as quality, latency, pricing, or specific capabilities. Instead of embedding those decisions throughout an application’s codebase, Google is moving that responsibility into the gateway layer. If the feature continues to expand after Public Preview, it could make multi-model applications easier to manage, especially for teams already building on Vertex AI and Google Cloud services. Developers interested in testing the feature can find configuration examples and deployment instructions in the official Google Cloud API Gateway documentation and the related product announcement.