> ## Documentation Index
> Fetch the complete documentation index at: https://lava.so/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Provider Routing

> Abstract provider logic, implement fallback strategies, and route requests across AI providers

Build flexible AI applications that work with multiple providers by abstracting provider logic, implementing fallback strategies, and routing requests based on cost, speed, or feature requirements.

<Info>
  **Multi-provider architecture provides resilience.** By supporting multiple AI providers, your application can gracefully handle rate limits, outages, and pricing changes without code changes.
</Info>

<Tip>
  **For automatic failover, use built-in fallbacks.** The [rewrite endpoint](/gateway/rewrite-proxy#model-fallbacks) supports automatic provider failover via the `x-lava-fallbacks` header — no try/catch required. The patterns on this page are for cases where you need custom routing logic beyond simple failover.
</Tip>

## Provider Configuration Pattern

Create a configuration-driven approach instead of hardcoding providers:

```typescript theme={null}
// config/providers.ts
export interface ProviderConfig {
  name: string;
  baseUrl: string;
  models: string[];
  features: string[];
  priority: number;
}

export const PROVIDERS: Record<string, ProviderConfig> = {
  openai: {
    name: 'OpenAI',
    baseUrl: 'https://api.openai.com/v1/chat/completions',
    models: ['gpt-4o', 'gpt-4o-mini', 'o3-mini'],
    features: ['streaming', 'function-calling', 'vision'],
    priority: 1
  },
  anthropic: {
    name: 'Anthropic',
    baseUrl: 'https://api.anthropic.com/v1/messages',
    models: ['claude-opus-4-5-20250514', 'claude-sonnet-4-5-20250514', 'claude-haiku-3-5-20241022'],
    features: ['streaming', 'function-calling', 'vision'],
    priority: 2
  },
  google: {
    name: 'Google',
    baseUrl: 'https://generativelanguage.googleapis.com/v1beta/models',
    models: ['gemini-2.5-flash', 'gemini-2.0-flash'],
    features: ['streaming', 'vision', 'multimodal'],
    priority: 3
  },
  groq: {
    name: 'Groq',
    baseUrl: 'https://api.groq.com/openai/v1/chat/completions',
    models: ['llama-3.3-70b-versatile', 'llama-3.1-8b-instant'],
    features: ['streaming', 'low-latency'],
    priority: 4
  }
};
```

For a complete provider abstraction class that wraps this configuration with unified request/response handling, see the full example in the [Node.js SDK](/sdk/nodejs) reference.

## Provider Switching with Configuration

### Environment-Based Selection

Switch providers via environment variables:

```typescript theme={null}
// lib/get-provider.ts
import { PROVIDERS } from '@/config/providers';

export function getProviderConfig(providerName?: string) {
  const name = providerName || process.env.AI_PROVIDER || 'openai';
  const config = PROVIDERS[name];

  if (!config) {
    throw new Error(`Unknown provider: ${name}`);
  }

  return config;
}
```

**Environment configuration:**

```bash theme={null}
# .env.production
AI_PROVIDER=openai

# .env.development
AI_PROVIDER=groq  # Use faster Groq for development
```

### Dynamic Provider Selection

Choose provider based on request characteristics:

```typescript theme={null}
export function selectProvider(request: {
  requiresVision?: boolean;
  requiresFunctionCalling?: boolean;
  prioritizeSpeed?: boolean;
  prioritizeCost?: boolean;
}): string {
  // Vision required
  if (request.requiresVision) {
    return 'google';  // Gemini Pro Vision
  }

  // Speed priority (low latency)
  if (request.prioritizeSpeed) {
    return 'groq';  // Ultra-fast inference
  }

  // Cost priority
  if (request.prioritizeCost) {
    return 'groq';  // Most cost-effective
  }

  // Default: OpenAI for quality
  return 'openai';
}

// Usage
const providerName = selectProvider({ prioritizeSpeed: true });
const config = PROVIDERS[providerName];
```

### Model-Based Routing

Route to providers based on desired model:

```typescript theme={null}
export function getProviderForModel(modelName: string): string {
  for (const [providerKey, config] of Object.entries(PROVIDERS)) {
    if (config.models.includes(modelName)) {
      return providerKey;
    }
  }
  throw new Error(`No provider found for model: ${modelName}`);
}

// Usage
const providerName = getProviderForModel('claude-opus-4-5-20250514');
const config = PROVIDERS[providerName];
```

## Fallback Strategies

### Sequential Fallback

Try providers in priority order until one succeeds:

```typescript theme={null}
export async function completionWithFallback(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  model?: string
): Promise<any> {
  // Sort providers by priority
  const sortedProviders = Object.entries(PROVIDERS)
    .sort(([, a], [, b]) => a.priority - b.priority)
    .map(([key]) => key);

  const errors: Array<{ provider: string; error: string }> = [];

  for (const providerName of sortedProviders) {
    try {
      console.log(`Attempting provider: ${providerName}`);
      const config = PROVIDERS[providerName];

      const response = await fetch(
        `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${forwardToken}`,
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({
            model: model || config.models[0],
            messages,
            temperature: 0.7
          })
        }
      );

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      console.log(`Success with provider: ${providerName}`);
      return await response.json();
    } catch (error) {
      const errorMessage = error instanceof Error ? error.message : 'Unknown error';
      errors.push({ provider: providerName, error: errorMessage });
      console.warn(`Provider ${providerName} failed:`, errorMessage);
      continue;
    }
  }

  // All providers failed
  throw new Error(
    `All providers failed:\n${errors.map(e => `- ${e.provider}: ${e.error}`).join('\n')}`
  );
}

// Usage
try {
  const response = await completionWithFallback(
    forwardToken,
    [{ role: 'user', content: 'Hello' }]
  );
  console.log('Response:', response);
} catch (error) {
  console.error('All providers exhausted:', error);
}
```

### Smart Retry with Exponential Backoff

Add exponential backoff for transient errors before falling back to the next provider:

```typescript theme={null}
async function completionWithRetry(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  maxRetries = 3
): Promise<any> {
  const sortedProviders = Object.keys(PROVIDERS)
    .sort((a, b) => PROVIDERS[a].priority - PROVIDERS[b].priority);

  for (const providerName of sortedProviders) {
    let retries = 0;

    while (retries < maxRetries) {
      try {
        const config = PROVIDERS[providerName];
        const response = await fetch(
          `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
          {
            method: 'POST',
            headers: {
              'Authorization': `Bearer ${forwardToken}`,
              'Content-Type': 'application/json'
            },
            body: JSON.stringify({
              model: config.models[0],
              messages
            })
          }
        );

        if (!response.ok) {
          throw new Error(`HTTP ${response.status}`);
        }

        return await response.json();
      } catch (error) {
        const errorMessage = error instanceof Error ? error.message : '';

        // Check if error is retryable (rate limit, timeout)
        const isRetryable = errorMessage.includes('429') ||
                           errorMessage.includes('timeout') ||
                           errorMessage.includes('503');

        if (!isRetryable || retries >= maxRetries - 1) {
          // Non-retryable error or max retries reached, try next provider
          break;
        }

        // Exponential backoff: 1s, 2s, 4s
        const delay = Math.pow(2, retries) * 1000;
        console.log(`Retrying ${providerName} in ${delay}ms...`);
        await new Promise(resolve => setTimeout(resolve, delay));

        retries++;
      }
    }
  }

  throw new Error('All providers and retries exhausted');
}
```

### Cost-Aware Routing

Fallback to cheaper providers when budget is a concern:

```typescript theme={null}
interface ProviderWithCost extends ProviderConfig {
  estimatedCostPer1kTokens: number;
}

async function costAwareFallback(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  maxCostUsd: number
): Promise<any> {
  // Try cheapest providers first
  const providersByCost = Object.entries(PROVIDERS)
    .sort(([, a], [, b]) =>
      (a as ProviderWithCost).estimatedCostPer1kTokens -
      (b as ProviderWithCost).estimatedCostPer1kTokens
    );

  for (const [providerName, config] of providersByCost) {
    try {
      const response = await fetch(
        `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
        {
          method: 'POST',
          headers: {
            'Authorization': `Bearer ${forwardToken}`,
            'Content-Type': 'application/json'
          },
          body: JSON.stringify({
            model: config.models[0],
            messages
          })
        }
      );

      if (!response.ok) throw new Error(`HTTP ${response.status}`);
      return await response.json();
    } catch (error) {
      continue;
    }
  }

  throw new Error(`No provider within budget: $${maxCostUsd}`);
}
```

## Feature Detection

Check provider capabilities before using advanced features:

```typescript theme={null}
function hasFeature(providerName: string, feature: string): boolean {
  const config = PROVIDERS[providerName];
  return config?.features.includes(feature) || false;
}

// Usage
if (hasFeature('openai', 'function-calling')) {
  // Include function definitions in request
}

if (hasFeature('google', 'vision')) {
  // Include image URLs in request
}
```

### Graceful Feature Degradation

Fallback when features aren't supported:

```typescript theme={null}
async function completionWithFeatureFallback(
  forwardToken: string,
  messages: Array<{ role: string; content: string }>,
  options: { functions?: any[] } = {}
): Promise<any> {
  const preferredProvider = 'openai';

  try {
    if (options.functions && !hasFeature(preferredProvider, 'function-calling')) {
      console.warn('Function calling not supported, removing from request');
      delete options.functions;
    }

    const config = PROVIDERS[preferredProvider];
    const response = await fetch(
      `https://api.lava.so/v1/forward?u=${encodeURIComponent(config.baseUrl)}`,
      {
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${forwardToken}`,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({
          model: config.models[0],
          messages,
          ...(options.functions ? { tools: options.functions } : {})
        })
      }
    );

    if (!response.ok) throw new Error(`HTTP ${response.status}`);
    return await response.json();
  } catch (error) {
    // Fallback to another provider without advanced features
    const fallbackConfig = PROVIDERS['groq'];
    const response = await fetch(
      `https://api.lava.so/v1/forward?u=${encodeURIComponent(fallbackConfig.baseUrl)}`,
      {
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${forwardToken}`,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({
          model: fallbackConfig.models[0],
          messages
        })
      }
    );

    if (!response.ok) throw new Error(`HTTP ${response.status}`);
    return await response.json();
  }
}
```

## Troubleshooting

<AccordionGroup>
  <Accordion title="Fallback doesn't trigger when primary fails" icon="arrow-right-arrow-left">
    **Check:**

    * Exception is being caught properly in try/catch
    * Provider priority order is correct (lower number = higher priority)
    * Error is thrown (not just logged) when provider fails

    **Debug:**

    ```typescript theme={null}
    for (const providerName of sortedProviders) {
      try {
        console.log('Trying provider:', providerName);
        const response = await makeRequest(providerName);
        console.log('Success!');
        return response;
      } catch (error) {
        console.error('Provider failed:', providerName, error);
        // IMPORTANT: Must continue, not return/throw here
        continue;
      }
    }
    ```
  </Accordion>

  <Accordion title="Feature detection returns wrong capabilities" icon="list-check">
    **Issue:** Provider claims to support feature but request fails

    **Reasons:**

    * Provider configuration outdated (features changed)
    * Feature available but different API format required
    * Feature requires specific model (not all models support all features)

    **Solution:**
    Add model-specific feature checks:

    ```typescript theme={null}
    function hasFeature(providerName: string, feature: string, model?: string): boolean {
      const config = PROVIDERS[providerName];
      if (!config?.features.includes(feature)) {
        return false;
      }

      // Model-specific feature support
      if (feature === 'vision' && model) {
        return model.includes('vision') || model.includes('4');
      }

      return true;
    }
    ```
  </Accordion>

  <Accordion title="Cost estimates inaccurate across providers" icon="dollar-sign">
    **Problem:** Estimated costs don't match actual Lava costs

    **Explanation:**

    * Lava costs include provider base cost + merchant fee + service charge
    * Provider pricing varies by model and usage
    * Costs are only accurate AFTER request completes

    **Solution:**
    Track usage from response body and calculate cost based on your pricing:

    ```typescript theme={null}
    const usage = data.usage?.total_tokens || 0;
    const requestId = response.headers.get('x-lava-request-id');

    // Calculate cost based on your configured pricing
    console.log('Tokens used:', usage, 'Request ID:', requestId);
    ```
  </Accordion>

  <Accordion title="Response parsing fails for a new provider" icon="code">
    **Cause:** Provider uses different response format than expected

    **Solution:**
    Handle multiple response formats:

    ```typescript theme={null}
    function extractContent(data: any): string {
      // Try each known format
      if (data.choices?.[0]?.message?.content) return data.choices[0].message.content;
      if (data.content?.[0]?.text) return data.content[0].text;
      if (data.candidates?.[0]?.content?.parts?.[0]?.text) {
        return data.candidates[0].content.parts[0].text;
      }

      // Log unknown format for debugging
      console.error('Unknown response format:', JSON.stringify(data, null, 2));
      throw new Error('Unknown response format');
    }
    ```
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Streaming" icon="stream" href="/gateway/forward-proxy#streaming">
    Handle streaming responses across different providers
  </Card>

  <Card title="Forward Proxy & Authentication" icon="route" href="/gateway/forward-proxy">
    Understand URL encoding and provider routing
  </Card>

  <Card title="Supported Providers" icon="layer-group" href="/gateway/supported-providers">
    See all supported providers and their features
  </Card>

  <Card title="Webhooks" icon="webhook" href="/integration/webhooks">
    Receive real-time notifications for usage events
  </Card>
</CardGroup>