Rate Limits & Fallback Mechanics

Rate Limits: Per-Key Ceilings

Unlike standard API providers that impose inflexible account-wide tiers, InfinityRouter allows you to configure granular ceilings on individual API keys:

• Requests Per Minute (RPM): Caps the total request throughput for a specific deployment.
• Tokens Per Minute (TPM): Limits overall token velocity to prevent budget spikes.
• Concurrent In-Flight Requests: Prevents parallel worker storms from exhausting account balances.

You can configure these limits in the API Keys dashboard section.

Handling 429 Rate Limit Responses

When a request exceeds configured ceilings, InfinityRouter returns an HTTP 429 Too Many Requests status code with a Retry-After header indicating how many seconds to wait:

{
  "error": {
    "message": "Requests per minute limit exceeded for this API key.",
    "type": "rate_limited",
    "request_id": "req_091bc8274a"
  }
}

Recommended Exponential Backoff Implementation (Python)

Implement jittered exponential backoff to handle transient rate limits cleanly:

import os
import random
import time
from openai import OpenAI, RateLimitError, APIConnectionError

client = OpenAI(
    base_url=os.environ.get("INFINITY_BASE_URL", "https://infinityrouter.qd.je/v1"),
    api_key=os.environ.get("INFINITY_API_KEY"),
    max_retries=0, # Manage retries with explicit jitter
)

def completion_with_retry(messages, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(
                model="claude-sonnet-5",
                messages=messages,
            )
        except RateLimitError as e:
            if attempt == max_attempts - 1:
                raise
            # Exponential backoff with random jitter
            delay = (2 ** attempt) + random.uniform(0.1, 1.0)
            print(f"Rate limited. Retrying in {delay:.2f}s...")
            time.sleep(delay)
        except APIConnectionError as e:
            if attempt == max_attempts - 1:
                raise
            time.sleep(1.0)

Automated Multi-Upstream Failover

For every canonical model (such as claude-sonnet-5 or gpt-4o), InfinityRouter maintains multiple healthy downstream upstream routes.

If an upstream provider experiences elevated latency, HTTP 502/503 errors, or capacity shortages, InfinityRouter automatically retries the completion against an alternative healthy upstream supplier within milliseconds, preventing customer-facing downtime.