Ops & reliability

Concurrency & rate limiting

Two provider wrappers, same shape as withRetry/withFallback/withTimeout — compose all of them freely. Both queue calls beyond the limit rather than rejecting.

concurrency.ts

import { withConcurrencyLimit, withRateLimit, withRetry, anthropic } from "samai-sdk";

// Caps in-flight calls — a queue, not a rejection.
const capped = withConcurrencyLimit(anthropic({ apiKey: "..." }), { maxConcurrent: 5 });

// Caps requests per time window — a token-bucket limiter, refills continuously.
const throttled = withRateLimit(anthropic({ apiKey: "..." }), { maxRequests: 60, intervalMs: 60_000 });

// Compose with retries — wrapping the limit AROUND retry means retries of the same call
// count against the limit too (usually what you want).
const provider = withConcurrencyLimit(
  withRetry(anthropic({ apiKey: "..." }), { maxRetries: 2 }),
  { maxConcurrent: 5 }
);

Use withConcurrencyLimit() to stay under a provider's hard concurrent-request cap when running many agents (or a generateObjectBatch()) at once. Use withRateLimit() to stay under a published requests-per-minute limit before it turns into 429s that withRetry then has to spend time recovering from.

Order matters

Wrapping withConcurrencyLimit() around withRetry() means each retry attempt of a call also counts against the concurrency cap — usually what you want, since it prevents a burst of retries from silently exceeding your provider's limit.