All documentation

Evaluate / guide 23

Resilience and chaos testing

Exercise runtime policy fallback, retries, and cost ceilings against realistic failure conditions before they happen in production.

Copy or share.

Resilience and chaos testing

Both SDKs ship a chaos harness that drives your configured runtime policy through a weighted mix of failure conditions — timeouts, rate limits, server errors, and cost overruns — against real HTTP calls. Use it to verify fallback chains, retry backoff, and cost ceilings actually behave the way your policy declares before an incident forces the question.

Run it from the dashboard

The Resilience tab under a project's Improve section (/dashboard/<projectId>/resilience) runs the same fallback-chain, retry, and cost-ceiling logic as a policy simulation against your project's actual runtime policy — no real provider calls, no API keys, no cost. Configure iteration count and failure mix, run the test, and see per-failure-mode success rate, average latency, and fail-open counts without leaving the browser. Use this for a quick policy sanity check; use the CLI harness below when you need to exercise real provider latency and network conditions.

What it tests

Each iteration picks a failure mode at random from a configurable mix and drives it through orchestrate():

  • success — a real call to the configured baseUrl, exercising the happy path and abort wiring.
  • timeout — sleeps past latencyBudgetMs to confirm the client aborts and fails over.
  • 429 — simulates a rate limit response to confirm retry/backoff engages.
  • 500 — simulates a server error to confirm fallback-chain traversal.
  • cost_overrun — returns a response that trips maxCostPerRun/maxCostPerDay to confirm the cost ceiling actually blocks.

The harness reports success rate, average latency, and fail-open count per failure mode, so you can confirm enforce mode fails closed (or intentionally fails open) exactly where you expect.

Run it (TypeScript)

bash
npx tsx packages/ts-sdk/harness/chaos.ts --iterations 20 --run-id chaos-smoke

Flags: --iterations, --latency-budget, --max-cost-run, --max-cost-day, --base-url, --api-key, --run-id. Defaults target OpenAI with a gpt-4o → claude-sonnet-4 → gpt-4o-mini fallback chain and a 40/15/15/15/15 success/timeout/429/500/cost-overrun mix.

Run it (Python)

bash
python packages/python-sdk/harness/chaos.py --iterations 20 --run-id chaos-smoke

The Python harness mirrors the TypeScript one: same failure modes, same weighted selection, same runtime-policy shape.

When to run this

  • Before raising maxCostPerRun/maxCostPerDay in production, to confirm the ceiling still blocks under load.
  • After changing a fallback chain, to confirm the new chain is actually reachable under 429/500 conditions.
  • As part of a pre-release check alongside test:regions and test:sdk:ts/test:sdk:python, when the deployment touches orchestration or retry logic.

This is a local/CI harness, not a hosted feature — runs are not persisted to the dashboard. Point --base-url at a non-production endpoint; iterations make real HTTP calls and can incur provider cost under the success failure mode.

PreviousEvaluation release gates