api-benchmark
由 polarjunction 提供
Coding Agents & IDEs简介
Benchmark token generation speed across multiple LLM API providers.
描述
该技能的功能
Benchmarks token generation speed across multiple LLM API providers by measuring TTFT, tokens-per-second throughput, and total generation time. Configuration is read from ~/.openclaw/openclaw.json, with support for anthropic-messages, openai-completions, and openai-responses API formats.
Measures TTFT separately from total generation time, giving a more accurate picture of perceived latency than simple end-to-end timing scripts.
何时使用
- Comparing TTFT between OpenAI and Anthropic for a latency-sensitive app
- Verifying a new API key and endpoint are reachable before deploying
- Measuring throughput differences between two versions of the same model
- Checking whether a self-hosted LLM endpoint matches a hosted provider's speed
- Running repeated benchmark runs to catch API performance regressions
示例工作流
User asks: Compare token generation speed across all configured providers
- 1
Reads provider definitions from ~/.openclaw/openclaw.json
- 2
Runs preflight connectivity check on all targets
- 3
Sends short, medium, and long prompts to each provider
- 4
Records TTFT, TPS, and total time per run
- 5
Outputs results as a formatted comparison table
Table showing TTFT, tokens/sec, and total time for each configured provider and model
前置要求
- API keys for each LLM provider to benchmark (e.g., ANTHROPIC_API_KEY, OPENAI_API_KEY)
安装
clawhub install polarjunction/api-benchmark运行 CLI 命令,或把这段提示词粘贴到 Claude Code、Cursor、Codex 或你在用的任意编程智能体里。