본문으로 건너뛰기
TakoAPI
스킬로 돌아가기

api-benchmark

polarjunction 제작

Coding Agents & IDEs
—아직 평가 없음
내 평가

요약

Benchmark token generation speed across multiple LLM API providers.

설명

이 스킬이 하는 일

Benchmarks token generation speed across multiple LLM API providers by measuring TTFT, tokens-per-second throughput, and total generation time. Configuration is read from ~/.openclaw/openclaw.json, with support for anthropic-messages, openai-completions, and openai-responses API formats.

Measures TTFT separately from total generation time, giving a more accurate picture of perceived latency than simple end-to-end timing scripts.

사용 시점

  • Comparing TTFT between OpenAI and Anthropic for a latency-sensitive app
  • Verifying a new API key and endpoint are reachable before deploying
  • Measuring throughput differences between two versions of the same model
  • Checking whether a self-hosted LLM endpoint matches a hosted provider's speed
  • Running repeated benchmark runs to catch API performance regressions

예시 워크플로

입력

User asks: Compare token generation speed across all configured providers

에이전트
  1. 1

    Reads provider definitions from ~/.openclaw/openclaw.json

  2. 2

    Runs preflight connectivity check on all targets

  3. 3

    Sends short, medium, and long prompts to each provider

  4. 4

    Records TTFT, TPS, and total time per run

  5. 5

    Outputs results as a formatted comparison table

출력

Table showing TTFT, tokens/sec, and total time for each configured provider and model

요구 사항

  • API keys for each LLM provider to benchmark (e.g., ANTHROPIC_API_KEY, OPENAI_API_KEY)

설치

clawhub install polarjunction/api-benchmark

CLI 명령을 실행하거나, 프롬프트를 Claude Code, Cursor, Codex 등 사용하는 코딩 에이전트에 붙여넣으세요.

링크