Kimi K3 Inference Provider Benchmark

Komodor builds the modern agent runtime, Evaluation Factory and AI SRE for production AI agents. We use the Evaluation Factory to test and improve our own agents and our customers' agents. This report uses it to answer two questions: can an open-weight model replace a closed one for an AI SRE agent, and does it matter which provider serves it?

Starting point
An AI SRE agent, Claude Code, running on a closed model, Claude Sonnet 5. That is our baseline: 45.1% of incidents fixed end to end on SREGym-Lite, a public benchmark of 17 broken Kubernetes clusters, the same as its published leaderboard score.
The switch
We swapped the model for Kimi K3, an open-weight model, and kept everything else. Open weights can be served by many providers at about the same list price, so we ran Kimi K3 through six of them: AWS Bedrock, Baseten, Fireworks, Modal, Nebius and Together AI.
Why
An open model is only worth switching to if it is good enough and the provider behind it is cheap, fast and reliable. A price sheet answers none of that, so we measured success rate, cost and speed on real incidents, for each provider, against the baseline.
How
Each provider got 51 runs: 17 incidents, 3 runs each. Every run starts a fresh Kubernetes cluster, injects a fault into the app running on it, and gives the agent 30 minutes to find the root cause and fix it. SREGym grades each run. Agent version, prompt, tools and grading are identical for every Kimi K3 provider; only the provider changes. Full method

Leaderboard

Sorted by E2E success. Click a column header to re-sort. Green is better and red is worse within each column; bold marks the best Kimi K3 provider.

E2E success
end-to-end: share of runs where the agent found the root cause and fixed the cluster
Cost / run
agent and judge tokens for one run, at list price
Session time
time the agent worked on one run (excludes cluster setup and grading)
LLM call, TTFT
time for one model call to finish, and to its first token (TTFT)
Quantization
serving precision the provider reports for Kimi K3 (hover for the source)
Tokens / run
prompt tokens (cached or not) plus output tokens billed
Baseline
Claude Sonnet 5, the closed model we start from; same harness and incidents, stock Claude Code settings
p50, p95
median, 95th percentile

Cost vs. session time

One point per provider; the green corner is cheaper and faster. Labels show E2E success. Grey is the Claude Sonnet 5 baseline.

E2E success
% of 51 runs, with 95% confidence interval · higher is better
Cost per run
USD · lower is better
Session time
Median minutes per run · lower is better

Findings

Detailed results

All tables are sortable.

Method

Full details and the evidence for each choice: METHOD.md.

Run on
The Komodor Evaluation Factory, part of KAOP (Komodor Agentic Operations Platform): the system we use to evaluate and improve our own AI SRE agents and our customers' agents.
Benchmark
SREGym 1.1, SREGym-Lite cohort: 17 incidents, each a production-style fault injected into a live microservice app on a fresh Kubernetes cluster. 3 runs per incident, 51 runs per provider, as on the public leaderboard.
Agent
SREGym's stock Claude Code agent, CLI pinned to 2.1.302, with a 30-minute limit per run. Same prompt and tools for every Kimi K3 provider; the Sonnet 5 baseline kept stock Claude Code settings.
Isolation
One Kubernetes pod per run, on its own on-demand node, with a fresh kind (Kubernetes-in-Docker) cluster inside.
Routing
A LiteLLM proxy in each pod sends the agent's calls to the provider's native Chat Completions API and logs every call: latency, time to first token, tokens, cache hits.
Held constant
Sampling and reasoning settings left at model defaults (providers translate them differently, so none are sent); the model's earlier reasoning passed back every turn; Claude Code's per-turn token counter turned off so prompt caching works; the benchmark's own 9-tool list.
Grading
SREGym's own. The LLM judge grades the diagnosis; by default it is the agent's model, so each provider's Kimi K3 runs are judged by Kimi K3 on that same provider. The oracle checks that the live cluster is fixed and involves no model. E2E success means both passed. Infrastructure failures are re-run (up to 3 times); agent timeouts count as failures.
Cost
Tokens each provider reported for every call (agent and judge) × its published list price. Checked against the bills of Nebius, Baseten, Together AI and AWS (for an earlier batch of Bedrock runs): our totals came out 4–10% below what they charged. Fireworks was not checked.
Baseline
Claude Sonnet 5 on AWS Bedrock, on the same harness and incidents with stock Claude Code settings (token counter on, all 21 tools, judged by Sonnet 5), scored 45.1% E2E success, matching its published SREGym leaderboard result.