Evaluation sourcesOfficial evaluation
DeepSWE v1.1
DataCurve / 编程 · 在统一的编程助手环境里改仓库、修缺陷，比较的是写代码的模型本身。
On HOTAIRanked
Evidence budget3.6%
Upstream data09/22 14:27
Last successful sync10/01 20:05
What and how it measures
只取 pass@1；统一 harness 内按预先固定的推理档位优先级选代表配置。
How this evidence is used
编程与设计预算中的 3.6%；固定受控系统与版本。
Evaluation results
按固定规则，每个公开模型采用一套代表配置。匿名测试型号不展示，保留原榜名次，每项最多 30 个。
| Source rank | Source model | Raw score | Representative configuration |
|---|---|---|---|
| 2 | gemini-3-8-flashGoogle | 73.8% | High 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 3 | claude-opus-5Anthropic | 73.6% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 4 | gpt-6-astraOpenAI | 73.2% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 9 | gpt-5-6-solOpenAI | 72.7% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 13 | claude-fable-5Anthropic | 69.7% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 14 | gpt-5-6-terraOpenAI | 69.6% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 16 | glm-5-3Z.ai | 69.0% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 19 | kimi-k3Moonshot AI | 68.5% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 21 | gpt-5-6-lunaOpenAI | 67.2% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 22 | gpt-5-5OpenAI | 67.0% | xHigh 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 24 | grok-4-6xAI | 66.7% | xHigh 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 27 | gemini-3-7-flashGoogle | 65.3% | High 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 30 | glm-5-3-flashZ.ai | 63.4% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 31 | deepseek-v4-proDeepSeek | 62.8% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 35 | claude-opus-4-8Anthropic | 59.0% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 37 | qwen3-8-maxAlibaba | 57.5% | xHigh 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 39 | muse-spark-1-2Meta | 54.9% | xHigh 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 42 | claude-sonnet-5Anthropic | 53.8% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 43 | grok-4-5xAI | 53.8% | High 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 46 | deepseek-v4-flashDeepSeek | 53.3% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 46 | muse-spark-1-1Meta | 53.3% | xHigh 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 48 | gpt-5-4OpenAI | 51.8% | xHigh 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 53 | gemini-3-6-flashGoogle | 46.7% | High 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 56 | glm-5-2Z.ai | 43.8% | Max 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 61 | gemini-3-5-flashGoogle | 36.1% | High 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 63 | kimi-k2-7-codeMoonshot AI | 30.5% | 完整系统 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 65 | claude-sonnet-4-6Anthropic | 29.9% | High 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
| 68 | gemini-3-1-pro-previewGoogle | 11.7% | High 推理 · mini-swe-agent · deepswe-v1.1:mini-swe-agent |
Evaluation limits and attribution
仍是模型＋mini-swe-agent 的系统表现；大版本变化必须隔离快照。
Data license: 官方公开结构化结果；仅管理员私有观察，公开前复核许可
成绩由 DataCurve 发布，原始分数与 HOTAI 共识分使用不同尺度，不能直接相加。