Knowledge Bases and Evaluation Organizations: SWE-bench
Knowledge Bases and Evaluation Organizations · SWE-bench. Official leaderboard for fixing real bugs at repository level. This source has 7 entries admitted under this category's review policy; each links directly to the original.
Authority tiers in this category: A2–B. Admission policy: All core resources are included. Each secondary source is admitted according to its own database, methodology, versions and leaderboard structure; there is no cross-organization scoring scheme.
SWE-bench Official Leaderboards
Shows leaderboards for fixing bugs in real code repositories, including Verified, Multilingual, Multimodal, Lite and Full.
Evidence use: Supports comparisons of bug-fix rates tied to the dataset, agent scaffold and date.
Limitations: Scores are determined jointly by the model and the scaffold. When top scores are close, ranking differences may not be significant.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
SWE-bench: a benchmark for resolving real GitHub issues
Defines the task of producing a test-passing patch given a codebase and an issue, with containerized evaluation and several datasets.
Evidence use: Supports describing SWE-bench tasks, data sources and reproduction mechanism.
Limitations: Mainly measures repository tasks that can be verified automatically. Does not cover the full range of software development work.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
SWE-bench Dataset Catalog
Lists the size and purpose of the Full, Lite, Verified, Multimodal and Multilingual datasets.
Evidence use: Supports distinguishing SWE-bench variants and their sample scopes.
Limitations: Scores from different variants cannot be compared with each other. The dataset may change with maintenance.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
SWE-bench Multimodal
Requires agents to fix real code repository issues using screenshots or interface information in the issue.
Evidence use: Supports evaluating multimodal software engineering agents.
Limitations: Scores are determined jointly by the model, the agent scaffold and the run budget. Results from different dataset variants cannot be compared with each other.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
SWE-bench Multilingual
Covers real GitHub fix tasks in nine languages other than Python.
Evidence use: Supports evaluating how coding agents generalize across programming languages.
Limitations: Scores are determined jointly by the model, the agent scaffold and the run budget. Results from different dataset variants cannot be compared with each other.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
SWE-bench Lite
Provides an easier-to-run filtered subset for quickly comparing ability to fix real repositories.
Evidence use: Supports low-cost reproduction and evaluation during development.
Limitations: Scores are determined jointly by the model, the agent scaffold and the run budget. Results from different dataset variants cannot be compared with each other.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
SWE-bench Evaluation Harness
Explains data download, prediction format, container builds and test execution.
Evidence use: Supports reproducing SWE-bench submissions and checking run conditions.
Limitations: Scores are determined jointly by the model, the agent scaffold and the run budget. Results from different dataset variants cannot be compared with each other.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24