Knowledge Bases and Evaluation Organizations: METR
Knowledge Bases and Evaluation Organizations · METR. Pre-deployment evaluations and task time-horizon curves. This source has 8 entries admitted under this category's review policy; each links directly to the original.
Authority tiers in this category: A2–B. Admission policy: All core resources are included. Each secondary source is admitted according to its own database, methodology, versions and leaderboard structure; there is no cross-organization scoring scheme.
Measuring AI Ability to Complete Long Software Tasks
Uses the time human experts need to complete a task as a difficulty scale to study how reliably models complete long software tasks.
Evidence use: Supports the definition, methodology and historical trend of the task time-horizon metric.
Limitations: Tasks are mainly software, machine learning and cybersecurity. Trend extrapolation is not a capability promise.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Task-Completion Time Horizons of Frontier AI Models
Continuously publishes frontier model results on the task time-horizon curve, with explanations and FAQs.
Evidence use: Supports 50% time-horizon results for a specified date, model and evaluation suite.
Limitations: Time horizon refers to the estimated time for a human to complete a task, not model run time. Intervals become less stable as a suite nears saturation.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Frontier Risk Report (February–March 2026)
A comprehensive report sharing frontier models' task time horizons, monitoring evasion, sabotage strategies and qualitative evaluation results.
Evidence use: Supports the capabilities, risks and uncertainties observed in that round of pre-deployment evaluation.
Limitations: Covers specific models and settings provided by labs. Qualitative cases cannot summarize all behavior.
Related concepts: AI Agent · AI Red Teaming · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
METR General Capability Evaluations
Introduces METR's approach to evaluating autonomous completion of complex tasks, information gathering and tool use.
Evidence use: Supports understanding the design and limits of METR's general capability evaluations.
Limitations: Results depend on model access, scaffolding and the task set. Research findings cannot be extrapolated to all real-world work performance.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
RE-Bench: evaluating AI R&D capabilities
Has model agents and human experts complete real machine learning R&D tasks under controlled time budgets.
Evidence use: Supports comparing frontier agents' AI R&D capabilities with human baselines.
Limitations: Results depend on model access, scaffolding and the task set. Research findings cannot be extrapolated to all real-world work performance.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
MALT Evaluation Integrity Dataset
Provides human-reviewed agent traces of evaluation integrity behaviors such as reward hacking and sandbagging.
Evidence use: Supports training and testing evaluation monitors.
Limitations: Results depend on model access, scaffolding and the task set. Research findings cannot be extrapolated to all real-world work performance.
Related concepts: AI Agent · AI Red Teaming
Link last checked: 2026-09-24
AI Tools and Experienced Open-Source Developer Productivity
Uses a randomized controlled design to measure the productivity impact of AI tools on experienced developers working on real open-source tasks.
Evidence use: Supports causal statements about the productivity impact on developers for a specific population and period.
Limitations: Results depend on model access, scaffolding and the task set. Research findings cannot be extrapolated to all real-world work performance.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Algorithmic vs. Holistic Evaluation
Compares how automatically verifiable tasks and holistic human review measure agent capabilities differently.
Evidence use: Supports analyzing how evaluation methods affect conclusions about capability trends.
Limitations: Results depend on model access, scaffolding and the task set. Research findings cannot be extrapolated to all real-world work performance.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24