Knowledge Bases and Evaluation Organizations: LMArena (formerly LMSYS / Arena)
Knowledge Bases and Evaluation Organizations · LMArena (formerly LMSYS / Arena). Model arena and open evaluation. This source has 10 entries admitted under this category's review policy; each links directly to the original.
Authority tiers in this category: A2–B. Admission policy: All core resources are included. Each secondary source is admitted according to its own database, methodology, versions and leaderboard structure; there is no cross-organization scoring scheme.
Arena：How It Works
Explains anonymous model battles, user voting, data sharing and how the leaderboard is formed.
Evidence use: Supports describing Arena's data sources, voting process and role as a public evaluation.
Limitations: User preference is not the same as factual correctness. The sample distribution changes with the user base.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Arena Leaderboard Policy
Sets out how model listing, pre-release models, transparency, data releases and methodology changes are handled.
Evidence use: Supports explaining Arena's inclusion criteria, timeliness and auditability.
Limitations: The policy is revised over time. Public data is only part of the platform's data.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Agent Arena：Causal Evaluation of Agents in the Real World
Uses real agent interaction traces, randomized component selection and causal methods to assess signals such as task success, steerability and recovery from tool errors.
Evidence use: Supports Agent Arena's metrics, causal estimates and what the leaderboard means.
Limitations: The distribution of real user tasks cannot be fully controlled. An aggregate signal is not the same as an individual capability.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Arena Ranking Method
Explains pairwise voting, Bradley-Terry ranking, uncertainty intervals and leaderboard grouping.
Evidence use: Supports explaining Arena scores, ranking differences and statistical significance.
Limitations: User preference is affected by the population, prompts and presentation. Dynamic leaderboards and methodology changes should be dated.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Arena-Rank
Builds a reproducible, open model ranking method from Arena battle data.
Evidence use: Supports studying the agreement between automatic evaluation and real user preferences.
Limitations: User preference is affected by the population, prompts and presentation. Dynamic leaderboards and methodology changes should be dated.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Search Arena
Evaluates web search and answer generation systems through anonymous pairwise comparison.
Evidence use: Supports comparing end-to-end user preference for search-augmented systems.
Limitations: User preference is affected by the population, prompts and presentation. Dynamic leaderboards and methodology changes should be dated.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Code Arena
Compares code models and agent systems using real coding requests and user votes.
Evidence use: Supports comparing preferences for code generation on user tasks.
Limitations: User preference is affected by the population, prompts and presentation. Dynamic leaderboards and methodology changes should be dated.
Related concepts: AI Agent · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Style Control for Arena Leaderboards
Analyzes and controls the influence of style factors such as answer length and formatting on user votes.
Evidence use: Supports separating the contributions of content quality and presentation style to rankings.
Limitations: User preference is affected by the population, prompts and presentation. Dynamic leaderboards and methodology changes should be dated.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Factuality in Arena
Studies the relationship between Arena preference rankings and the factual accuracy of answers.
Evidence use: Supports judging how far preference leaderboards reflect factuality.
Limitations: User preference is affected by the population, prompts and presentation. Dynamic leaderboards and methodology changes should be dated.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
Image Arena Improvements
Explains data quality, matching and ranking improvements in the image arena.
Evidence use: Supports explaining human preference leaderboards for image generation models.
Limitations: User preference is affected by the population, prompts and presentation. Dynamic leaderboards and methodology changes should be dated.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24