Knowledge Bases and Evaluation Organizations: Stanford CRFM / HELM
Knowledge Bases and Evaluation Organizations · Stanford CRFM / HELM. Foundation model transparency and evaluation. This source has 12 entries admitted under this category's review policy; each links directly to the original.
Authority tiers in this category: A2–B. Admission policy: All core resources are included. Each secondary source is admitted according to its own database, methodology, versions and leaderboard structure; there is no cross-organization scoring scheme.
HELM：Holistic Evaluation of Language Models
Evaluates foundation model capabilities in language, multimodal, medical, image and other areas with uniform scenarios, metrics and reproducible configurations.
Evidence use: Supports the methods, scenarios, metrics and model results of each HELM sub-leaderboard.
Limitations: Sub-leaderboards use different definitions. Some evaluations use model judges.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
2025 Foundation Model Transparency Index
Uses a set of transparency indicators to assess what major foundation model developers disclose about upstream resources, the model and downstream use.
Evidence use: Supports comparing developers' transparency performance and the indicator framework.
Limitations: Scores measure disclosure, not model quality or safety. Comparisons are valid only within the same version of the index.
Related concepts: AI Governance and Regulation · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HELM Capabilities
Evaluates general capabilities of language models across multiple scenarios and metrics.
Evidence use: Supports side-by-side comparison of model capabilities under a uniform configuration.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HELM Audio
Comprehensively evaluates audio understanding and audio language models.
Evidence use: Supports scenario-based capability comparison of audio models.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HELM Lite
Covers a broad range of language model capability scenarios at lower cost.
Evidence use: Supports reproducible, lightweight, broad-coverage evaluation.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HELM Classic
Continues the original HELM paper's multi-scenario, multi-metric and perturbation evaluations.
Evidence use: Supports studying multidimensional performance including capability, robustness, fairness and efficiency.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HEIM：Holistic Evaluation of Text-to-Image Models
Evaluates text-to-image models on alignment, quality, originality, risk and other dimensions.
Evidence use: Supports multidimensional, reproducible comparison of text-to-image models.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HELM Instruct
Evaluates how well models follow natural language instructions using absolute scoring.
Evidence use: Supports comparing task completion and response quality of instruction-tuned models.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HELM MMLU
Reproduces MMLU evaluation with uniform prompts and run configuration.
Evidence use: Supports MMLU comparisons with prompt differences removed.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
VHELM：Vision-Language Model Evaluation
Evaluates vision-language models across visual question answering, recognition, reasoning and risk scenarios.
Evidence use: Supports uniform side-by-side comparison of multimodal models.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
HELM Image2Struct
Evaluates how well vision-language models extract structured information from charts, documents and interface images.
Evidence use: Supports comparing document and interface understanding capabilities.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
MedHELM v2.0.0
Evaluates medical language models with clinically relevant tasks and multidimensional metrics.
Evidence use: Supports versioned comparison of model capabilities in medical settings.
Limitations: Sub-leaderboards use different scenarios, metrics and versions. The "latest" page updates on a rolling basis; citations should record the access date.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24