Knowledge Bases and Evaluation Organizations: MLCommons
Knowledge Bases and Evaluation Organizations · MLCommons. Public benchmarks for training, inference and safety. This source has 14 entries admitted under this category's review policy; each links directly to the original.
Authority tiers in this category: A2–B. Admission policy: All core resources are included. Each secondary source is admitted according to its own database, methodology, versions and leaderboard structure; there is no cross-organization scoring scheme.
MLCommons Benchmark Suites
Brings together MLCommons benchmark suites for training, inference, client devices, storage, algorithms and AI safety.
Evidence use: Supports MLCommons' benchmark landscape, positioning and official version entry points.
Limitations: The catalog is not a single evaluation result. Different suites cannot be compared directly.
Related concepts: Model Evaluation and Benchmarks · LLM Inference Optimization
Link last checked: 2026-09-24
MLPerf Training
Measures the performance of hardware and software training systems by the time needed to reach a set quality target, and publishes rules, reference implementations and results.
Evidence use: Supports performance comparisons of training systems under a specified version, model and configuration.
Limitations: Results depend on version, division and hardware configuration. Open and closed divisions cannot be compared with each other.
Related concepts: Distributed Training and Parallelism Strategies · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
MLPerf Inference
Measures throughput, latency and power when running models in different deployment scenarios, with submission and reproduction rules.
Evidence use: Supports inference performance comparisons under specified hardware, software stack and scenario.
Limitations: Must be pinned to a scenario and version. Submitted results do not represent all real workloads.
Related concepts: LLM Inference Optimization · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
MLCommons AILuminate
Evaluates the safety and jailbreak resilience of general-purpose generative AI systems using hazard categories, a public practice set and a private official test set.
Evidence use: Supports comparing models' safety grades and hazard categories under an explicit version and language.
Limitations: Mainly covers single-turn content harms. The official test set is private; verification depends on the official process.
Related concepts: Model Evaluation and Benchmarks · AI Red Teaming
Link last checked: 2026-09-24
AILuminate Safety Methodology
Specifies hazard categories, prompt construction, response grading, aggregation and the safety grading method.
Evidence use: Supports precise interpretation of AILuminate safety scores and their limits.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: Model Evaluation and Benchmarks · AI Red Teaming
Link last checked: 2026-09-24
AILuminate General Purpose AI Chat Official Results
Publishes safety grades and hazard category results for general-purpose chat models on the official private test set.
Evidence use: Supports comparisons of model safety results tied to language and version.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
AILuminate Security and Jailbreak v0.5 Whitepaper
Introduces a collection of jailbreak attacks for generative AI, with scoring and safety evaluation design.
Evidence use: Supports analyzing the coverage and methodology of jailbreak evaluation.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: AI Red Teaming · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
MLPerf Endpoints
Measures throughput, latency and service behavior of online model endpoints under controlled load.
Evidence use: Supports performance comparisons of hosted inference services under specified load.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: LLM Inference Optimization · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
MLPerf Client
Tests performance and experience of language, image and agent workloads on personal computing devices.
Evidence use: Supports cross-device and cross-workload comparisons of edge AI systems.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: LLM Inference Optimization
Link last checked: 2026-09-24
MLPerf Automotive
Defines performance and power evaluations for automotive scenarios such as perception and the cockpit.
Evidence use: Supports standardized comparisons of automotive AI hardware and software stacks.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: Model Evaluation and Benchmarks
Link last checked: 2026-09-24
AlgoPerf Training Algorithms Benchmark
Compares how efficiently training algorithms reach target quality under a fixed workload and resource budget.
Evidence use: Supports comparing the training efficiency of optimization algorithms under uniform conditions.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: Distributed Training and Parallelism Strategies · Model Evaluation and Benchmarks
Link last checked: 2026-09-24
MLPerf Storage
Covers AI storage workloads such as training data, checkpoints, vector databases and KV caches.
Evidence use: Supports comparing storage performance and scalability for AI infrastructure.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: Distributed Training and Parallelism Strategies · LLM Inference Optimization
Link last checked: 2026-09-24
MLPerf Mobile
Compares vision, language and generative AI inference performance on smartphones and other mobile devices.
Evidence use: Supports comparing on-device inference performance and energy efficiency on mobile devices.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: LLM Inference Optimization
Link last checked: 2026-09-24
MLPerf Tiny
Defines tiny machine learning inference tasks and measurement rules for ultra-low-power microcontrollers.
Evidence use: Supports comparing performance and energy efficiency of TinyML devices.
Limitations: Must be pinned to the suite, version, division and submission configuration. Results from different workloads and levels of formality cannot be compared directly.
Related concepts: LLM Inference Optimization
Link last checked: 2026-09-24