New Product: AI Evaluations

At LMArena, our mission is to improve the reliability of AI applications and systems. The majority of these applications involve a human in the loop: from coding assistants to customer-support bots to everyday chat tools. Studying those interactions at scale is critical to making AI more effective.

As more users join the platform, we’re building the infrastructure and methods to capture this complexity and turn real-world interactions into reliable insights the ecosystem can learn from.

Today, we’re introducing a commercial product: AI Evaluations. This service offers enterprises, model labs, and developers comprehensive evaluation services grounded in real-world human feedback, showing how models actually perform in practice. By providing analytics based on community feedback, we are giving them the tools they need to understand strengths, weaknesses, and tradeoffs—and, ultimately, to improve their models and AI applications for everyone.

LMArena’s AI Evaluations include:

• Comprehensive, in-depth evaluations based on feedback from our community
• Auditability through representative samples of feedback data
• Service-level agreements (SLA) with committed delivery timelines for evaluation results

These features are designed to support both commercial labs and open source teams looking to better understand how their models perform in the wild.

AI evaluation is critical for the entire ecosystem. Labs use LMArena feedback to build more reliable models. Developers use it to choose the best tools for their work. And the community benefits from AI that performs better in real-world use.

LMArena has already logged 250M+ real conversations, 2M+ monthly votes, and has 3M+ monthly users. We operate at a scale that makes our signal uniquely trustworthy. That’s why leading labs are turning to us—not just for transparency, but to accelerate progress for everyone.

As we expand into commercial evaluations, LMArena’s core values remain unchanged:

• All models—commercial, open source, public, or awaiting release—are evaluated using the same community-driven methodology, with no preferential treatment.