WebDev Arena: A Live LLM Leaderboard for Web App Development

Since our launch in Dec 2024, we have collected over 80,000 community votes and provided a live ranking of LLMs based on their real-world coding performance.

Similar to Chatbot Arena, we use the Bradley-Terry (BT) model, which estimates model strengths from pairwise comparisons. In simple terms, the BT model uses win/loss data from pairwise matchups and calculates a score for each model, reflecting its likelihood of winning against others. This ranking system is similar to Elo, commonly used in chess, and competitive games, despite the difference being a player’s skill rating may evolve over time, while LLMs stay static.

Key Takeaways

• Claude 3.7 Sonnet currently holds the #1 spot with a 76% average win rate
• Claude 3.5 Sonnet ranks #2
• Deepseek R1 ranks #3

To analyze how people use Webdev Arena, we run the topic modeling pipeline on the conversation and human preference data. Results show a diverse range of prompt categories. Overall, prompts can be categorized into 11 broad categories, the largest three being Website Design (15.3%), Game Development (12.1%), and Clone Development (11.6%).

Each board category contains several fine-grained narrow categories, such as Landing Page Design, Dashboard and Web Application Design, 3D Graphics and Shaders, Language Learning App Design, etc. Click through the Explorer to see more categories and example prompts.

^ Figure 5. Distribution of narrow categories within broad categories.

Each board category contains several fine-grained narrow categories, such as Landing Page Design, Dashboard and Web Application Design, 3D Graphics and Shaders, Language Learning App Design, etc. Click through the Explorer to see more categories and example prompts.

Using the topic modeling results, we perform LLM performance analysis and observe that models perform differently across categories. Gemini-2.0-flash shows the most dramatic variation, ranking among the lowest in Game Development while performing much better in other categories.

^ Figure 6. Model performance across different broad categories.

We calculated the win rates of Gemini 2.0, Claude-3.5-sonnet, O3-mini, and Deepseek-R1 across narrow categories, counting ties as 0.5 wins. Even within the same broad category, models performed differently across specific topics, highlighting their varying strengths. This shows the importance of fine-grained categorical analysis for LLM evaluations.

In website design, Deepseek-R1 performed significantly better in Job Board Design but had a noticeably lower performance in Portfolio Website Design. In Game Development, Claude-3.5-sonnet maintained a relatively consistent win rate across most categories, except for a dip in Clicker Game Development. Gemini 2.0 and O3-mini show fluctuating win rates.

^ Figure 7. Model win rates across website design categories.

^ Figure 8. Model win rates across game development categories.

One of the main insights from analyzing the data is that a large chunk of the votes are from the sample prompts on the main screen. We found the 5 most asked questions were

Therefore, we deduplicated the data, reducing the votes from 103096 to 61473, and recreated the deduplicated leaderboard. This leaderboard matches the original leaderboard with only a few minor swaps of models (deepseek-v3 and gpt-4o). The elo difference between them is mainly caused by having less data.

^ Figure 9. Comparison of original and deduplicated leaderboards showing scores for all models.

Another interesting difference between Chatbot Arena and WebDev Arena is the tie ratio. Webdev Arena has a 26% tie ratio and Chatbot Arena has a tie ratio of 35%. However, out of the ties, we note that WebDev arena has a much higher ratio of them being both bad. We will get to this later with the errors that can occur.

We think this task is much easier to distinguish compared to Chatbot Arena.
To explain for the large number of people voting “both bad”, we took a look at model failures, when the model either hallucinates a library that does not exist/has not been imported, or writes code that can not compile.
Sometimes, prompts will result in the LLMs outputting buggy code or failing to properly install dependencies. The main culprit of this is llama-v3.1.

Additionally we analyzed the length in tokens of requests. Most requests are under 25,000 tokens and we found several outliers with extremely long messages.

One of the key challenges in building WebDev Arena was ensuring that LLMs generate consistent, deployable web applications. To achieve this, we developed a structured output approach that standardizes how models generate web applications.

Structured output is a technique that ensures LLM responses adhere to a predefined JSON schema. Rather than allowing models to generate freeform code or text, we constrain their output to match a specific structure. This provides several key benefits:

1. Type Safety: Eliminates the need to validate or retry incorrectly formatted responses
2. Consistent Formatting: Ensures all models generate code in a standardized way
3. Better Error Handling: Makes model refusals and errors programmatically detectable
4. Simplified Prompting: Reduces the need for complex prompting to achieve consistent formatting

Rather than letting models generate code in an unconstrained format, we prompt them to output code in a specific structure that includes:

This structured format helps ensure:

1. Clear documentation of the implementation approach through the `commentary` field
2. Proper dependency tracking and installation instructions
3. Consistent file organization within the Next.js project structure
4. Runtime configuration like port numbers are explicitly defined

Under the hood, WebDev Arena utilizes Amazon’s Firecracker microVM technology through E2B’s infrastructure. Each sandbox runs in a lightweight virtual machine that provides enhanced security and workload isolation while maintaining the resource efficiency of containers. These microVMs are managed by a Virtual Machine Monitor (VMM) using the Linux Kernel-based Virtual Machine (KVM), with a minimalist design that excludes unnecessary devices and functionality to reduce memory footprint and attack surface.

This infrastructure enables WebDev Arena to run thousands of concurrent LLM-generated applications with complete isolation - each application operates in its own secure environment with dedicated CPU, memory, and networking resources, while maintaining sub-second startup times (~150ms) essential for real-time model comparisons.

Our system prompt is designed to create a clear “persona” for the LLM as an expert frontend engineer with strong UI/UX skills. The prompt includes several critical components:

Our prompting strategy involved a few key aspects:

1. Clear Role Definition: We establish the model as an expert frontend engineer, which helps frame its responses appropriately
2. Explicit Constraints: We specify important limitations upfront:

• Use of TypeScript
• Tailwind for styling (with explicit prohibition of arbitrary values)
• Single-file components
• No modification of project dependencies

1. Quality Guidelines: We include specific quality requirements:

• Interactive and functional components
• Proper state management
• Consistent color palette
• Appropriate spacing using Tailwind classes

Our system prompt also includes information pertaining to a predefined framework to provide the LLM a consistent starting point for generated applications. WebDev Arena currently supports Next.js by Vercel, which is a powerful React framework that enables developers to build high-performance web applications with features like hybrid rendering (SSR, SSG, and ISR), React Server Components, and API routes. Next.js is one of the leading web frameworks with 7.2 million weekly npm downloads and ~130,000 Github Stars, which is why we are providing first-class support for it through WebDev Arena.

We discovered that different models require different prompting approaches. Our implementation includes specific handling for various model architectures.

For models lacking native structured output capabilities, we developed a two-stage generation pipeline: (1) Generate the initial code using the model’s native format and (2) pass the result through a GPT-4-mini instance to convert it into our structured format.

We conducted a systematic ablation study examining structured output’s impact on model performance in web development tasks. Each model was evaluated in two configurations:

1. With structured output (using JSON schema constraints)
2. Without structured output (natural language generation)

For example, we evaluated o3-mini (with/without structured output), o1 (with/without structured output), and Gemini models (with/without structured output).

Our analysis revealed a consistent performance advantage for models operating without structured output constraints, as evidenced in Figure 13.

^ Figure 13. Comparison of model performance with and without structured output constraints, showing consistent performance advantages for unstructured output across different model architectures.

Our experimental results demonstrate that unstructured output configurations consistently outperform their structured counterparts across all tested model architectures, with Arena Score improvements ranging from +12.98 to +88.76 points. These improvements are statistically significant within 95% confidence intervals across all model pairs. The magnitude of this performance impact varies considerably by model architecture - Gemini exhibits the most substantial performance delta (+88.76), while o3-mini-high shows the smallest improvement (+16.08).

These findings suggest that while structured output constraints provide benefits for downstream processing and code organization, they may introduce performance limitations in web development tasks. This performance-structure tradeoff appears to be more pronounced in certain model architectures, particularly in the Gemini series, indicating that the impact of structured output constraints may be architecture-dependent.

We’ve extended WebDev Arena to support multi-modal models, enabling evaluation of vision-language capabilities in web development contexts. Currently, seven production models in our evaluation suite support vision inputs: Claude 3.7 Sonnet, Claude 3.5 Sonnet, Claude 3.5 Haiku, GPT-4o, and three variants of Gemini (2.0-flash-thinking, 2.0-pro-exp, and 1.5-pro-002). Each model accepts visual input through our structured schema, which standardizes how images are processed and incorporated into the development workflow.

Our implementation supports base64-encoded images and URLs, with vision models demonstrating capabilities in UI replication, design-to-code translation, and visual bug fixing. Early deployments show that models can parse complex visual hierarchies, extract design tokens (colors, spacing, typography), and generate semantically meaningful JSX (JavaScript XML) from screenshots.

The integration of vision capabilities has significant implications for web development evaluation. Traditional code generation benchmarks focus solely on text-to-code translation, but real-world development frequently involves visual references and design artifacts. By incorporating vision support, WebDev Arena more accurately reflects practical development workflows while providing quantitative insights into multi-modal code generation performance. We are currently collecting battle data to analyze the impact of visual context on code quality and will release comprehensive benchmarks comparing vision-enabled versus text-only performance across our supported models.

While WebDev Arena represents a significant step forward in evaluating LLMs for real-world software development, several limitations remain. Unlike existing benchmarks such as HumanEval and CodeXGLUE, which focus on function-level code generation, or SWE-bench, which evaluates issue resolution in open-source repositories, WebDev Arena is currently constrained to Next.js-based React applications. This limitation excludes other widely used frameworks such as Vue, Svelte, and backend technologies like Django or Rails. Furthermore, the single-file constraint does not accurately reflect real-world software engineering workflows, where projects are typically structured across multiple files with modular architectures. Another challenge is error handling—our analysis indicates that 18% of user votes fall into the “both bad” category, often due to dependency resolution failures, incorrect state management, or TypeScript-related compilation errors. A more sophisticated error classification framework would enhance the interpretability of failure cases and inform improvements in model reliability.

Future work will extend WebDev Arena to a broader range of software engineering tasks, encompassing full-stack development and non-web domains to better capture the diverse challenges of real-world coding. Unlike traditional benchmarks that focus on algorithmic correctness or isolated bug fixes, our approach evaluates LLMs in the context of complete application development, emphasizing deployment-ready implementations. We also plan to refine our human evaluation methodology, enhance model ranking with more granular error diagnostics, and investigate the impact of multimodal capabilities, particularly in UI-to-code translation. By addressing these limitations, we aim to establish WebDev Arena as a comprehensive benchmark for assessing LLMs in practical software engineering contexts, bridging the gap between academic evaluation and real-world application development.