Which AI Ranks Best? Chatbot Arena Uses Millions of Votes

As companies such as OpenAI, Google, and Meta release increasingly advanced artificial intelligence products, the need for effective evaluation methods has grown. Crowdsourced rankings have become a practical approach to determining the best AI tools, with LMSYS’s Chatbot Arena emerging as a leading real-time metric for this purpose. Unlike traditional benchmarks that evaluate AI models based on general capabilities—like solving math problems or programming challenges—there is no universal standard for assessing large language models (LLMs) such as OpenAI’s GPT-4o, Meta’s Llama 3, Google’s Gemini, and Anthropic’s Claude. Small variations in datasets, prompts, and formatting can significantly impact performance, making it challenging to compare LLMs fairly when companies set their own evaluation criteria. Jesse Dodge, a senior scientist at the Allen Institute for AI in Seattle, highlights this issue, noting that even minute differences in benchmark scores can be overlooked by users.

The difficulty in comparing LLMs is compounded by how closely leading models perform on many benchmarks. Tech executives often claim superiority with differences as small as 0.1%, which may go unnoticed by everyday users. To address this, community-driven leaderboards that incorporate human insights have gained popularity. These platforms, including the Chatbot Arena, leverage public votes to gauge AI performance in real-time. The Chatbot Arena, an open-source project created by LMSYS and the University of California, Berkeley’s Sky Computing Lab, has become a significant player in this space. It ranks over 100 AI models based on nearly 1.5 million human votes. Visitors to the site compare responses from two anonymous AI models and vote on which performs better. The platform evaluates models across various categories, such as long queries, coding, instruction following, and different languages including English, French, Chinese, Japanese, and Korean.

As AI tools become more prevalent, the need for effective evaluation methods intensifies. Benchmarks serve as crucial goals for researchers developing models, but not all human capabilities are easily quantifiable. Vanessa Parli, director of research at Stanford University’s Institute for Human-Centered AI, emphasizes the importance of evaluating traits like bias, toxicity, and truthfulness, especially for sensitive sectors such as healthcare. She notes that while benchmarks are essential, they are not perfect and can be manipulated, making it necessary to explore innovative evaluation methods. “The benchmarks aren’t perfect, but as of right now, that’s the primary mechanism we have to evaluate the models,” Parli says. She cautions that researchers can exploit current benchmarks, which can lead to AI models quickly reaching saturation. “We need to get creative in the development of new ways to evaluate AI models,” she adds.

Evaluating intelligence presents its own challenges, as there is no universally accepted definition of intelligence, even in humans. The nature and extent of animal intelligence remain subjects of debate among scientists. Current AI benchmarks focus on specific tasks, but as research progresses towards artificial general intelligence (AGI)—an AI that excels across a broad range of tasks—more general assessments will be necessary. Chatbot Arena stands out for its reliance on human judgment to evaluate AI models. Jesse Dodge trusts its rankings more than other methods due to the direct human feedback involved. Parli agrees that assessments like those from Chatbot Arena can capture nuanced aspects of AI performance that are harder to quantify, such as user preferences. However, she stresses that Chatbot Arena should not be the sole evaluation tool, as it does not cover all critical factors organizations must consider when assessing AI models. By incorporating human insight, Chatbot Arena provides a valuable perspective on AI performance, but comprehensive evaluation will require a combination of methods to address all relevant criteria.