Thousands tune into a livestream of Claude 3.7 Sonnet navigating the game on its own better than its predecessor. Anthropic tells PCMag it's a more effective way of measuring AI progress.

AI researchers have used all sorts of video games, from Street Fighter to Pictionary, to test new models — often more for amusement than utility. But Anthropic said that Pokémon proved to be a useful benchmark for Claude 3.7 Sonnet, which can effectively “think” through the sorts of puzzles the game contains.

A new Twitch channel called ClaudePlaysPokemon uses Claude's LLM capabilities to see if the AI can become a true Pokemon Master.

As I write this, it’s hunting a Mankey outside of Viridian City in Pokémon Red or Blue (I can’t tell which). It’s using the new Claude 3.7 Sonnet “hybrid reasoning model” and shows its thinking in a side window.

Anthropic’s AI Claude 3.7 Sonnet played Pokémon Red on Twitch, showcasing its decision-making. Could AI agents play games for humans someday?