How Machine Learning and Reinforcement Learning Have Transformed Chess and Strategic Games

The 2024 World Chess Championship, a contest between teenage prodigy Gukesh (India) and reigning champion Ding Liren (China), exemplifies the razor-thin margins of excellence at the highest levels of competitive chess. After eight games, the score is tied with one victory each, the rest being hard-fought draws. This scenario highlights not only the immense skill and preparation required but also how hard it is to gain an edge at the highest levels of strategic analysis.
For me, strategic games have been a lifelong passion. From the simplicity of early board games to the complexity of modern games like chess, poker and online games (TFT). I have always been intrigued by the interplay of skill, strategy, and logic. With the rise of artificial intelligence, the way we approach and understand strategic games has been transformed. This blog explains how these technologies have elevated our ability to master such games, focusing on chess while exploring other domains such as Go and poker, and examining the challenges of imperfect-information games.

The Evolution of AI in Chess: From Heuristics to Neural Networks

Chess, often described as the "game of kings," has been a testing ground for artificial intelligence since the mid-20th century. Early chess programs relied on brute force calculations and handcrafted heuristics. These engines evaluated millions of positions by assigning static values to pieces and board configurations, mimicking human logic.

The turning point came in 1997 when IBM's Deep Blue defeated World Champion Garry Kasparov. Deep Blue combined vast computational power with an extensive database of opening moves and endgame strategies. While groundbreaking, it was still rule-based, unable to "learn" or adapt dynamically.

The advent of machine learning (ML) revolutionised chess engines. Unlike earlier programs, modern engines like Stockfish and Leela Chess Zero rely on deep learning and neural networks. These systems can evaluate positions with greater nuance, identifying patterns and strategies without explicit programming. The landmark innovation was DeepMind's AlphaZero, which taught itself chess by playing millions of games against itself, using reinforcement learning (RL) to iteratively improve.

How Reinforcement Learning Works in Chess AI

Reinforcement learning is a branch of machine learning where an agent learns by interacting with an environment, making decisions to maximise cumulative rewards over time. In chess, the environment is the board, the actions are the moves, and the rewards are determined by the game’s outcome—win, loss, or draw. Unlike supervised learning, which requires labelled datasets, reinforcement learning allows the agent to discover optimal strategies through trial and error, guided by feedback from its interactions.

AlphaZero exemplifies the power of reinforcement learning in chess. Beginning with no prior knowledge beyond the basic rules of the game, it explored millions of possible move sequences by playing against itself. Each game helped it refine its understanding of what constituted strong or weak play. Using a deep neural network, AlphaZero evaluated board positions and made predictions about the likelihood of winning from those positions. It combined these predictions with a search algorithm, effectively balancing short-term tactical calculations with long-term strategic considerations.

The iterative nature of reinforcement learning allowed AlphaZero to discover creative strategies and unconventional moves that had never been considered by human players or traditional chess engines. For example, its handling of pawn structures and positional sacrifices revealed an ability to think far beyond the immediate material advantages that typically dominate human play. This capability is not only a testament to the effectiveness of reinforcement learning but also a glimpse into the unique insights AI can contribute to strategic games.

AI's Success Beyond Chess: Strategic Games in Focus

1. Go: The Ultimate Challenge

The ancient Chinese game of Go is exponentially more complex than chess. Its 19x19 grid allows for more possible board configurations than there are atoms in the observable universe. This complexity made Go a significant challenge for AI, one that was conquered in 2016 by AlphaGo.

AlphaGo combined deep neural networks with Monte Carlo Tree Search (MCTS), a probabilistic algorithm. Its victory over Lee Sedol, one of the world's top Go players, was historic, showcasing AI's ability to devise creative, human-like strategies.

2. Shogi: Chess's Japanese Cousin

In shogi, or Japanese chess, captured pieces can be reused, creating a dynamic and unpredictable game state. DeepMind's AlphaZero mastered shogi using the same RL framework as chess, achieving superhuman performance despite the game's added complexity.

3. Poker: Navigating Uncertainty

While chess and Go are "perfect information" games, where all information is visible to both players, poker introduces hidden information and the element of bluffing. AI's success in poker is exemplified by systems like DeepStack and Libratus, which combine reinforcement learning with advanced game theory. These algorithms simulate opponents' strategies and use probabilistic reasoning to handle the uncertainty of hidden cards.

Imperfect-Information Games: The Frontier of AI Research

While chess, Go, and shogi are "perfect-information" games where all players have complete visibility of the board state, games like poker introduce a fascinating layer of complexity through imperfect information. In poker, players must navigate uncertainty—each opponent’s cards are hidden, and success often relies on psychological tactics like bluffing. This introduces a host of challenges for artificial intelligence, as it must contend not only with the strategic complexity of the game itself but also with the unpredictability of human behaviour and hidden variables.

The presence of hidden information fundamentally alters the dynamics of game-playing for AI. In perfect-information games, AI systems can calculate exhaustive game trees to predict all possible outcomes from a given position. However, in poker, the sheer number of potential game states and the incomplete knowledge about opponents' cards make such exhaustive calculations impractical.

To overcome these challenges, researchers have developed innovative methods that allow AI to approximate optimal strategies under uncertainty. One breakthrough approach involves training AI agents through fictitious self-play. This technique enables the AI to simulate opponents with diverse strategies, refining its own behaviour through repeated interactions. Another significant advancement is the use of counterfactual regret minimisation, which helps the AI learn to minimise long-term regret for suboptimal decisions by evaluating alternative actions it could have taken during gameplay.

Frameworks like Recursive Belief-based Learning (ReBeL) represent the cutting edge of AI research in this domain. ReBeL integrates reinforcement learning with principles of game theory, allowing AI to make informed decisions even when dealing with incomplete information. By modelling beliefs about opponents' strategies and adjusting its own accordingly, ReBeL pushes the boundaries of what AI can achieve in environments characterised by uncertainty.

Despite these advances, imperfect-information games remain a challenging frontier for AI. They require algorithms not only to calculate probabilities and optimise strategies but also to adapt dynamically to the unpredictable and often irrational behaviours of human opponents. These challenges ensure that research in this area continues to provide valuable insights into decision-making under uncertainty, with implications far beyond the poker table.

Conclusion: From Chess Boards to Real-World Impact

The 2024 World Chess Championship reminds us of the human brilliance and creativity that define strategic games. Yet, as Gukesh and Ding Liren compete, they are accompanied by a silent collaborator: the ever-evolving power of AI. From AlphaZero's unprecedented success to breakthroughs in poker and beyond, machine learning has transformed our understanding of strategy, offering tools that surpass human ability.

While perfect-information games have been largely conquered, the challenges of imperfect-information environments will persist into the future, definitely not to be conquered until our computational capacity takes a significant leap forward… perhaps a quantum leap.

Join the IoA and prepare for the future of business

Sign up now to access the benefits of IoA Membership including 1400+ hours of training and resources to help you develop your data skills and knowledge. There are two ways to join: