Playing an AI at your level: Elo, handicaps and honest bots

Stockfish running on a laptop is far stronger than any human. Playing it at full strength is not a lesson; it is a very polite form of demolition. To learn from playing a computer, you need one that plays roughly at your level, and that turns out to be a harder problem than making it strong.

What Elo actually measures

The Elo system, developed by the physicist Arpad Elo and adopted by FIDE in 1970, does not measure skill directly. It measures expected results. The difference between two ratings predicts what share of the points each player should score over many games.

Roughly:

• Equal ratings: each player expects about 50%.
• 100 points apart: the stronger player expects about 64%.
• 200 points apart: about 76%.
• 400 points apart: about 91%.

After each game, ratings move up or down depending on how the result compared with that expectation. Beat someone rated far above you and you gain a lot; lose to them and you lose little.

Two consequences are worth knowing. First, ratings are only comparable within the same pool: a rating from one website or federation does not translate exactly to another. Second, ratings are uncertain, especially for new players. Systems like Glicko and Glicko-2, used by many online platforms, track that uncertainty explicitly and show a rating as provisional until enough games have been played.

Why a matched opponent teaches more

Learning happens best at the edge of your ability. Against a much weaker opponent, your mistakes go unpunished and you learn nothing about them. Against a much stronger one, you lose without understanding why, because you are outplayed everywhere at once.

Against an opponent close to your level:

• Your mistakes get punished often enough that you notice them.
• You get real chances to win, which rewards good plans and keeps you motivated.
• Games are long enough to reach middlegames and endgames, where most of the learning is.

The 400-point table above makes the point numerically: if you only score about one game in ten, you are not practising winning positions or converting advantages at all.

The problem with weakening an engine

The obvious way to make an engine weaker is to limit how deeply it searches, or to make it occasionally play a random move. Both produce opponents that feel strange. A depth-limited engine plays near-perfect short tactics and then misses a simple two-move threat that any beginner would see. A randomised engine plays like a grandmaster for ten moves and then hangs its queen for no reason.

Humans make mistakes differently. A beginner misses pieces hanging in plain sight and forgets about back-rank mates. A club player sees one- and two-move tactics but misjudges long-term plans and trades. A strong player rarely blunders material but can misplay a complicated middlegame under pressure. A good training opponent should make the kinds of mistakes a real player at that level makes, so that the lessons you learn transfer to games against people.

Modern engines, including Stockfish, offer a strength setting that picks weaker moves in a more controlled way, aiming at a target rating. It is not a perfect imitation of human play, but it is a large improvement over crude depth limits, and it can be calibrated against rated opponents.

Why we name bots after players, not difficulty

"Easy", "Medium" and "Hard" say how the bot feels to you, and that changes as you improve. They also mean different things in every app. So in Walnut Chess the preset opponents are named for the class of player they play like:

• Novice AI is aimed at someone who has learned the rules and a few basic ideas: beatable, and not always alert to threats.
• Club AI is aimed at a typical club player: sensible development and simple tactics, but it can be outplanned.
• Master AI is aimed at a very strong human: it rarely drops material and punishes loose play.

If you want finer control, you can set an exact target rating instead. The point of the names is that they describe the opponent, not your feelings about it. When you start beating Club AI regularly, you know something concrete about your chess.

How to pick your level

A good rule of thumb is to choose an opponent you beat somewhere between a third and a half of the time. If you win almost every game, step up. If you almost never win, step down; a few wins are useful too, because converting a winning position is a skill that needs practice.

It also helps to play the same level for a while rather than jumping around. Consistent opposition makes your progress visible and lets you notice patterns in your losses.

Make the games count

Playing a bot is most useful when it is deliberate:

• Play slower than you would online. Take the time to find candidate moves and do a quick blunder check before each move.
• Use help sparingly. Hints and takebacks are great for learning, but a game where you took back every mistake does not tell you much about your level. Try some games with help and some without.
• Review afterwards. Even five minutes on the biggest swing in the evaluation helps. Our review routine takes about twenty.
• Replay critical positions. Set up a position you misplayed and try again against the same bot.

A different kind of AI goal

For decades the aim of chess AI was to beat the strongest humans. That is settled. The aim that matters for most players now is different: an opponent that is beatable, behaves like a person, and helps you understand your mistakes. As we discuss in game AI beyond chess, games have always needed both kinds of AI. The strong kind finds the truth. The well-designed kind makes it fun to learn.