Our Self Improving Agent hit Top 500 on Pokemon Showdown
Launching Scorecard's Pokeharness, an experiment to learn how to measure and improve an AI agent over time.
The Scorecard teamSep 30, 20269 min read
Just after midnight on Sep 26, our self-improving Pokémon agent won 24 of 28 games, climbed from 1269 to 1674 Elo, and hit rank 423. Then it proceeded to lose 24 of the next 32 and fell back to 1356.
You might be surprised to learn that the agent didn’t change at all that night. Its self-improvement loop runs on a long cycle, so from Sep 24 to Sep 26 it played 702 games as the exact same agent, winning 350 and losing 352 before its next update. A great night of wins is a fluke. Agents are sometimes brilliant and sometimes mind-bogglingly dumb, and making one consistently good is what agent builders are chasing.
Turns out real life is a great eval
We run three agents (more on that later) that play on the live servers around the clock. The neat part about this is that people do things no synthetic dataset would think to include. One opponent this week spent a game pushing its evasion as high as it goes, a strategy that usually makes people forfeit out of frustration. It happens that agents have more patience than your (presumably) very patient grandmother, so that game ran about four times longer than normal.
Luck is in every game too. Damage varies by up to 15% from roll to roll, moves miss, critical hits land, and some attacks make the target flinch and lose its turn. Here is one loss from the first day.
Between luck, matchups and opponents of every skill level, a rating can move hundreds of points in a day without the agent getting any better or worse. One of our agents is a control agent: it never changes, so it shows how much a rating moves on its own. Yet this week its rating ranged from 1061 to 1554, and overnight on Sep 25 it fell 453 points in nine hours.
The control agent’s rating after each of its 2,441 games since noon on Sep 24. Ladder cut from the public Reg M-C top-500 list.
Scorecard helps teams make their AI agents better. The usual recipe starts with experts annotating what the agent did. Teams then use those annotations to hill-climb on the prompts and harness, or to fine-tune the model with Reinforcement Learning from Human Feedback (RLHF). It works, but expert time is scarce, and models are starting to get good enough to review their own work.
So we want to know how far an agent gets when another agent annotates its games, and what a human expert’s annotations add on top, measured by a score we don’t control. Pokémon Showdown is an open-source battle simulator with a public ranked ladder, where every ranked game moves an Elo rating against a huge pool of real players. Because the code is open, the agent and its coaches can check type matchups and speeds against the same engine that runs the games, and every game comes with a full turn-by-turn log.
Our goal is a top-500 rank held for a full day, not one reached for a couple of hours on a lucky streak. The bar moves as the people on the ladder improve: 500th place needed 1560 on Sep 25 and needs 1591 today. All three agents are chasing it, each on its own account: the control agent that never changes, one annotated by a separate Claude agent, and one annotated by Yash, our in-house Pokémon expert.
controlNever changesplays as Scorecard-Pokemon
self-improving ↻Annotated by Claudeplays as SC-Control *
human-annotated ↻Annotated by Yashplays as SC-SME
Improver agentRewrites each agent’s playbook from its annotations
* Yes, the self-improving agent’s account is called SC-Control.
The experiment’s setup
Every agent plays the same six Pokémon, based on a team that took second at a major official tournament in Baltimore. The agent picks which four to bring each game but never changes the team.
Charizard
Charizardite Y · Blaze · Timid
HP 2SpA 32Spe 32
Heat Wave
Hurricane
Weather Ball
Protect
Golisopod
Golisopite · Emergency Exit · Adamant
HP 32Atk 32Def 2
Leech Life
Drill Run
Iron Head
Protect
Politoed
Mystic Water · Drizzle · Modest
HP 32Def 2SpA 32
Muddy Water
Ice Beam
Weather Ball
Protect
Archaludon
Leftovers · Stamina · Bold
HP 32Def 1SpA 1SpD 29Spe 3
Dragon Pulse
Snarl
Electro Shot
Protect
Grimmsnarl
Light Clay · Prankster · Sassy
HP 32Def 19SpD 15
Light Screen
Reflect
Parting Shot
Spirit Break
Farigiraf
Sitrus Berry · Armor Tail · Bold
HP 29Def 21SpD 16
Psychic
Trick Room
Protect
Grass Knot
One Claude Code session plays the whole battle. It can call a Pokédex, a type chart and a speed calculator, plus any skills, memories and tools the improver agent has written for it.
Showdown client. Holds the websocket to the live server and plays one battle at a time.
Turn builder. New log lines, the request JSON, opponent usage stats on turn one.
Claude Code session. Opus 5.5 in fast mode, one session per battle. System prompt: harness rules plus the agent’s playbook. Tools: pokedex, type_matchup, speed_check, load_skill, recall, and the agent’s own sandboxed tools.
Decision checks. Schema, repairs, and a first-legal-move fallback after 35 seconds.
Out./choose to Showdown, a trace to Scorecard, and a feed to the live page.
Every game is saved in Scorecard as a trace of every decision, and annotations attach to the exact decision they’re about. Here is the game from the luck chart. Yash’s note sits on turn 2.
I think Electro Shot is free, double up into Politoed is obvious, so we should protect.
Completion
{
"risk": "Aerodactyl is at 1 HP and Taunted Archaludon, so its Rock Slide is now just chip damage. Sneasler's Close Combat on Archaludon is the only big threat, and it's single-target. No double-KO is possible.",
"choice": "move 1, move 3 +1",
"thoughts": "Muddy Water to finish off the 1-HP Aerodactyl, and a +1 Electro Shot straight into Sneasler."
}
Prompt
How an agent improves
Both improving agents run the same loop with the same improver agent. They differ in who annotates their games, how many annotated games the improver waits for, and who approves the change.
1
Play
Each agent plays ranked games around the clock on its own account.
2
Record
Each game becomes one Scorecard record. control stops here.
3
Review
A reviewer marks decisions good or bad and says why.
self-improvingA Claude annotator, hourly, up to 15 losses.
human-annotatedYash watches replays and writes notes.
4
Teach
Daily at 3:30 PM PT, the improver agent edits the playbook once enough of the current version’s games are reviewed: 20 for the self-improving agent, 5 for the human-annotated one.
5
Ship
The next version goes live.
self-improvingMerges on its own.
human-annotatedYash approves a pull request.
The annotations in step 3 are where the agents differ most. A Claude annotator reviews the self-improving agent’s losses. Yash annotates the human-annotated agent’s games as he watches the replays. Here is one game from each.
Yashhuman-annotated agent · loss vs a 1297 player · Sep 24
Turn 0👍
Bring Politoed, Archaludon, Farigiraf, Golisopod
Yeah I think this is a good call as a whole.
Turn 1👍
Politoed: Muddy Water · Archaludon: Electro Shot
I do like the Electro Shot, we’re not pressured so that’s good. I would swap Poli to keep weather pressure up, prob into Farig to block Fake Out (obvious setup for Tailwind).
Turn 2👎
Politoed: Muddy Water · Archaludon: Electro Shot → Sneasler
I think Electro Shot is free, double up into Politoed is obvious, so we should protect.
Turn 5👎
Farigiraf: Psychic · Archaludon: Electro Shot
Yeah it’s kinda over here as a whole, should’ve set up Trick Room before.
SummaryWe should’ve kept Politoed alive to keep up rain pressure, and then also swap in Farig when Fake Out is predicted.
Claudeself-improving agent · loss vs a 1328 player · Sep 30
Turn 2👎
Farigiraf: Psychic · Golisopod: Drill Run, both into Chandelure
Both went into Chandelure, which had attacked on turn 1 with Protect unused. It Protected, and the one free Trick Room turn was lost while the fresh Mega Garchomp-Z took no damage. […] This breaks the strategy’s own rule: never more than one action into a slot with Protect available.
Turn 6👍
Charizard: Hurricane · Politoed: Weather Ball
Holding the Charizard Mega for one turn to fire rain Hurricane (100% accurate, x2) KO’d Grapploct. […] That was good weather sequencing.
SummaryWe lost because their Mega, Garchomp-Mega-Z, took almost no damage while our Trick Room was up.
Yash writes like a player talking through a replay. Claude writes longer, checks the numbers, and points out when the playbook itself is wrong.
Both kinds of note go into the same brief for the improver agent in step 4. Here’s what the improver reads, what it’s told to do, and what it’s allowed to change.
In · the brief
Every decision flagged as a mistake in the current version’s games, with the move the agent made
Mistakes counted by category
Praised decisions to keep, up to 25
Situations that show up far more in losses than in wins
→
Improver agent · Claude Code
Find the 1 to 3 patterns that cost the most games
Decide why the playbook failed: a rule was missing, wrong, or there but ignored
Check game facts against the simulator’s own data
Make the smallest edit that fixes it and log the evidence
→
Out · the agent’s folder
strategy.mdrules for every game, always in the prompt
skills/playbooks the agent loads for certain positions
memory/facts from past games it can look up
tools/code for sums it keeps getting wrong
CHANGELOG.mdeach change and its evidence
Off limitsthe harness, the other agents, the version number
What differs by agentThe improver’s instructions for the human-annotated agent add one line: “The human reviewer’s notes are the ground truth: follow their judgment even where you would have played differently.”
Next month is about collecting enough data to make our findings meaningful. Each version will stay live for about a week instead of a day, long enough for a real change to show through the noise. The improvement loop will be refined, introducing regression checks such as testing each new playbook against a fixed set of past mistakes before it reaches the ladder. Yash will review more games, so the human-annotated agent isn’t steering on a handful of notes. And as the data rolls in, we’ll improve our stats and analytics to surface more interesting findings.
Everything is public. You can watch every game live, see the agent’s reasoning as it plays, and follow the progress on the stats page.
Stay tuned for part two: whether any agent held the top 500 for a full day, and whether a few annotations from an expert beat a lot of annotations from Claude.