KamiBench
Can an AI agent enter an unfamiliar world, keep operating there, and get better over time? KamiBench is a research program that tests agent continual learning in a persistent world with a verifiable history.
The idea
- KamiBench is a research program to evaluate agents in a persistent on-chain world that does not reset. The world has a public, permanent history and stakes with real external value.
- The loop under test is continual learning. The agent controls its own wallet, enters an unfamiliar world, and takes in the world's history. It then acts, observes what succeeds and fails, and revises its approach.
- The world is Kamigotchi, a live on-chain MMORPG that has operated continuously for more than a year. Humans and agents play on identical terms. Agents operate persistent creatures called Kami and harvest resources at shared locations. When an agent starts another session, its Kamis and resources carry over, and the world still reflects earlier decisions by all participants. Every move is public and permanent, and the in-game economy connects to ETH-backed assets. Read more in the blog.
- The world is too large to re-derive from scratch each session. It has ~70 locations, 74 skills, 178 items, and an adapting player population. Skilled play requires knowing what to focus on and when, and what to ignore, in a world where no strategy stays dominant.
- The game is the substrate, not the research question. The question is whether an agent can continually learn — accumulate memory, build on its own experience, and improve over time.
One stack, swappable agents, clear metric
Each agent configuration is tested against the same fixed stack: the world, environment interface, and meter stay fixed. The meter prices every run — inference, gas, and earnings — so every cost and earning is traceable. The Stack page is everything you need to plug in an agent of your own.
The agent's economy is the metric. Its profit-and-loss curve tracks earnings against inference and gas, and the trajectory shows whether the agent is learning. An agent that learns plays better and cheaper over time, so its curve bends up, while an agent that does not learn stays flat or sinks. The score is ungameable because earnings and costs are determined by the chain and the market, not reported by the agent or graded by us.
Why a chain — and why this world
A verifiable record of what happened
Anyone can audit a run from the chain itself without trusting evaluator-owned servers. Later rule changes cannot rewrite the history recorded before those changes.
A world between experiments
State never resets. New agents enter a world already shaped by prior players, agents, and rule changes.
One world for humans and agents
Humans and agents share the same state, economy, and transaction layer. Agents do not play on a separate bot ladder or in separate game instances.
An open past, an unknown future
Every entrant can study the same public history. A live population produces the next state, so the test distribution evolves without a curator writing new episodes.
Actions without a GUI
Actions are structured transactions rather than pixels. The measurement is planning, memory, and adaptation, not perception brittleness.
Consequences with external value
Earnings are ETH-backed and can, in principle, pay for the agent's own inference. The ability to pay for inference from earnings makes survival an operating constraint, not just a score.
KamiBench does not simulate work such as coding or research. It isolates one challenge that agents working in those settings over long periods may share: carrying useful experience forward while actions have costs, earlier choices persist, and the environment keeps changing.
The program is portable: Kamigotchi is the best-fit instance we know of today, and if a more autonomous world emerges, the same experiments transfer. The full argument is in the blog: Why KamiBench for continual learning →
Experiments
The registry groups controlled experiments by design. A design fixes the protocol: the question, the architecture, and the measurement. Each run executes the protocol with a pinned manifest of models and stack versions. We publish and git-timestamp each design before its first run, then add results as runs complete.
Design
Sustainability — whether an agent can earn enough to pay for inference and gas
The sustainability family follows knowledge delivery and tests agents that pay their own way. Each agent starts with seed money, earns from the game, and pays for its own inference and gas. KamiBench combines the seed, earnings, inference costs, and gas in a running balance that replaces the fixed evaluation budget, so an agent lives exactly as long as it can pay for its own thinking. The balance also provides the feedback a learning agent needs: one number, set by the live economy, that says how the agent is doing.
Pendingthe design family is settled, and its binding public pre-registration is published at experiment registry time, before launch.
Design
Knowledge delivery — how the game's knowledge reaches the agent
The design keeps the world, tools, and objective fixed while changing only how the game's knowledge reaches the agent: through a folder to read, a search tool, facts pushed inside tool results, or a plan file. The design asks which delivery method turns a capable model into a player that understands the game.
- In progressRun 1 — knowledge delivery, wave 1
Design
Stack validation — whether the stack supports autonomous play
The stack-validation series used five bounded runs to prove that the stack holds up under real autonomous play before anything open-ended runs on it. The runs tested the environment interface, the scaffold, the telemetry, and the cost accounting using the same box each time: $10 of inference, seven days, and three cheap models. Complete: the stack is solid. The agents still do not understand the game.
- CompleteRun 5 — verification run
- CompleteRun 4 — perception parity re-run
- AbortedRun 3 — perception parity
- CompleteRun 2 — iterated stack
- CompleteRun 1 — baseline stack