Skip to content
noeiranoeiranoeira

Procgen & Craftax

BetaWorks, but under active development — APIs and results may shift. Complete and tested, but newer than the rest of the catalogue.

Procedurally generated environments, where every episode draws a fresh level from a seed. These measure generalization rather than mastery: an agent that memorises one layout scores nothing.

Procgen — all 16 games

Section titled “Procgen — all 16 games”

A complete pure-Mojo port of OpenAI’s Procgen benchmark, with roughly 86 tests covering it.

bigfishbossfightcaveflyerchaser
climbercoinrundodgeballfruitbot
heistjumperleapermaze
minerninjaplunderstarpilot

Each generates its level from a seed, so the standard protocol works: train on a fixed set of levels, evaluate on unseen ones, and the gap is the generalization measure.

from noeira.envs.procgen.games import CoinRun, Maze, StarPilot
Terminal window
pixi run mojo run -I . examples/procgen/coinrun_render_demo.mojo
pixi run -e nvidia mojo run -I . examples/procgen/coinrun_rainbow_training_gpu.mojo

The substrate lives in noeira/envs/procgen/{core,games} — a shared engine (tile maps, sprites, collision, asset atlases) with per-game logic on top, which is how 16 games fit in a manageable amount of code.

Craftax

Section titled “Craftax”

An open-world survival and crafting benchmark — a 2D Minecraft-like world with a deep achievement tree: gather wood, craft tools, mine stone, descend, survive.

Two variants:

  • Craftax Classic — the Crafter-equivalent surface world
  • Craftax Full — multi-level dungeons, more mechanics
Craftax Classicplayable demo
Terminal window
pixi run mojo run -I . examples/craftax_classic/play_classic.mojo
pixi run -e nvidia mojo run -I . examples/craftax_classic/ppo_training_gpu.mojo

Craftax is a different kind of hard from the control tasks. Reward is tied to achievements that require long action sequences discovered in order — you cannot craft a pickaxe before collecting wood — so it stresses exploration and long-horizon credit assignment rather than fine motor control. Sprite-based pixel observations are available alongside the symbolic state.

Why these matter here

Section titled “Why these matter here”

Most of the catalogue tests whether an algorithm can learn a task. These test whether it learned the task or the level.

That distinction is sharp on procedurally generated worlds: an agent overfitting to specific layouts shows near-perfect training returns and chance-level evaluation returns. It is the cleanest available signal that a representation generalises, which makes these the right benchmarks for world-model and representation-learning work.

Next

Section titled “Next”