Algorithms

StableTested and in active use. Breaking changes will be called out. Maturity varies sharply by algorithm — see the table at the bottom.

The learning half of the stack, organised by family. Families 1–6 are reinforcement learning, ordered by increasing complexity. The last three are what a physical robot needs besides reward: learning from demonstrations, mixing demonstrations into RL, and policies that take a task after training. Each family shares an interface, a training driver and a set of failure modes, so reading one page usually transfers to its siblings.

The families

1 · Tabular and linear

No neural network, no GPU. Exact, fast, and the right place to verify an environment or an idea before scaling.

Page	Covers
TD methods	Q-Learning · SARSA · Expected SARSA · Double Q · n-step · SARSA(λ) · Monte Carlo · Dyna-Q · priority sweeping · replay variants
Function approximation	tile coding · polynomial and RBF features
Classical policy gradient	REINFORCE · actor-critic · linear A2C/PPO · linear DDPG/TD3/SAC

2 · Value-based

Learn a value network over discrete actions and act greedily on it.

Page	Covers
DQN	Q-network · replay · target network · Double · Dueling · Noisy
C51 and Rainbow	distributional value learning · the six Rainbow components

3 · Policy gradient

Learn the policy directly. On-policy: collect fresh data, update, discard.

Page	Covers
A2C	advantage actor-critic — a single-epoch PPO
PPO	clipped surrogate · GAE · multi-epoch · discrete and continuous

4 · Continuous control

Off-policy actor-critic over a replay buffer. The workhorses for robotics-style tasks.

Page	Covers
DDPG	deterministic actor · single critic · the overestimation problem
TD3	twin critics · delayed updates · target smoothing
SAC	maximum entropy · auto-tuned temperature — the recommended default
REDQ	N-critic ensemble · high update-to-data ratio

5 · Model-based

Learn a model of the world and use it — for data, for planning, or to train a policy inside it.

Page	Model used for
MBPO	generating synthetic training data
TD-MPC2	planning at decision time (MPPI)
DreamerV3	training a policy in imagination
Dreamer 4	as DreamerV3, with a transformer — experimental

6 · Planning — the zero-series

Monte-Carlo tree search, with self-play as the training signal.

Page	Dynamics
AlphaZero	a known simulator
MuZero	a learned latent model
EfficientZero V2	learned, plus consistency and sampled actions

These four — the zero-series plus TD-MPC2 — are the agents that search rather than evaluate a policy. The search itself is factored out into noeira.planners: MPPI and CEM over action sequences, MCTS over a tree. If you are tuning simulation counts, elite fractions or horizons, that section is where the behaviour is described.

7 · Imitation and VLAs

Learn a policy by regression onto recorded demonstrations — from a teleoperated arm, a scripted expert or a published dataset. No reward.

Page	Covers
Imitation learning	the shared loop: record → import → train → evaluate → deploy
Behaviour cloning	an MLP fitted to low-dimensional observations — the baseline
ACT	action chunking with a CVAE transformer, from camera images
SmolVLA	a 450M vision-language-action model, flow-matching head — experimental

8 · From demonstrations

Reinforcement learning that starts from demonstrations and corrections instead of from nothing.

Page	Covers
HIL-SERL and DAgger	SAC with a pinned demo buffer (RLPD 50/50), a BC term, intervention recording

9 · Zero-shot

Train once without a task; pick the policy for a reward supplied afterwards.

Page	Covers
Forward-Backward	FB representations · FB-CPR · the BFM-Zero reproduction on the Unitree G1

Choosing one

Discrete actions, want it to work — DQN, or Rainbow if you need the sample efficiency.

Continuous actions — SAC. Its auto-tuned temperature removes the hyperparameter people most often get wrong.

Cheap simulation, want reliability — PPO. More environment steps than SAC, far harder to destabilise, and the batched GPU driver makes the sample cost affordable.

Sample efficiency above all — the model-based tier. Budget time for it.

You have demonstrations, not a reward — start with behaviour cloning as the baseline, then ACT if the task needs cameras.

RL that never finds the rare success — put demonstrations in its replay: HIL-SERL.

Two-player games — AlphaZero if you have the rules, which for board games you do.

Learning how any of this works — start with TD methods on GridWorld. Twenty lines, seconds to run, and you already know the right answer.

Maturity

Algorithm	Status	Notes
Tabular and linear	stable	well tested, exact
DQN · Double · Dueling · Noisy	stable
C51 · Rainbow	stable	mind the support range
A2C · PPO · PPO-discrete	stable
DDPG · TD3 · SAC	stable	SAC is the most exercised path in the repo
REDQ	beta	LayerNorm critic effectively required
MBPO	beta	needed Q-explosion fixes
TD-MPC2	beta	works; MPPI planning is intricate
DreamerV3	beta	full Atari Pong reproduction achieved
AlphaZero · MuZero · EfficientZero V2	beta	GPU MCTS has sharp edges
Dreamer 4	experimental	known open issue — see its page
Behaviour cloning	beta	a baseline; gated fit and reload, weak closed-loop rates
ACT	beta	gated against the reference and LeRobot; no real-arm success rate yet
HIL-SERL · DAgger	beta	first lift in simulation; not yet run on the real arm
SmolVLA	experimental	first grasp-and-place on the arm (20 Sep 2026); no success rate measured
Forward-Backward · FB-CPR	experimental	released BFM-Zero actor reproduced in our G1; our own training trails the reference

The facade pattern

Every deep agent has the same shape: one package per algorithm exposing an XxxAgent over a trainer plus a shared training driver.

from noeira.deep_agents.dqn import DQNAgent
from noeira.deep_agents.sac import SACAgent
from noeira.deep_agents.ppo import PPOAgent
from noeira.deep_agents.muzero import MuZeroAgent
from noeira.deep_agents.dreamerv3 import DreamerV3Agent

Imitation and zero-shot code follows the same one-package-per-algorithm layout (noeira.deep_agents.act, .smolvla, .bc, .hil_serl, .fb, plus the shared .demos), but not the XxxAgent facade: their entry points are the drivers named on each page.

Networks arrive as compile-time parameters, so an agent does not know what architecture it is optimizing — see compile-time composition.

Tabular and linear agents live in a separate package with a simpler interface:

from noeira.agents import QLearningAgent, SARSAAgent, TiledQLearningAgent

Writing your own agent — the facade pattern in detail
Environments — what to train on
Planners — the search the model-based agents run at decision time
Imitation learning — policies from demonstrations
Traits — the interfaces involved