Classical policy gradient

StableTested and in active use. Breaking changes will be called out.

Everything on the TD methods page learns a value and derives a policy from it. These learn the policy directly, which is what makes continuous actions and stochastic policies natural.

They use linear function approximation, so the algorithm is visible without a network in the way — and they are the direct ancestors of PPO, A2C and the continuous-control family.

from noeira.agents import (
    REINFORCEAgent, REINFORCEWithEntropyAgent,
    ActorCriticAgent, ActorCriticLambdaAgent,
    A2CAgent, PPOAgent, PPOAgentWithMinibatch,
)

Papers

• Simple statistical gradient-following algorithms for connectionist reinforcement learningWilliams · 1992 · Machine Learning 8 · linkREINFORCE.
• Policy Gradient Methods for Reinforcement Learning with Function ApproximationSutton, McAllester, Singh, Mansour · 1999 · NeurIPS 1999 · linkThe policy gradient theorem.

REINFORCE

The original policy gradient. Run a full episode, then push up the probability of every action taken, weighted by the return that followed it.

Agent	Notes
REINFORCEAgent	Monte-Carlo policy gradient
REINFORCEWithEntropyAgent	plus an entropy bonus

Unbiased and very high variance — the whole episode’s return is credited to every action in it, including the ones that had nothing to do with it. That variance is the problem every later method attacks.

The entropy variant adds a bonus for keeping the policy stochastic, which prevents premature collapse onto a mediocre deterministic policy. The same idea, formalised, is SAC’s maximum-entropy objective.

Actor-critic

Agent	Notes
ActorCriticAgent	one-step actor-critic
ActorCriticLambdaAgent	with eligibility traces

Replace the Monte-Carlo return with a learned value estimate. The critic gives a baseline, so the actor is updated by the advantage — how much better the action was than the state’s average — rather than the raw return.

Much lower variance, at the cost of bias from the critic’s own error. That bias/variance dial is exactly what GAE’s λ exposes in PPO.

A2C and PPO (linear)

Agent	Notes
A2CAgent	advantage actor-critic
PPOAgent	clipped surrogate
PPOAgentWithMinibatch	with minibatch updates

compute_gae is shared with the deep PPO implementation, so the advantage estimation is literally the same code.

pixi run mojo run -I . examples/cartpole/cartpole_policy_gradient.mojo

Name collision

noeira.agents.PPOAgent is the linear one; noeira.deep_agents.ppo.PPOAgent is the neural one. The same applies to DDPGAgent, TD3Agent and SACAgent, which exist in both packages. Check the import path.

Continuous control (linear)

DDPGAgent, TD3Agent and SACAgent also exist in linear form. They work on Pendulum and are a good way to see each algorithm’s structure — twin critics, target smoothing, entropy temperature — before networks are involved.

pixi run mojo run -I . examples/solve_pendulum.mojo

The through-line

Reading these in order is the cheapest way to understand the deep algorithms:

Linear	Deep counterpart	What changed
REINFORCE	—	baseline added → actor-critic
Actor-critic	A2C	neural critic, batched envs
Linear PPO	PPO	neural nets, GAE, minibatch epochs
Linear SAC	SAC	twin critics, auto-tuned temperature

The algorithms are the same. What the deep versions add is a function approximator that scales — and every failure mode that comes with it.

Next

• PPO — the deep version
• A2C — the simplest deep policy gradient
• TD methods — the value-based side