DDPG

StableTested and in active use. Breaking changes will be called out. Works, but TD3 and SAC supersede it. Use those for results.

Deep Deterministic Policy Gradient is the ancestor of the off-policy continuous-control family. A deterministic actor, a single critic, and Gaussian noise for exploration.

from noeira.deep_agents.ddpg import DDPGAgent

The idea

For discrete actions, DQN picks the best action by taking a max over the network’s outputs. With continuous actions that maximisation is itself an optimization problem — you cannot enumerate the options.

DDPG’s answer: train a network to output the maximising action. The actor is updated by pushing its actions toward higher critic values, backpropagating through the critic into the policy.

actor loss  = −Q(s, π(s))          # maximise the critic's estimate
critic loss = (Q(s,a) − y)²        # y = r + γ·Q′(s′, π′(s′))

That trick — differentiate through the critic to improve the actor — is what every algorithm in this family inherits.

environment interactionlearning updatetarget network

TD targety = r + γ · Q′(s′, π′(s′))

Why the chain rule is the whole algorithm

The deterministic policy gradient theorem says the objective’s gradient with respect to the actor’s parameters factors through the action:

∇_θ J = 𝔼_s [ ∇_a Q(s, a)|_(a = π(s)) · ∇_θ π(s) ]

Read it right to left. The actor produces an action; the critic reports which direction in action space increases value; the chain rule carries that direction back into the actor’s weights. The critic is not just a baseline here, as it is in A2C — it is the source of the gradient. If the critic’s action-space slope is wrong, the actor is not slowed down, it is pointed the wrong way.

That distinction explains why this family is so much more sensitive to critic quality than policy-gradient methods, and why every successor (TD3, SAC, REDQ) attacks the critic rather than the actor.

Exploration is external: Gaussian noise added to the actor’s output, since a deterministic policy explores nothing on its own. The original paper used temporally correlated Ornstein-Uhlenbeck noise; uncorrelated Gaussian noise was later shown to work as well or better, and is what this implementation uses — noise_scale, default 0.1, in units of the action scale.

Building it

from noeira.deep_agents.ddpg import DDPGAgent
from noeira.deep_agents.training.blocks import UniformSampleGpuStep

var agent = DDPGAgent[
    "gpu",
    UniformSampleGpuStep[OBS_DIM, ACT_DIM, BATCH, REPLAY_CAPACITY],
    ActorNet,
    CriticNet,
](
    ctx=ctx,
    actor_lr=1e-4,      # note: lower than the critic's
    critic_lr=1e-3,
    gamma=0.99,
    tau=0.005,
    noise_scale=0.1,    # exploration σ, in units of action_scale
)

The actor is a deterministic network ending in Tanh scaled to the action range — not a StochasticActor, which is what SAC uses.

The asymmetric learning rates are deliberate, and they are the one place DDPG’s defaults differ from TD3’s and SAC’s 3e-4/3e-4. Since the actor’s gradient is manufactured by the critic, a fast actor chasing a slow critic compounds whatever the critic has got wrong. TD3 replaces this soft mitigation with an explicit one — updating the actor every other step — and can then afford equal rates.

pixi run mojo run -I . examples/pendulum_ddpg.mojo
pixi run mojo run -I . examples/half_cheetah/ddpg_half_cheetah_training.mojo

The problem

DDPG is brittle, and understanding why explains the two algorithms that follow.

The critic is trained on targets built from its own estimates at the actor’s chosen action. Wherever the critic happens to overestimate, the actor moves toward that region — because that is where the critic says the value is high. The next target is then computed there, reinforcing the error.

This is not noise that averages out. The max-like structure of the update selects for optimistic error. A run can look healthy for a long time and then collapse.

Fujimoto et al. made the mechanism precise. Write the critic’s error at the actor’s action as ε, with mean zero. Because the actor is trained to maximise Q, it is drawn to states and actions where ε happens to be positive — so the expectation of ε conditioned on the actor’s own choice is not zero:

𝔼[ Q(s, π(s)) ]  >  𝔼[ Q^π(s, π(s)) ]

The bias enters the target, the target trains the critic, the critic re-aims the actor. Each of the three fixes in TD3 breaks one link of that chain: the twin-min attacks the bias itself, the delay slows the loop, and target smoothing removes the narrow peaks the actor was exploiting.

TD3 fixes this with three targeted changes; SAC fixes it differently, with a stochastic policy and an entropy term.

When to use it

Honestly: to understand the family. TD3 is DDPG with the known fixes applied and is strictly better; SAC is better again on most tasks.

DDPG remains useful as a baseline — if a new environment trains under TD3 but not DDPG, that is informative about how much the stabilisation matters there.

TD3 — the three fixes
SAC — the recommended default
Traits — the environment interface for continuous actions