Actor-Critic Methods

Define the $k$-step advantage estimator:

$\hat{A}_t^{(k)} = \sum_{l=0}^{k-1} \gamma^l \delta_{t+l} = -V_\phi(s_t) + r_t + \gamma r_{t+1} + \cdots + \gamma^{k-1} r_{t+k-1} + \gamma^k V_\phi(s_{t+k})$

The GAE estimator is the $\lambda$-weighted average:

$\hat{A}_t^{\text{GAE}} = (1-\lambda) \sum_{k=1}^{\infty} \lambda^{k-1} \hat{A}_t^{(k)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}$

When $V_\phi = V^\pi$ (perfect critic), all estimators are unbiased. When $V_\phi \neq V^\pi$, the bias of the $k$-step estimator is $O(\gamma^k \|V_\phi - V^\pi\|_\infty)$. longer rollouts reduce bias from an imperfect critic. The variance grows with $k$ because more random rewards are included. $\lambda$ interpolates between these extremes.

A perfect critic gives zero-variance advantage estimates via one-step TD. An imperfect critic introduces bias, which you can reduce by relying more on actual observed rewards (larger $\lambda$). GAE provides a smooth dial between trusting the critic ($\lambda = 0$) and trusting the data ($\lambda = 1$). In practice, $\lambda = 0.95$ is a common default.

Without GAE, you must choose between one-step TD (biased but stable) and Monte Carlo returns (unbiased but noisy). GAE gives a principled middle ground and is used in every modern policy gradient implementation, including PPO.

Define the probability ratio $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$. The PPO-Clip objective is:

$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\Big( r_t(\theta) \hat{A}_t, \;\; \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \Big) \right]$

where $\epsilon$ (typically 0.1 to 0.2) limits how far the new policy can deviate from the old one in a single update.

Effect of the clipping:

• When $\hat{A}_t > 0$ (good action): the objective is $\min(r_t \hat{A}_t, (1+\epsilon)\hat{A}_t)$, capping the benefit of increasing the action probability
• When $\hat{A}_t < 0$ (bad action): the objective is $\min(r_t \hat{A}_t, (1-\epsilon)\hat{A}_t)$, capping the penalty reduction from decreasing the action probability

TRPO solves a constrained optimization problem (maximize surrogate subject to KL constraint), which requires second-order methods. PPO approximates this with a simple clipping trick: if the policy ratio moves too far from 1, the gradient is zeroed out, preventing destructively large updates. The result is a first-order algorithm that is nearly as stable as TRPO and much simpler to implement.

PPO is the algorithm used in RLHF for training instruction-following language models. Its simplicity (no KL constraint optimization, no conjugate gradients) makes it the default choice whenever policy optimization is needed. The clipping mechanism is the key innovation. It provides a soft trust region without the computational overhead of TRPO.

In maximum entropy RL, the objective augments the standard return with an entropy term:

$J(\pi) = \sum_{t=0}^{\infty} \gamma^t \mathbb{E}\left[ r_t + \alpha \, H[\pi(\cdot|s_t)] \right]$

The corresponding soft Bellman equation is:

$V^\pi(s) = \mathbb{E}_{a \sim \pi}\left[ Q^\pi(s,a) - \alpha \log \pi(a|s) \right]$

$Q^\pi(s,a) = R(s,a) + \gamma \mathbb{E}_{s' \sim P}\left[ V^\pi(s') \right]$

The optimal policy is the Boltzmann distribution: $\pi^*(a|s) \propto \exp(Q^*(s,a)/\alpha)$.

SAC adds a reward for being uncertain. The agent is encouraged to find all good actions, not just the single best one. This leads to better exploration and more robust policies. The temperature $\alpha$ controls how much randomness the policy retains. high $\alpha$ means more exploration, low $\alpha$ approaches the standard (deterministic) optimal policy.

SAC achieves state-of-the-art performance on continuous control benchmarks (robotic locomotion, manipulation). Its key advantages: (1) it is off-policy (uses a replay buffer, so it is sample-efficient), (2) the entropy bonus provides automatic exploration, and (3) it avoids the brittle hyperparameter tuning of epsilon-greedy or noise-based exploration.