Implementing and understanding actor-critic algorithms

Overview

Actor-critic methods combine the benefits of policy gradient methods (the actor) with value function approximation (the critic). The actor learns a policy, while the critic evaluates states to reduce variance in policy gradient estimates.

The Algorithm

Following Sergey Levine's lecture, the batch actor-critic algorithm works as follows:

Key Components

Actor: The policy network \(\pi_\theta(a|s)\) that maps states to action distributions.

Critic: The value network \(V_\phi(s)\) that estimates expected returns from a state.

Advantage: The advantage function \(A(s,a) = Q(s,a) - V(s)\) tells us how much better an action is compared to the average.

Why Actor-Critic?

• Lower variance: Using a learned baseline (critic) reduces variance compared to REINFORCE
• Online learning: Can update after each step, not just at episode end
• Continuous actions: Works well with continuous action spaces

Implementation: actorCritic.py

Dealbreaker - Lan et al.

Machine Learning based education problem is student-response modeling i.e., developing principled statistical models that (i) accurately predict unobserved student responses to questions and (ii) identify the latent concepts that govern correct or incorrect responses. The Rasch Model: simple yet effective for analyzing student-response data. This model characterizes the probability of a correct response as a function of two scalar parameters: the student's ability and the question's difficulty. Affine functions - characterize a student's probability of success on a question as an affine function of the student's knowledge on underlying concepts. Affine models allow weak knowledge of a concept to be erroneously covered up by strong knowledge of other potentially unrelated concepts. Affine models also fail to capture more complicated nonlinear dynamics underlying student responses.

Deep Q-Networks and Deep Recurrent Q-Learning

We read the DQN paper - Playing Atari with Deep Reinforcement Learning by Min et. al and a variation of the DQN algorithm, DQRN - Deep Recurrent Q-Learning for Partially Observable MDPs - by Hausknecht & Stone.

Here is an annotated version of the paper with some useful side notes and external references DQN and DQRN.

Some of the questions that came up while discussing these papers:

• How does the \(\gamma \) affect the policy that is learnt? Also, is there an optimal \(\gamma \) for every particular game?
• Further, how does one decide on the correct in a more strategic game like Dota versus in Cartpole. The main argument being that in cartpole, you always want to give your immediate rewards a lot of weight given the nature of the game. However, in a more strategic game, that is not necessarily the case.

Here's a cartpole demo with help from external implementations here. I have added some experiments surrounding the questions that came up during the reading group.