Implementing and Understanding Actor Critic Algorithms

I was following Seregey Levine's lecture and using the following pseudocode on page 19 to implement the algorithm: