Introduction to PolicyBased Reinforcement Learning
A policy is a function that returns a probability distribution. The probability distribution is calculated over the actions based on the given state that the agent can take. Another function named value determines the expected return for an agent acting according to a particular policy and starting at a given state.
In the policybased methods, we directly learn the policy function instead of learning a value function. It means that instead of learning the expected sum of rewards given a state and an action, we directly learn the function that maps state to action. In other words, in policybased reinforcement learning, we directly manipulate the policy to find the optimal policy.
Pseudocode to ActorCritic Method
Follow the listed steps for the pseudocode of the actorcritic model algorithm.

From the actorcritic network, get the policy.

Now, using this policy sample {s_t, a_t}

Since in the actorcitric model, the advantage function is generated by the critic network, so using it evaluates the critic function. Let the critic function be A_t. It can also be called a t (Temporal Difference error).

Using the equation below, evaluate the gradient.

The next step now is to update, the parameter for the policy.
Implementation and building CartPole Game
Before implementing and building the CartPole game, let us first begin with defining a CartPole game and knowing what a CartPole game is.
What is the CartPole game?
It is a game that consists of a pole attached to a cart that has to move on a frictionless track. For this movement, the agent has to apply force on it. Whenever the pole is upright, you receive a reward for that time stamp. So, to be successful in the game, the agent has to learn how to save the pole from falling over.
Implementing the game
Letâ€™s implement the game, but first, we need to set up our environment. For the environment import the required packages and configure the global changes in the system. Then define the model and implement the game.
import gym
import numpy as npy
import tensorflow as tsf
from tensorflow import keras
from tensorflow.keras import layers
# Configuration for the setup
seed = 42
gamma = 0.99 # Discount factor for past rewards
max_steps_per_episode = 10000
env = gym.make("CartPolev0") # Creation of the environment
env.seed(seed)
eps = npy.finfo(npy.float32).eps.item() # Smallest number such that 1.0 + eps != 1.0
num_inputs = 4
num_actions = 2
num_hidden = 128
inputs = layers.Input(shape=(num_inputs,))
common = layers.Dense(num_hidden, activation="relu")(inputs)
action = layers.Dense(num_actions, activation="softmax")(common)
critic = layers.Dense(1)(common)
model = keras.Model(inputs=inputs, outputs=[action, critic])
optimizer = keras.optimizers.Adam(learning_rate=0.01)
huber_loss = keras.losses.Huber()
action_probs_history = []
critic_value_history = []
rewards_history = []
running_reward = 0
episode_count = 0
while True: # Run until solved
state = env.reset()
episode_reward = 0
with tsf.GradientTape() as tape:
for timestep in range(1, max_steps_per_episode):
state = tsf.convert_to_tensor(state)
state = tsf.expand_dims(state, 0)
# Predicting the action probabilities and estimated future reward from the environment state
action_probs, critic_value = model(state)
critic_value_history.append(critic_value[0, 0])
# Sampling action from the action probability distribution
action = npy.random.choice(num_actions, p=npy.squeeze(action_probs))
action_probs_history.append(tsf.math.log(action_probs[0, action]))
# Applying sampled action in our environment
state, reward, done, _ = env.step(action)
rewards_history.append(reward)
episode_reward += reward
if done:
break
# Updating the running reward to check the condition
running_reward = 0.05 * episode_reward + (1  0.05) * running_reward
# Calculating the expected value from the rewards at each timestep.
#  These are the labels for our critic
returns = []
discounted_sum = 0
for r in rewards_history[::1]:
discounted_sum = r + gamma * discounted_sum
returns.insert(0, discounted_sum)
# Normalizing
returns = npy.array(returns)
returns = (returns  npy.mean(returns)) / (npy.std(returns) + eps)
returns = returns.tolist()
# Calculating the loss values
history = zip(action_probs_history, critic_value_history, returns)
actor_losses = []
critic_losses = []
for log_prob, value, ret in history:
# The actor must be updated to predict an action which will leads to high rewards (compared to the critic's estimate) with higher probability.
diff = ret  value
actor_losses.append(log_prob * diff) # actor loss
# We must update the critic to predict a better estimate of the future rewards.
critic_losses.append(
huber_loss(tsf.expand_dims(value, 0), tsf.expand_dims(ret, 0))
)
# Backpropagation
loss_value = sum(actor_losses) + sum(critic_losses)
grads = tape.gradient(loss_value, model.trainable_variables)
optimizer.apply_gradients(zip(grads, model.trainable_variables))
# Clearing the loss and reward history
action_probs_history.clear()
critic_value_history.clear()
rewards_history.clear()
# Log details
episode_count += 1
if episode_count % 10 == 0:
template = "running reward: {:.2f} at episode {}"
print(template.format(running_reward, episode_count))
if running_reward > 195: # Condition to consider the task solved
print("Solved at episode {}!".format(episode_count))
break
Frequently Asked Questions
What is the temporal difference?
The temporal difference is a class of modelfree that deals with predicting a variableâ€™s expected value in any sequence of states.
What is a policy?
A policy is a function that returns a probability distribution. The probability distribution is calculated over the actions based on the given state that the agent can take.
Define actor and critic in the actorcritic model.
The part of the agent responsible for the recommended action is called the actor. The part of agent responsible for the estimated rewards of the future is called the critic.
What is the CartPole game about?
It is a game that consists of a pole attached to a cart that has to move on a frictionless track. To be successful in the game, the agent has to learn how to save the pole from falling over.
Conclusion
In this article, we have discussed the Actor cryptic model. We started with policybased reinforcement learning, then learned the pseudo code and implementation of the actorcritic model.
We hope this blog was useful and helped you enhance your knowledge of the actorcritic model, and if you want to learn more, check out our articles on Introduction to Reinforcement Learning and Four Types of Learnings in Machine Learning. Do upvote our blogs if you find them useful.
Happy Coding!