Finding Gradient in Tensorflow using tf.GradientTape

What is tf.gradientTape?

GradientTape() is used to record operations for automatic differentiation

Syntax: tensorflow.GradientTape( persistent, watch_accessed_variables)

Persistent (optional): It can either be True or False with the default value False. It defines whether persistent gradient tape is created or not.

watch_accessed_variables: It is a boolean defining whether the tape will automatically watch any (trainable) variables accessed while the tape is active or not.

Example 1

# Importing the library
import tensorflow as ttf
  
x = ttf.constant(5.0)
  
# Using GradientTape
with ttf.GradientTape() as CN:
  CN.watch(x)
  y = x * x * x
  
# Computing gradient
res  = CN.gradient(y, x)
  
# Printing result
print("result: ",res) 

Output

Example 2 

# Importing the library
import tensorflow as ttf
  
p = ttf.constant(4.0)
  
# Using GradientTape
with ttf.GradientTape() as CN:
  CN.watch(p)
  
  # Using nested GradientTape for calculating higher order derivative
  with ttf.GradientTape() as CN:
    CN.watch(x)
    y = p * p * p
  # Computing first order gradient
  first_order_derivative = gg.gradient(y, p)
  
# Computing Second order gradient
second_order_derivative  = gfg.gradient(first_order, p) 
  
# Printing result
print("first_order_derivative: ",first_order)
print("second_order_derivative: ",second_order)

Output 

Various Methods in tf.GradientTape

Introduction 

We will track the computations and compute gradients with tf.gradientTape as follows:   

p = tf.Variable(2.0, trainable=True)
with tf.GradientTape() as tape:
    y = p**3


print(tape.gradient(y, p).numpy()) 

Output

• GradientTape doesn't track constants by default so we will instruct it with: tape.watch(variable)
• We perform some computations ranging from cubing it to passing it to the neural networks. 
   

If, at any point, we want to use multiple variables in our calculations, we will pass a list or tuple to those variables.  

Automatically Watched Variables 

GradientTape automatically watches all trainable variables, so if x is a trainable variable instead of a constant, there is no need to instruct the tape to watch it. 

import tensorflow as ttf
p = ttf.constant(2.0)
with ttf.GradientTape() as tape:
    tape.watch(p)
    y = p**3
print(tape.gradient(y, p).numpy())

Output

If we will re-run this by replacing the first line with:

x = tf.constant(4.0)

or

x = tf.Variable(4.0, trainable=False)

The code would raise an error, as GradientTape wouldn't be watching x.

Watch_accessed_variables = False 

We can set the GradientTape watch variable to false if we want so that it cannot watch all training variables automatically.  

import tensorflow as ttf
p = ttf.Variable(2.0, trainable=True)
with ttf.GradientTape(watch_accessed_variables=False) as tape:
    y = p**3


print(tape.gradient(y, p))

We can get fine control over what variables we want to watch by disabling watch_accessed_variables 

If you have a lot of trainable variables and are not optimizing them all at once, you should disable watch_accessed_variables to protect yourself from mistakes.

Output

Persistent = True 

If we run the given code: 

import tensorflow as tf
x = tf.Variable(5.0, trainable=True)
y = tf.Variable(3.0, trainable=True)
with tf.GradientTape() as tape:
    y1 = x ** 2
    y2 = y ** 3
print(tape.gradient(y1, x).numpy())
print(tape.gradient(y2, y).numpy())

Expected Output should be:

10

27 

But actually when we call tapeGradient second time, will raise an error, to resolve this error we need to set the persistent = true. 

import tensorflow as tf
x = tf.Variable(5.0, trainable=True)
y = tf.Variable(3.0, trainable=True)
with tf.GradientTape(persistent=True) as tape:
    y1 = x ** 2
    y2 = y ** 3
                                                                                                                                                                                                                                                                                                                                             
print(tape.gradient(y1, x).numpy())
print(tape.gradient(y2, y).numpy())

Output 

Stop_Recording() 

It will temporarily pause the recoding, leading to greater computation speed. 

import tensorflow as ttf
p = ttf.Variable(5.0, trainable=True)
with ttf.GradientTape() as tape:
    y = p**2
    with tape.stop_recording():
        print(tape.gradient(y, p).numpy()) 

Frequently Asked Questions 

What is TensorFlow? 

A Python-based library called TensorFlow is used to build machine learning programs. It is a basic toolkit for performing complicated math. It provides users with the flexibility to create experimental learning architectures. 

Name three Working components of TensorFlow architecture. 

Preprocessing the data -> Building the model >Training the model 
 

What is gradient descent? 

To put it simply, it is a numerical method for identifying the inputs to an equation system that minimizes its Output. This set of equations represents our model in the context of machine learning; the inputs are the model's unobserved parameters, and the Output is a loss function that must be minimized to reflect the amount of error between the model and our data. 

What are the variables in TensorFlow? 

Variables in TensorFlow are objects that can change their values in response to state changes. At the time of execution, it prints the most recent value. It begins when tf.variable.initializer initializes a variable.

What is mnist Dataset? 

Handwriting recognition data from the MNIST (Modified National Institute of Standards and Technology database) to support visual image recognition. 

Conclusion

In this blog, we discussed the introduction to automatic differentiation, and then we discussed the tf.gradientTape and the various methods in tf.gradientTape. 

Refer to our guided paths on Coding Ninjas Studio to learn more about DSA, Competitive Programming, JavaScript, System Design, etc. Enroll in our courses and refer to the mock test and problems available; look at the Top 150 Interview Puzzles,  interview experiences, and interview bundle for placement preparations. Read our blogs on aptitude, competitive programming, interview questions, IT certifications, and data structures and algorithms for the best practice.