## Introduction

The activation function is the most critical element in determining whether or not a neuron will be activated and transmitted to the next layer in a neural network. This implies that throughout the prediction phase, it will determine if the neuron's input to the network is meaningful or not. There are several activation functions to select from, and it can be tough to figure out which one would work best for their purposes.

In this blog, we'll look at different activation functions and give instances of when they should be employed in various types of neural networks. But before that, let's get hold of the fundamentals of activation functions.

Also Read, __Resnet 50 Architecture__

## What is the Activation Function?

Activation Functions are a part of neural networks; thus, understanding the basics of neural networks before understanding activation functions is necessary. We will understand the architecture using a quick summary, and further, we will use the knowledge acquired in understanding the activation function.

The image given below is a neural network along with its supporting components. We can see several neurons being connected, forming a significant web-like connection of several interconnected neurons, which are characterized based on three things, i.e., weight bias and activation function.

Source: __Link__

To learn about neural networks, you can follow up with the __blog__ that describes the above diagram and provides a more extensive overview of neural networks and their basics.

Getting back to activation functions, the above diagram can be visualized in the following way to the working of activation functions.

Source:__ Link__

From the above diagram, we can conclude about activation function that It's a Neural Network function that calculates the weighted total and adds bias to it. It also causes non-linearity in a neuron's output due to the no-linear change of input induced by the activation function, making the neural networks are capable of training and executing complex operations.

Talking about feedforward propagation and backpropagation of the activation function. The Activation Function is simply a "gate" between input feeding the existing neuron and its output flowing to the next layer in feedforward propagation.

When we talk about backpropagation by altering the network's weights and biases, backpropagation attempts to minimize the cost function. The amount of modification concerning parameters such as the activation function, weights, bias, and so on is determined by the cost function gradients.