Boltzmann Machine - Introduction

Objective of Boltzmann Machine

The fundamental goal of a Boltzmann Machine is to optimize a problem solution. The Boltzmann Machine's job is to optimize the weights and quantities associated with that particular challenge.

Architecture of Boltzmann Machine

In Boltzmann Machine, we have two layers: the hidden layer and the visible layer.

In this network, the blue neurons (h₁, h₂, h₃)  are the hidden layer, and the white neurons (v₁, v₂, v₃, v₄) are visible. Visible neurons are nothing but the input layer. Each neuron is connected to every neuron, and even the input neurons are connected. The weights of neurons are used to represent the cost function of neurons. For example, the weight between v₁ and v₄, namely W_1,4 will define the cost function of V₁ and V₄ neurons. Remember, Boltzmann Machine is only accountable for finding the relation between the input layers using the features, not the output.

Training Algorithm

Because Boltzmann machines have fixed weights, there will be no training algorithm because the weights in the network do not need to be updated. However, before we can test the network, we must first set the weights and determine the consensus function CF.

The Boltzmann machine features bidirectional connections on its units U_i and U_j

• We're thinking about the fixed weight, w_ij.
• w_ij ≠ 0 if U_i and U_j are connected.
• There is also symmetry in weighted interconnectivity, i.e.,  w_ij = w_ji
• There is also w_ii which is self-connection between units.
• For any unit U_i, its state u_i would be either 1 or 0.
   

The primary goal of the Boltzmann Machine is to maximize the Consensus Function CF, which can be calculated using the following relation.

When the status changes from 1 to 0 or 0 to 1, the change in consensus can be described by the following relation:

Here, u_i represents the current state of U_i

The following relationship describes the variance in coefficient (1 - 2ui).

In general, unit Ui does not change its state, but if it does, the information is stored locally in the unit. The network's consensus would increase as a result of this adjustment.

The following relation expresses the probability of the network accepting a change in the unit's state.

Here, T is the controlling parameter. It will decrease as CF reaches the maximum value.

Testing Algorithm

Step 1 − To begin the training, use the following commands.

• Weights expressing the problem's restriction
• T is a control parameter.
   

Step 2 − When the stopping condition is not met, proceed to steps 3-8.

Step 3 − Steps 4–7 must be completed.

Step 4 −Assume that one of the states has changed the weight and randomly pick the integers I and J between 1 and n.

Step 5 −Calculate the difference in consensus as follows:

Step 6 −Determine the likelihood that this network will accept the state change.

Step 7 −Accept or reject the following change:

Case I − if R < AF, accept the change.

Case II − if R ≥ AF, reject the change.

R represents a random number between 0 and 1.

Step 8 − Reduce the temperature of the control parameter as follows

T new = ⁡0.95T old

 

Step 9 − Check for the following possible stopping conditions:

• The temperature has reached a predetermined level.
• For a set number of iterations, there is no change in state.

Energy-Based Models

The Boltzmann Distribution is employed in the Boltzmann Machine's sampling distribution. The equation governs the Boltzmann distribution.

Pi = e(-∈i/kT)/ ∑e(-∈j/kT)         
∈i - Energy of system in state i
Pi - probability of system being in state i
k - Boltzmann constant
T - Temperature of the system
∑e(-∈j/kT) - Sum of values for all possible states of the system

 

The Boltzmann Distribution explains multiple states of the system; hence, Boltzmann machines use this distribution to construct different states of the machine. According to the preceding equation, as the system's energy grows, the chance of the system being in the state I decrease. As a result, the system is most stable in its lowest energy state (a gas is most stable when it spreads). In Boltzmann machines, the system's energy is defined in synaptic weights. Once trained and the weights are established, the system will always try to reach the lowest energy state by altering the weights.

There are three types of Boltzmann machines

Restricted Boltzmann machine

A restricted term means we cannot connect the same type of layer. In other words, the two neurons in the input or hidden layer cannot communicate even though the hidden and visible layers can be linked. Because there is no output layer in this machine, the question of how we will identify, update the weights, and measure whether our prediction is correct or not arises. All of the questions have a single answer: Restricted Boltzmann Machine.Geoffrey Hinton (2007) proposed the RBM algorithm, which learns probability distributions from sample training data inputs. It has widespread use in supervised and unsupervised machine learning applications such as feature learning, dimensionality reduction, classification, collaborative filtering, and topic modeling                  

Deep Boltzmann machine

As shown in Fig, a deep Boltzmann machine is a model with additional hidden layers and directionless connections between the nodes. DBM learns features from raw data in a hierarchical manner, and the features recovered in one layer are applied as hidden variables as input to the subsequent layer. To define the training information, weight initialization, and adjustment parameters, the DBM training method must be modified. According to the DBM, temporal complexity limitations will occur when the parameters are set to ideal. Montavon et al presented a centering optimization strategy to make the learning process more robust and for midsized DBM to construct a generative, quicker, and discriminative model.                         

Deep Belief Network

The Deep Belief Network is one of the types of Boltzmann machine. It is a generative model which uses multiple stacks of the deep architecture of the Restricted Boltzmann Machine. Each restricted Boltzmann machine performs a non-linear transformation on the input neurons and produces the outputs that serve as the input for the consecutive model. Deep Belief Networks can act supervised or unsupervised as they have a generative model. Due to it, Deep Belief Networks has a lot of flexibility, and it is easier to expand.  

Applications of Boltzmann Machine

Boltzmann Machine is used for solving various types of real-world problems:

A search problem

In the Boltzmann machine, the wights of the neuron network are fixed. They are used to represent the cost function of an optimization problem. The Boltzmann machine's stochastic dynamics allow the sample of the binary vectors and represent an excellent solution to the optimization problem.

A learning problem

The Boltzmann machine is given a set of binary vectors for learning problems. It finds the weights of each network so that it can provide a better solution. To solve this problem, the Boltzmann machine takes minute updates to the importance, and each update needs to solve various search problems. The Boltzmann device has the first to solve multiple minor search problems to solve a learning problem.

HandWritten Digit Recognition

Many applications, including data entry, office computerization, check verification, and criminal evidence, use it as a prevalent problem. Additionally, it has drawbacks, including inconsistent writing styles, size and shape inconsistencies, and picture noise that alters the topology of the numerals. For digit recognition in this, a hybrid RBM-CNN algorithm is applied. First, RBM deep learning techniques are used to extract the features. The CNN deep learning system is then fed the retrieved features for categorization. The ability to extract features from input data is a strong suit of RBMs. By introducing hidden units in an unsupervised manner, it is created in a way that it can extract the discriminative characteristics from vast and complex datasets.

Implementation of the Boltzmann Machine

#importing necessary libraries
import numpy as np

#importing the inbuilt Boltzmann Machine
from sklearn.neural_network import BernoulliRBM

#Defining dataset
X = np.array([[1, 1, 1], [0, 1, 0], [0, 0, 0], [1, 0, 1]])
model = BernoulliRBM(n_components=2)
model.fit(X)

 

Output

BernoulliRBM(n_components=2)

Check out this article - Padding In Convolutional Neural Network

Frequently Asked Questions

How many layers are there in the Boltzmann Machine?

There are two layers: the hidden layer and the visible layer.

Are Boltzmann machines still used?

RBMs are generally not used currently.

What does the Boltzmann machine do?

Recurrent neural networks with Boltzmann machines have nodes that make binary decisions with some bias. Boltzmann machines can be connected to create more complex systems, like deep belief networks.

What is the other name of the Boltzmann machine? 

A stochastic Hopfield network with hidden units is another name for a Boltzmann machine.

Where is the Boltzmann machine used?

The machine is used to solve computational problems.

Conclusion

The Boltzmann Machines can be considered an advanced neural network. They possess numerous qualities and many uses currently being applied in modern technology. This article understood what a Boltzmann Machine is and its architecture. We also explored its application, implementation, and how it can be created using the Python library.

Here are a few key websites that will aid your exploration of the Boltzmann Machine.

• Restricted Boltzmann Machine
• Types of Boltzman Machine
• Dense in Deep Learning
• Instruction Format in Computer Architecture

 

You can also consider our Online Coding Courses such as the Machine Learning Course to give your career an edge over others.

Do upvote our blogs if you find them helpful and engaging!

Happy Learning!!