Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Table of contents
1.
Introduction
2.
Understanding Eyeball Tracking
3.
What is OpenCV?
4.
How does it work?
5.
Eye Tracking Techniques with OpenCV
5.1.
Pupil Tracking
5.2.
Gaze Estimation
5.3.
Feature-based Tracking
5.4.
Machine Learning-based Approaches
6.
Implementation
6.1.
Code
6.2.
Python
6.3.
Output
7.
Applications
7.1.
Human-Computer Interaction (HCI)
7.2.
Usability Testing and Market Research
7.3.
Neuroscience and Psychology
7.4.
Virtual and Augmented Reality (VR/AR)
8.
Challenges and Future Scope
9.
Frequently Asked Questions
9.1.
Does eye tracking using OpenCV require specialised hardware?
9.2.
Is eye tracking using OpenCV suitable for mobile or embedded systems?
9.3.
Can OpenCV track multiple eyes simultaneously?
9.4.
What kind of accuracy can we achieve with eye tracking using OpenCV?
9.5.
Are there any privacy concerns associated with eye tracking using OpenCV?
10.
Conclusion
Last Updated: Mar 27, 2024
Easy

Eye Ball Tracking with OpenCV

Author Juhi Pathak
1 upvote

Introduction

Eye-tracking technology has emerged as a powerful tool in various domains. They range from psychology to human-computer interaction (HCI). Eye tracking is commonly seen in Virtual Reality. By analysing eye movements, researchers gain valuable insights. They capture the cognitive processes, focus, and user behaviour.

Eye Ball Tracking with OpenCV

This article will cover Eye Ball Tracking with OpenCV- its procedure, techniques, applications, challenges, and future scope of eyeball tracking using OpenCV.

Understanding Eyeball Tracking

Eyeball tracking is a technique that involves analysing eye movements. It measures gaze patterns as well. The human eye exhibits several distinct movements. It displays fixations, saccades, and smooth pursuits. These movements can reveal a lot about an individual's attention. Researchers get to know about one's load and interest. Eye tracking systems use specialised hardware to track eye movement better.

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

What is OpenCV?

OpenCV is Open Source Computer Vision Library. It is a versatile machine-learning software library. It has a wide range of tools and algorithms for visual analysis. This makes it a powerful resource for developing CV applications. It is written in C++ and has bindings for Python and other languages. OpenCV plays a significant role in eye tracking. It provides the necessary tools and algorithms for eyeball tracking. They process eye images and extract relevant information.

How does it work?

OpenCV plays a significant role in eye tracking by providing the necessary tools and algorithms for processing eye images and extracting relevant information. Here's how OpenCV helps in eye tracking:
 

  • Image Processing

OpenCV offers an extensive set of image-processing functions. These functions enable the extraction of features from eye images. They include image filtering, thresholding, edge detection, and morphological operations. By applying these techniques, OpenCV can perform various tasks. It enhances eye images, segment relevant regions, and remove noise.
 

  • Feature Detection

OpenCV provides algorithms for detecting distinctive features in the eyes. These features include the pupil, iris, eye corners, and other key points. Feature detection is essential in eyeball tracking. It allows the identification of specific regions within the eye. OpenCV's feature detection algorithms can track eyes across video frames—for example- Kanade-Lucas-Tomasi (KLT) tracker.
 

  • Machine Learning Integration

We can integrate OpenCV with machine learning algorithms. This results in improved accuracy and robustness of eye-tracking systems. Machine learning techniques can learn from large datasets and detect eye features. OpenCV provides functions for training and utilising machine learning models. It allows for the development of advanced eye-tracking algorithms.
 

  • Gaze Estimation

Gaze estimation is a crucial aspect of eye tracking. OpenCV can estimate the direction of a person's gaze. It combines geometric and trigonometric calculations. It analyses the positions of the pupil and other eye features. This determines the point on a screen that the person is looking at. This ability enables gaze-based interaction in apps. This is present in HCI and VR.
 

OpenCV provides an extensive toolkit for eye-tracking applications. Its rich features make it vital for making efficient eye trackers. By leveraging its power, developers can gain varied insights.

Eye Tracking Techniques with OpenCV

There are multiple approaches to eye tracking using OpenCV. Each has its strengths and limitations. Following are a few popular techniques:

Pupil Tracking

Pupil tracking is one of the fundamental steps in eye tracking. OpenCV provides algorithms for detecting and tracking the pupil region. It is dependent on image processing techniques. They may be thresholding. Contour detection and morphological operations are other methods. These algorithms analyse the brightness and shape of the pupil. This is done to estimate its accurate position.

Gaze Estimation

Gaze estimation determines the direction in which a person is looking. Geometric calculations perform pupil tracking. This enables gaze-based interaction. OpenCV can estimate the gaze point on a screen or in a 3D space. HCI, VR, and AR applications use this technique.

Feature-based Tracking

OpenCV's feature-based tracking algorithms can track distinctive eye features. For instance, corners or key points. By monitoring these features across video frames, gaze patterns turn observable. This method analyses eye movements while reading or object recognition. An example is the Kanade-Lucas-Tomasi (KLT) tracker.

Machine Learning-based Approaches

OpenCV with ML algorithms improves the robustness of eye-tracking systems. The accuracy is further increased. CNN and other DL models help to automate the detection of eye features. These approaches learn from large datasets. They can handle variations in lighting, head movements, and occlusions.

Implementation

1. Install Python

Make sure you have Python installed on your computer. You can download Python from the official website (https://www.python.org/downloads/) and follow the installation instructions for your operating system.
 

2. Install OpenCV

To use the computer vision capabilities required for eye tracking, you need to install the OpenCV library. Open a terminal or command prompt and run the following command to install OpenCV using pip (Python package manager):

pip install opencv-python


3. Download the Haar Cascades

The eye tracking code uses pre-trained Haar cascades for face and eye detection. These files contain the patterns that help the computer recognize faces and eyes. You need to download two XML files, and here's how:

a. Download the 'haarcascade_frontalface_default.xml' file.

b. Download the 'haarcascade_eye.xml' file.


After downloading, save these XML files to your local machine in a folder of your choice. Remember the path where you save them, as you'll need it in the code.
 

4. Connect a Webcam (Optional)

If you want to use a live webcam feed for eye tracking, make sure you have a webcam connected to your computer. If you don't have a webcam, you can use a pre-recorded video file for testing.
 

5. Open a text editor

Open a text editor like Notepad (Windows), TextEdit (Mac), or any code editor like Visual Studio Code, PyCharm, or Sublime Text.
 

6. Copy the Code

Copy the provided Python code for eye tracking into the text editor.
 

7. Modify the File Paths (Optional)

If you saved the 'haarcascade_frontalface_default.xml' and 'haarcascade_eye.xml' files in a different folder, make sure to update the respective paths in the code:

face_cascade = cv2.CascadeClassifier('path/to/haarcascade_frontalface_default.xml')
eye_cascade = cv2.CascadeClassifier('path/to/haarcascade_eye.xml')

 

8. Save the File

Save the code file with a ".py" extension, for example, "eye_tracking.py".
 

9. Run the Code

Open a terminal or command prompt and navigate to the folder where you saved the Python code. Type the following command to execute the code:

python eye_tracking.py

 

10. Enjoy Eye Tracking

Once the code starts running, you should see a window displaying the video feed from your webcam (if you're using one). The code will detect faces and eyes in real-time and draw rectangles around them.
 

11. Stop the Code

To stop the eye tracking, press the 'q' key on your keyboard. The video feed window will close, and the code will stop running.

 

That's it! You've successfully run the eye tracking code and experienced the magic of computer vision with OpenCV. Have fun exploring this exciting field and discovering what else computers can "see" in the world around us!

Code

  • Python

Python

import cv2

# For the face and eye detected load the Haar cascades
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
eye_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_eye.xml')

# Function to detect eyes and draw red dots at the center of the eyes.
def detect_eyes_in_face(gray_frame, colored_frame):
	# Detect faces in the grayscale frame
	faces = face_cascade.detectMultiScale(gray_frame, scaleFactor=1.3, minNeighbors=5)

	# Iterate through the detected faces
	for (x, y, w, h) in faces:
		# Extract the region of interest (ROI) for face and eyes
		roi_gray = gray_frame[y:y + h, x:x + w]
		roi_color = colored_frame[y:y + h, x:x + w]

		# Detect eyes in the ROI
		eyes = eye_cascade.detectMultiScale(roi_gray)

		# Iterate through the detected eyes
		for (ex, ey, ew, eh) in eyes:
			# Draw a green rectangle around each eye
			cv2.rectangle(roi_color, (ex, ey), (ex + ew, ey + eh), (0, 255, 0), 2)

			# Calculate the center of the eye
			center = (x + ex + ew // 2, y + ey + eh // 2)

			# Draw a red dot at the center of the eye
			cv2.circle(colored_frame, center, 2, (0, 0, 255), -1)

	# Return the colored frame with the eye tracking annotations
	return colored_frame

def main():
	# Display mode options
	print("Choose mode:")
	print("1. Webcam mode")
	print("2. Static image mode")

	# Get user choice
	choice = input("Enter your choice (1 or 2): ")

	if choice == '1':
		# Open webcam
		webcam_capture = cv2.VideoCapture(0)

		print("To quit webcam eyeball tracking window, press 'Q'.")

		while True:
			# Read a frame from the webcam
			ret, frame = webcam_capture.read()

			# Convert the frame to grayscale
			gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

			# Perform eye tracking and display the result
			result_frame = detect_eyes_in_face(gray_frame, frame)
			cv2.imshow('Eyeball Tracking - Webcam Mode', result_frame)

			# Break the loop when 'q' is pressed
			if cv2.waitKey(1) & 0xFF == ord('q'):
				break

		# Release the webcam and close all windows
		webcam_capture.release()
		cv2.destroyAllWindows()

	elif choice == '2':
		# Get the path to the static image
		image_path = 'image.png'

		# Read the static image from file
		static_frame = cv2.imread(image_path)

		# Check if the image was read successfully
		if static_frame is None:
			print("Error: Unable to read the image file.")
			return

		# Convert the static frame to grayscale
		gray_static_frame = cv2.cvtColor(static_frame, cv2.COLOR_BGR2GRAY)

		# Perform eye tracking on the static image and display the result
		result_static_frame = detect_eyes_in_face(gray_static_frame, static_frame)
		cv2.imshow('Eyeball Tracking - Static Image Mode', result_static_frame)
		cv2.waitKey(0)
		cv2.destroyAllWindows()

	else:
		print("Invalid choice. Please enter '1' for Webcam mode or '2' for Static image mode.")

	print("Thanks for using Eyeball Tracking!!!")

if __name__ == '__main__':
	main()

Output

The output will look like this when the code is used for a static image.

output

Applications

Eye tracking using OpenCV has found applications in various fields:

Human-Computer Interaction (HCI)

Eye tracking enhances HCI by enabling natural interaction methods. It improves user convenience and usability. Users can control devices and trigger actions through eye movements. Navigation across interfaces is possible as well. Eye tracking in HCI can give valuable cues to user preferences.

Usability Testing and Market Research

Eye tracking provides insights into user attention and engagement. By analysing gaze, researchers can find the potency of interfaces. Assessment of websites, adverts, and products becomes possible. This leads to better design decisions. Usability testing benefits from objective measurements. Market research is another area affected by it. Patterns of visual focus are responsible for its calculations.

Neuroscience and Psychology

Eye tracking helps to study perception in cognitive neuroscience. It gives details about visual attention and decision-making. OpenCV enables researchers to conduct accurate experiments. It provides a thorough understanding of other measures. They may include physiologic factors.

Virtual and Augmented Reality (VR/AR)

Eye tracking is crucial for creating immersive VR and AR experiences. By tracking the user's gaze with accuracy, virtual content is rendered. This is usable in real-time. This will optimize computational resources and enhance realism. Eye tracking in VR/AR enables dynamic adjustments. The user adapts to the virtual environment based on focus. It improves immersion and interaction.

Challenges and Future Scope

Despite its many benefits, eyeball tracking OpenCV faces challenges. Accurate calibration is crucial for precise gaze estimation. Robustness in different lighting conditions can be challenging to achieve. Head movements and occlusions can affect tracking accuracy. Ongoing research and advancements continue to address these limitations.
 

In the future, eye tracking with OpenCV holds promising potential. It can link with ML algorithms to develop more accurate tracking models. Using sensors could give a better knowledge of human cognition. For example- using the EEG sensor. The development of affordable systems can democratise this research. It would bring its benefits to a wider range of applications.

Frequently Asked Questions

Does eye tracking using OpenCV require specialised hardware?

Eye tracking using OpenCV can use various types of hardware. Some specialised devices, like IR cameras, can give precise results. OpenCV can work with standard webcams capable of capturing eyes. The choice of hardware depends on the requirements and desired accuracy.

Is eye tracking using OpenCV suitable for mobile or embedded systems?

Yes, we can implement it on mobile or embedded systems. OpenCV offers platform-specific optimizations. It supports various operating systems, including mobile platforms. By leveraging hardware acceleration, we can implement this. It requires multi-threading and efficient algorithms. Eye-tracking applications can work for other constrained environments too.

Can OpenCV track multiple eyes simultaneously?

Yes, OpenCV can track multiple eyes simultaneously. By implementing suitable techniques, it can track numerous features. They could be pupils or irises in a given image or video stream. This fosters eye tracking of multiple people. It can be simultaneous monitoring of eye movements as well.

What kind of accuracy can we achieve with eye tracking using OpenCV?

The accuracy of eye tracking depends on various factors. These include image quality, calibration, and tracking algorithms. The specific application requirements play a significant role. With optimised algorithms, OpenCV can achieve high levels of accuracy. We must consider that absolute accuracy may vary. This could be based on personal differences or other factors.

Are there any privacy concerns associated with eye tracking using OpenCV?

Privacy concerns can arise when implementing eye tracking using OpenCV. It happens if the system involves capturing sensitive personal data. It is important to follow ethical guidelines. Informed consent of the user is a must. Developers must ensure the secrecy of collected data. Transparency and user awareness must be a priority. This turns into a necessity to avoid privacy concerns.

Conclusion

In this article, we discussed Eye Ball Tracking with OpenCV. We got to know about the procedure, various techniques, applications, challenges and future scope of eyeball tracking using OpenCV. Now that you have learnt about it, you can also refer to other similar articles.

You may refer to our Guided Path on Code Ninjas Studios for enhancing your skill set on DSACompetitive ProgrammingSystem Design, etc. Check out essential interview questions, practice our available mock tests, look at the interview bundle for interview preparations, and so much more!

Happy Learning, Ninja!

Previous article
Feature Detection with Haar-Cascade
Next article
Smoothing Images
Live masterclass