Cloud Vision

Detect text in images

The process of detecting text in a photograph and then enclosing it with a rectangular bounding box is known as text detection. Image-based or frequency-based techniques can be used to detect text.

Image-based techniques are used to divide images into sections. Each segment is made up of pixels with similar properties that are linked together. To categorize and structure the content, statistical attributes of connected components are used. To categorize the components as text or non-text, machine learning techniques such as support vector machines and convolutional neural networks are used. An example of text detection is shown below.

Optical Character Recognition (OCR)

Text may be detected and extracted from images using the Vision API. Annotation features that facilitate optical character recognition (OCR) are as follows:

• TEXT_DETECTION: It extracts the text out of any image and outputs it (e.g., photos of street views or sceneries). The model is somewhat more resilient in reading words of diverse types because it was originally created to be useful in a variety of lighting circumstances, but only at a more sparse level. 
  The whole strings, as well as the individual words and their respective bounding boxes are all included in the JSON file that was returned.
  
   

• DOCUMENT_TEXT_DETECTION: Although it extracts text from images, the response is optimized to work best with dense text and documents. The JSON contains information on pages, blocks, paragraphs, words, and breaks.

Text detection requests

When a text recognition request is made, the input image is searched for all possible glyphs or characters, and then each string is analyzed.

Detect text in a local image

By including the contents of the local image file as a base64-encoded string in the body of your request, you may use the Vision API to perform feature detection on the local image file.

Implementing in Node.js

const vision = require('@google-cloud/vision');
// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO for developer, Uncomment the following line before running the sample.
 */
// const fileName = 'Local image file, e.g. /path/to/image.png';

// Performs text detection on the local file
const [result] = await client.textDetection(fileName);
const detections = result.textAnnotations;
console.log('Text:');
detections.forEach(text => console.log(text));

Detect text in a remote image

Without sending the contents of the image file in the body of your request, the Vision API can conduct feature detection directly on an image file that is stored in Google Cloud Storage or online.

Implementing in Node.js

// Import Google Cloud client libraries
const vision = require('@google-cloud/vision');

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO for developer, Uncomment the following lines before running the sample.
 */
// const bucketName = 'Bucket where the file resides, e.g. my-bucket';
// const fileName = 'Path to file within bucket, e.g. path/to/image.png';

// Performs text detection on the gcs file
const [result] = await client.textDetection(`gs://${bucketName}/${fileName}`);
const detections = result.textAnnotations;
console.log('Text:');
detections.forEach(text => console.log(text));

Detect Text in Files

Text from PDF and TIFF files kept in cloud storage can be detected and transcribed using the Vision API.

The files:asyncBatchAnnotate function, which does an offline (asynchronous) request and delivers its status using the operations resources, must be used to request document text detection from PDF and TIFF.

A JSON file is produced in the designated Cloud Storage bucket and used to store the output from a PDF/TIFF request.

Limitations

Up to 2000-page PDF/TIFF files are supported by the Vision API. Larger files will produce an error message.

Document text detection requests

Currently, only files kept in Cloud Storage buckets may detect PDF/TIFF documents. Similar JSON files are saved to a Cloud Storage bucket as responses.

gs://cloud-samples-data/vision/pdf_tiff/census2010.pdf, Source: Census Of India 2011.
 

Implementing in Node.js

// Imports the Google Cloud client libraries
const vision = require('@google-cloud/vision').v1;

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO for developer, Uncomment the following lines before running the sample.
 */
// Bucket where the file resides
// const bucketName = 'my-bucket';
// Path to PDF file within bucket
// const fileName = 'path/to/document.pdf';
// The folder to store the results
// const outputPrefix = 'results'

const gcsSourceUri = `gs://${bucketName}/${fileName}`;
const gcsDestinationUri = `gs://${bucketName}/${outputPrefix}/`;

const inputConfig = {
  // Supported mime_types are: 'application/pdf' and 'image/tiff'
  mimeType: 'application/pdf',
  gcsSource: {
    uri: gcsSourceUri,
  },
};
const outputConfig = {
  gcsDestination: {
    uri: gcsDestinationUri,
  },
};
const features = [{type: 'DOCUMENT_TEXT_DETECTION'}];
const request = {
  requests: [
    {
      inputConfig: inputConfig,
      features: features,
      outputConfig: outputConfig,
    },
  ],
};

const [operation] = await client.asyncBatchAnnotateFiles(request);
const [filesResponse] = await operation.promise();
const destinationUri =
  filesResponse.responses[0].outputConfig.gcsDestination.uri;
console.log('Json saved to: ' + destinationUri);

Detect Handwriting in Images

The ability of a computer to read and interpret intelligible handwritten input from sources such as paper documents, pictures, touch screens, and other devices is known as handwriting recognition (HWR), sometimes known as handwritten text recognition (HTR).

The Vision API can detect and extract text from images:

• DOCUMENT_TEXT_DETECTION: Although it extracts text from images, the response is optimized to work best with dense text and documents. The JSON contains information on pages, blocks, paragraphs, words, and breaks.

Document text detection requests

Detect document text in a local image

By including the contents of the local image file as a base64-encoded string in the body of your request, you may use the Vision API to perform feature detection on the local image file.

Implementing in Node.js

// Imports the Google Cloud client library
const vision = require('@google-cloud/vision');

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO for developer, Uncomment the following line before running the sample.
 */
// const fileName = 'Local image file, e.g. /path/to/image.png';

// Read a local image as a text document
const [result] = await client.documentTextDetection(fileName);
const fullTextAnnotation = result.fullTextAnnotation;
console.log(`Full text: ${fullTextAnnotation.text}`);
fullTextAnnotation.pages.forEach(page => {
  page.blocks.forEach(block => {
    console.log(`Block confidence: ${block.confidence}`);
    block.paragraphs.forEach(paragraph => {
      console.log(`Paragraph confidence: ${paragraph.confidence}`);
      paragraph.words.forEach(word => {
        const wordText = word.symbols.map(s => s.text).join('');
        console.log(`Word text: ${wordText}`);
        console.log(`Word confidence: ${word.confidence}`);
        word.symbols.forEach(symbol => {
          console.log(`Symbol text: ${symbol.text}`);
          console.log(`Symbol confidence: ${symbol.confidence}`);
        });
      });
    });
  });
});

Detect document text in a remote image

Without sending the contents of the image file in the body of your request, the Vision API can conduct feature detection directly on an image file that is stored in Google Cloud Storage or online.

Implementing in Node.js

// Imports the Google Cloud client libraries
const vision = require('@google-cloud/vision');

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO for developer, Uncomment the following lines before running the sample.
 */
// const bucketName = 'Bucket where the file resides, e.g. my-bucket';
// const fileName = 'Path to file within bucket, e.g. path/to/image.png';

// Read a remote image as a text document
const [result] = await client.documentTextDetection(
  `gs://${bucketName}/${fileName}`
);
const fullTextAnnotation = result.fullTextAnnotation;
console.log(fullTextAnnotation.text);

Detect faces in Cloud Vision

Face detection identifies several faces in an image together with the main facial characteristics that go with them, including emotional mood or headwear use. Facial Recognition for an individual is not supported.

Image credit: Himanshu Singh Gurjar on Unsplash (annotations added).

Face detection requests

Consult the API documentation before creating a call to the Vision API. You will be asking the pictures resource to annotate your image in this scenario. An object with a requests list represents a request to this API. Each item in this list has two pieces of information:

• Data for the base64-encoded image
   
• a list of the attributes you want to be noted next to that picture.
   

Detect Faces in a local image

By including the contents of the local image file as a base64-encoded string in the body of your request, you may use the Vision API to perform feature detection on the local image file.

Implementing in Node.js

// Imports the Google Cloud client library
const vision = require('@google-cloud/vision');

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO for developer, Uncomment the following line before running the sample.
 */
// const fileName = 'Local image file, e.g. /path/to/image.png';

const [result] = await client.faceDetection(fileName);
const faces = result.faceAnnotations;
console.log('Faces:');
faces.forEach((face, i) => {
  console.log(`  Face #${i + 1}:`);
  console.log(`    Joy: ${face.joyLikelihood}`);
  console.log(`    Anger: ${face.angerLikelihood}`);
  console.log(`    Sorrow: ${face.sorrowLikelihood}`);
  console.log(`    Surprise: ${face.surpriseLikelihood}`);
});

Detect Faces in a remote image

Without sending the contents of the image file in the body of your request, the Vision API can conduct feature detection directly on an image file that is stored in Google Cloud Storage or online.

Implementing in Node.js

// Imports the Google Cloud client libraries
const vision = require('@google-cloud/vision');

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO for developer, Uncomment the following lines before running the sample.
 */
// const bucketName = 'Bucket where the file resides, e.g. my-bucket';
// const fileName = 'Path to file within bucket, e.g. path/to/image.png';

// Performs face detection on the gcs file
const [result] = await client.faceDetection(`gs://${bucketName}/${fileName}`);
const faces = result.faceAnnotations;
console.log('Faces:');
faces.forEach((face, i) => {
  console.log(`  Face #${i + 1}:`);
  console.log(`    Joy: ${face.joyLikelihood}`);
  console.log(`    Anger: ${face.angerLikelihood}`);
  console.log(`    Sorrow: ${face.sorrowLikelihood}`);
  console.log(`    Surprise: ${face.surpriseLikelihood}`);
});

Frequently Asked Questions

What is OCR API?

Optical Character Recognition (OCR) API allows you to extract text from image files and PDF documents and save it in JSON, CSV, Excel, or other file formats.
 

Which algorithm is used to detect the text in images?

An optical character recognition (OCR) algorithm is used to convert images into text.
 

What is the Google vision?

With the help of the Google Cloud Vision API, application developers can easily add vision detection features like image labeling, face and landmark identification, optical character recognition (OCR), and the tagging of explicit information to their product.

Conclusion

We explored the Cloud Vision API in this blog, as well as optical character recognition (OCR), detecting text in files and images, detecting handwriting in images, and face detection using the cloud vision API.
If you think the blog has helped you with an overview of Cloud Vision API, and if you like to learn more, check out our articles Cloud Computing, Cloud Computing Technologies, Cloud Computing Infrastructure, and Overview of a log-based metric.

Refer to our Coding Ninjas Studio Guided Path to learn Data Structures and Algorithms, Competitive Programming, JavaScript, System Design, and even more! You can also check out the mock test series and participate in the contests hosted by Coding Ninjas Studio! But say you're just starting and want to learn about questions posed by tech titans like Amazon, Microsoft, Uber, and so on. In such a case, for placement preparations, you can also look at the problems, interview experiences, and interview bundle.

Do upvote our blogs if you find them helpful and engaging!

Happy Coding!