Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Last Updated: Mar 27, 2024
Leveraging ChatGPT - GenAI as a Microsoft Data Expert
Speaker
Prerita Agarwal
Data Specialist @
23 Jul, 2024 @ 01:30 PM

Introduction

Aren't you excited about this interesting name called Pub/Sub? Pub/Sub in GCP means Publish/Subscribe.

Pub/sub in GCP

Let's find out together what is  Pub/Sub in GCP and how it is related to the cloud.

Must Recommended Topic, procedure call in compiler design

Pub/Sub in GCP

Services(like communication with Subscribers etc) can communicate asynchronously using Pub/Sub, with latencies of about 100 milliseconds.

  • Data integration pipelines and streaming analytics use Pub/Sub to receive and deliver data. It works well as a queue to parallelize activities and a messaging-oriented middleware for service integration.
     
  • You may build systems of event producers and consumers, known as publishers and subscribers, with the help of pub/sub. Publishers communicate with subscribers asynchronously by broadcasting events rather than synchronous remote procedure calls (RPCs).
     
  • Without considering how or when these events should be processed, publishers deliver events to the Pub/Sub service. Following that, Pub/Sub in GCP distributes events to all the services that react to them. 
     
  • Publishers in systems communicating through RPCs must wait for subscribers to get data. However, the asynchronous integration of Pub/Sub in GCP boosts the system's overall flexibility and robustness.
Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Real-Life Examples or Use Cases

Enterprise Event Bus 

You can build a real-time data sharing bus that spans your entire company, distributing business events, database updates, and analytics events.

Data Streaming from IoT Devices, Services, or Apps

For instance, a SaaS (software as a service)service might be able to publish a real-time events feed. Alternatively, a home sensor can use a Dataflow pipeline to transmit data to Pub/Sub for usage in other Google Cloud applications.

Refreshing Distributed Caches

For instance, refreshing distributed caches Invalidation events might be published by an application to update the IDs of objects that have changed.

Loads Balancing for Dependability

On Compute Engine, for instance, a service's models could be deployed in various zones while subscribing to the same topic. The other zones can automatically take over the load if a zone's service fails.

Ingesting Server Events and User Interactions 

You might transmit user interaction events from end-user apps or server events from your system to Pub/Sub for utilization. 

  • After that, you can utilize a stream processing tool that sends the events to databases, like Dataflow. 
  • These databases include BigQuery, Cloud Bigtable, and Cloud Storage. 
  • You can simultaneously collect events from numerous clients using Pub/Sub.

Dispersal of Events in Real-Time 

Events can be made available in either their raw or processed form to various applications throughout your team and business for real-time processing. 

Pub/Sub supports the "enterprise event bus" and event-driven application architecture patterns. 

Many Google systems that export events to  Pub/Sub in GCP can be integrated via Pub/Sub.

Data Replication Between Databases 

To distribute change events from databases, Pub/Sub is frequently utilized. In BigQuery and other data storage systems, a view of the database state and state history may be created using these events.

Parallel Workflows and Processing 

Connecting to Cloud Functions using Pub/Sub messages may easily divide various tasks across numerous employees. Compressing text files, sending email notifications, testing AI models, and reformatting photographs are a few examples of such jobs.

Types of Pub/Sub Services

Two varieties of  Pub/Sub in GCP services exist:

Pub/Sub Service

For most users and apps, this messaging service is the default option. Along with intelligent capacity management, it provides the most excellent reliability and the broadest range of integrations. Pub/Sub in GCP ensures that all data is replicated synchronously to at least two zones and that a third zone will receive best-effort replication.

Pub/Sub Lite Service

Pub/Sub Lite service is a different but comparable messaging service created at a lesser cost. Compared to Pub/Sub, it provides less reliability. It provides both regional and zonal topic storage. 

Using the Console

Before You Start 

  1. Create an account if you're new to Google Cloud to assess how well our solutions work in practical situations. Additionally, new users receive $300 in complimentary credits to run, test, and deploy workloads.
     
  2. Establish a console project.
    - Select or create a project.
    - Make that project's Pub/Sub API available.
    These resources are always available for viewing and management in the console.
     
  3. The Google Cloud CLI should be installed and started.

Make a Topic

  1. Navigate to the Pub/Sub topics page in the console.
  2. Then select “Create topic”.
  3. Enter a specific topic name, such as my topic, in the “Topic ID” field.
  4. Then select “Create topic”.


You've just finished creating a Pub/Sub topic.

Subscription Addition

To add a subscription to the newly established topic:

  1. Start with clicking on the Subscriptions tab.
  2. Again click on “create a subscription”.
  3. Enter a name for the subscription in the “Subscription ID” box, such as MySub.
  4. For the remaining parameters, use their default settings.
  5. Press Create.

Publish a Note about the Subject

To post a message on the subject:

  1. Navigate to the Pub/Sub topics page in the console.
  2. Then click on the topic ID.
  3. At the bottom of the page, select the Messages tab and then click Publish Message.
  4. In the Message field, type ‘Hello World.’
  5. Press Publish.

Take the Message out of the Subscription

  1. Navigate to the Pub/Sub subscriptions page in the console.
  2. On the subscription ID, click.
  3. Select the Messages tab.
  4. Click on Pull.

Clean Up

Please follow these instructions to prevent your Google Cloud account from being charged for the resources used on this page.

  1. Navigate to the console's Pub/Sub topics page.
  2. The topic you created is checked in the checkbox next to it.
  3. To permanently remove the topic, click Delete.
  4. Display each subscription and click the trash can button to delete any remaining subscriptions.

Stream Processing with Dataflow

Dataflow is a fully-managed service for reliably and expressively converting and enhancing data in both stream (real-time) and batch modes. The Apache Beam SDK, which contains a wealth of windowing and session analysis primitives and an ecosystem of source and sink connectors, offers a streamlined pipeline building environment. This short tutorial demonstrates how to utilize Dataflow for:

  • Read messages posted to a Pub/Sub topic 
  • Group them according to timestamps.
  • Messages are sent to cloud storage.

Ordering Messages

This section explains how to receive messages in order.

Receiving Messages in Order

You can activate message ordering and get the messages in the order that the Pub/Sub service receives them if the messages have the same ordering key and are in the same region.

  • Each message is delivered at least once using Pub/Sub in GCP. Therefore messages may be resent. When you get messages in sequence, and the Pub/Sub service redelivers one of the messages with an ordering key, it also redelivers the following messages with the same ordering key to keep the order. 
     
  • The Pub/Sub service resends these messages in the same order they were initially received.
     
  • Every message that follows with the same ordering key, including acknowledged messages, is redelivered by the Pub/Sub service after redelivering the first message with an ordering key. 
     
  • It may not be the case if a subscription enables message ordering and a dead-letter topic because Pub/Sub makes its best effort to forward messages to dead-letter topics. You must recognize emails that have been redistributed once again. There is no assurance that messages will continue to be sent even if prior ones are not acknowledged.

Publish Communications on Topics

This section offers details on message publication.

Messages are created and sent by a publisher program to a topic. Existing subscribers can receive messages at least once and receive best-effort ordering from Pub/Sub.

Generally speaking, a publisher application goes like this:

  • Make a message with your info in it.
  • To publish the message to the specified subject, send a request to the Pub/Sub server.

Publish Messages on Topics

This section offers details on message publication.

Messages are created and sent by a publisher program to a topic. Existing subscribers can receive messages at least once and receive best-effort ordering from Pub/Sub.

Generally speaking, a publisher application goes like this:

  • Make a message with your info in it.
  • To publish the message to the specified subject, send a request to the Pub/Sub server.

Overview of Subscription Type

Pull subscriptions

Pull delivery involves sending queries to the Pub/Sub server from your subscriber application to get messages.

The pull mode can be used for either synchronous or asynchronous message processing. The client library in which the pull mode runs can use any of the two service APIs, Pull or StreamingPull. Depending on your needs, the client library may be either a high-level or low-level client library.

Push Subscriptions

When pushing messages, Pub/Sub sends queries to your subscriber application.

The characteristics of a push subscription

The upcoming properties can be specified when configuring a push subscription:

  • Endpoint URL (required)
  • Enable authentication
  • Audience

BigQuery Subscriptions

  • As messages are received, a BigQuery subscription writes them to an existing BigQuery table. 
     
  • A subscriber client does not need to be configured individually. 
     
  • To create, edit, list, detach, or delete a BigQuery subscription, use the console, the Google Cloud CLI, the client libraries, the Pub/Sub API, or the Pub/Sub API client libraries.
     

The benefits of the BigQuery subscription over direct data ingestion pipelines, which frequently employ Dataflow to write to BigQuery, include:

  1. Simple deployment: A single workflow in the console, Google Cloud CLI, client library, or Pub/Sub API can be used to set up a BigQuery subscription.
     
  2. Low-cost offer: Eliminate the extra costs and delays associated with comparable Pub/Sub pipelines that employ Dataflow activities. This cost optimization is helpful for message systems that don't need additional processing before storage.
     
  3. Reduces monitoring: The multi-tenant Pub/Sub service includes BigQuery subscriptions, so you do not need to run additional monitoring tasks.

Frequently Asked Questions

What is a pub/sub?

In serverless and microservices architectures, publish/subscribe messaging—also known as pub/sub-messaging is a type of asynchronous service-to-service communication.

How does Pub/Sub in GCP operate?

Using a pub/sub architecture, messages can be distributed asynchronously throughout several application sections.

Is Pub/Sub a message queue?

The message pattern used by message brokers to communicate across the various microservices' components is called the Pub/Sub queue (or Pub/Sub). It is utilized to offer asynchronous and program-to-program communication between the micro-services.

When should you not use a pub/sub?

The optimum method for streaming media is a synchronous point-to-point connection between the two ends. Pub/Sub is ineffective for Internet-based VoIP(voice over Internet Protocol )or video telephony traffic.

Can Pub/Sub be scaled?

The system is made horizontally scalable, meaning it can manage an increase in topics, subscriptions, or messages by adding more instances of servers already running.

Conclusion

In this article, we have discussed the Pub/Sub in GCP, also known as Publish/Subscribe, examples, types, streaming processing with dataflow, ordering message, etc.

Also, read some more articles related to GCP:

Refer to our guided paths on Coding Ninjas Studio to learn more about DSA, Competitive Programming, JavaScript, System Design, etc. Enroll in our courses and refer to the mock test and problems available; look at the Top 150 Interview Puzzles interview experiences, and interview bundle for placement preparations.

Thank you

Do upvote our blog to help other ninjas grow. 

Happy Learning Ninja!

Topics covered
1.
Introduction
2.
Pub/Sub in GCP
3.
Real-Life Examples or Use Cases
3.1.
Enterprise Event Bus 
3.2.
Data Streaming from IoT Devices, Services, or Apps
3.3.
Refreshing Distributed Caches
3.4.
Loads Balancing for Dependability
3.5.
Ingesting Server Events and User Interactions 
3.6.
Dispersal of Events in Real-Time 
3.7.
Data Replication Between Databases 
3.8.
Parallel Workflows and Processing 
4.
Types of Pub/Sub Services
4.1.
Pub/Sub Service
4.2.
Pub/Sub Lite Service
5.
Using the Console
5.1.
Before You Start 
5.2.
Make a Topic
5.3.
Subscription Addition
5.4.
Publish a Note about the Subject
5.5.
Take the Message out of the Subscription
5.6.
Clean Up
6.
Stream Processing with Dataflow
7.
Ordering Messages
7.1.
Receiving Messages in Order
8.
Publish Communications on Topics
9.
Publish Messages on Topics
10.
Overview of Subscription Type
10.1.
Pull subscriptions
10.2.
Push Subscriptions
10.3.
BigQuery Subscriptions
11.
Frequently Asked Questions
11.1.
What is a pub/sub?
11.2.
How does Pub/Sub in GCP operate?
11.3.
Is Pub/Sub a message queue?
11.4.
When should you not use a pub/sub?
11.5.
Can Pub/Sub be scaled?
12.
Conclusion