Services(like communication with Subscribers etc) can communicate asynchronously using Pub/Sub, with latencies of about 100 milliseconds.
Data integration pipelines and streaming analytics use Pub/Sub to receive and deliver data. It works well as a queue to parallelize activities and a messaging-oriented middleware for service integration.
You may build systems of event producers and consumers, known as publishers and subscribers, with the help of pub/sub. Publishers communicate with subscribers asynchronously by broadcasting events rather than synchronous remote procedure calls (RPCs).
Without considering how or when these events should be processed, publishers deliver events to the Pub/Sub service. Following that, Pub/Sub in GCP distributes events to all the services that react to them.
Publishers in systems communicating through RPCs must wait for subscribers to get data. However, the asynchronous integration of Pub/Sub in GCP boosts the system's overall flexibility and robustness.
Real-Life Examples or Use Cases
Enterprise Event Bus
You can build a real-time data sharing bus that spans your entire company, distributing business events, database updates, and analytics events.
Data Streaming from IoT Devices, Services, or Apps
For instance, a SaaS (software as a service)service might be able to publish a real-time events feed. Alternatively, a home sensor can use a Dataflow pipeline to transmit data to Pub/Sub for usage in other Google Cloud applications.
Refreshing Distributed Caches
For instance, refreshing distributed caches Invalidation events might be published by an application to update the IDs of objects that have changed.
Loads Balancing for Dependability
On Compute Engine, for instance, a service's models could be deployed in various zones while subscribing to the same topic. The other zones can automatically take over the load if a zone's service fails.
Ingesting Server Events and User Interactions
You might transmit user interaction events from end-user apps or server events from your system to Pub/Sub for utilization.
After that, you can utilize a stream processing tool that sends the events to databases, like Dataflow.
These databases include BigQuery, Cloud Bigtable, and Cloud Storage.
You can simultaneously collect events from numerous clients using Pub/Sub.
Dispersal of Events in Real-Time
Events can be made available in either their raw or processed form to various applications throughout your team and business for real-time processing.
Pub/Sub supports the "enterprise event bus" and event-driven application architecture patterns.
Many Google systems that export events to Pub/Sub in GCP can be integrated via Pub/Sub.
Data Replication Between Databases
To distribute change events from databases, Pub/Sub is frequently utilized. In BigQuery and other data storage systems, a view of the database state and state history may be created using these events.
Parallel Workflows and Processing
Connecting to Cloud Functions using Pub/Sub messages may easily divide various tasks across numerous employees. Compressing text files, sending email notifications, testing AI models, and reformatting photographs are a few examples of such jobs.
Types of Pub/Sub Services
Two varieties of Pub/Sub in GCP services exist:
Pub/Sub Service
For most users and apps, this messaging service is the default option. Along with intelligent capacity management, it provides the most excellent reliability and the broadest range of integrations. Pub/Sub in GCP ensures that all data is replicated synchronously to at least two zones and that a third zone will receive best-effort replication.
Pub/Sub Lite Service
Pub/Sub Lite service is a different but comparable messaging service created at a lesser cost. Compared to Pub/Sub, it provides less reliability. It provides both regional and zonal topic storage.
Using the Console
Before You Start
Create an account if you're new to Google Cloud to assess how well our solutions work in practical situations. Additionally, new users receive $300 in complimentary credits to run, test, and deploy workloads.
Establish a console project. - Select or create a project. - Make that project's Pub/Sub API available. These resources are always available for viewing and management in the console.
The Google Cloud CLI should be installed and started.
Make a Topic
Navigate to the Pub/Sub topics page in the console.
Then select “Create topic”.
Enter a specific topic name, such as my topic, in the “Topic ID” field.
Then select “Create topic”.
You've just finished creating a Pub/Sub topic.
Subscription Addition
To add a subscription to the newly established topic:
Start with clicking on the Subscriptions tab.
Again click on “create a subscription”.
Enter a name for the subscription in the “Subscription ID” box, such as MySub.
For the remaining parameters, use their default settings.
Press Create.
Publish a Note about the Subject
To post a message on the subject:
Navigate to the Pub/Sub topics page in the console.
Then click on the topic ID.
At the bottom of the page, select the Messages tab and then click Publish Message.
In the Message field, type ‘Hello World.’
Press Publish.
Take the Message out of the Subscription
Navigate to the Pub/Sub subscriptions page in the console.
On the subscription ID, click.
Select the Messages tab.
Click on Pull.
Clean Up
Please follow these instructions to prevent your Google Cloud account from being charged for the resources used on this page.
Navigate to the console's Pub/Sub topics page.
The topic you created is checked in the checkbox next to it.
To permanently remove the topic, click Delete.
Display each subscription and click the trash can button to delete any remaining subscriptions.
Stream Processing with Dataflow
Dataflow is a fully-managed service for reliably and expressively converting and enhancing data in both stream (real-time) and batch modes. The Apache Beam SDK, which contains a wealth of windowing and session analysis primitives and an ecosystem of source and sink connectors, offers a streamlined pipeline building environment. This short tutorial demonstrates how to utilize Dataflow for:
Read messages posted to a Pub/Sub topic
Group them according to timestamps.
Messages are sent to cloud storage.
Ordering Messages
This section explains how to receive messages in order.
Receiving Messages in Order
You can activate message ordering and get the messages in the order that the Pub/Sub service receives them if the messages have the same ordering key and are in the same region.
Each message is delivered at least once using Pub/Sub in GCP. Therefore messages may be resent. When you get messages in sequence, and the Pub/Sub service redelivers one of the messages with an ordering key, it also redelivers the following messages with the same ordering key to keep the order.
The Pub/Sub service resends these messages in the same order they were initially received.
Every message that follows with the same ordering key, including acknowledged messages, is redelivered by the Pub/Sub service after redelivering the first message with an ordering key.
It may not be the case if a subscription enables message ordering and a dead-letter topic because Pub/Sub makes its best effort to forward messages to dead-letter topics. You must recognize emails that have been redistributed once again. There is no assurance that messages will continue to be sent even if prior ones are not acknowledged.
Publish Communications on Topics
This section offers details on message publication.
Messages are created and sent by a publisher program to a topic. Existing subscribers can receive messages at least once and receive best-effort ordering from Pub/Sub.
Generally speaking, a publisher application goes like this:
Make a message with your info in it.
To publish the message to the specified subject, send a request to the Pub/Sub server.
Publish Messages on Topics
This section offers details on message publication.
Messages are created and sent by a publisher program to a topic. Existing subscribers can receive messages at least once and receive best-effort ordering from Pub/Sub.
Generally speaking, a publisher application goes like this:
Make a message with your info in it.
To publish the message to the specified subject, send a request to the Pub/Sub server.
Overview of Subscription Type
Pull subscriptions
Pull delivery involves sending queries to the Pub/Sub server from your subscriber application to get messages.
The pull mode can be used for either synchronous or asynchronous message processing. The client library in which the pull mode runs can use any of the two service APIs, Pull or StreamingPull. Depending on your needs, the client library may be either a high-level or low-level client library.
Push Subscriptions
When pushing messages, Pub/Sub sends queries to your subscriber application.
The characteristics of a push subscription
The upcoming properties can be specified when configuring a push subscription:
Endpoint URL (required)
Enable authentication
Audience
BigQuery Subscriptions
As messages are received, a BigQuery subscription writes them to an existing BigQuery table.
A subscriber client does not need to be configured individually.
To create, edit, list, detach, or delete a BigQuery subscription, use the console, the Google Cloud CLI, the client libraries, the Pub/Sub API, or the Pub/Sub API client libraries.
The benefits of the BigQuery subscription over direct data ingestion pipelines, which frequently employ Dataflow to write to BigQuery, include:
Simple deployment: A single workflow in the console, Google Cloud CLI, client library, or Pub/Sub API can be used to set up a BigQuery subscription.
Low-cost offer: Eliminate the extra costs and delays associated with comparable Pub/Sub pipelines that employ Dataflow activities. This cost optimization is helpful for message systems that don't need additional processing before storage.
Reduces monitoring: The multi-tenant Pub/Sub service includes BigQuery subscriptions, so you do not need to run additional monitoring tasks.
Frequently Asked Questions
What is a pub/sub?
In serverless and microservices architectures, publish/subscribe messaging—also known as pub/sub-messaging is a type of asynchronous service-to-service communication.
How does Pub/Sub in GCP operate?
Using a pub/sub architecture, messages can be distributed asynchronously throughout several application sections.
Is Pub/Sub a message queue?
The message pattern used by message brokers to communicate across the various microservices' components is called the Pub/Sub queue (or Pub/Sub). It is utilized to offer asynchronous and program-to-program communication between the micro-services.
When should you not use a pub/sub?
The optimum method for streaming media is a synchronous point-to-point connection between the two ends. Pub/Sub is ineffective for Internet-based VoIP(voice over Internet Protocol )or video telephony traffic.
Can Pub/Sub be scaled?
The system is made horizontally scalable, meaning it can manage an increase in topics, subscriptions, or messages by adding more instances of servers already running.
Conclusion
In this article, we have discussed the Pub/Sub in GCP, also known as Publish/Subscribe, examples, types, streaming processing with dataflow, ordering message, etc.