Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Last Updated: Mar 27, 2024

Streaming Live Tailing

Leveraging ChatGPT - GenAI as a Microsoft Data Expert
Speaker
Prerita Agarwal
Data Specialist @
23 Jul, 2024 @ 01:30 PM

Introduction

Streaming logs allow you to view log entries in real-time, while live tailing allows you to view log entries in real-time and is available as Cloud Logging API method entries.tail and as the gcloud CLI command gcloud alpha logging tail.

When you analyze and view your logs using Logs Explorer, gcloud logging read, or the API method entries.list, you are viewing log entries that Cloud Logging has stored. When you live tail or stream log entries, you are viewing log entries as your applications write them to the Cloud Logging API.

Stream logs in the Logs Explorer

In Logs Explorer, you can see your logs data in real-time with the help of Stream logs. When you utilize Stream logs, you can add a query to stream only those logs that match the query. To stream logs, follow the steps below:

  1. In the Google Cloud console, go to the Logs Explorer.
     
  2. Enter any query in the Query text box and click on Stream logs.
     
  3. Only the logs that match the query are displayed in the Query results pane as Logging writes the logs' data. Logging displays the most recent log entries that have been stored if no query is given. Until you choose the scroll bar on the logs panel, logs stream continuously. A button restart streaming appears when it has stopped.
Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Use live tailing in the Google Cloud CLI

Live tailing lets you view your log entries in real time as Cloud Logging writes them, by using either the Google Cloud CLI or the Cloud Logging API.

Installing gcloud alpha logging tail

To use gcloud alpha logging tail, you must have Python 3 and the grpcio Python package installed.

Follow the steps to install gcloud alpha logging tail:

  • Check that you have the Google Cloud CLI installed. 
     
  • Check that you're using version 302.0.0 or greater of the gcloud CLI.
gcloud version

 

  • Install the gcloud CLI alpha components:
gcloud components install alpha

 

  • For MacOS, Linux, and Cloud Shell users:
    • Install gRPC client libraries:
sudo pip3 install grpcio

 

  • Specify the environment variable CLOUDSDK_PYTHON_SITEPACKAGES to any value:
export CLOUDSDK_PYTHON_SITEPACKAGES=1

 

  • Use the following commands to set your Cloud project ID and to authenticate:
gcloud config set project PROJECT_ID
gcloud auth login

 

  • Get the project ID.
     
  • Verify that gcloud alpha logging tail is installed by running the following command:
gcloud alpha logging tail

 

  • The command displays the following message:
Initializing tail session.

 

  • You are now viewing the log entries for your Cloud project as Logging writes them.

Log entries during a live-tail session.

Buffering and ordering

Because Logging can get log entries out of chronological order, live tailing gives a buffer-window setting so you can balance the trade-off in viewing the log entries as they are being written and in ascending order. You can set the buffer window from 0 to 60 seconds.

Note the given characteristics of the buffer window:

  • The default buffer window is 2 seconds.
  • Ingestion of the log entries is delayed by logging for the period of the buffer window.
  • Logging returns the log entries as they are received if a log entry is written outside of the buffer window.

You can choose between viewing logs as they are being digested and viewing the entries out of order when configuring the buffer window.

Buffer window and tradeoff table

Limits and quotas

The following table lists the limits and quotas for live tailing:

limits and quotas vs value table

Client limitations

Your client might be unable to consume entries as rapidly as they are created for a Cloud project that generates many of them. In this instance, logging restricts the overall volume of data sent while giving the most recent ones a top priority. The number of entries that could not be displayed because of client limits is returned by Logging after the tail session.

Use live tailing with client libraries.

Live tailing lets you view your log entries in real-time as Cloud Logging writes them. 

Code in Go:

import (
        "context"
        "fmt"
        "io"


        logging "cloud.google.com/go/logging/apiv2"
        loggingpb "google.golang.org/genproto/googleapis/logging/v2"
)


// tailLogs creates a channel to stream log entries that were recently ingested for a project
func tailLogs(projectID string) error {
        // projectID := "your_project_id"


        ctx := context.Background()
        client, err := logging.NewClient(ctx)
        if err != nil {
                return fmt.Errorf("NewClient error: %v", err)
        }
        defer client.Close()


        stream, err := client.TailLogEntries(ctx)
        if err != nil {
                return fmt.Errorf("TailLogEntries error: %v", err)
        }
        defer stream.CloseSend()


        req := &loggingpb.TailLogEntriesRequest{
                ResourceNames: []string{
                        "projects/" + projectID,
                },
        }
        if err := stream.Send(req); err != nil {
                return fmt.Errorf("stream.Send error: %v", err)
        }


        // read and print two or more streamed log entries
        for counter := 0; counter < 2; {
                resp, err := stream.Recv()
                if err == io.EOF {
                        break
                }
                if err != nil {
                        return fmt.Errorf("stream.Recv error: %v", err)
                }
                fmt.Printf("received:\n%v\n", resp)
                if resp.Entries != nil {
                        counter += len(resp.Entries)
                }
        }
        return nil

Frequently Asked Questions

How long are GCP logs kept?

GCP logs are kept for 30 days.

What is Google Cloud Platform?

Google Cloud Platform is a Google cloud platform that allows users to access cloud systems and computing services. GCP gives a wide range of cloud computing services in the storage, compute, database, migration, and networking domains.

What are the GCP cloud storage libraries and tools?

Google Cloud Platform Console, which performs primary object and bucket operations.

GustilCommand-line Tool, which gives a command line interface for cloud storage. Cloud Storage Client Libraries provide programming support for various languages such as Java, Ruby, and Python.

Conclusion

I hope this article gave you insights into the streaming live tailing in the google cloud platform.

Refer to our guided paths on Coding Ninjas Studio to learn more about DSA, Competitive Programming, System Design, JavaScript, etc. Enroll in our courses, refer to the mock test and problems available, interview puzzles, and look at the interview bundle and interview experiences for placement preparations.

We hope this blog has helped you increase your knowledge regarding AWS Step functions, and if you liked this blog, check other links. Do upvote our blog to help other ninjas grow. Happy Coding!"

Thank You Image
Topics covered
1.
Introduction
2.
Stream logs in the Logs Explorer
3.
Use live tailing in the Google Cloud CLI
3.1.
Installing gcloud alpha logging tail
3.2.
Buffering and ordering
4.
Use live tailing with client libraries.
5.
Frequently Asked Questions
5.1.
How long are GCP logs kept?
5.2.
What is Google Cloud Platform?
5.3.
What are the GCP cloud storage libraries and tools?
6.
Conclusion