Table of contents
1.
Introduction
2.
How to write logs
3.
Log Analytics
3.1.
Log Analytics features
4.
Working of Log Analytics
5.
Benefits of employing logging agents to ingest logs
6.
Drawbacks of using logging agents 
7.
Frequently Asked Questions
7.1.
What kinds of hosting and computing services are there?
7.2.
List some of the GCP's data analytics offerings.
7.3.
What is Google Cloud Messaging?
8.
Conclusion
Last Updated: Mar 27, 2024

Log Analytics Concept

Author Shivani Singh
0 upvote
Career growth poll
Do you think IIT Guwahati certified course can help you in your career?

Introduction

A word related to structured logging has been around for a while. As additional log analysis and managing solutions hit the market, the practice gains popularity. Structured logging has replaced many well-known proprietary formatting systems. 

cloud logging intro image

Structured logging is the process of giving application logs a standardized, preset message format that enables them to be handled as data sets rather than text.

Structured logging is defined in a wide variety of ways. The concept of representing each log entry as an object or record so it can be processed and searched by automated tools is at the core of structured logging. The ingested structured logs are often parsed and saved as objects using a preset text or binary format, such as JSON, XML, or Protocol Buffers (protobuf). From the beginning, Google Cloud uses structured logging. The LogEntry structure in Google Cloud serves as a log description. With Log Explorer or through Logging APIs, the logs can be searched for and examined. It is possible to use Log Analytics for more intricate analysis or to export log entries to BigQuery.

How to write logs

Depending on the log management tool, the local logging framework, and the application environment, different approaches are used when an application sends logs to a remote destination. Cloud Logging makes the Logging v2 API available, which can be accessed directly over REST or gRPC or by utilizing special Logging framework adapters. 

Additionally, Google offers logging client libraries for usage by developers. These choices import logs into the Cloud Logging backend directly. Additionally, it is possible to "assign" the task of log ingestion to logging agents running on one of the Google Cloud platforms. 

AppEngine, GKE, Cloud Run, Cloud Function, Cloud Build, DataProc, and DataFlow are the platforms with logging agents. On GCE instances, the agent must be explicitly installed before it can be set up to allow log capturing as well. By simply outputting logs to stdout or stderr while an application is operating on the platform with the logging agent, it is possible to ingest them into Cloud Logging. 

The agents are capable of capturing both structured and plain text payloads. It is recommended to format the supported structured payload as a single-line JSON string. The agent can parse and save special fields from the payload in the context information fields (e.g. httpRequest). 

The following details are inaccessible when entered into the special fields:

  • When writing a structured payload to stdout, it is not feasible to alter the log's name or destination. 
  • The resource field is filled in by the agent automatically. This field cannot be customized in any way.

Log Analytics

With a new user interface that is designed specifically for analyzing log data, Log Analytics gives you the analytical power of BigQuery right in Cloud Logging. SQL may be used to perform sophisticated log analysis with Log Analytics. Log Analytics also makes your logs data readily accessible in BigQuery to aid in the dismantling of data silos. Your logs can be correlated with other commercial data kept in BigQuery, providing you with a deeper understanding of your Google Cloud services.

Log Analytics features

  • Configuration at the bucket level makes it simple to apply log analytics to the data from your ingested logs.
  • A new user interface: Use BigQuery standard SQL to query your log data in a new Log Analytics UI. The results of your query are presented in a format that is best for examining log data.
  • Simple storage: Logging manages a BigQuery dataset in which your log data is saved.
  • Access from BigQuery: Utilize a new bucket setup option in BigQuery to quickly retrieve your log data.
workflow diagram for log analytics features

Working of Log Analytics

The data from the logs are made available in the Log Analytics interface in Logging when you build a log bucket and enable Log Analytics on it. You can avoid routing and managing a separate copy of the data in BigQuery by turning on Log Analytics. With Log Analytics, you can use the same kind of standard SQL that BigQuery offers to analyze your log data. Using the Logging features you are accustomed to, you may still query and analyze the data as usual.

You may set up a view of the data in the Logs Analytics-enabled bucket directly in BigQuery if you wish to mix and use your log data with other data in BigQuery. BigQuery can be used with the same query that you use for Log Analytics.

Benefits of employing logging agents to ingest logs

The perks of employing logging agents are the following: 

  • It's easy to print to stdout. It is not necessary to use extra libraries or to be concerned about possible retrying procedures, communication latency, or error handling.
  • It moves quickly. Any API call requires a certain amount of time-based on the network latency, the backend latency, and other variables. The application should remain operational throughout the time needed to finish the API call, even when the communication portion is executed asynchronously.
  • It is free of resources. Use of resources like CPU, memory, and the network is required for API requests. It is different in cloud environments. Managed environments, like Cloud Run, will assign specific resources (such as CPU) for the deployment that are separate from those utilized by the logging agent.

Drawbacks of using logging agents 

The disadvantage of using logging agents are stated below: 

  • It may not be as simple to implement the formatting requirements for a single-line JSON string as it may seem. The responsibility for proper formatting falls on the application's developers unless a specific library is employed.
  • The logging agent is not present in all contexts and platforms. There is also no way that works across all platforms to identify the logging agent. Without the logging agent, running such an application on the platform will result in the logs being sent to stdout rather than being ingested by the Cloud Logging backend.
  • For applications that run in one Google Cloud project but need to import logs into another project, restrictions like the inability to control the log destination may be crucial.

Frequently Asked Questions

What kinds of hosting and computing services are there?

According to your needs, Google Cloud Platform offers a variety of hosting and computing services. 

Computing without servers: The Cloud Functions service from Google Cloud offers a serverless execution environment known as FaaS. (Function as a service). Only the code needs to be written; Google Cloud will take care of the infrastructure. Events start these processes off.

Application Platform: Virtual Machines and Containers and combining hosting and computational capabilities.

List some of the GCP's data analytics offerings.

Numerous Data Analytics services are available through Google Cloud like BigQuery, DataProc, Dataflow, Pub/Sub, Cloud Data Fusion, Data Catalog, Cloud Composer, Google, Data Studio, and Dataprep.

Cloud Life Sciences enables the life sciences community to manage, process, and transform biomedical data at scale.

What is Google Cloud Messaging?

It is a mobile notification service that enables outside application developers to send alert information from servers they control to applications. Since April 2018, Firebase Cloud Messaging has taken its place and is no longer active.

Conclusion

In this blog, we discussed structured logging, and how to write logs. We also discussed log analytics and its features. We also saw the working of log analytics. In the end, we discussed the pros and cons of using logging agents. 

For more content, Refer to our guided paths on Coding Ninjas Studio to upskill yourself.

Do upvote our blogs if you find them helpful and engaging!

Happy Learning!

thank you image
Live masterclass