Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Table of contents
1.
Introduction
2.
Basic Concepts
3.
Access control with IAM 
3.1.
Predefined roles
3.2.
Custom roles
3.3.
Google Cloud console permissions
3.4.
Command-line permissions
3.5.
Log-routing permissions
4.
Audit logs for Cloud Logging
4.1.
Available audit logs
4.2.
Audit log format
4.3.
Log name
4.4.
Service name
4.5.
Enable audit logging
4.6.
View logs
4.7.
Route audit logs
5.
Available logs
5.1.
Google Cloud platform logs
5.2.
User-written logs
5.2.1.
Agent logs
5.3.
Component logs
5.4.
Security logs
5.4.1.
Cloud Audit logs
5.4.2.
Access Transparency logs
5.5.
Multi-cloud and hybrid-cloud logs
5.6.
Understand and manage logs
5.6.1.
Log entry structure
5.6.2.
View logs
5.6.3.
Route logs
6.
Google Cloud platform logs
6.1.
Index of Google Cloud platform logs
6.2.
Log entry structure
7.
Frequently Asked Questions
7.1.
What is Cloud Logging?
7.2.
What is sinks?
7.3.
What is a Cloud Audit log?
8.
Conclusion
Last Updated: Mar 27, 2024

Cloud Logging in GCP

Author Nagendra
0 upvote
Master Python: Predicting weather forecasts
Speaker
Ashwin Goyal
Product Manager @

Introduction

The Google Cloud operations product line includes Cloud Logging. It features log storage, the Logs Explorer user interface, and an API for managing logs programmatically. By using logging, you may read and write log entries, query your logs, and manage how your logs are routed, stored, and used.

This blog describes the specifics of Google Cloud Logging, Access Control with IAM, Audit Logs, Google Cloud Logs, and the methods for creating query log entries. Without further ado, let's get started with the basic concepts of Cloud Logging.

Basic Concepts

The following are the basics concepts and terminologies used in Cloud Logging in GCP

  • Log entries
    A log entry details a specific event or transaction that occurs in a computer system or records its status. Your own code, the Google Cloud services it utilises, third-party apps, and the infrastructure that Google Cloud relies on all contribute to the creation of log entries.
     
  • Logs
    Within a Google Cloud resource, such as a Cloud project, a log is a named collection of log entries. Only when log entries are present, do logs truly exist.
     
  • Retention period
    The retention period refers to how long Cloud Logging keeps log items on file. The entries are then removed. In Logging Quotas and Limits, the retention times for various log categories are specified.
     
  • Query
    In the Logging query language, a query is an expression that returns the log entries that match the expression. To select and see log entries, such as those from a certain VM instance or those coming in a specific time period with a specific severity level, queries are used in the Logs Explorer and the Logging API.
     
  • Filter
    An expression in the Logging query language is referred to as a filter and is used in sinks to direct logs that match the expression to a storage location. Additionally, filters are used to send logs that match that expression to Cloud Monitoring when constructing metrics based on logs.
     
  • Monitored resources
    Each log entry includes the name of a resource that is being watched as a way to identify its source. Individual Google Kubernetes Engine containers, database instances, individual Compute Engine VM instances, and more are examples.
     
  • Log Router
    All logs are delivered to the Cloud Logging API, where they go through the Log Router, including audit logs, platform logs, and user-written logs. 
     
  • Sinks
    Log routing in Cloud Logging is managed by sinks. You may use sinks to either exclude log entries from being stored in Logging or to redirect some or all of your logs to supported destinations. The log entries to be routed through a sink are chosen by a filter and are sent to a destination.
     
  • Logs-based metrics
    A feature of cloud monitoring is metrics. The amount of log items that match a given query constitutes the value of a metric that is based on logs.
     
  • Access control
    Granting principals certain Identity and Access Management roles or permissions regulates who has access to logs.
     

Let's look at the details of access control with IAM.

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Access control with IAM 

Your ability to access logs data in the Logging API, the Logs Explorer, and the Google Cloud CLI is determined by IAM permissions and roles. A role is a group of privileges. A principal must be given a role; you cannot directly grant them permissions. Every privilege that a role possesses is granted to a principal when you assign them a role. The single principal can have many roles assigned to them.

Predefined roles

For granular access to particular Google Cloud resources and to prevent unauthorised access to other resources, IAM offers predefined roles. These roles are created and maintained by the Google Cloud, and whenever new functionality is added to Logging, for example, Google Cloud immediately updates the roles' permissions.

Consider the following when determining whether permissions and responsibilities apply to the use cases of your principals:

  • You are given all logging-related rights by roles/logging.admin (Logging Admin).
     
  • Principals have read-only access to the majority of Logging's functionality thanks to roles/logging.viewer (Logs Viewer).
     
  • All the permissions provided by roles/logging.viewer are included in privateLogViewer (Private Logs Viewer), in addition to the capability of reading Data Access audit logs in the _Default bucket.
     
  • Principals are given authorization to read logs, resource keys, and values using a log view, as well as to download logs using roles/logging.viewAccessor (Logs View Accessor).
     
  • If there are any restricted LogEntry fields in a specific bucket, roles/logging.fieldAccessor (Logs Field Accessor) allows principals authority to access them.
     
  • Principals are given the bare minimum of rights needed to write logs to the Logging API by roles/logging.logWriter (Logs Writer). Viewing rights are not provided by this position.
     
  • The minimal permissions necessary for a sink's service account to route logs to a particular bucket are granted by roles/logging.bucketWriter (Logs Buckets Writer).
     
  • Principals are given authorization to build or edit logging settings, such as sinks, buckets, views, log-based metrics, or exclusions, through roles/logging.configWriter (Logs Configuration Writer). Add roles/logging.viewer to the Logs Explorer to utilise it for these operations.
     
  • Roles/logging.viewer and roles/viewer (Project Viewer) are identical. All Logging functionalities, with the exception of seeing Data Access audit logs that are in the _Default bucket, are accessible to principals via read-only access under the role.
     
  • The permissions of roles/logging.viewer are included in roles/editor (Project Editor), along with the ability to write log entries, remove logs, and generate log-based analytics. Principals are prohibited from creating sinks or reading Data Access audit logs that are located in the _Default bucket by the role.
     
  • Principals have complete access to Logging, including the ability to read Data Access audit logs, thanks to roles/owner (Project Owner).
     

Custom roles

Follow these steps to create a custom role with logging permissions:

  • Make your selection from the list of API permissions, then follow the instructions to establish a custom role to allow access to the Logging API.
     
  • Choose from the Console permissions permission groups to establish a custom role that grants access to the Logs Explorer, then follow the on-screen directions.
     
  • See the Command-line permissions section of this page to learn how to build a custom role that allows users to access gCloud logging.
     

Google Cloud console permissions

The permissions required to utilise the Logs Explorer are listed in the following table:

Let's look at the command line permissions.

Command-line permissions

IAM permissions have power over gcloud logging commands.Principals must have the serviceusage.services.use permission in order to use any of the gcloud logging commands.

Additionally, a principal needs to be assigned the IAM role that goes with the use case and resource assigned to the log. See command-line interface permissions for further information. Let's look at the log routing permissions.

Log-routing permissions

Configuring sinks is connected with managing exclusion filters. The logging.sinks.* permissions cover all rights relevant to controlling sinks, including setting exclusion filters. Add the logging.sinks.* permissions to the role rather than the logging.exclusions.* permissions when building a new role that contains permissions to manage exclusion filters. After your log entries have been forwarded to one of the available destinations (Cloud Storage, BigQuery, or Pub/Sub), access to the log copy is solely governed by IAM permissions and roles on those destinations.

Let's dive into the details of Audit logs.

Audit logs for Cloud Logging

To assist you in determining "Who did what, where, and when?", regarding your Google Cloud resources, Google Cloud services keep audit logs.
The audit logs for resources that are present directly within your Google Cloud projects are included. The audit logs for the entity itself are contained in other Google Cloud resources including files, organisations, and billing accounts.

Available audit logs

The following audit log categories are accessible for logging:

  • Admin Activity audit logs
     
  • It comprises "admin write" activities that store settings or metadata.
     
  • Audit logs for admin activity cannot be turned off.
     
  • Data Access audit logs
     
  • It contains "admin read" actions that read configuration or metadata. comprises "data write" and "data read" actions that read or write data supplied by the user. You must specifically enable Data Access audit logs in order to receive them.
     

Audit log format

The following objects are included in audit log entries:

  • The actual log entry, an object of type LogEntry. The following fields are useful:
    • The resource ID and audit log type are both contained in the logName.
       
    • The resource includes the audited operation's aim.
       
    • The time of the audited operation is contained in the timeStamp.
       
    • The audited data is contained in the protoPayload.
       
  • The protoPayload field of the log entry contains an AuditLog object that contains the audit logging information.
     
  • A service-specific object that contains optional service-specific audit data. This object is stored in the AuditLog object's serviceData field for older integrations and in the metadata field for later integrations.
     

Log name

The names of the Cloud Audit Logs logs contain resource identifiers that show which Google Cloud project or other entity owns the audit logs and if they contain information about Admin Activity, Data Access, Policy Denied, or System Events.

Service name

The service name for logging audit logs is logging.googleapis.com.

Enable audit logging

Audit logs for admin activity are never disabled; they are always enabled.
Except for Data Access audit logs for BigQuery, which cannot be disabled, data access audit logs are disabled by default and are not written unless expressly enabled.

View logs

You must know the audit log name, which contains the resource identifier of the Cloud project, folder, billing account, or organisation for which you wish to view audit logging information, in order to conduct a query for audit logs. You can also include additional indexed LogEntry fields, like resource.type, to your query.

You may access your audit log entries for your Cloud project, folder, or organisation via the Google Cloud console using the Logs Explorer:

  • Navigate to the Logging> Logs Explorer page in the Google Cloud console.
     
  • Choose a project, folder, or organisation that is already in the Cloud.
     
  • Make the following changes in the Query builder pane:
    • Choose the Google Cloud resource you want to view the audit logs for under Resource type.
       
    • Choose the audit log type you want to view under Log name:
      • Select activity to view the audit logs for admin activity.
         
      • Select data_access to view the audit logs for Data Access.
         
      • Select system_event to view the audit logs for System Events.
         
      • Select policy to view audit logs for Policy Denied.
         

If you don't see these choices, then the Cloud project, folder, or organisation doesn't have any audit logs of that kind.

Route audit logs

The same methods used to route other types of logs can also be used to route audit logs to supported destinations. You may want to route your audit logs for the following reasons:

  • You can route the copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub if you want to preserve them for a longer time or utilise more robust search tools. You can route to different applications, different repositories, and outside parties via Pub/Sub.
     
  • You can establish aggregated sinks that can route logs from any or all Cloud projects in the business in order to manage your audit logs throughout the entire organisation.
     
  • You can build sinks that exclude the Data Access audit logs from Logging if your enabled Data Access audit logs are pushing the Cloud projects beyond your log allotments.
     

Let's look at the details of available logs in Google Cloud.

Available logs

Google Cloud, virtual machine (VM) instances that are running the Logging agent, and other cloud service providers are just a few of the sources that Logging receives, indexes, and stores log information from.
The types of logs that Logging supports are roughly categorised in the following areas. The log categories aren't mutually exclusive; they're only there to help you explain the logging data that's at your disposal.

Google Cloud platform logs

The platform logs for Google Cloud are service-specific logs that can aid in problem-solving and better understanding the Google Cloud services you're utilising.
Depending on whatever Google Cloud monitored resources you're utilising in your Cloud project, folder, or organisation, the Google Cloud platform logs that are available to you in Cloud Logging change.

User-written logs

Users can write their own logs to Cloud Logging in a variety of ways, including utilising the logging agent, the Cloud Logging API, or the Cloud Logging client libraries. Users' written programs and services are mentioned in user-written logs.

As a counterexample, App Engine publishes logs to Cloud Logging on behalf of the user's software components. Because App Engine is writing the logs to Cloud Logging on the user's behalf in this counterexample, the component logs aren't regarded as user-written logs.

Agent logs

A process called the Logging agent gathers logs from user applications and writes them to the Cloud Logging API.

Component logs

Platform and user-written logs combine in component logs. These logs might have a similar function to platform logs, but their log entry structure is different. A few software components, such the fluentd logging agent or Knative logging on Google Kubernetes Engine, produce logs while running on your virtual machine or computing hardware but are provided by Google. The logs that the software component generates occasionally offer a service to the users of the component.
For instance, GKE offers software components that customers can utilise to run their own virtual machines or data centres. The user's Cloud project receives logs from their GKE instances, but GKE uses the logs or their metadata to offer user support.

Security logs

The following information outlines the two types of security-related logs that Cloud Logging offers: Cloud Audit Logs and Access Transparency Logs.

Cloud Audit logs

For each Cloud project, folder, and organization, Cloud Audit Logs provide the following audit logs:

  • Admin Activity audit logs
     
  • System Event audit logs
     
  • Data Access audit logs
     
  • Policy Denied audit logs
     

Your Google Cloud resources' administrative changes and data accesses are audited using the Cloud Audit Logs.

Access Transparency logs

You may view activity logs of how Google employees accessed your Google Cloud content thanks to Access Transparency. You may monitor compliance with the legal and regulatory standards for your organisation using Access Transparency logs.

Multi-cloud and hybrid-cloud logs

Logs from other cloud service providers, such as Microsoft Azure and Amazon Web Services, can be ingested via Cloud Logging (AWS). They are referred to as "multi-cloud" logs.
As well as supporting logs from your on-premises infrastructure and applications, cloud logging. The logs that Cloud Logging receives when you expand it to include your on-premises resources are referred to as "hybrid-cloud" logs.

Understand and manage logs

Basic information on using and managing logs in Cloud Logging is provided in the sections that follow:

Log entry structure

A single data type, LogEntry, which defines certain common data for all log entries as well as the payloads that Google Cloud services can contain, is used by Cloud Logging to represent logs.

These details are used to identify each log entry:

  • Log name. This comprises the LOG ID log type identifier as well as the name of the Cloud project, folder, or organisation that houses the log entry.
     
  • The resource that the log entry was created from. This is made up of a resource type from the list of monitored resources and label values that are specific to that instance.
     
  • A Timestamp.
     
  • A payload, which can be one of textPayload, jsonPayload, or protoPayload (for some Google Cloud services).
     

You can usually determine which of the aforementioned log categories a log entry belongs to by looking at the LOG ID string in the logName field of each log entry:

  • It is a Cloud Audit Logs, or Access Transparency log entry if LOG ID contains the string cloudaudit.googleapis.com.
     
  • It is a platform log entry if LOG ID is preceded by serviceName.googleapis.com.
     
  • If LOG ID doesn't start with a service prefix, the log entry was probably created by a user.
     

View logs

Your log entries can be viewed in a variety of ways:

  • See Using the Logs Explorer for instructions on how to read logs using the Google Cloud console.
     
  • See the entries.list function to read log entries using the Logging API.
     
  • See Reading log entries for information on reading logs using the Google Cloud CLI.
     

Route logs

You have control over the storage and routing of your log entries. To decide which log entries to reject, which log entries to store in logging, and which log entries to include in exports to other destinations, the Log Router compares each log item to the set of rules.

Let's look at the details of the Google Cloud Platform logs.

Google Cloud platform logs

Google Cloud services send log entries to logging, which collects, indexes, and stores them. The platform logs for Google Cloud are service-specific logs that can aid in problem-solving and better understanding the Google Cloud services you're utilising.
Depending on whatever Google Cloud resources you're using in your Cloud project or organisation, different Google Cloud platform logs may be visible to you in Cloud Logging.

Index of Google Cloud platform logs

The tables that follow list some of the Google Cloud platform logs that Cloud Logging supports. See Log entry structure on this page to comprehend the information in the tables.

  • AlloyDB
    Logging API service name: alloydb.googleapis.com 
     
  • Assured Workloads
    Logging API service name: assuredworkloads.googleapis.com
     
  • Backup for GKE
    Logging API service name: gkebackup.googleapis.com
     
  • BigQuery Data Transfer API
    Logging API service name: bigquerydatatransfer.googleapis.com
     
  • Chat
    Logging API service name: chat.googleapis.com
     
  • Cloud Database Migration Service
    Logging API service name: datamigration.googleapis.com
     
  • Cloud Deploy
    Logging API service name: clouddeploy.googleapis.com
     
  • Cloud Functions
    Logging API service name: cloudfunctions.googleapis.com
     
  • Cloud Integrations
    Logging API service name: integrations.googleapis.com
     
  • Cloud Load Balancing
    Logging API service name: loadbalancing.googleapis.com
     
  • Cloud Logging API
    Logging API service name: logging.googleapis.com
     
  • Cloud Monitoring API
    Logging API service name: monitoring.googleapis.com
     
  • Cloud Run
    Logging API service name: run.googleapis.com
     
  • Cloud SQL
  • Logging API service name: cloudsql.googleapis.com
     
  • Cloud Setup
    Logging API service name: cloudsetup.googleapis.com
     
  • Dataproc
    Logging API service name: dataproc.googleapis.com
     
  • Dataproc Metastore
    Logging API service name: metastore.googleapis.com
     
  • Dialogflow
    Logging API service name: dialogflow-runtime.googleapis.com
     
  • Event Threat Detection
    Logging API service name: threatdetection.googleapis.com
     
  • Firestore/Datastore
    Logging API service name: datastore.googleapis.com
     
  • GKE On-prem (Anthos)
    Logging API service name: anthos.googleapis.com
     
  • Google Binary Authorization API
    Logging API service name: binaryauthorization.googleapis.com
     
  • Google Workspace Add-ons
    Logging API service name: gsuiteaddons.googleapis.com
     
  • Immersive Stream for XR
    Logging API service name: stream.googleapis.com
     
  • Live Stream API
    Logging API service name: livestream.googleapis.com
     
  • Network Analyzer
    Logging API service name: networkanalyzer.googleapis.com
     
  • Partner Data Ingestion
    Logging API service name: partnerdataingestion.googleapis.com
     
  • Recommender API
    Logging API service name: recommender.googleapis.com
     
  • SaaS Lifecycle Management
    Logging API service name: saasmanagement.googleapis.com
     
  • Secret Manager API
    Logging API service name: secretmanager.googleapis.com
     
  • Serial Console
    Logging API service name: serialconsole.googleapis.com
     
  • Storage Transfer
    Logging API service name: storagetransfer.googleapis.com
     
  • Traffic Director
    Logging API service name: trafficdirector.googleapis.com
     
  • Vertex AI API
    Logging API service name: aiplatform.googleapis.com
     
  • Workflows
    Logging API service name: workflows.googleapis.com
     
  • reCAPTCHA Enterprise
    Logging API service name: recaptchaenterprise.googleapis.com
     

Let's look at the detailed structure of log entry.

Log entry structure

A single data type, LogEntry, which defines certain common data for all log entries as well as the payloads that Google Cloud services can contain, is used by Cloud Logging to represent logs. See the Logging API reference for the LogEntry class for more information on the contents of log entries.

The platform logs' index draws attention to the following traits of log entries:

  • Log ID: The logName's URL-encoded identity. The log ID is cloudresourcemanager.googleapis.com percent 2Factivity, for instance, if the broader log name is organizations/1234567890/logs/cloudresourcemanager.googleapis.com% 2Factivity.
     

It should be noted that a log entry is for the Google Cloud platform if the log ID string is prefixed with serviceName.googleapis.com in the logName field.

  • Monitores Resource: The resource from which the log entry was generated is known as a monitored resource. This consists of an instance-specific label value and a resource type from the list of Monitored resources.
     
  • Payload: TextPayload or JSONPayload are two different ways to represent the payload of a log entry.
     

Let's continue our discussion about Cloud Logging in GCP in our next blog.

Frequently Asked Questions

What is Cloud Logging?

Cloud Logging is a fully managed service that enables you to store, search, analyse, monitor, and alert on logging data and events from Google Cloud and Amazon Web Services.

What is sinks?

Sinks regulate the way that Cloud Logging routes logs. Sinks allow you to send some or all of your logs to recognised destinations.

What is a Cloud Audit log?

The Cloud Audit Logs feature Admin Activity logs that capture administrative events and Data Access logs that track user access to your cloud data.

Conclusion

In this article, we have extensively discussed the details of Cloud Logging in GCP along with details of Access control with IAM , Audit logs , and details of Google Cloud Logs.

We hope that this blog has helped you enhance your knowledge regarding Cloud Logging in GCP, and if you would like to learn more, check out our articles on Google Cloud Certification. You can refer to our guided paths on the Coding Ninjas Studio platform to learn more about DSADBMSCompetitive ProgrammingPythonJavaJavaScript, etc. To practice and improve yourself in the interview, you can also check out Top 100 SQL problemsInterview experienceCoding interview questions, and the Ultimate guide path for interviews. Do upvote our blog to help other ninjas grow. Happy Coding!!

Thank You Image
Next article
Cloud Logging in GCP-Part 2
Live masterclass