The Google Cloud operations product line includes Cloud Logging. It features log storage, the Logs Explorer user interface, and an API for managing logs programmatically. By using logging, you may read and write log entries, query your logs, and manage how your logs are routed, stored, and used.
This blog describes the specifics of Google Cloud Logging, Access Control with IAM, Audit Logs, Google Cloud Logs, and the methods for creating query log entries. Without further ado, let's get started with the basic concepts of Cloud Logging.
Basic Concepts
The following are the basics concepts and terminologies used in Cloud Logging in GCP
Log entries A log entry details a specific event or transaction that occurs in a computer system or records its status. Your own code, the Google Cloud services it utilises, third-party apps, and the infrastructure that Google Cloud relies on all contribute to the creation of log entries.
Logs Within a Google Cloud resource, such as a Cloud project, a log is a named collection of log entries. Only when log entries are present, do logs truly exist.
Retention period The retention period refers to how long Cloud Logging keeps log items on file. The entries are then removed. In Logging Quotas and Limits, the retention times for various log categories are specified.
Query In the Logging query language, a query is an expression that returns the log entries that match the expression. To select and see log entries, such as those from a certain VM instance or those coming in a specific time period with a specific severity level, queries are used in the Logs Explorer and the Logging API.
Filter An expression in the Logging query language is referred to as a filter and is used in sinks to direct logs that match the expression to a storage location. Additionally, filters are used to send logs that match that expression to Cloud Monitoring when constructing metrics based on logs.
Monitored resources Each log entry includes the name of a resource that is being watched as a way to identify its source. Individual Google Kubernetes Engine containers, database instances, individual Compute Engine VM instances, and more are examples.
Log Router All logs are delivered to the Cloud Logging API, where they go through the Log Router, including audit logs, platform logs, and user-written logs.
Sinks Log routing in Cloud Logging is managed by sinks. You may use sinks to either exclude log entries from being stored in Logging or to redirect some or all of your logs to supported destinations. The log entries to be routed through a sink are chosen by a filter and are sent to a destination.
Logs-based metrics A feature of cloud monitoring is metrics. The amount of log items that match a given query constitutes the value of a metric that is based on logs.
Access control Granting principals certain Identity and Access Management roles or permissions regulates who has access to logs.
Let's look at the details of access control with IAM.
Access control with IAM
Your ability to access logs data in the Logging API, the Logs Explorer, and the Google Cloud CLI is determined by IAM permissions and roles. A role is a group of privileges. A principal must be given a role; you cannot directly grant them permissions. Every privilege that a role possesses is granted to a principal when you assign them a role. The single principal can have many roles assigned to them.
Predefined roles
For granular access to particular Google Cloud resources and to prevent unauthorised access to other resources, IAM offers predefined roles. These roles are created and maintained by the Google Cloud, and whenever new functionality is added to Logging, for example, Google Cloud immediately updates the roles' permissions.
Consider the following when determining whether permissions and responsibilities apply to the use cases of your principals:
You are given all logging-related rights by roles/logging.admin (Logging Admin).
Principals have read-only access to the majority of Logging's functionality thanks to roles/logging.viewer (Logs Viewer).
All the permissions provided by roles/logging.viewer are included in privateLogViewer (Private Logs Viewer), in addition to the capability of reading Data Access audit logs in the _Default bucket.
Principals are given authorization to read logs, resource keys, and values using a log view, as well as to download logs using roles/logging.viewAccessor (Logs View Accessor).
If there are any restricted LogEntry fields in a specific bucket, roles/logging.fieldAccessor (Logs Field Accessor) allows principals authority to access them.
Principals are given the bare minimum of rights needed to write logs to the Logging API by roles/logging.logWriter (Logs Writer). Viewing rights are not provided by this position.
The minimal permissions necessary for a sink's service account to route logs to a particular bucket are granted by roles/logging.bucketWriter (Logs Buckets Writer).
Principals are given authorization to build or edit logging settings, such as sinks, buckets, views, log-based metrics, or exclusions, through roles/logging.configWriter (Logs Configuration Writer). Add roles/logging.viewer to the Logs Explorer to utilise it for these operations.
Roles/logging.viewer and roles/viewer (Project Viewer) are identical. All Logging functionalities, with the exception of seeing Data Access audit logs that are in the _Default bucket, are accessible to principals via read-only access under the role.
The permissions of roles/logging.viewer are included in roles/editor (Project Editor), along with the ability to write log entries, remove logs, and generate log-based analytics. Principals are prohibited from creating sinks or reading Data Access audit logs that are located in the _Default bucket by the role.
Principals have complete access to Logging, including the ability to read Data Access audit logs, thanks to roles/owner (Project Owner).
Custom roles
Follow these steps to create a custom role with logging permissions:
Make your selection from the list of API permissions, then follow the instructions to establish a custom role to allow access to the Logging API.
Choose from the Console permissions permission groups to establish a custom role that grants access to the Logs Explorer, then follow the on-screen directions.
See the Command-line permissions section of this page to learn how to build a custom role that allows users to access gCloud logging.
Google Cloud console permissions
The permissions required to utilise the Logs Explorer are listed in the following table:
Let's look at the command line permissions.
Command-line permissions
IAM permissions have power over gcloud logging commands.Principals must have the serviceusage.services.use permission in order to use any of the gcloud logging commands.
Additionally, a principal needs to be assigned the IAM role that goes with the use case and resource assigned to the log. See command-line interface permissions for further information. Let's look at the log routing permissions.
Log-routing permissions
Configuring sinks is connected with managing exclusion filters. The logging.sinks.* permissions cover all rights relevant to controlling sinks, including setting exclusion filters. Add the logging.sinks.* permissions to the role rather than the logging.exclusions.* permissions when building a new role that contains permissions to manage exclusion filters. After your log entries have been forwarded to one of the available destinations (Cloud Storage, BigQuery, or Pub/Sub), access to the log copy is solely governed by IAM permissions and roles on those destinations.
Let's dive into the details of Audit logs.
Audit logs for Cloud Logging
To assist you in determining "Who did what, where, and when?", regarding your Google Cloud resources, Google Cloud services keep audit logs. The audit logs for resources that are present directly within your Google Cloud projects are included. The audit logs for the entity itself are contained in other Google Cloud resources including files, organisations, and billing accounts.
Available audit logs
The following audit log categories are accessible for logging:
Admin Activity audit logs
It comprises "admin write" activities that store settings or metadata.
Audit logs for admin activity cannot be turned off.
Data Access audit logs
It contains "admin read" actions that read configuration or metadata. comprises "data write" and "data read" actions that read or write data supplied by the user. You must specifically enable Data Access audit logs in order to receive them.
Audit log format
The following objects are included in audit log entries:
The actual log entry, an object of type LogEntry. The following fields are useful:
The resource ID and audit log type are both contained in the logName.
The resource includes the audited operation's aim.
The time of the audited operation is contained in the timeStamp.
The audited data is contained in the protoPayload.
The protoPayload field of the log entry contains an AuditLog object that contains the audit logging information.
A service-specific object that contains optional service-specific audit data. This object is stored in the AuditLog object's serviceData field for older integrations and in the metadata field for later integrations.
Log name
The names of the Cloud Audit Logs logs contain resource identifiers that show which Google Cloud project or other entity owns the audit logs and if they contain information about Admin Activity, Data Access, Policy Denied, or System Events.
Service name
The service name for logging audit logs is logging.googleapis.com.
Enable audit logging
Audit logs for admin activity are never disabled; they are always enabled. Except for Data Access audit logs for BigQuery, which cannot be disabled, data access audit logs are disabled by default and are not written unless expressly enabled.
View logs
You must know the audit log name, which contains the resource identifier of the Cloud project, folder, billing account, or organisation for which you wish to view audit logging information, in order to conduct a query for audit logs. You can also include additional indexed LogEntry fields, like resource.type, to your query.
You may access your audit log entries for your Cloud project, folder, or organisation via the Google Cloud console using the Logs Explorer:
Navigate to the Logging> Logs Explorer page in the Google Cloud console.
Choose a project, folder, or organisation that is already in the Cloud.
Make the following changes in the Query builder pane:
Choose the Google Cloud resource you want to view the audit logs for under Resource type.
Choose the audit log type you want to view under Log name:
Select activity to view the audit logs for admin activity.
Select data_access to view the audit logs for Data Access.
Select system_event to view the audit logs for System Events.
Select policy to view audit logs for Policy Denied.
If you don't see these choices, then the Cloud project, folder, or organisation doesn't have any audit logs of that kind.
Route audit logs
The same methods used to route other types of logs can also be used to route audit logs to supported destinations. You may want to route your audit logs for the following reasons:
You can route the copies of your audit logs to Cloud Storage, BigQuery, or Pub/Sub if you want to preserve them for a longer time or utilise more robust search tools. You can route to different applications, different repositories, and outside parties via Pub/Sub.
You can establish aggregated sinks that can route logs from any or all Cloud projects in the business in order to manage your audit logs throughout the entire organisation.
You can build sinks that exclude the Data Access audit logs from Logging if your enabled Data Access audit logs are pushing the Cloud projects beyond your log allotments.
Let's look at the details of available logs in Google Cloud.
Available logs
Google Cloud, virtual machine (VM) instances that are running the Logging agent, and other cloud service providers are just a few of the sources that Logging receives, indexes, and stores log information from. The types of logs that Logging supports are roughly categorised in the following areas. The log categories aren't mutually exclusive; they're only there to help you explain the logging data that's at your disposal.
Google Cloud platform logs
The platform logs for Google Cloud are service-specific logs that can aid in problem-solving and better understanding the Google Cloud services you're utilising. Depending on whatever Google Cloud monitored resources you're utilising in your Cloud project, folder, or organisation, the Google Cloud platform logs that are available to you in Cloud Logging change.
User-written logs
Users can write their own logs to Cloud Logging in a variety of ways, including utilising the logging agent, the Cloud Logging API, or the Cloud Logging client libraries. Users' written programs and services are mentioned in user-written logs.
As a counterexample, App Engine publishes logs to Cloud Logging on behalf of the user's software components. Because App Engine is writing the logs to Cloud Logging on the user's behalf in this counterexample, the component logs aren't regarded as user-written logs.
Agent logs
A process called the Logging agent gathers logs from user applications and writes them to the Cloud Logging API.
Component logs
Platform and user-written logs combine in component logs. These logs might have a similar function to platform logs, but their log entry structure is different. A few software components, such the fluentd logging agent or Knative logging on Google Kubernetes Engine, produce logs while running on your virtual machine or computing hardware but are provided by Google. The logs that the software component generates occasionally offer a service to the users of the component. For instance, GKE offers software components that customers can utilise to run their own virtual machines or data centres. The user's Cloud project receives logs from their GKE instances, but GKE uses the logs or their metadata to offer user support.
Security logs
The following information outlines the two types of security-related logs that Cloud Logging offers: Cloud Audit Logs and Access Transparency Logs.
Cloud Audit logs
For each Cloud project, folder, and organization, Cloud Audit Logs provide the following audit logs:
Admin Activity audit logs
System Event audit logs
Data Access audit logs
Policy Denied audit logs
Your Google Cloud resources' administrative changes and data accesses are audited using the Cloud Audit Logs.
Access Transparency logs
You may view activity logs of how Google employees accessed your Google Cloud content thanks to Access Transparency. You may monitor compliance with the legal and regulatory standards for your organisation using Access Transparency logs.
Multi-cloud and hybrid-cloud logs
Logs from other cloud service providers, such as Microsoft Azure and Amazon Web Services, can be ingested via Cloud Logging (AWS). They are referred to as "multi-cloud" logs. As well as supporting logs from your on-premises infrastructure and applications, cloud logging. The logs that Cloud Logging receives when you expand it to include your on-premises resources are referred to as "hybrid-cloud" logs.
Understand and manage logs
Basic information on using and managing logs in Cloud Logging is provided in the sections that follow:
Log entry structure
A single data type, LogEntry, which defines certain common data for all log entries as well as the payloads that Google Cloud services can contain, is used by Cloud Logging to represent logs.
These details are used to identify each log entry:
Log name. This comprises the LOG ID log type identifier as well as the name of the Cloud project, folder, or organisation that houses the log entry.
The resource that the log entry was created from. This is made up of a resource type from the list of monitored resources and label values that are specific to that instance.
A Timestamp.
A payload, which can be one of textPayload, jsonPayload, or protoPayload (for some Google Cloud services).
You can usually determine which of the aforementioned log categories a log entry belongs to by looking at the LOG ID string in the logName field of each log entry:
It is a Cloud Audit Logs, or Access Transparency log entry if LOG ID contains the string cloudaudit.googleapis.com.
It is a platform log entry if LOG ID is preceded by serviceName.googleapis.com.
If LOG ID doesn't start with a service prefix, the log entry was probably created by a user.
View logs
Your log entries can be viewed in a variety of ways:
See Using the Logs Explorer for instructions on how to read logs using the Google Cloud console.
See the entries.list function to read log entries using the Logging API.
See Reading log entries for information on reading logs using the Google Cloud CLI.
Route logs
You have control over the storage and routing of your log entries. To decide which log entries to reject, which log entries to store in logging, and which log entries to include in exports to other destinations, the Log Router compares each log item to the set of rules.
Let's look at the details of the Google Cloud Platform logs.
Google Cloud platform logs
Google Cloud services send log entries to logging, which collects, indexes, and stores them. The platform logs for Google Cloud are service-specific logs that can aid in problem-solving and better understanding the Google Cloud services you're utilising. Depending on whatever Google Cloud resources you're using in your Cloud project or organisation, different Google Cloud platform logs may be visible to you in Cloud Logging.
Index of Google Cloud platform logs
The tables that follow list some of the Google Cloud platform logs that Cloud Logging supports. See Log entry structure on this page to comprehend the information in the tables.
AlloyDB Logging API service name: alloydb.googleapis.com
Assured Workloads Logging API service name: assuredworkloads.googleapis.com
Backup for GKE Logging API service name: gkebackup.googleapis.com
BigQuery Data Transfer API Logging API service name: bigquerydatatransfer.googleapis.com
Chat Logging API service name: chat.googleapis.com
Cloud Database Migration Service Logging API service name: datamigration.googleapis.com
Cloud Deploy Logging API service name: clouddeploy.googleapis.com
Cloud Functions Logging API service name: cloudfunctions.googleapis.com
Cloud Integrations Logging API service name: integrations.googleapis.com
Cloud Load Balancing Logging API service name: loadbalancing.googleapis.com
Cloud Logging API Logging API service name: logging.googleapis.com
Cloud Monitoring API Logging API service name: monitoring.googleapis.com
Cloud Run Logging API service name: run.googleapis.com
Cloud SQL
Logging API service name: cloudsql.googleapis.com
Cloud Setup Logging API service name: cloudsetup.googleapis.com
Dataproc Logging API service name: dataproc.googleapis.com
Dataproc Metastore Logging API service name: metastore.googleapis.com
Dialogflow Logging API service name: dialogflow-runtime.googleapis.com
Event Threat Detection Logging API service name: threatdetection.googleapis.com
Firestore/Datastore Logging API service name: datastore.googleapis.com
GKE On-prem (Anthos) Logging API service name: anthos.googleapis.com
Google Binary Authorization API Logging API service name: binaryauthorization.googleapis.com
Google Workspace Add-ons Logging API service name: gsuiteaddons.googleapis.com
Immersive Stream for XR Logging API service name: stream.googleapis.com
Live Stream API Logging API service name: livestream.googleapis.com
Network Analyzer Logging API service name: networkanalyzer.googleapis.com
Partner Data Ingestion Logging API service name: partnerdataingestion.googleapis.com
Recommender API Logging API service name: recommender.googleapis.com
SaaS Lifecycle Management Logging API service name: saasmanagement.googleapis.com
Secret Manager API Logging API service name: secretmanager.googleapis.com
Serial Console Logging API service name: serialconsole.googleapis.com
Storage Transfer Logging API service name: storagetransfer.googleapis.com
Traffic Director Logging API service name: trafficdirector.googleapis.com
Vertex AI API Logging API service name: aiplatform.googleapis.com
Workflows Logging API service name: workflows.googleapis.com
reCAPTCHA Enterprise Logging API service name: recaptchaenterprise.googleapis.com
Let's look at the detailed structure of log entry.
Log entry structure
A single data type, LogEntry, which defines certain common data for all log entries as well as the payloads that Google Cloud services can contain, is used by Cloud Logging to represent logs. See the Logging API reference for the LogEntry class for more information on the contents of log entries.
The platform logs' index draws attention to the following traits of log entries:
Log ID: The logName's URL-encoded identity. The log ID is cloudresourcemanager.googleapis.com percent 2Factivity, for instance, if the broader log name is organizations/1234567890/logs/cloudresourcemanager.googleapis.com% 2Factivity.
It should be noted that a log entry is for the Google Cloud platform if the log ID string is prefixed with serviceName.googleapis.com in the logName field.
Monitores Resource: The resource from which the log entry was generated is known as a monitored resource. This consists of an instance-specific label value and a resource type from the list of Monitored resources.
Payload: TextPayload or JSONPayload are two different ways to represent the payload of a log entry.
Cloud Logging is a fully managed service that enables you to store, search, analyse, monitor, and alert on logging data and events from Google Cloud and Amazon Web Services.
What is sinks?
Sinks regulate the way that Cloud Logging routes logs. Sinks allow you to send some or all of your logs to recognised destinations.
What is a Cloud Audit log?
The Cloud Audit Logs feature Admin Activity logs that capture administrative events and Data Access logs that track user access to your cloud data.
Conclusion
In this article, we have extensively discussed the details of Cloud Logging in GCP along with details of Access control with IAM , Audit logs , and details of Google Cloud Logs.
46+ registered 