Viewing Logs in Cloud logging

How to use Log Explorer 

To start with the log explorer, follow the steps below:

• To view logs, make sure you have the appropriate Identity and Access Management role.
• If you use Amazon Web Services Elastic Compute Cloud (AWS EC2), your log entries will be stored in the AWS connector project, which connects your AWS account to Google Cloud services.
• It's worth noting that if you use BindPlane to ingest on-premise and hybrid cloud logs, you'll find them under the resource type generic node.
• Go to Logs Explorer and Choose the correct Cloud project.

Log Explorer Interface 

The log explorer interface is very attractive and user-friendly. The GUI is really great and very much easy and understandable for users. It allows you to retrieve logs, parse and analyze log data, and refine your query parameters.

Let us see each of the options in detail below:

Action Toolbar

Options include visiting the Legacy Logs Viewer, providing feedback, and viewing a summary of new Logs Explorer features.

The following benefits are given by the action toolbar.

• Refine your search by limiting it to logs in your current Cloud project or one or more storage views. See Refine scope for more information on scoping.
• View links to relevant documentation and topics to learn more.
• Share link: To make it easier to share a query, create a shortened URL of the current query and copy it to your clipboard. The copied URL contains the absolute time range that corresponds to the current time range of your query.

Query Pane

The following tasks are performed using the query pane:

• Query tab: Use the following features to create and refine queries:
• Time-range selector: Limit query results based on a time range. 
• Find log entries that fit your search words or phrases using the search-text box. 
• Build queries using the Resource, Log name, and Severity filter menus. See Use filter menus for more information.
• Query-editor field: Use the Logging query language to create advanced queries. 
• View queries that you have recently run in the Recent tab.
• Saved tab: View your saved queries as well as queries shared by other Cloud project users.
• View suggested queries based on the resources in your Cloud project
• View and run Google-provided queries based on your use cases in the Library tab. 
• Save queries so that they can be viewed and run from the Saved tab.
• Clear query: Clears and resets the selections you made in the Query pane when creating a query.
• Stream logs: View log entries as they are ingested by Logging. 
• Run query: After you've built your queries in the Query pane, you can run them.

Results Toolbar

The following tasks are performed by the results toolbar.

• Show Logs field pane: This pane can be quickly shown or hidden from your Logs Explorer page layout.
• Show Histogram pane: This pane can be quickly shown or hidden from your Logs Explorer page layout.
• Create metric: Create a log-based metric using the current query expression.
• Create an alert: Create a log-based alert using your current query expression.
• Jump to now: Forcibly refresh your query results to include the current time. If the time-range selector is used with a custom range and an end time is specified, the query is run with a one-hour default time range. Otherwise, it refreshes the query with the current start date or duration and runs it.

Log Fields Pane

The Log fields pane provides a high-level summary of log data and a faster way to refine a query. It displays log entries divided into dimensions that correspond to fields in these entries. The Log fields pane displays values and their frequency in descending frequency order for each field.

Analyze logs using the Log fields pane: Add field-value pairs from the Log fields pane to the expression in the Query pane to narrow and refine a query. To do so, in the Log fields pane, click on the value of a field. Using the AND operator adds the field-value pair to the expression. The query is then executed.

Histogram

The Histogram pane displays the distribution of logs over time. When you run a query, the histogram regenerates, making it easier to see trends in your log data and troubleshoot issues. Click the Histogram button in the Results toolbar to show or hide the Histogram pane.

Query Results

You can investigate the log entries that fit your query representations and the time range you've chosen using the histogram's time controls or the time-range selector.

The following tasks are performed by query results:

• Query results: View the structured JSON formatted log entries returned by your query. 
• Change the time zone in which logs are displayed.
• Download history: Logs from your query results can be downloaded.
• Hide logs that are similar: Hide a large number of similar logs from the query results.
• Data traces: View trace details and use the trace to refine your query. View trace data for more information.
• Log summary should be hidden: Remove the line with the log summary from the query results.
• Expanding or collapsing nested log fields: Expanding or collapsing nested fields
• To copy to the clipboard: Make a copy of the log entry in JSON format.
• Share the link to a log entry: Copy a link to a log entry. 
• In the Summary column header, click the Edit button: Add a log field's value to the summary lines of your log entries. You can also hide the default summary fields entirely to see your entries in raw text. 
• Log entry for the pin: A log entry can be pinned to the Query results and Histogram panes.
• Scrolling the cursor: Scroll through the logs in the Query results; the URL changes to include the cursor. Timestamp, which indicates the time of the most recent log displayed in the current Query results pane.

Pin log Entries

To pin a log entry, perform the following steps:

• Hover your mouse over the log entry you want to save.
• Push the pin icon push pin.
• When you pin a log entry, its background darkens and the pin icon push pin appears.

View trace data: The latency and trace icons appear when a log entry includes both the trace and the latency-related field. When a log entry contains only the trace field, the trace icon is displayed.

Click the trace icon to view the trace data associated with the log entry. You can choose from the following options:

• View trace information: Displays the parent span and child traces, as well as trace details. To see more information about the trace, go to Cloud Trace and click View in Trace.
• Display all logs associated with this trace: The query is refined and executed by appending the trace field set to the identifier of the trace associated with the log entry.
• Display only traced requests

View Monitoring data

The following tasks are performed here:

• View monitoring details: This command displays a details panel for a GKE resource. 
• View in Monitoring: navigates to the resource's Monitoring page.
• View in GKE or View in Compute Engine: opens the resource's details page in the GKE or Compute Engine user interface.

Method: entries.list

This method is used to obtain log entries originating from a project/folder/organization/billing account.

HTTP request: POST

Request body: The request body contains data with the following structure:

JSON representation

{
  "projectIds": [
    string
  ],
  "resourceNames": [
    string
  ],
  "filter": string,
  "orderBy": string,
  "pageSize": integer,
  "pageToken": string
  }

 

Response body: If successful, the response body contains data with the following structure:

Results returned from entries. list.

JSON representation

{
  "entries": [
    {
      object (LogEntry)
    }
  ],
  "nextPageToken": string
}

Authorization Scopes

It needs one of the following authorization scopes:

• https://www.googleapis.com/auth/logging.read
• https://www.googleapis.com/auth/logging.admin
• https://www.googleapis.com/auth/cloud-platform.read-only
• https://www.googleapis.com/auth/cloud-platform

View logs in sink destinations

This section describes how to locate log entries routed from Cloud Logging to supported destinations.

Cloud Storage

Step 1: In the Google Cloud console, navigate to Cloud Storage Browser

Step 2: Navigate to the Cloud Storage Browser.

Step 3: Choose the Cloud Storage bucket that will serve as your routing destination.

Routing frequency

Hourly batches of log entries are saved to Cloud Storage buckets. It could take up to 3 hours for the first entries to appear.

Logs organization

Logging writes a set of files to a Cloud Storage bucket when you route logs to it.

The files are arranged in directory hierarchies according to log type and date. In the LogEntry reference, the log type, referred to as [LOG ID], can be a simple name like syslog or a compound name like appengine.googleapis.com/request log. If these logs were stored in a bucket called my-gcs-bucket, the directories would be named as follows: 

my-gcs-bucket/syslog/YYYY/MM/DD/

my-gcs-bucket/appengine.googleapis.com/request_log/YYYY/MM/DD/

Logs from multiple types of resources can be stored in a single Cloud Storage bucket. Each file is about 3.5 GiB in size.

BigQuery

Step 1: Navigate to the Google Cloud console's BigQuery page:

Step 2: Choose the dataset that will be the sink's destination.

Step 3: Choose a table from the dataset. You can view the log entries on the Details tab, or you can query the table to get your data.

Table organization

Logging creates dated tables to retain the routed log entries when you route logs to a BigQuery dataset. Log entries are placed in tables with names based on the log names and timestamps of the entries. 

Pub/Sub

Do the following to view your routed logs as they are streamed through Pub/Sub:

Step 1: Navigate to the Google Cloud console's Pub/Sub-page:

Step 2: Find or create a subscription to the log sink's topic and pull a log entry from it. It is possible that you will have to wait for a new log entry to be published.

Shutdown of Legacy Logs Viewer

The Logs Explorer is an updated version of the Logging interface that allows you to retrieve, view, and analyze logs from your queries quickly and efficiently.

Commonly used Features of Log Explorer

The Logs Explorer gives you the most recent versions of all Legacy Logs Viewer features.

The following information will help you locate the most commonly used log-viewing features:

Search for text across all log fields

Fill in the search query with text. To find log entries that contain a phrase, use quotation marks.

Construct queries with filter menus

To quickly build queries without attempting to know the Logging query language, use the Resource, Log name, and Severity filter menus.

Quick time-range options

To quickly query over seconds, minutes, hours, or days, open the relocated time-range selector. Use the preset values or enter your own time span.

Jump to a specific time

Open the time-range selector and select Jump to time to jump to a specific time in the Query results pane. Enter a specific date and time, or paste an ISO 8601 timestamp while the date or time field is focused.

Set your regional preferences

To change the format of the date and time that appears in the Logs Explorer, open the time-range selector and select Enter custom range > Change date & time format. Your favored date and time format appears in the Logs Explorer after you update your preferences in the Language & region menu.

Frequently Asked Questions

What is GKE Autopilot?

Since its inception, Kubernetes, developed by Google, has become the de-facto standard for container orchestration within enterprises. Organizations that require the highest levels of stability, security, and scalability for their applications use Google Kubernetes Engine (GKE).

What exactly is the Google Distributed Cloud?

A variety of Google Cloud services, such as databases, machine learning, data analytics, and container management services, can be used to move or update apps and process data locally. Depending on the organization, distributed Cloud products may be operated in one of four locations: Google's network edge, Operator/Customer/Client data center. Google Distributed Cloud products are available in all four locations.

What exactly is a bucket of Google Cloud Storage?

Google Cloud Storage is based on the same slashing technology that powers Google products worldwide, making it simple to store, access, and secure your data. With Google Cloud Storage, you can save and control access to any amount of data, whether for an individual or a group.

Conclusion

To conclude this blog, firstly we discussed the features of log explorer. Then we looked at how to use the log explorer and its interface. We also discussed method entries. List and view logs in sink destination. Last but not the least, we discussed the shutdown of the legacy logs viewer. 

For more content, Refer to our guided paths on Coding Ninjas Studio to upskill yourself.
Check out this problem - Largest Rectangle in Histogram

Do upvote our blogs if you find them helpful and engaging!

Happy Learning!