Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Last Updated: Mar 27, 2024

Cloud Logging in GCP-Part 2

Leveraging ChatGPT - GenAI as a Microsoft Data Expert
Speaker
Prerita Agarwal
Data Specialist @
23 Jul, 2024 @ 01:30 PM

Introduction

In the previous blog  we had looked at the details of Cloud Logging in GCP along with details of Access control with IAM, Audit logs, and details of Google Cloud Logs. This blog explains the details of Cloud Logging in GCP along with detailed steps for writing query log entries with the gcloud CLI, using python script and Cloud Logging for Compute Engine VMs .

 Without further ado , let's get started.

Write and query log entries with the gcloud CLI

Let's get started with writing the log entries with the gcloud CLI.

Prerequisites

To finish this quickstart, you need a Google Cloud project that has billing enabled. If you don't have a cloud project or your cloud project doesn't have billing enabled, take the following actions:

  • Create an account if you're new to Google Cloud to see how well our products work in practical situations. Additionally, new users receive $300 in complimentary credits to run, test, and deploy workloads.
     
  • Choose or create a Google Cloud project from the project selector page in the Google Cloud dashboard.
     
  • Make sure your Cloud project's billing is enabled. Find out how to determine whether billing is enabled for a project.
     

Getting started

A set of commands called gcloud logging in the gcloud CLI offers a command-line interface to the Cloud Logging API.In this quickstart, the gcloud CLI commands can be executed using the Cloud Shell environment. In the Cloud Shell environment, the gcloud CLI is already installed.

Make sure the correct Cloud project is being used by the gcloud CLI and follow the following steps:

  • Click Activate Cloud Shell in the Google Cloud console.
     
  • The Cloud Shell launches in a window and shows a greeting. 
     
  • Run the following command, substituting PROJECT ID for the project ID of the Cloud project you want to use in place of the one listed in the welcome message:
gcloud config set project PROJECT_ID

Write log entries by using the gcloud CLI

Logging enables both organised and unstructured data in log entries. JSON data structures make up structured data, such as "weather": "Sunny." A string of text, such as "The first entry," is an example of unstructured data. The gcloud CLI is used in the following steps to create log entries with unstructured data and structured data:

  • Create a log entry in the my-test-log containing unstructured data as follows:
gcloud logging write my-test-log "The first entry."

You see the following message when the command has finished: Created log entry

  • Add a structured log entry to the log called "my-test-log" like follows:
gcloud logging write --payload-type=json my-test-log '{ "message": The second entry", "weather": "Sunny"}'

Structured data log entries must contain the —payload-type=json flag. Logging treats the payload as unstructured data if this field is left blank.

Let's look at the procedure to list long entries by using the gcloud CLI.

List log entries by using the gcloud CLI

Using the gcloud CLI, you can get log entries from Logging and display them. Run the following command, for instance, to retrieve and display log entries with a resource type of global:

gcloud logging read "resource.type=global"

Let's look at the procedure to list long entries by using APIs Explorer.

List log entries by using APIs Explorer

Refer to Using the APIs Explorer to execute Logging API calls without creating any code. Do the following to read a list of log items from Logging:

  • Visit the entries.list API method reference page here.
     
  • Set up and execute the API command:

The text after PROJECT ID should be changed:

"resourceNames": [
"projects/PROJECT_ID"
],
"filter": "resource.type=global",
"orderBy": "timestamp desc"
  • Copy the revised content from the previous step, and then paste it into APIs Explorer's Request body area.
     
  • Choose Execute.
     

Let's look at the procedure to view long entries.

View log entries in the Logs Explorer

You can use the Logs Explorer to view log entries in the Google Cloud console. The majority of Cloud projects keep a lot of logs; you can choose certain log entries by writing a query.

Follow these steps to view the log entries you made using the Logs Explorer:

  • Logs Explorer may be found in the Google Cloud console.
     
  • In the Google Cloud navigation bar, make sure your Cloud project is selected. Select your Cloud project using the drop-down menu, if necessary.
     
  • Go to the Resource menu and choose Global.
     
  • Wait a few minutes, then refresh the website if you don't see the Global menu item or your log entries. The time it takes for Logging to receive log entries varies.
     
  • Click the Global Menu right of a log entry to access its information.
     

The information for the initial log entry is kept in textPayload. Structured data that is saved in jsonPayload is found in the second log entry. The weather and the key message are both included in the structured payload.

Let's look at the procedure to query long entries.

Query log entries in the Logs Explorer

Using the query editor and, with structured logs, by the key and value, you can query log entries. For instance, carry out the following actions to show all log entries that include the word simple:

  • Go to the Logs Explorer in the Google Cloud console.
     
  • Go to the Resource menu and choose Global.
     
  • Enter the string simple in quotation marks in the query editor. Only the log entry is displayed in the logs list : A simple entry.
     
  • Remove the query string you added and click Run query after you have seen your log. Both log entries are displayed again.
     

Do the following to display all structured data log entries with weather as the key and partly in the value field:

  • The line resource is present in the query editor. type="global". Input the command as follows:
jsonPayload.weather:partly
  • Click Run search. 
     

The outcome is the single log entry : The Second Entry.

Lets dive into the details of write and query log entries using python script.

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Write and query log entries using a Python script

Follow the procedures to write and query log entries using a Python script.

Getting started

To finish this quickstart, utilise the Cloud Shell environment. The Cloud Shell comes with Python already installed. Follow the following steps:

  • Open the Cloud Shell, then check the setup of your Cloud project:
    • Click Activate Cloud Shell in the Google Cloud console.
       
  • The set Cloud project ID is repeated in the greeting. Run this command, substituting [PROJECT ID] for the ID of your project if this isn't the Cloud project you wish to use:
 gcloud config set project [PROJECT_ID]

Let's look at the details of cloning the source.

Clone source

Follow the following to set up your Cloud Shell for this quickstart:

  • Clone the python-logging project on GitHub:
git clone https://github.com/googleapis/python-logging
  • Snippets directory should now be selected:
  cd python-logging/samples/snippets

Let's look at the details of writing log entries.

Write log entries

The Python client libraries are used by the snippets.py script to write log entries to Logging. The following log entries are written by the script when the write option is provided on the command line:

  • A record that contains unstructured data but lacks a severity rating.
     
  • A record that has unstructured data and an ERROR severity level.
     
  • A JSON formatted data entry without a specified severity level.
     

Run the snippets.py script with the write option to add new log entries to the my-log log:

python snippets.py my-log write

Let's look at the details to view log entries.

View log entries

Run the snippets.py script with the list option to see the log entries in the Cloud Shell:

python snippets.py my-log list

Let's look at the details of deleting log entries.

Delete log entries

Run the snippets.py script with the option delete to eliminate every entry in the log my-log:

python snippets.py my-log delete

Lets dive into Cloud Logging for Compute Engine VMs

Cloud Logging for Compute Engine VMs

Follow the following procedure for Cloud Logging for Compute Engine VMs:

Create a Compute Engine VM instance

Because you install and set up software for this quickstart, we advise you to build a Compute Engine VM instance. Follow the following steps:

  • Go to Compute in the Google Cloud console, and then pick Compute Engine.
     
  • Click Create instance to start a virtual machine.
     
  • For your instance, complete the fields as follows:
    • Enter quickstart-vm in the Name field.
       
    • Select Small in the Machine type field.
       
    • Make sure Debian GNU/Linux is configured on the boot drive.
       
    • Select Allow HTTP traffic as well as Allow HTTPS traffic in the Firewall area.
       
    • Leave the default settings for the remaining fields.
       
  • Press Create. When your VM is prepared, it shows up in the Instances tab's list of instances.
     

Let's look at the procedure to install Apache Web Server.

Install Apache Web Server

Follow the following steps to install an Apache Web Server on your Compute Engine VM instance:

  • Select SSH under Connect in the Connect column to launch a terminal to your instance.
  • Run the following command to update the package lists on your instance:
sudo apt-get update
  • Run the following command to install an Apache2 HTTP Server:
sudo apt-get install apache2 php7.0
  • Use the URL http://EXTERNAL_IP to launch your browser and access your Apache2 HTTP server, where EXTERNAL_IP is the external IP address of your VM.

Let's look at the details to Install and configure the Ops Agent

Install and configure the Ops Agent

Install the Ops Agent using the terminal in order to gather logs and metrics from your Apache Web Server:

  • Click SSH in the Connect column to launch a terminal to your virtual machine instance.
     
  • Run the following command to install the Ops Agent:
    Command:
curl -sSO
https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install
The following command should be copied and then pasted into the terminal:

Command:

set -e
sudo tee /etc/google-cloud-ops-agent/config.yaml > /dev/null << EOF
metrics:
  receivers:
    apache:
      type: apache
  service:
    pipelines:
      apache:
        receivers:
          - apache
logging:
  receivers:
    apache_access:
      type: apache_access
    apache_error:
      type: apache_error
  service:
    pipelines:
      apache:
        receivers:
          - apache_access
          - apache_error
EOF


sudo service google-cloud-ops-agent restart
sleep 60

The configuration to gather and ingest logs and analytics from the Apache Web Server is created by the preceding operation.

Let's look at the steps to view the log.

View your logs

To see logs in the Google Cloud console, utilise the Logs Explorer. You can create a query to pick out specific log entries.

Follow the following to examine your logs in the Google Cloud console:

  • Select Logging from the navigation menu in the Google Cloud console, then click Logs Explorer.
     
  • Enable Show query in the Query pane. Open the query editor.
     
  • Expand the list of Cloud projects using the Cloud project selector to find your Cloud project ID, then copy it into the ID field.
     
  • The following query phrase, which only shows the apache access and apache error logs, should be copied and pasted into the query editor, replacing PROJECT ID with the project ID you copied in the previous step:
resource.type="gce_instance"
logName=("projects/PROJECT_ID/logs/apache_access" OR
"projects/PROJECT_ID/logs/apache_error")

 

  • To run a query, either click Run or hit Enter. The query's results are shown in the Query results window.

 

Must Read Apache Server

Frequently Asked Questions

What is the gcloud CLI?

A collection of tools for creating and managing Google Cloud resources is called the Google Cloud CLI. These tools enable you to automate a variety of typical platform operations via scripts, other automation methods, or the command line.

What is Ops Agent?

The Ops Agent is the principal agent for gathering telemetry from your Compute Engine instances. The Ops Agent combines logging and metrics into a single agent by using Fluent Bit for logs, which allows high-throughput logging, and the OpenTelemetry Collector for measurements.

What is structured logging?

Structured logging is the process of giving application logs a standardised, preset message format that enables them to be handled as data sets rather than text.

Conclusion

In this article, we have extensively discussed the details of Cloud Logging in GCP along with detailed steps for writing query log entries with the gcloud CLI, using python script and Cloud Logging for Compute Engine VMs . 

We hope that this blog has helped you enhance your knowledge regarding Cloud Logging in GCP, and if you would like to learn more, check out our articles on Google Cloud Certification. You can refer to our guided paths on the Coding Ninjas Studio platform to learn more about DSADBMSCompetitive ProgrammingPythonJavaJavaScript, etc. To practice and improve yourself in the interview, you can also check out Top 100 SQL problemsInterview experienceCoding interview questions, and the Ultimate guide path for interviews. Do upvote our blog to help other ninjas grow. Happy Coding!!

Thank You Image
Topics covered
1.
Introduction
2.
Write and query log entries with the gcloud CLI
2.1.
Prerequisites
2.2.
Getting started
2.3.
Write log entries by using the gcloud CLI
2.4.
List log entries by using the gcloud CLI
2.5.
List log entries by using APIs Explorer
2.6.
View log entries in the Logs Explorer
2.7.
Query log entries in the Logs Explorer
3.
Write and query log entries using a Python script
3.1.
Getting started
3.2.
Clone source
3.3.
Write log entries
3.4.
View log entries
3.5.
Delete log entries
4.
Cloud Logging for Compute Engine VMs
4.1.
Create a Compute Engine VM instance
4.2.
Install Apache Web Server
4.3.
Install and configure the Ops Agent
4.4.
View your logs
5.
Frequently Asked Questions
5.1.
What is the gcloud CLI?
5.2.
What is Ops Agent?
5.3.
What is structured logging?
6.
Conclusion