Cloud Logging in GCP-Part 2

Write and query log entries using a Python script

Follow the procedures to write and query log entries using a Python script.

Getting started

To finish this quickstart, utilise the Cloud Shell environment. The Cloud Shell comes with Python already installed. Follow the following steps:

• Open the Cloud Shell, then check the setup of your Cloud project:
  • Click Activate Cloud Shell in the Google Cloud console.
     
• The set Cloud project ID is repeated in the greeting. Run this command, substituting [PROJECT ID] for the ID of your project if this isn't the Cloud project you wish to use:

 gcloud config set project [PROJECT_ID]

Let's look at the details of cloning the source.

Clone source

Follow the following to set up your Cloud Shell for this quickstart:

• Clone the python-logging project on GitHub:

git clone https://github.com/googleapis/python-logging

• Snippets directory should now be selected:

  cd python-logging/samples/snippets

Let's look at the details of writing log entries.

Write log entries

The Python client libraries are used by the snippets.py script to write log entries to Logging. The following log entries are written by the script when the write option is provided on the command line:

• A record that contains unstructured data but lacks a severity rating.
   
• A record that has unstructured data and an ERROR severity level.
   
• A JSON formatted data entry without a specified severity level.
   

Run the snippets.py script with the write option to add new log entries to the my-log log:

python snippets.py my-log write

Let's look at the details to view log entries.

View log entries

Run the snippets.py script with the list option to see the log entries in the Cloud Shell:

python snippets.py my-log list

Let's look at the details of deleting log entries.

Delete log entries

Run the snippets.py script with the option delete to eliminate every entry in the log my-log:

python snippets.py my-log delete

Lets dive into Cloud Logging for Compute Engine VMs

Cloud Logging for Compute Engine VMs

Follow the following procedure for Cloud Logging for Compute Engine VMs:

Create a Compute Engine VM instance

Because you install and set up software for this quickstart, we advise you to build a Compute Engine VM instance. Follow the following steps:

• Go to Compute in the Google Cloud console, and then pick Compute Engine.
   
• Click Create instance to start a virtual machine.
   
• For your instance, complete the fields as follows:
  • Enter quickstart-vm in the Name field.
     
  • Select Small in the Machine type field.
     
  • Make sure Debian GNU/Linux is configured on the boot drive.
     
  • Select Allow HTTP traffic as well as Allow HTTPS traffic in the Firewall area.
     
  • Leave the default settings for the remaining fields.
     
• Press Create. When your VM is prepared, it shows up in the Instances tab's list of instances.
   

Let's look at the procedure to install Apache Web Server.

Install Apache Web Server

Follow the following steps to install an Apache Web Server on your Compute Engine VM instance:

• Select SSH under Connect in the Connect column to launch a terminal to your instance.
• Run the following command to update the package lists on your instance:

sudo apt-get update

• Run the following command to install an Apache2 HTTP Server:

sudo apt-get install apache2 php7.0

• Use the URL http://EXTERNAL_IP to launch your browser and access your Apache2 HTTP server, where EXTERNAL_IP is the external IP address of your VM.

Let's look at the details to Install and configure the Ops Agent

Install and configure the Ops Agent

Install the Ops Agent using the terminal in order to gather logs and metrics from your Apache Web Server:

• Click SSH in the Connect column to launch a terminal to your virtual machine instance.
   
• Run the following command to install the Ops Agent:
  Command:

curl -sSO
https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install
The following command should be copied and then pasted into the terminal:

Command:

set -e
sudo tee /etc/google-cloud-ops-agent/config.yaml > /dev/null << EOF
metrics:
  receivers:
    apache:
      type: apache
  service:
    pipelines:
      apache:
        receivers:
          - apache
logging:
  receivers:
    apache_access:
      type: apache_access
    apache_error:
      type: apache_error
  service:
    pipelines:
      apache:
        receivers:
          - apache_access
          - apache_error
EOF


sudo service google-cloud-ops-agent restart
sleep 60

The configuration to gather and ingest logs and analytics from the Apache Web Server is created by the preceding operation.

Let's look at the steps to view the log.

View your logs

To see logs in the Google Cloud console, utilise the Logs Explorer. You can create a query to pick out specific log entries.

Follow the following to examine your logs in the Google Cloud console:

• Select Logging from the navigation menu in the Google Cloud console, then click Logs Explorer.
   
• Enable Show query in the Query pane. Open the query editor.
   
• Expand the list of Cloud projects using the Cloud project selector to find your Cloud project ID, then copy it into the ID field.
   
• The following query phrase, which only shows the apache access and apache error logs, should be copied and pasted into the query editor, replacing PROJECT ID with the project ID you copied in the previous step:

resource.type="gce_instance"
logName=("projects/PROJECT_ID/logs/apache_access" OR
"projects/PROJECT_ID/logs/apache_error")

 

• To run a query, either click Run or hit Enter. The query's results are shown in the Query results window.

 

Must Read Apache Server

Frequently Asked Questions

What is the gcloud CLI?

A collection of tools for creating and managing Google Cloud resources is called the Google Cloud CLI. These tools enable you to automate a variety of typical platform operations via scripts, other automation methods, or the command line.

What is Ops Agent?

The Ops Agent is the principal agent for gathering telemetry from your Compute Engine instances. The Ops Agent combines logging and metrics into a single agent by using Fluent Bit for logs, which allows high-throughput logging, and the OpenTelemetry Collector for measurements.

What is structured logging?

Structured logging is the process of giving application logs a standardised, preset message format that enables them to be handled as data sets rather than text.

Conclusion

In this article, we have extensively discussed the details of Cloud Logging in GCP along with detailed steps for writing query log entries with the gcloud CLI, using python script and Cloud Logging for Compute Engine VMs . 

We hope that this blog has helped you enhance your knowledge regarding Cloud Logging in GCP, and if you would like to learn more, check out our articles on Google Cloud Certification. You can refer to our guided paths on the Coding Ninjas Studio platform to learn more about DSA, DBMS, Competitive Programming, Python, Java, JavaScript, etc. To practice and improve yourself in the interview, you can also check out Top 100 SQL problems, Interview experience, Coding interview questions, and the Ultimate guide path for interviews. Do upvote our blog to help other ninjas grow. Happy Coding!!