Overview of log based metric

Overview of log-based metric types

Only included logs are used to construct system log-based metrics. Log-based metrics set by the user are calculated from both included and excluded logs.

A log-based metric's data is derived from log entries received after the metric is created. A metric is not retrospectively supplied with data from existing Logging log entries.

Each time a matched log entry is received, logging gathers information for a log-based metric. At a pace of one data point every minute, logging adds a new data point to the metric's time series, making the data available to Cloud Monitoring.

Each data point in the time series of a log-based metric indicates simply the new information (the delta) received since the preceding data point.

User-defined log-based metrics might be of the counter or distribution metric kinds. Most system-defined log-based metrics are counters. However, others are Boolean.

The sections that follow discuss the properties of counter-type and distribution-type metrics.

Counter metrics

The amount of log items that match a specified filter is counted using counter metrics. You can, for example, perform the following:

• Count the number of log entries that include a specific error message.
• Count the number of times each user performs an action by searching for log messages that match the following pattern:

... named OPERATION by the user USERNAME...

You may later query, "How many times did sally call the update operation?", "How many individuals called the read operation?", "How many times did george call an operation?" and so on by extracting USERNAME and OPERATION and utilizing them as values for two labels.

Configure counter metrics

Create a counter metric

You may use regular expressions to establish a filter for the log entries you wish to count in your statistic. It should be noted that the length of a filter cannot exceed 20,000 characters.

To establish a counter metric in the Google Cloud console for your Google Cloud project, follow these steps:

1. Navigate to Logging > Logs-based Metrics page: 
2. Click the Create Metric. The Create logs metric panel is displayed.
3. Select Metric type: Choose Counter as the Metric type.
4. In the Details section, fill in the following fields:
   Log metric name: Select a name that is distinct from the other logs-based metrics in your Cloud project. Some name limitations apply; for more information, see Troubleshooting in further sections of this page.
   Description: Enter a metric description.
   Units: Leave this field empty or enter 1.
5. In the Filter selection area, define your metric filter. Using the logging query language, create a filter that captures the log items you want to count in your statistic. You may also use regular expressions to construct filters for your metrics. Click Preview logs to bring up a screen with log items that match your filter.
6. (Optional) In the Labels area, add a label. 
7. To construct the metric, click Create metric.

New metric latency

Your new metric is immediately visible in the Logs Explorer list of metrics as well as the applicable Monitoring menus. Data is often accessible in about a minute.

Inspect counter metrics

To investigate a specific metric in your Google Cloud project or to list the user-defined logs-based metrics in your Cloud project, perform the following:

1. Navigate to the following page: Logs-Based Metrics
2. The user-defined logs-based metrics in the current Cloud project are displayed in the User-defined metrics pane:
3. To examine the data in a logs-based metric, pick View in Metrics Explorer from the option in the metric's row.

Update counter metrics

You can adjust the description, filter, and names of fields referenced in a user-defined logs-based metric. You may edit the regular expressions used to extract data for the metric and its labels and add new labels to it.

You cannot alter the names, kinds, or labels of user-defined logs-based metrics or delete existing labels in a logs-based metric.

To modify a logs-based measure, perform the following steps:

1. Navigate to the following page: Logs-Based Metrics
2. Click Edit metric in the menu for the logs-based metric you want to change.
3. Modify the metric's allowable items.
4. Click Update Metric.

Delete counter metrics

To remove a user-defined logs-based metric, perform the following steps:

1. Navigate to the following page: Logs-Based Metrics
2. Select the metric to be deleted and click the Delete option.

Alternatively, under the menu of the logs-based metric, you wish to delete, click Delete metric.

Distribution metrics

Distribution metrics collect numerical data from log entries that meet a filter. The metrics include a time series of distribution objects, each of which has the following information:

• The number of values in the distribution is counted.
• The average of the values
• The total of the squared deviations: Sumi=1..n(xi–mean) 2
• A collection of histogram buckets with the number of values in each. You can use the default bucket layout or create your own.

Distribution metrics are frequently used to measure latencies. A latency value is retrieved from someplace in the log entry and added to the distribution when each log entry is received. The cumulative distribution is written to Cloud Monitoring at regular intervals.

Configure distribution metrics 

Creating a distribution metric

You may use regular expressions to establish a filter for the log entries you wish to count in your statistic. It should be noted that the length of a filter cannot exceed 20,000 characters.

To establish a distribution metric in the Google Cloud console for your Google Cloud project, follow these steps:

1. Navigate to Logging > Logs-based Metrics page: 
2. Click the Create Metric. The Create logs metric panel is displayed.
3. Select Metric type: Choose Distribution as the Metric type.
4. In the Details section, fill in the following fields:
   Log metric name: Select a name that is distinct from the other logs-based metrics in your Cloud project. Some name limitations apply; for more information, see Troubleshooting in further sections of this page.
   Description: Enter a metric description.
   Units: (Optional) You can input units such as s, ms, and so on for distribution metrics.
5. In the Filter selection area, define your metric filter.
   Using the logging query language, create a filter that captures the log items you want to count in your metric. You may also use regular expressions to construct filters for your metrics.
   Field name: Enter the value of the distribution in the log input area. As you type, options are shown to you. For example:
   protoPayload.latency
   Extraction expression: (Optional) You can leave this field empty if the Field name always includes a numeric value converted to type double. In the absence of a regular expression, supply one that extracts the numeric distribution value from the field value. Example. Assume your latency log entry field comprises a number followed by ms, which stands for milliseconds. The regexp below selects the number without the unit suffix:  ([0-9.]+) The parenthesis, also known as a regexp capture group, indicates which part of the text match will be extracted.
   (Histogram buckets) Advanced: (Optional) By selecting Advanced, you may access a part of the form where you can define custom bucket layouts. If no bucket layouts are specified, a default bucket layout is supplied. Click Preview logs to bring up a screen with log items that match your filter.
6. (Optional) In the Labels area, add a label. See Create a label for details on how to do so.
7. To construct the metric, click Create metric.

Histogram buckets

A histogram that counts the number of values that fall within specific ranges is included in distribution metrics (buckets). A distribution metric can contain up to 200 buckets.

Each bucket contains two boundary values, L and H, determining the bucket's lowest and highest values. The bucket's width is H - L. As there can be no gaps between buckets, the lower border of one bucket corresponds to the upper boundary of the preceding bucket, and so on. A bucket contains its lower border so that the boundaries do not fall into more than one bucket; its upper boundary belongs to the next bucket.

All bucket layouts may be customized by specifying the boundary values between each bucket in increasing order. The underflow bucket is the first bucket, and it counts data below the first limit. The overflow bucket is the last bucket, and it counts values more than or equal to the last boundary. The values in the other buckets are more than or equal to their lower boundary and less than their upper boundary. There are n+1 buckets if there are n boundary values. There are n-1 finite buckets, excluding the underflow and overflow buckets.

There are three approaches to set the boundaries between histogram buckets for distribution metrics. You may either define a formula for the boundary values or list them:

• Linear(offset, width, i): The width of each bucket is the same. The boundaries are defined as offset + width * i, where i = 0, 1, 2,..., N.
• Exponential(scale, growth_factor, i): Bucket widths rise as the scale, growth_factor, and i increase. The boundaries are defined as scale * growth_factori, where i=0, 1, 2,..., N.
• Explicit: In the bounds array, you list all the buckets' boundaries. The bucket i has the following boundaries: 
  Upper bound: bounds[i] for (0 <= i < N-1)
  Lower bound: bounds[i - 1] for (1 <= i < N)

New metric latency

Your new metric is immediately added to the list of metrics and the appropriate Monitoring menus. However, the metric may take up to a minute to begin collecting data for the matching log entries.

Inspect distribution metrics

To inspect a specific metric in your Google Cloud project or to list the user-defined logs-based metrics in your Cloud project, perform the following:

Navigate to the following page: Logs-Based Metrics

The user-defined logs-based metrics in the current Cloud project are displayed in the User-defined metrics pane:

To examine the data in a logs-based metric, select View in Metrics Explorer from the menin the metric's row.

Update distribution metrics

You can adjust the description, filter, and names of fields referenced in a user-defined logs-based metric. You may edit the regular expressions used to extract data for the metric and its labels and add new labels to it.

You cannot alter the names, types, or labels of user-defined logs-based metrics or delete existing labels in a logs-based metric.

To modify a logs-based metric, perform the following steps:

• Navigate to the following page: Logs-Based Metrics
• Click Edit metric in the menu for the logs-based metric you want to change.
• Change the metric's allowable components.
• Click the Update Metric option.

Delete distribution metrics

To remove a user-defined logs-based metric, perform the following steps:

1. Navigate to the following page: Logs-Based Metrics
2. Select the metric to be deleted and click the Delete option.

Alternatively, under the menu of the logs-based metric, you wish to delete, click Delete metric.

Log-based metrics labels

Labels enable log-based metrics to incorporate multiple time series—one for each label value. All log-based metrics provide specific default labels.

Specifying extractor expressions allows you to create additional user-defined labels in both counter-type and distribution-type metrics. An extractor expression instructs Cloud Logging on extracting a label's value from log entries. The label's value can be one of the following:

• The LogEntry object's entire contents of a named field.
• A part of a named field corresponds to a regular expression (regexp).

Labels can be extracted from LogEntry built-in fields like httpRequest.status or payload fields like textPayload, jsonPayload, or protoPayload.

Limitations of user-defined labels

User-defined labels are subject to the following limitations:

• Each metric can have up to ten user-defined labels.
• You cannot delete a label after it has been created.
• You can change the extractor expression and description of an existing label.
• A label's name or value type cannot be changed after it has been created.
• A label value's first 1,024 characters are retained.
• Each log-based metric is restricted to about 30,000 active time series, determined by the number of available labels, including default labels.

Default labels

All log-based metrics include the following labels:

Resource labels: All metrics require a monitored resource object to identify the source of time series data. Each resource type has a name and one or more labels. VM instances, Cloud SQL databases, load balancers, and other resources are examples of resource categories.

In Cloud Monitoring, the resource and its labels are listed separately from other metric labels, but they have the same effect: they add time series to the metric.

log: In log entries, this label stores the value of the LOG ID component of the logName field.

severity: This label stores the severity field value in log entries. Only system log-based metrics include the severity label by default.

Create a label

When you create the metric, you create user-defined labels. Labels can be assigned to both counter metrics and distribution metrics. The system log-based metrics cannot be labeled.

1. The Create logs metric panel offers a labeling option when you create a log-based metric.
2. Click the Add label option.
   Tip: To view the fields and values included within a log entry, perform the following:
   Click Preview logs in the Filter selection section.
   Select a log entry in the View logs window and click the expander > next to it.
   Click the Expand nested fields option.
3. In the Labels section, fill in the following fields:
   Label name: Give the label a name. For instance, ID.
   The name must fulfill the following requirements:
   Be no longer than 100 characters.
   Match the regular expression [a-zA-Z][a-zA-Z0-9_]*.
   Include more than simply the string "log."
   Description: Describe the label in detail. Try to be as precise as possible about the desired log values' format. Instance number, for example.
   Label type: Select a label type from String, Boolean, or Integer.
   Field name: Enter the log entry field's name containing the label's value. As you type, options are shown to you. 
   Regular expression: If the value of your label is the entire contents of the field, you may leave this field empty. Alternatively, you may supply a regexp capture group that extracts the label value from the field value.
4. To finish creating the label, click Done. Repeat these steps to add more labels.
5. Click Create metric to finish creating the measure.

Configure charts and alerts

View logs-based metrics charts

Metrics Explorer is the quickest method to view a graph of your logs-based metrics:

1. In the Google Cloud console, go to Logs-based metrics:
2. Select View in Metrics Explorer from the Moreoption of the metric you want to view.

Specify logs-based metrics

If you want to narrow the charted data further, the Metrics Explorer's Metric tab allows you to build your query and dynamically show the results. The following fields are available for customization:

• Find resource type and metric:  The metric and resource type are predefined for you. You can change these settings if you like. The resource type cannot be specified if there is no data for the metric.
• Filter (optional): Choose which metrics to filter on.
• Group by(optional): Select metric labels to divide the time series.
• Aggregator (optional): Choose an aggregation function for combining several time series.
• Advanced options (optional): You can further specify your aggregate or customize your chart's legend.
• Add metric: After you've created your first query, you may create more queries to display more metrics in the charting area.

Create a logs-based metric chart

Create a monitoring dashboard chart that displays a measure based on logs. A counter metric or a distribution metric can be used. Carry out the following actions:

1. Go to Cloud Monitoring:
2. In the left-side navigation menu, go to Dashboards > Create Dashboard. You see a blank dashboard with an add chart button. To add a chart to an existing dashboard, go to Dashboards > [DASHBOARD NAME].
3. On the Charts page, click Add Chart. Fill out the form:

• Chart title: Give the chart a name.
• Resource type: Choose the Metric option.
• Metric: Choose the logs-based metric you want to display.
• Chart type: There is an option to build a heatmap for distribution metrics:

Other options for modifying the charted data may be found on the Charts page. You may, for example, filter data based on metrics and resource designations.

Charts for distribution metrics

In addition to the capabilities shared by all charts, distribution logs-based metrics-based charts contain the following features:

They may be represented graphically as heatmaps, with the intensity of the color in a certain bucket range on the y-axis corresponding to the number of data samples in that bucket. 

They can be shown as percentile lines by selecting an alignment option for the 5th, 50th, 95th, or 99th percentile values.

View logs related to charts

If a Monitoring chart contains data, there is a View Logs option in its Moremenu. By selecting this option, you will be sent to the Logs Explorer for the time period shown in the chart.

View Logs isn't just for log-based metrics; it also allows you to look at log entries related to the metrics presented in the chart.

Create an alerting policy on a counter metric

An alerting policy describes a collection of situations that you wish to monitor. When you develop an alerting policy, you must also set its requirements, such as what is watched and when an alert should be triggered.

To construct an alerting strategy for a counter statistic based on logs, do the following:

1. In the Google Cloud console, go to Logs-based metrics:
2. Choose the measure you wish to investigate. Create alert from metric under the Moremenu of the metric.
3. The Conditions pane appears with the pre-populated log-based measure you selected in the Target dialog.
4. Fill in the Threshold value in the Configuration dialog.
5. When you click Save, the Create new alerting policy screen with your completed condition appears.
6. Optional: Fill out the Notifications field. If you complete this part, when an incident is produced, a notification is delivered to each notification channel. This notification contains links to the incident as well as Logs Explorer.
7. Optional: Fill up the Documentation section. Notifications contain the documentation you select. The message usually contains instructions on how to respond to the alert notification.
8. Enter a name for the alerting policy in the Name this policy section.
9. Click on Save.

Troubleshoot log-based metrics

Cannot view or create metrics

Log-based metrics are only applicable to one Google Cloud project. They cannot be created for Logging buckets or other Google Cloud resources like billing accounts or organizations. Log-based metrics are computed solely for logs received in the Cloud project.

You must have the appropriate Identity and Access Management rights to build metrics.

Metric is missing logs data

Missing data in log-based metrics can occur for a variety of causes, including:

• New log entries may not meet the filter on your metric. A log-based metric obtains information from matched log entries received after the measure is generated. Logging does not use earlier log entries to backfill the metric.
• New log entries may lack the required information, or the data may be in an unsuitable format for extraction by your distribution metric. Check the spelling of your field names and regular expressions.
• It is possible that your metric counts will be delayed. Even when countable log items display in the Logs Explorer, updating the log-based metrics in Cloud Monitoring may take up to 10 minutes.
• Because they are time-stamped too far in the past or future, the displayed log entries may be counted late or not at all. If a log entry is received via Cloud Logging more than 24 hours ago or 10 minutes from now, it will not be counted in the log-based measure. The system log-based metric records the number of late-arriving items for each log-based metrics error count.
  logging.googleapis.com/logs_based_metrics_error_count.
• After the arrival of log entries that the metric could count, the log-based metric was produced. Log-based metrics assess log entries as they are fed into Logging; they do not evaluate log entries that have already been placed in Logging.

Resource type is "undefined" in Cloud Monitoring

Some Cloud Logging monitored-resource types do not translate to Cloud Monitoring monitored-resource types directly. For example, when you initially build an alert or chart using a log-based statistic, the resource type may be "undefined."

False-positive alerts or alerts that aren't triggered

Since the alignment period for the alert is too short, you may receive false-positive alerts or alerts that aren't triggered by log-based metrics. When an alert employs less than logic or the warning is based on a percentile condition for a distribution measure, an alignment period that is too short might cause difficulties.

Since log entries might be transmitted to Logging late, false-positive warnings can arise. In rare circumstances, the log fields timestamp and receiveTimestamp might have different minutes. Furthermore, there is an inherent latency between when log entries are created and when Logging receives them when it ingests logs. This implies that the overall count for a specific log entry may not be available until some time after the log entries have been produced. This is why an alert based on less than logic or a percentile criterion for a distribution measure might generate a false-positive alert: not all log entries have been accounted for.

However, log-based measurements are always finally consistent. Log-based metrics become consistent over time because a log entry matching a log-based metric might be transmitted to Logging with a timestamp substantially older or younger than the log's receiveTimestamp.

This means the log-based metric can get log items with earlier timestamps after Logging has previously received log entries with the same timestamp. As a result, the metric value must be revised.

Alert rules for log-based metrics should utilize alert conditions with alignment periods larger than two minutes to ensure alert accuracy even for on-time data. An alignment period of ten minutes is advised for log entries transmitted to Logging, with delays measured in minutes to balance timeliness and accuracy.

Metric has too many time series

The number of time series in a metric is determined by the number of possible label value combinations. The cardinality of the measure is the number of time series that must not exceed 30,000.

Because a time series may be generated for any combination of label values, if you have one or more labels with a large number of values, you can easily reach 30,000 time series. High-cardinality measurements should be avoided.

As the cardinality of a metric grows, the metric may be throttled, and some data points may be missed. Because of the enormous amount of time series that the chart must analyze, charts displaying the metric might be slow to load. You may also incur expenses for API requests to access time series data.

To avoid creating metrics with a high cardinality:

• Make sure your label fields and extractor regular expressions match values with low cardinality.
• Avoid extracting text messages that can alter, without boundaries, as label values.
• Avoid obtaining numerical numbers with infinite cardinality.
• Only extract values from labels with known cardinalities, such as status codes with a known set of values.

Metric name is invalid

When you create a counter or distribution metric, give it a name that is distinct from the other log-based metrics in your Cloud project.

Metric-name strings are limited to 100 characters and can only contain the following characters:

• A-Z
• a-z
• 0-9
• The special characters _-.,+!*',()%\/.

The forward slash character / indicates a hierarchy of parts inside the metric name and cannot be the initial character.

Label values are truncated

User-defined label values shall not exceed 1,024 bytes.

Cannot delete a custom log metric

You use the Google Cloud interface to remove a custom log-based metric. The delete request is unsuccessful, and the deletion window shows an error message. While performing this procedure, an unexpected error occurred.

Try the following to solve the problem:

• Refresh the Google Cloud console's logs-based metrics page. The error message might be shown as a result of an internal timing problem.
• Identify and remove any alerting policies that monitor the metric based on logs. Delete the logs-based metric after ensuring an alerting strategy is not monitored. It is impossible to remove log-based metrics monitored by an alerting policy.

Frequently Asked Questions

How do you do log metrics?

By selecting the Generate new Measure option from your graph, you may generate a log-based metric from your log analytics queries. To construct a new query, go to the Generate Metrics page of the logs setup section of the Datadog app.

What is a log metric filter?

Metric filters specify which phrases and patterns should be searched for in log data supplied to CloudWatch Logs. CloudWatch Logs use these metric filters to convert log data into numerical CloudWatch metrics that you can graph or trigger an alarm on.

What are metrics used for?

Metrics are quantitative evaluation metrics often used to compare and measure performance or production. Metrics may be applied in a variety of contexts. Internal management and external stakeholders rely extensively on metrics in their financial analyses of firms.

Conclusion

In this article, we have extensively discussed the Overview of log-based metrics. Our discussion mainly focused on description, types, counter metrics, distribution metrics, labels, charts and alerts, and troubleshooting in log-based metrics.

We hope this blog has helped you enhance your Google cloud knowledge. To learn more about Google cloud concepts, refer to our articles on All about GCP Certifications: Google Cloud Platform | Coding Ninjas Blog. 

Check out this problem - First Missing Positive  

Refer to our guided paths on the Coding Ninjas Studio platform to learn more about DSA, DBMS, Competitive Programming, Python, Java, JavaScript, etc. 

Refer to the links problems, top 100 SQL problems, resources, and mock tests to enhance your knowledge.

For placement preparations, visit interview experiences and interview bundle.

Do upvote our blog to help other ninjas grow. Happy Coding!