Manage incidents for log-based alerts

Introduction

Log-based alerts is a facility available in google cloud using which the user can be notified whenever a specific message appears in the included logs. For example, if a user wishes to know when an audit log records some particular data-access message, the user can simply create a log-based alert that will notify the user when a match with the message appears. Log-based alerts aren't a good fit always like they don't operate on excluded logs, or they can be used to derive counts from your logs. 

Managing Incidents for Log-based alerts

An incident for log-based alerts is a record of the triggering of an alerting policy. Whenever a condition of an alerting policy has been met, Cloud Monitoring opens that Incident. Necessary information required to investigate the cause of the alert is present in the Incident itself. When the log-based alerting policy is first triggered, Monitoring opens the Incident and sends the user a notification.

Finding Incidents 

• Click on the Navigation menu and then choose Monitoring from the Google cloud console toolbar.
• Select Alerting from the Monitoring Navigation Pane. The summary pane lists the number of open incidents, whereas the incidents pane displays the most recent incidents.
• If you wish to view details of a specific incident, simply select that Incident from the list, and the incident details page will open.

Filtering incidents

If the user wants to filter the incidents, the user can enter a value on the filter bar. Only those incidents that match the filter mentioned by the user get listed in the incidents table. In the case of multiple filters, only those incidents are displayed that satisfy all the filters.

To add a filter, follow the below steps.

• Click on the filter table, which is available on the Incidents page. Select a filter property. A filter includes:
  • State of the Incident
  • Alerting policy's name
  • When was the Incident opened or closed
• Either choose a value from the secondary menu or enter a value in the filter bar.

Investigating Incidents

If the user wishes to view the details of an incident, then the user must have the identity and the access management role of role/Monitoring.viewer. Once the Incident has been found, go to the incident details page for that Incident. On either the Alerting or Incidents page, click on the incident summary in the incidents table if you wish to view the details.

The page provides the following information:

Status information like

• Name of the alerting policy which caused the Incident
• What's the status of the Incident: open, acknowledged, or closed?
• Period of Time for which the Incident was open

Information about alerting policy that caused the Incident:

• The condition pane is used to identify the Condition in the alerting policy which caused the Incident. The condition name is always "Log match condition" for log-based alerting policies created by using Logs explorer.
• Based on the configuration of the Condition in the alerting policy, the message pane provides a brief explanation of the cause. This pane is always populated.
• The documentation pane displays the documentation template for notification that was used while creating the alerting policy. This may include information about what the alerting policy monitors and tips for mitigation.

Labels report the labels and the values for the monitored resources, which are included in the log entry that triggered the alerting policy. Such information can help the user to identify exactly which monitored resource caused the Incident.

Managing Incidents

An incident can be in either of the following states:

Open

The log-based alerting policy was triggered, and the Incident is still open. If an already open incident gets triggered again, then a new incident isn't opened.

Acknowledge

If the Incident was open and has been manually marked as acknowledged. It is used to represent the incidents being investigated.

Closed

Either it was manually closed or automatically closed after the auto close period expired.

Closing Incidents

The user can either let Monitoring close an incident or can close the Incident themselves.

When the auto-close duration for the alerting policy expires, the Monitoring automatically closes the Incident. By default, the auto-close is set to 7 days, while the minimum auto-close duration, which can be set, is 30 minutes.

To close an incident manually, follow the below steps:

• Click on See all incidents on the Incidents pane of the Alerting dashboard.
• From the incidents page, find the Incident which is to be closed and follow the below steps:
• Click on the more options and choose close this Incident 
• Click on the close Incident after opening the details page for the Incident