Monitors and notifiers

Learn how to setup monitors and notifiers to get alerts when queries match certain conditions.

This page introduces monitors and notifiers, and explains how you can use them to generate automated alerts from your event data.

A monitor is a background task that periodically runs a query that you define. For example, it counts the number of error messages in your logs over the previous 5 minutes. A notifier defines how Axiom notifies you about the monitor output. For example, Axiom can send you an email.

You can use the following types of monitor:

• Match monitors filter for key events and send them to you.
• Threshold monitors aggregate event data over time. When the results cross a threshold, Axiom sends you an alert.

Match monitors

Match monitors allow you to continuously filter your log data and send you matching events. Axiom sends a notification for each matching event. By default, the notification message contains the entire matching event in JSON format. When you define your match monitor using APL, you can control which event attributes to include in the notification message.

Axiom recommends using match monitors for alerting purposes only. A match monitor can send 10 notifications per minute and 500 notifications per day.

To create a match monitor, follow these steps:

1. Click the Monitors tab on the top of the page, and then click New monitor.
2. Click Match monitor.
3. Name your monitor and add a description.
4. Click Add notifier, and then select the notifiers that define how you want to receive notifications for this monitor. For more information, see Notifiers.
5. To define your query, use one of the following options:
   • To use the visual query builder, click Simple query builder. Select the filters, and then click Run query to preview the recent events that match your filters. To preview matching events over a specific period, select the time range.
   • To use Axiom Processing Language (APL), click Advanced query language. Write a query using the where operator to filter for events, and then click Run query to preview the results. To transform matching events before sending them to you, use the extend and the project operators. Don’t use aggregations in your query. For more information, see Introduction to APL.
6. When the preview displays the events that you want to match, click Create. You cannot create a match monitor if more than 500 events match your query within the past 24 hours.

You have created a match monitor, and Axiom alerts you about every event that matches the filters you set. Each notification contains the event details as shown in the preview.

Threshold monitors

Threshold monitors allow you to periodically aggregate your event data, and compare the results of this aggregation to a threshold that you define. When the results cross the threshold, the monitor enters the alert state. The monitor remains in the alert state until the results no longer cross the threshold. A threshold monitor sends you a notification each time it enters or exits the alert state.

To create a threshold monitor, follow these steps:

1. Click the Monitors tab on the top of the page, and then click New monitor.
2. Click Threshold monitor.
3. Name your monitor and add a description.
4. Configure the monitor using the following options:
   • The threshold is the value to compare the results of the query to. This can be any numeric value.
   • The comparison operator is the rule to apply when comparing the results to the threshold. The possible values are above, above or equal, below, and below or equal.
   • The frequency is how often the monitor runs. This is a positive integer number of minutes.
   • The range is the time range for your query. This is a positive integer number of minutes. The end time is the time the monitor runs.
   • Alert on no data triggers the monitor when your query doesn’t return any data. Your query returns no data if no events match your filters and an aggregation used in the query is undefined. For example, you take the average of a field not present in any matching events.
   • You can group by attributes when defining your query. By default, your monitor enters the alert state if any of the values returned for the group-by attributes cross the threshold, and remains in the alert state until none of the values returned cross the threshold. To trigger the monitor separately for each group that crosses the threshold, enable Notify by group. At most one trigger notification is sent per monitor run. This option only has an effect if the monitor’s query groups by a non-time field.
5. Click Add notifier, and then select the notifiers that define how you want to receive notifications for this monitor. For more information, see Notifiers.
6. To define your query, use one of the following options:
   • To use the visual query builder, click Simple query builder. Click Visualize to select an aggregation method, and then click Run query to preview the results in a chart. The monitor enters the alert state if any points on the chart cross the threshold. Optionally, use filters to specify which events to aggregate, and group by fields to split the aggregation across the values of these fields.
   • To use Axiom Processing Language (APL), click Advanced query language. Write a query where the final clause uses the summarize operator, and then click Run query to preview the results. For more information, see Introduction to APL. If your query returns a chart, the monitor enters the alert state if any points on the chart cross the threshold. If your query returns a table, the monitor enters the alert state if any numeric values in the table cross the threshold. If your query uses the bin_auto function, Axioms displays a warning. To ensure that the monitor preview gives an accurate picture of future performance, use bin rather than bin_auto.
7. Click Create.

You have created a threshold monitor, and Axiom alerts you when the results from your query cross the threshold.

Notifiers

Set up a notifier to determine the channel Axiom uses to alert you about what’s happening with your data. For example, Axiom can send you an email.

To create a notifier, follow these steps:

1. Click the Monitors tab on the top of the page, and then click Manage notifiers on the right.
2. Click New notifier on the top right.
3. Name your notifier.
4. Select and configure the channel Axiom uses to alert you. For more information, see the types of notifiers below.
5. Click Create.

By adding a notifier to a monitor, you receive a notification with the following message:

• When a match monitor matches an event, the message contains the full event if you created the monitor using the simple query builder, or the output of the APL query if you created the monitor using APL.
• When a threshold monitor changes state, the message includes a relevant value from the query results. If you enable Notify by group, the notification message also contains the relevant group value.

Email

Create an email notifier to notify a list of email addresses.

Slack

Create Slack notifiers to notify specific channels in your Slack organization using a webhook.

For more information on creating an incoming webhook in Slack, see the Slack documentation.

PagerDuty

Create a PagerDuty notifier to use all the incident management features of PagerDuty with Axiom.

To configure a new “Service” in PagerDuty using the Events V2 API, go to Configuration > Services > Add New Service and create a new Service named ‘Axiom’ with all the default settings. Take note of the Integration Key provided and enter this when creating a PagerDuty notifier in Axiom.

You will also need to provide an API Access Key with the right permissions which can be generated at pagerduty.com/api_keys.

Custom Webhook

Create customizable webhook notifiers to connect to internal or external services.

The URL will receive a HTTP(s) POST with a content type of application/json, together with any other headers specified. The following variables are available when defining the content of your webhook:

.Action — This has value “open” when the notification corresponds to a match monitor matching or a threshold monitor triggering, and has value “closed” when the notification corresponds to a threshold monitor resolving.

.MonitorID — The unique identifier for the monitor associated with the notification.

.Body — The message body associated with the notification. When the notification corresponds to a match monitor, this is the matching event data. When the notification corresponds to a threshold monitor, this provides information about the value that gave rise to the monitor triggering or resolving.

.Description — The description of the monitor associated with the notification.

.QueryEndTime — The end time applied in the monitor query that gave rise to the notification.

.QueryStartTime — The start time applied in the monitor query that gave rise to the notification.

.Timestamp — The time the notification was generated.

.Title — The name of the monitor associated with the notification.

.Value — When the notification corresponds to a threshold monitor, this is the value that gave rise to the monitor triggering or resolving.

.MatchedEvent — When the notification corresponds to a match monitor, this is the event that gave rise to the monitor matching.

You can use the Go Template syntax to interact with these variables when defining a custom body for your webhook.

A sample webhook body is shown below. This applies the default template shown on creating a custom webhook to a sample notification generated from a threshold monitor triggering.

{
  "action": "Open",
  "event": {
    "monitorID": "CabI3w142069etTgd0",
    "body": "Current value of 57347 is above or equal to the threshold value of 0",
    "description": "",
    "queryEndTime": "2024-06-28 14:55:57.631364493 +0000 UTC",
    "queryStartTime": "2024-06-28 14:45:57.631364493 +0000 UTC",
    "timestamp": "2024-06-28 14:55:57 +0000 UTC",
    "title": "Axiom Monitor Test Triggered",
    "value": 57347,
    "matchedEvent": null
  }
}

Opsgenie

Create an Opsgenie notifier to use all the incident management features of OpsGenie with Axiom.

For more information on configuring an integration in Opsgenie, see the Opsgenie documentation.

Discord

Create a Discord notifier to notify specific channels in your Discord server. You need to provide Axiom with a Discord token and channel identifier to create a Discord notifier.

• To get a token, go to discord.com/developers/applications and create a new application.
• Click Bot > Add Bot > Reset Token to get your Discord token.
• Go to OAuth2 > URL Generator, check the Bot scope and the Send Messages permission, then copy and open the generated URL to add the bot to your server.
• To get the channel ID, go to User Settings > Advanced and enable developer mode. Then right-click a channel and click Copy ID.
• Confirm if the check box of the Discord Bot has the proper allow channel access permissions from your settings.

Discord Webhook

Create a Discord Webhook notifier to notify specific channels in your Discord server using a webhook.

For more information on creating a webhook in Discord, see the Discord documentation.

Microsoft Teams

Create a Microsoft Teams notifier to send a notification to a specific channel in your Microsoft Teams instance using a webhook.

For more information on creating an incoming webhook in Microsoft Teams, see the Microsoft documentation.

Snooze a monitor

Both monitors and notifiers can be snoozed by clicking the ‘alarm clock’ icon when configuring the monitor or notifier. Snoozing a monitor will prevent it from running until the snooze time is elapsed. Snoozing a notifier will prevent it from sending messages until the snooze time has elapsed.

Monitor examples

This section presents example monitor configurations for some common alerting use cases.

Notify on all occurrences of error

To receive a notification on all occurrences of an error, create a match monitor where the filter conditions match the events reporting the error.

To receive only certain attributes in the notification message, use the project operator.

Notify when error rate above threshold

To receive a notification when the error rate exceeds a threshold, create a threshold monitor with an APL query that identifies the rate of error messages.

For example, logs in your dataset ['sample_dataset'] have a status.code attribute that takes the value ERROR when a log is about an error. In this case, the following example query tracks the error rate every minute:

['sample_dataset']
| extend is_error = case(['status.code'] == 'ERROR', 1, 0)
| summarize avg(error) by bin(_time, 1m)

Other options:

• To trigger the monitor when the error rate is above or equal to 0.01, set the threshold value to 0.01 and the comparison operator to above or equal.
• To run the monitor every 5 minutes, set the frequency to 5.
• To keep the monitor in the alert state until 10 minutes have passed with the per-minute error rate remaining below your threshold value, set the range to 10.

Notify when number of error messages above threshold

To receive a notification when the number of error message of a given type exceeds a threshold, create a threshold monitor with an APL query that counts the different error messages.

For example, logs in your dataset ['sample_dataset'] have a error.message attribute. In this case, the following example query counts errors by type every 5 minutes:

['sample_dataset']
| summarize count() by ['error.message'], bin(_time, 5m)

Other options:

• To trigger the monitor when the count is above or equal to 10 for any individual message type, set the threshold to 10 and the comparison operator to above or equal.
• To run the monitor every 5 minutes, set the frequency to 5.
• To run the monitor the query with a range of 10 minutes, set the range to 10.

By default, the monitor enters the alert state when any of the counts returned by the query cross the threshold, and remains in the alert state until no counts cross the threshold. To alert separately for each message value instead, enable Notify by group.