Overview

The App arbitrarily defines 6 alerts checks each alert should pass.

These checks are either automatically performed by App’s main report or manually reviewed by Splunk admins through an interactive dashboard.

Alert checks

Check	Definition	Type
Source	Target data source must be indexed	Manual
Index	If applicable, target index(es) must be specified in the search query	Automatic
Runtime	Runtime must be lower than the interval between one run to the next	Automatic
Alignment	Alert schedule must be coordinated with search time range	Automatic
Delay	Alert must be scheduled with at least one minute of delay	Automatic
Structure	Search query must be correctly structured	Manual

Automatic checks

Index

When there is no index specified in a search query, Splunk searches in all available allowed indexes. This is not optimal in terms of resource usage and it is best practice to specify index(es) to be searched within the query. Searches that use alternate search commands in which index has not to be specified (e.g. dbxquery, inputlookup) are not taken into account (i.e. such queries are marked as having index specified). Resource

Note

Add any custom command to the Search commands lookup to have it considered as an alternate search command.

Runtime

When Splunk takes so much time to execute the query that search job is not finished when the alert’s next run launches.

Alignment

Alert schedule must be coordinated with search time range. For instance, an alert running every 5 minutes should have a time range of 5 minutes to avoid duplicate alerts and for better use of resources. Resource

Note

Alignement check assumes that the interval between two alert runs remains even. While it should be the case to avoid overlapping, an uneven cron interval might be needed in some specific scenarios. This is not covered in this check just yet. In the mean time, it is possible to whitelist a particular alert check.

Delay

It is better practice to leave some delay on alerts by configuring a latest time of at least 1 minute. Resource

Manual checks

Source

Is there any data at all when you run alert’s base search (i.e. query’s first line)?

Structure

This a way more subjective check whose goal is to make sure search queries are properly written considering searches best practices. Resource

KV Store lookup

The Update KV Store lookup report is the core function of the App.

It checks for all enabled and scheduled alerts, perform the automatic checks and save results into a KV Store lookup.

See Update KV Store lookup report

Inventory dashboard

This dashboard loads KV Store lookup entries and lets Splunk admins review each alert independently.

During the review the admin will address alert manual checks and save results to the KV Store through interactive buttons.

See Review Alerts

Concurrency dashboard

The goal of this dashboard is to help resolve alert spreading issues.

Whith a growing number of alerts, there could be plenty of alerts launching at the same schedule.

This could be limited by the maximum concurrent scheduled searches Splunk scheduler can run.

Hence, the idea is to represent the number of alerts launched over time against this concurrency limit so it becomes easy to spot too busy schedules.

See Improve Spreading