Regular Expression log search
Regular expressions, or regex, can be used independently or with any of the search functionality in the basic search documentation to provide advanced capability. A regex search is placed within two slashes (“/”).
Regular expressions have powerful capabilities for searching complex data and custom log formats.
- Unlike Keyword searches, regex can perform partial and case-insensitive matching.
- A regex search can be used to search events with special characters like slashes ("/"), double quotes (").
- Regex can be combined with KVP and JSON searching to match fields with variable values.
For example, if you were sending your Apache Access Logs in JSON format, you could use the remoteIP key to find any values that represent IPs from the country of Guine a (All IPs in Guinea are in the range 197.149.192.0 – 197.149.225.225). You could create the following regex:
Using Regular Expressions to search logs
Regular Expression operators
Regular Expressions use special characters to enable searching for more advanced patterns. These characters are *, +, ., \,[, ], (, ), {, }, ^, $. If you need to use special characters as ordinary characters, you have to escape them with a backward slash \.
Syntax
Log searches use Google RE2 Syntax. Regex only supports the use of the regular expression operators. For help troubleshooting regex, visit regexr.com.
Match something a number of times
| Name | Example | Description |
|---|---|---|
| Any | a* | Star matches zero or more of the previous character. |
| At least one | a+ | Plus matches at least one repetition of the previous character. |
| Exactly | a{x} | Matches the exact number of the previous character. |
| From, to | a{x,y} | Matches the number of the previous character within the range. |
| Up to | a{,y} | Matches up to the limit of the previous character. |
| At least | a{x,} | Matches at least the limit of the previous character. |
Matching a character set
| Name | Example | Description |
|---|---|---|
| Any character | . | Dot matches any single character. |
| Any digit | \d | Matches a digit character, that is 0-9. |
| Any whitespace | \s | Matches any whitespace character. |
| Anything but a digit | \D | Matches any character that is not a digit. |
| Anything but a whitespace | \S | Matches any character except for whitespace. |
| Given set | [abc] | Matches any of the characters specified. |
| Anything but the given set | [^abc] | Matches any character except for those specified. |
Log searches also support named capture groups.
Regular Expression flags
Flags can change the default behavior of a regex search. They are specified at the end of a regex search after the close slash “/”.
- /i case-insensitive - Disables case sensitivity, default operation is case sensitive.
- /m multiline- Enables the special characters for start (^) and end ($) to match individual lines of a multi-line log event. The default operation is to only match the start and end of the log event.
- /s New lines - Enables the special character (.) to match new lines.
- /U ungreedy - By default, a regex search tries to match the maximum number of characters. Using this flag causes the search to be ungreedy, which means it tries to match the fewest number of characters that satisfy the search parameters.
Regular expression examples
Regex can be used independently or with any of the search functionality in the basic search documentation to provide advanced capability. A regex search is placed within two slashes (“/”) and can include optional flags such as “i”.
| Example | Description |
|---|---|
| /Null/ | Events that contains Null, such as NullPointerException |
| /error/i | Events that contains error, case insensitive, such as Error, ERROR |
| /Exception “.*” at/ | Events that contains exception trace with a name |
| /20[01]/ | Events that contains 200 or 201 |
| ab*c | Matches strings ac, abc, abbc |
| ab+c | Matches strings abc, abbc, but not ac |
| ab{2}c | Matches abbc, but not abc or abbbc |
| ab{1,3}c | Matches abc, abbc, and abbbc |
| ab{,2}c | Matches ac, abc, and abbc |
| ab{2,}c | Matches abbc, abbbc, but not abc |
| a.c | Matches strings abc, acc, adc, but not ac |
| a\d | Matches a0, a1, a2 |
| a\sb | Matches a b |
| a\D | Matches strings ab, ac, but not a0 |
| a\Sc | Matches strings abc, a0c, but not a c |
| /completed/i | Matches strings completed as well as Completed, compLeted, and ComplETED |
| field=/regexp/ | Field’s value matches the regular expression |
| field!=/regexp/ | Field’s value does not match the regular expression |
Regular Expression field extraction
Regex grouping and naming allows you to identify values in your log events and give these values a name, similar to having a Key value pair in your log events. You can use this named capture group to perform more complex search functions.
Benefits
This functionality gives you the ability to identify key pieces of information in your logs which are not in a Key Value Format such that search functions can be applied to the values in your logs. By assigning a name to the identified value(s), these values can be used with our advanced search functions such as GroupBy() or for calculating values such as counts, sums, averages or unique instance counts. They can also be used for comparisons when creating alerts. This means you can create a type of Key Value Pairing out of non-Key Value Pair log formats.
- Uses standard RE2 RegEx syntax for named capture groups
- Is not dependent on any log type or structure
- Removes requirement for data to be in KVP formats
- Can be used in queries and saved for creating dashboard items
- Can be used when creating Tags and alerts
Using Regular Expression field extraction
A regex named capture group is declared by using the following syntax in your expression:
The result returned from the query to the right of the ‘>’ is assigned the name enclosed in the ‘< >’.
Consider the sample log events below that contain a specific value such as ‘total sale’:
The Regular Expression to find the value following ‘total sale’ and assign this value to a named variable called ‘saleValue’ is:
Consider that you would like to calculate the average value of ‘total sale’ for the last 24 hours. You can find the Average ‘saleValue’ by using the following query:
In this example, ‘saleValue’ is the name that the digits that follow ‘total sale’ are assigned because of the regular expression. Once assigned to saleValue, an average calculation can be applied to these numbers.
Regular Expression field extraction is extremely useful in this scenario because the value is not in a Key Value Format (KVP), making it hard to tell most systems what value to use. By using regex named capture group syntax, it is easy to identify the value and assign it a name. This name is used as part of the search query. It is also possible to save the query and then use it for creating a dashboard item.