JSON Structure
JavaScript Object Notation (JSON) structured logs are regularly produced by software components and are promoted as a logging best practice. JSON formatted logs are easily produced and consumed by a variety of software components and are human readable.
Nested JSON
You can use the JSON hierarchy for queries and alerts. For example, if you’re sending logs, you can create nested JSON like the following:
You need to use dot notation for nested objects and integers for array positions.
JSON entries whose depth is more than 10 levels are not supported. They are not parsed. Nested keys are resolved fully before they are stored; for example, current.members is not a key.
Example queries
Here are some example queries using the dot notation and array positions you may find helpful.
- You want to find when the volume is blaring.
- You want to find when current band is Rednex.
- You want to find a member of the Dubliners named “Ciaran”.
Here are more examples of queries and the resulting matches based on the log message example above.
| JSON dot notation | matches |
|---|---|
| current.band | rednex |
| current.song | cotton eyed joe |
| current.members | “” |
| current.members.0 | “” |
| current.members.1.firstname | Urban |
| current.members.2.lastname | Lundstrom |
Alerting
You can set up an Alert if someone tries to stick some Nickelback on with a pattern like “next.band”=”nickelback”. This tags the messages or sends an email based on the Tag or Alert setup.
Notes
Any pre-existing Alerts or saved queries may need to be updated. If there are any Alerts or queries that search against child objects, (e.g. where(band) in the example above), the change could break your queries.
We suggest updating your queries to support both old and new query. Using the example above, you could query for where(band OR current.band).
JSON and KVP parsing
JSON structured logs are regularly produced by software components and are promoted as a logging best practice. JSON formatted logs are easily produced and consumed by a variety of software components and are human readable. If you are not ready to make the move to JSON, using Key-Value Pairs in your logs provides similar benefits. Both formats allow you to take advantage of our advanced search and analytic capabilities.
Automatic parsing
Log searches automatically identify and index fields within your JSON and KVP log events. Keys are automatically highlighted and clickable. Clicking a field populates the search bar to allow for quick searching of across your data.
JSON parsing specification
The system automatically parses JSON formatted log events formatted in compliance with the JSON Specification (IETF RFC 4627). To take advantage of the prettification option, the entry must be a single object ({ … }) or array of JSON objects. JSON objects that appear embedded in unstructured data still have their keys recognized but you are not able to expand the JSON hierarchy.
If there are multiple keys which share the same name in a JSON object, the parser interprets the last key as the correct key.
KVP parsing specification
Log searches can automatically parse a wide variety of KVP formats in compliance with the specification detailed below.
Specification and term definitions
A key value pair consists of an identifier called a KEY, followed by a SEPARATOR, a VALUE, and has a prefix and suffix termed the KVDELIMITER
KVDELIMITER KEY {whitespace} SEPARATOR {whitespace} VALUE{UNIT} KVDELIMITER
|
Term |
Definition |
|---|---|
|
KVDELIMITER |
can be space (” “), comma (“,”), tab “\t” or “\r” or “\n” or “\r\n” |
|
KEY |
the allowed list of characters is any alphanumeric plus “_” “.” “$” “@” must have at least one non-numeric character from the allowed list of characters an ” or " on either side of a key results in anything between the quotations marks being interpreted as the KEY, e.g. “key[1]” will give a KEY of key[1] {whitespace} can be 0 or 1 spaces (Ascii 0x20 or ” “) |
|
SEPARATOR |
is “=” or “:” can be preceded by and/or followed by a space in order to appear in a key or value, the separator must be between a pair of quotes or escaped |
|
VALUE |
consists of any alphanumeric plus “_” “.” “$” “@” an ” or " on either side of a value results in anything between the quotations marks being interpreted as the VALUE, e.g. key1=”value description” gives a KEY of key1 and a VALUE of “value description” Placing quotes around the VALUE is required for parsing URLs or other items that contain slashes (/) a SEPARATOR in a value is treated such that the value (and the possible key preceding it) is not to be treated as a key value pair, e.g. so timestamps not broken up |
|
UNIT |
This is the alphabetical text or “%” which follows the last number in a Numeric only VALUE up to the next space or first non alphabetical character or up to 2 characters (whichever occurs first) – those 2 characters are considered as part of value for querying, i.e. 100ms is treated as 100. |
|
CHARACTER ESCAPING |
This is used to allow a KEY or VALUE to contain a character that would not otherwise be allowed, e.g. a space or quote or comma or bracket is not allowed in a KEY or VALUE unless escaped.This is supported in both a KEY and a VALUE by adding a backslash (“\”) in front of the character to be escaped or by including the entire KEY or VALUE in quotes (pairs of ” or pairs of ‘). As per the definition of KEY and VALUE, the \ is not an allowed character and only appears to support escaping, e.g a space can be included by either using quotes as in name=”David Tracey” or escaping as in name=David\ Tracey. Regarding quotes being in pairs of ” or pairs of ‘, this means that a ‘ does not finish an escaped sequence started with ” and it is possible for one type of quote to be appear within quotes of the other type, e.g. somekey = “David’s One” includes the ‘ in the value. Note that the \ is only treated as escaping a character in the process of determining a name or a value. |
A few things to keep in mind:
- “Key=value” “Key :value” “Key = value” “Key: value” are valid key value pairs.
- “-“, “+” are not allowed in KEY or VALUE (unless escaped) to avoid confusion with use in query language.
- This definition means a Key Value Pair must have both a key and a value the inclusion of “.” and “_” means that “cpu.time” and “disk_size” are both valid as a KEY or VALUE.
- UTF-8 Encoding is the supported encoding
Prettifying JSON formatting
JSON structured logs can be viewed as a flat log event or can be prettified to easily view the event hierarchy.
Advanced capabilities
Once events are stored in JSON or other KVP format, the advanced Log Search functions and shareable dashboards become simple to setup and use. For example, the following screen capture shows how you can use the ‘Group By’ function to visualize the distribution of status codes for a JSON formatted Apache Log.
