Query Syntax

The query string used for searching events is composed of a series of terms and operators. A term can be a single word — DefenseStorm or Agent — or a phrase, surrounded by double quotes — "DefenseStorm Agent" — which searches for all the words in the phrase, in the same order. Boolean operators (AND, OR, NOT) allow you to customize the search.

Searching

When performing a search in GRID, a single word or phrase is searched for in all available fields such as app_name, message, etc. Searches are not case sensitive but attention needs to be paid to spacing and required double quotes. Terms containing punctuation, such as 127.0.0.1, bro_dns, or microsoft-windows-search, are not considered a phrase and do not need to be placed in double quotes.

Single fields can be isolated and searched for without searching all of the fields. This is a preferred tactic for more specific searches. Be sure to separate the field and term with a colon(:).

For example, app_name:"DefenseStorm Agent"

A space is permitted between the colon and phrase: app_name: "DefenseStorm Agent" although not preferred. Also, app_name:"DefenseStorm Agent" is the same as app_name:"defensestorm agent" as capitalization is not acknowledged.

Watch for synonyms

Synonyms are not understood by the GRID. People may understand wi fi and "wi fi" and "wifi" and “Wi-Fi” to all be the same thing but the system does not. The query string parser would interpret your query as a search for wi fi as "wi OR fi". This likely is not specific enough for your intended search. Be sure what the query is designed for will in fact answer the intended demand.

Wildcards

Wildcard searches can be run on individual terms, using ? to replace a single character, and * to replace zero or more characters:

bro_d?s  would return events containing bro_dns and bro_dxs (if there were such a thing).

bro* would return events under all bro in app_name including bro_weird, bro_files, bro_ssl, etc. This particular search approach is good for confirming large quantities of events such as a firewall or other appliance traffic.

Boolean Operators

Boolean Operators are used to connect and define the relationship between your search terms and phrases. The three Boolean operators are AND, OR and NOT. 

Use AND to narrow your search: all of your search terms must be present in the retrieved records. AND can also be executed as + or &&.

app_name:"DefenseStorm Agent" AND category:Bad_Stuff AND event_id:911

app_name:"DefenseStorm Agent" +category:Bad_Stuff +event_id:911

app_name:"DefenseStorm Agent" && category:Bad_Stuff && event_id:911
(*note using && requires a space between && and the term whereas using + does not)

 These queries require that events contain all three terms in order to be returned.

Use OR to broaden your search by connecting two or more synonyms. OR can also be executed as a double pipe (||), not to be confused with two lowercase Ls as they look the same.

category:Bad_Stuff OR category:Worse_Stuff

category:Bad_Stuff || category:Worse_Stuff
(*note using || requires a space between || and the term)

This query requires either term in order to be returned.

Use NOT to exclude terms from your search results. NOT can also be executed as an exclamation point (!) or a minus sign (-).

category:Bad_Stuff NOT event_id:911

category:Bad_Stuff !event_id:911

category:Bad_Stuff -event_id:911
(*note using ! or - to denote NOT requires no space between it and the term)

This query requires any event_id fields be returned that are not event_id:911.

 

The familiar operators AND, OR, and NOT (also written &&, || and !) are supported and all variations of these can be combined in a single query. However, the effects of these operators are more complicated than is obvious because of the order of precedence, also known as the order of operations. There are predetermined rules that govern the order of which procedures are performed first in order to evaluate a query. NOT takes precedence over AND, which takes precedence over OR. 

NOT > AND > OR        which is the same as    ! > && > ||

While the + and - only affect the term to the right of the operator, AND and OR can affect the terms to the left and right. If this isn’t confusing enough, we have the added dilemma of remembering the spaces.

Another consideration is the use of parentheses which can ease the confusion with the other operators. Syntax contained within parentheses has the highest priority. 

Grouping

Multiple terms or phrases can be grouped together with parentheses to form sub-queries:

(virus OR malware) AND trojan

Reserved Characters

If you need to use any of the characters which function as operators in your query itself (and not as operators), then you should escape them with a leading backslash. For instance, to search for (1+1)=2, you would need to write your query as \(1\+1\)=2.

The reserved characters are: + - && || ! ( ) { } [ ] ^ " ~ * ? : \ /

Failing to escape these special characters correctly could lead to a syntax error which prevents your query from running.

Empty Query

If the query string is empty or only contains whitespaces the query string is interpreted as a no_docs_query and is expected to return no results.

Regular Expressions

Regex searching is supported but use this method against single fields. Regular expression patterns can be embedded in the query string by wrapping them in forward-slashes ("/"):

name:/joh?n(ath[oa]n)/

The supported regular expression syntax is explained later in this document in Appendix A.


Appendix A: Regular Expression Query

Regular Expression syntax

Regular expression queries are supported in the DefenseStorm GRID, under the Events view search. The Lucene regular expression engine is not Perl-compatible but supports a smaller range of operators. We are not going to explain regular expressions, just the supported operators.

An excellent resources for additional information is https://regexr.com.

Standard Operators

Standard operators are always enable through the DefenseStorm Console. For more detailed information about standard operators, see Standard Query Operators Overview from Microsoft.

Anchoring

Most regular expression engines allow you to match any part of a string. If you want the regexp pattern to start at the beginning of the string or finish at the end of the string, then you have to anchor it specifically, using ^ to indicate the beginning or $ to indicate the end.

Lucene’s patterns are always anchored. The pattern provided must match the entire string. 

For string "abcde":

ab.*  # match
abcd  # no match

Allowed characters

Any Unicode characters may be used in the pattern, but certain characters are reserved and must be escaped. The standard reserved characters are:

. ? + * | { } [ ] ( ) " \

If you enable optional features (see below) then these characters may also be reserved:

# @ & < >  ~

Any reserved character can be escaped with a backslash "\*" including a literal backslash character: "\\"

Additionally, any characters (except double quotes) are interpreted literally when surrounded by double quotes:

john"@smith.com"

Match any character

The period "." character can be used to represent any character. 

For string "abcde":

ab...  # match
a.c.e  # match

One-or-more

The plus sign "+" can be used to repeat the preceding shortest pattern once or more times. 

For string "aaabbb":

a+b+  # match
aa+bb+  # match
a+.+  # match
aa+bbb+  # match

 Zero-or-more

The asterisk "*" can be used to match the preceding shortest pattern zero-or-more times. 

For string "aaabbb":

a*b*  # match
a*b*c*  # match
.*bbb.*  # match
aaa*bbb*  # match 

Zero-or-one

The question mark "?" makes the preceding shortest pattern optional. It matches zero or one times.  When no parentheses (groupings) are used, this affects a single character to the left of it.

For string "aaabbb":

aaa?bbb?  # match
aaaa?bbbb?  # match
.....?.?  # match
aa?bb?  # no match

Min-to-max

Curly brackets "{}" can be used to specify a minimum and (optionally) a maximum number of times the preceding shortest pattern can repeat. The allowed forms are:

{5}  # repeat exactly 5 times
{2,5}  # repeat at least twice and at most 5 times
{2,}  # repeat at least twice

For string "aaabbb":

a{3}b{3}  # match
a{2,4}b{2,4}  # match
a{2,}b{2,}  # match
.{3}.{3}  # match
a{4}b{4}  # no match
a{4,6}b{4,6}  # no match
a{4,}b{4,}  # no match 

Grouping

Parentheses "()" can be used to form sub-patterns. The quantity operators listed above operate on the shortest previous pattern, which can be a group.  

For string "ababab":

(ab)+  # match
ab(ab)+  # match
(..)+  # match
(...)+  # no match
(ab)*  # match
abab(ab)?  # match
ab(ab)?  # no match
(ab){3}  # match
(ab){1,2}  # no match

Alternation

The pipe symbol "|" acts as an OR operator. The match succeeds if the pattern on either the left-hand side OR the right-hand side matches. The alternation applies to the longest pattern, not the shortest. 

For string "aabb":

aabb|bbaa  # match
aacc|bb  # no match
aa(cc|bb)  # match
a+|b+  # no match
a+b+|b+a+  # match
a+(b|c)+  # match

Character classes

Ranges of potential characters may be represented as character classes by enclosing them in square brackets "[]". A leading ^ negates (disallows) the character class. 

The allowed forms are:

[abc]  # 'a' or 'b' or 'c'
[a-c]  # 'a' or 'b' or 'c'
[-abc]  # '-' or 'a' or 'b' or 'c'
[abc\-] # '-' or 'a' or 'b' or 'c'
[^abc]  # any character except 'a' or 'b' or 'c'
[^a-c]  # any character except 'a' or 'b' or 'c'
[^-abc]  # any character except '-' or 'a' or 'b' or 'c'
[^abc\-] # any character except '-' or 'a' or 'b' or 'c'

Note that the dash "-" indicates a range of characters, unless it is the first character or if it is escaped with a backslash.

For string "abcd":

ab[cd]+  # match
[a-d]+  # match
[^a-d]+  # no match