Added formal grammar documentation for tiddler filters

editions/tw5.com/tiddlers/Release 5.0.8beta.tid

@@ -14,7 +14,7 @@ These are changes that might affect users upgrading from previous betas.
 
 !! Documentation updates
 
-* 
+* Improved documentation for TiddlerFilters, including a notation of the [[TiddlerFilter Formal Grammar]] contributed by Mike
 
 !! Improvements
 

editions/tw5.com/tiddlers/concepts/TiddlerFilters.tid

@@ -1,10 +1,10 @@
 created: 20130827080000000
-modified: 20140129214824811
+modified: 20140210141253282
 tags: concepts
 title: TiddlerFilters
 type: text/vnd.tiddlywiki
 
-Filters are used in TiddlyWiki to choose tiddlers by specifying simple match criteria.
+Filters are used in TiddlyWiki to choose tiddlers by specifying simple match criteria. See [[TiddlerFilter Formal Grammar]] for a technical explanation of the syntax.
 
 ! Examples
 

editions/tw5.com/tiddlers/dev/TiddlerFilter Formal Grammar.tid

@@ -0,0 +1,79 @@
+created: 20140210141217955
+modified: 20140210141312363
+tags: dev
+title: TiddlerFilter Formal Grammar
+type: text/vnd.tiddlywiki
+
+TiddlerFilters follow a formal grammar:
+
+''<filter-string>'' ::= ''<opt-whitespaces>'' ''<filter-operand>'' | ''<opt-whitespaces>'' ''<filter-operand>'' ''<filter-string>''
+
+Whitespace is matched with javascript "\s+", which matches space, tab, carriage return, new line, vertical tab, and form feed.
+
+''<opt-whitespaces>'' ::= ''<opt-whitespace>'' | ''<opt-whitespace>'' ''<opt-whitespaces>''
+
+''<opt-whitespace>'' ::= " " | "\t" | "0xD" | "0xA" | "0xB" | "0xC"
+
+''<filter-operand>'' ::= ''<opt-operation-prefix>'' ''<string-or-operator-list>''
+
+''<opt-operation-prefix>'' ::= "+" | "-" | ""
+
+''<string-or-operator-list>'' ::= ''<operation>'' | "\"" ''<string>'' "\"" | "'" ''<string>'' "'" | ''<string>''
+
+''<operation>'' ::= "[" ''<operator-list>'' "]"
+
+''<operator-list>'' ::= ''<operator>'' | ''<operator>'' ''<operator-list>''
+
+''<operator>'' ::= ''<opt-operator-prefix>''''<operator>''''<operand>''
+
+''<opt-operator-prefix>'' ::= "!" | ""
+
+''<operator>'' ::= ''<operator-name>'' | ''<operator-name>'' ":" ''<opt-operator-suffix>''
+
+''<operator-name>'' ::= "" | "is" | "has" | "each" | "field" ...
+
+''<opt-operator-suffix>'' ::= ''<string>'' | ""
+
+''<operand>'' ::= "[" ''<search-string>'' "]" | "{" ''<indirect-search-string>'' "}" | ''<regex>''
+
+''<regex>'' ::= "/" ''<string>'' "/" ''<opt-regex-args>''
+
+''<opt-regex-args>'' ::= "(" ''<regex-args>'' ")" | ""
+
+''<regex-args>'' ::= ''<regex-arg>'' | ''<regex-arg>'' ''<regex-args>''
+
+''<regex-arg>'' ::= "m" | "y" | "g" | "i"
+
+''<string>'' ::= ''<string-type-1>'' | ''<string-type-2>'' | ...
+
+At the end of parsing you end up with some or all of: 
+
+* ''<opt-operation-prefix>''
+* ''<opt-operator-prefix>''
+* ''<operator-name>''
+* ''<opt-operator-suffix>'', and 
+* ''<operand>''
+
+These are used differently by the different operators. For example, the field filter operator supports:
+
+* ''<opt-operator-prefix>'' to negate the result
+* ''<regex>'' or ''<string>'' operand (note that this must be explicitly supported by each filter operator)
+* ''<opt-operator-suffix>'' to specify a fieldname against which to filter
+
+NOTES:
+
+* The ''<string>'' is a terminal that generally supports single- or double- quoted strings which match, respectively, strings of non-single and non-double quotes. Unquoted strings include the extra exclusion of whitespace and square bracket characters.
+
+* In the case where the ''<string-or-operator-list>'' is NOT an ''<operation>'' it is treated as the operand passed to the default operator (see next bullet). It is not parsed as a full ''<operation>''.
+
+* If ''<operator-name>'' is the empty string then it will be set to "title" (i.e. the title filter operator is the default operator)
+
+* Results are collected and each operation is applied in turn. The ''<opt-operation-prefix>'' can be used to specify how the corresponding operation is used. Suppose T is the set of all tiddlers, R0 is the current set of results, and Fx is the xth operation. Then:
+
+** No prefix (""): R0 = R0 U Fx(T) (set union)
+** "-": R0 = R0 - Fx(T) (set difference)
+** "+": R0 = Fx(R0)
+
+Note that ''<filter-operand>''s are not commutative!
+
+* The parser was simplified by treating regex "/" as a "bracket" of sorts, meaning there could only be a start and end bracket. Thus the regex arguments, like i, are included in parenthesis immediately following the trailing "/".