TiddlyWiki5/editions/tw5.com/tiddlers/filters/syntax/Filter Expression.tid

created: 20150124182421000
modified: 20201102224444831
tags: [[Filter Syntax]]
title: Filter Expression
type: text/vnd.tiddlywiki

<$railroad text="""
[{:
  [: [[whitespace|"Filter Whitespace"]] ]
  ("+"|"~"|:-|"-"|"=")
  [[run|"Filter Run"]]
}]
"""/>

A <<.def "filter expression">> is the outermost level of the [[filter syntax|Filter Syntax]]. It consists of one or more [[runs|Filter Run]].

If a run has:

* no prefix, its output titles are [[dominantly appended|Dominant Append]] to the filter's output
* the prefix `=`, output titles are appended to the filter's output without de-duplication. <<.from-version "5.1.20">> 
* the prefix `-`, output titles are <<.em removed>> from the filter's output (if such tiddlers exist)
* the prefix `+`, it receives the filter output so far as its input; its output then <<.em "replaces">> all filter output so far and forms the input for the next run
* the prefix `~`, if the filter output so far is an empty list then the output titles of the run are [[dominantly appended|Dominant Append]] to the filter's output. If the filter output so far is not an empty list then the run is ignored. <<.from-version "5.1.18">>
* named prefixes for filter runs are available. <<.from-version "5.1.23">>
* named prefix `:filter`, it receives the filter output so far as its input. The next run is evaluated for each title of the input, removing every input title for which the output is an empty list. <<.from-version "5.1.23">>
* named prefix `:intersection` replaces all filter output so far with titles that are present in the output of this run, as well as the output from previous runs. Forms the input for the next run. <<.from-version "5.1.23">>

<<.tip "Compare named filter run prefix `:filter` with [[filter Operator]] which applies a subfilter to every input title, removing the titles that return an empty result from the subfilter">>

In technical / logical terms:

|!Run |!Equivalent named prefix |!Interpretation |!Output |
|`run` |`:or[run]` |de-duplicated union of sets |... OR run |
|`=run` |`:all[run]` |union of sets without de-duplication |... OR run |
|`+run` |`:and[run]` |accumulation of filter steps |... AND run |
|`-run` |`:except[run]` |difference of sets |... AND NOT run |
|`~run` |`:else[run]` |else |... ELSE run |
||`:intersection`|intersection of sets||

The input of a run is normally a list of all the non-[[shadow|ShadowTiddlers]] tiddler titles in the wiki (in no particular order). But the `+` prefix can change this:

|Prefix|Input|h
|`-`, `~`, `=`, `:intersection` or none| <$link to="all Operator">`[all[]]`</$link> tiddler titles, unless otherwise determined by the first [[filter operator|Filter Operators]]|
|`+`, `:filter`|the filter output of all previous runs so far|

Precisely because of varying inputs, be aware that both prefixes `-` and `+` do not behave inverse to one another!

For example, in both of the following, `$:/baz` will only be removed if it actually exists:

* <$link to="is Operator"> `foo bar $:/baz -[is[system]]`</$link>
* <$link to="prefix Operator">`foo bar $:/baz -[prefix[$:/]]`</$link>

To understand why, consider the input for both final runs with their `-` prefix.

In order to remove `$:/baz` in any case, existing or not, simply use the `+` prefix with [[negated filter operators|Filter Operators]]:

* <$link to="is Operator">`foo bar $:/baz +[!is[system]]`</$link>
* <$link to="prefix Operator">`foo bar $:/baz +[!prefix[$:/]]`</$link>

There is also a difference between the `:intersection` and `+` prefixes due to varying inputs.

The `+` prefix should be thought of as an "AND" in formal logic, e.g. "give me all titles that satisfy condition A ''and'' condition B". But it's not suitable for all cases; if condition B uses a filter operator that replaces its input, then it will be difficult to use the `+` prefix. For example, if you wanted to find out what tags two tiddlers have in common, you might try to write a filter expression like:

* <$link to="tags Operator">`[[field Operator]tags[]] +[[compare Operator]tags[]]`</$link>

But that won't work, because the second filter run will end up throwing away its input and replacing it with an input consisting of the single title `[[compare Operator]]`. So the result you'd get from that filter expression would be just the tags of the `compare Operator` tiddler.

For cases like this, the `:intersection` prefix is what you need. It takes the filter output so far, //sets it aside// in temporary storage, and starts the next filter run with all tiddler titles as input. Then once the latest filter run has completed, it takes the latest output, compares it to the set-aside output, and produces a new output that contains only titles that appeared in both the set-aside output and the latest output. So to get only the tags that the `field Operator` and `compare Operator` tiddlers have in common, you would write a filter expresison like this:

* <$link to="tags Operator">`[[field Operator]tags[]] :intersection[[compare Operator]tags[]]`</$link>