TiddlyWiki5/tiddlywiki5/tiddlers/TiddlyWikiArchitecture.tid

title: TiddlyWikiArchitecture
modifier: JeremyRuston

//This is a broad, conceptual overview of how TiddlyWiki works. See TiddlyWikiInternals for implementation details.//

!! Overview

TiddlyWiki is based on the idea of making information more useful by modelling it in the smallest meaningful semantic units. We call these units "tiddlers".  Structure coming from links, tags, and stories (sequences of tiddlers). TiddlyWiki's role is to provide a rich, useful and enjoyable interactive interface for manipulating tiddlers.

Tiddlers use a special wikitext notation that concisely represents text formatting and hypertext features. A cornerstone of wikitext is the ability to combine and embed tiddlers in various ways.

<<tiddler [[High Level Architecture.svg]]>>

!! Content Types

TiddlyWiki uses MIME types to indicate how the text of tiddlers should be interpreted. It uses standard MIME types such as {{{text/plain}}} and {{{text/html}}}, as well as these non-standard types for TiddlyWiki-specific formats:

* {{{text/x-tiddlywiki}}}: TiddlyWiki-format wiki text
* {{{application/x-tiddlywiki}}}: A TiddlyWiki HTML file containing tiddlers
* {{{application/x-tiddler}}}: A tiddler in TiddlyWeb-style tiddler file format
* {{{application/x-tiddler-html-div}}}: A tiddler in TiddlyWiki {{{<div>}}} format
In some situations //parameterised MIME types// are used to allow parameters to be specified for the conversion. For example, a tiddler can be rendered to an image of a particular size:
{{{
image/png({width:500})
}}}
!! Tiddlers

Tiddlers are a dictionary of name:value pairs called fields.

The only field that is required is the {{{title}}} field, but useful tiddlers also have a {{{text}}} field, and some or all of the standard fields {{{modified}}}, {{{modifier}}}, {{{created}}}, {{{creator}}}, {{{tags}}} and {{{type}}}.

Values can be a string, an array of strings, or a JavaScript {{{Date}}} object. Hardcoded in the system is the knowledge that the `tags` field is a string array, and that the `modified` and `created` fields are dates. All other fields are strings.

The {{{type}}} field identifies the representation of the tiddler text with a raw, unparameterised, MIME type.

!! WikiStore

Groups of uniquely titled tiddlers are contained in WikiStore objects.

The WikiStore also manages the plugin modules used for macros, and operations like serializing, deserializing, parsing and rendering tiddlers.

Each WikiStore is connected to another shadow store that is used to provide default content. Under usual circumstances, when an attempt is made to retrieve a tiddler that doesn't exist in the store, the search continues into its shadow store (and so on, if the shadow store itself has a shadow store).

!! WikiStore Events

Clients can register event handlers with the WikiStore object. Event handlers can be registered to be triggered for modifications to any tiddler in the store, or with a filter to only be invoked when a particular tiddler or set of tiddlers changes.



Whenever a change is made to a tiddler, the wikistore registers a nexttick handler (if it hasn't already done so). The nexttick handler looks back at all the tiddler changes, and dispatches matching event handlers. 

!! Parsing

TiddlyWiki parses the content of tiddlers to build an internal tree representation that is used for several purposes:
* Rendering a tiddler to other formats (e.g. converting wikitext to HTML)
* Detecting outgoing links from a tiddler, and from them...
* ...computing incoming links to a tiddler
* Detecting tiddlers that are orphans with no incoming links
* Detecting tiddlers that are referred to but missing

The parse tree is built when needed, and then cached by the WikiStore until the text changes.

TiddlyWiki5 uses multiple parsers:
* Wikitext ({{{text/x-tiddlywiki}}})
* JavaScript ({{{text/javascript}}})
* CSS ({{{text/css}}})
* JSON ({{{application/json}}})
* Recipe ({{{text/x-tiddlywiki-recipe}}})

The parsers support several useful conversions:
* Converting wikitext to HTML
* Converting Less-style CSS to real CSS, using tiddler references for variable names
* Whitelisting JavaScript for safe execution of untrusted code

In the future the architecture might also support:
* Converting JavaScript procedural images to PNGs, using a synthetic graphics context mocking the canvas
* Converting images from one size or format to another
* Minifying JavaScript

!! Compiling and Rendering

When the text of a tiddler is requested in a different format than its native type, TiddlyWiki5 compiles a JavaScript function that generates the new format from the text of the tiddler.

So, a simple tiddler in {{{application/x-tiddlywiki}}} format might read:
{{{
Hello World
}}}
The function to render it to {{{text/html}}} might look like this:
{{{
function() {
	return "<p>Hello World</p>";
}
}}}

The function can also include calls to the store to incorporate the values of other tiddlers. Consider this tiddler, called {{{HelloThere}}}:
{{{
Hello <<tiddler Who>>
}}}
And this one called {{{Who}}}:
{{{
World
}}}
The function to generate {{{HelloThere}}} in {{{text/html}}} might be:
{{{
function() {
	return ["<p>","Hello ", getTiddlerText("Who","text/html"), "</p>"].join(""); 
}
}}}

Now, the return value of this function can be cached until a tiddler in the dependency chain changes. The function itself can be cached until the tiddler itself changes, or a macro that it uses changes.

The dependency chain is calculated when a tiddler is compiled. Every tiddler that is directly referenced is accumulated (until the point at which it is concluded that it is simpler to mark the tiddler as being dependent on any other tiddler changing).

The render function is invoked with `this` pointing to a tiddler object. Parameters to the render function include:
* ''getTiddler(title)'' - a function to retrieve a specified tiddler
* ''renderTiddler(title,type,params)'' - a function to render a tiddler to a specified type
* ''forEachTiddler(sortField,excludeTag,callback)'' - a function to enumerate all tiddlers

Evaluated macro parameters would be parsed and checked for safeness, and then included in the compiled code. For example,
{{{
Hello <<echo {{2+2}}>>
}}}
Compiles to:
{{{
function() {
	return ["Hello ",(function(){
		return 2+2;
	})().toString()].join("");
}
}}}

The compilation process has several steps:
* First, the parse tree is used to generate a JavaScript tree
* The JavaScript tree is scanned to determine the tiddlers on which this one depends
* Finally, executable JavaScript text is generated by walking the JavaScript tree

!! Macros

Macros should be examples of javascript tiddlers that are invoked with parameters. The familiar ones all generate text/html output. For example:

	function tiddler_macro(args) {
		return getTiddlerText(args.tiddler,"text/html");
	}

These tiddlers should also have extractable and translateable docs, and perhaps some metadata about how the parameters should be parsed.

Maybe we can use the compiler to convert this form into the former? Yes, of course, we just lift and shift the parse tree from the macro definition into the point where it is used. Yay. So when we're walking the tree, we don't walk into the macro nodes children so much as into the definition tree.