This is an attempt to modernise TiddlyWiki's build system, which has been based on tools written in Ruby called Cook and Ginsu (see https://github.com/TiddlyWiki/cooker for details). They were first written in 2006 and have been heavily hacked since then.
This new version is written in JavaScript for node.js, with the intention that it can share code with TiddlyWiki itself.
The original goal was to achieve byte-for-byte compatibility with the old tools. However, so many bugs have been discovered in the old tools that the new goal is to achieve byte-for-byte compatibility with TiddlyWiki itself when it saves changes.
The command line options are processed in sequential order from left to right. Processing pauses during long operations, like loading a recipe file and all the subrecipes and tiddlers that it references. The following options are available:
`--servewiki` and `--servertiddlers` are for different purposes and should not be used together. The former is for TiddlyWiki core developers who want to be able to edit the TiddlyWiki source files in a text editor and view the results in the browser by clicking refresh; it is slow because it reloads all the TiddlyWiki JavaScript files each time the page is loaded. The latter is for experimenting with the new wikification engine.
You can use filepaths or URLs to reference recipe files and tiddlers. For example, this recipe cooks the latest TiddlyWiki components directly from the online repositories:
`.recipe` files are text files that list the components to be assembled into a TiddlyWiki. They link to a simple template file that contains the basic structure of the TiddlyWiki document with additional markers to identify parts of the file where ingredients are inserted. Recipes determine which tiddlers should be included in the file, and put together the individual JavaScript files making up the TiddlyWiki core code.
Each line of the recipe file lists an ingredient, prefixed with a tag that describes what to do with the ingredient. Tags either identify a marker within the template file or are special tags that initiate an action.
Recipe files contain lines consisting of a marker, a colon and the pathname of an ingredient:
marker: filepath
The filepath is interpreted relative to the directory containing the recipe file.
The special marker `recipe` is used to load a sub-recipe file.
The special marker `template` is used to identify the HTML template. The HTML template contains markers in two different forms:
<!--@@marker@@-->
<!--@@marker@@-->
The special marker `title` is automatically filled in with a plain text rendering of the WindowTitle tiddler.
The special marker `tiddler` is used for the main content tiddlers that will be encoded in TiddlyWiki `<DIV>` format.
The special marker `copy` is used to indicate files that should be copied alongside the finished TiddlyWiki file.
Tiddlers can be stored in text files in several different formats. Files containing single tiddlers can also have an auxiliary `.meta` file formatted as a sequence of name:value pairs:
<pre>Note that there is an embedded <pre> tag, and line feeds are not escaped.
And, weirdly, there is no HTML encoding of the body.</pre>
</div>
These `*.tiddler` files are therefore not quite the same as the tiddlers found inside a TiddlyWiki HTML file, where the body is HTML encoded in the expected way.
Older `*.tiddler` files more closely matched the store format used by TiddlyWiki at the time:
<divtiddler="AnotherExampleStyleSheet"modifier="JeremyRuston"modified="200508181432"created="200508181432"tags="examples">This is an old-school .tiddler file, without an embedded <pre> tag.\nNote how the body is "HTML encoded" and new lines are escaped to \\n</div>
These files are a straightforward array of hashmaps of name:value fields. Currently only these known fields are processed: `title`, `text`, `created`, `creator`, `modified`, `modifier`, `type` and `tags`.
`test.sh` contains a simple test that cooks the main tiddlywiki.com recipe and compares it with the results of the old build process (ie, running cook.rb and then opening the file in a browser and performing a 'save changes' operation). It also invokes `wikitest.js`, a wikification test rig that works off the data in `test/wikitests/`.
Here is a guide to the key modules making up tiddlywiki.js and their public APIs. The modules are listed in order of dependency; modules generally don't know about other modules later in the list unless specifically noted.
Some non-standard MIME types are used by the code:
* **text/x-tiddlywiki:** TiddlyWiki-format wiki text
* **application/x-tiddlywiki:** A TiddlyWiki HTML file containing tiddlers
* **application/x-tiddler:** A tiddler in TiddlyWeb-style tiddler file format
* **application/x-tiddler-html-div:** A tiddler in TiddlyWiki `<div>` format
Tiddlers are an immutable dictionary of name:value pairs called fields. Values can be a string, an array of strings, or a JavaScript date object.
The only field that is required is the `title` field, but useful tiddlers also have a `text` field, and some or all of the standard fields `modified`, `modifier`, `created`, `creator`, `tags` and `type`.
Hardcoded in the system is the knowledge that the 'tags' field is a string array, and that the 'modified' and 'created' fields are dates. All other fields are strings.
Create a Tiddler given a series of sources of fields which can either be a plain hashmap of name:value pairs or an existing tiddler to clone. Fields in later sources overwrite the same field specified in earlier sources.
The hashmaps can specify the "modified" and "created" fields as strings in YYYYMMDDHHMMSSMMM format or as JavaScript date objects. The "tags" field can be given as a JavaScript array of strings or as a TiddlyWiki quoted string (eg, "one [[two three]]").
Registers a function that knows how to serialise a tiddler into a representation identified by a file extension and a MIME type. The serializer is called with a tiddler to convert and returns the string representation:
Registers a function that knows how to deserialize one or more tiddlers from a block of text identified by a particular file extension and a MIME type. The deserializer is called with the text to convert and should return an array of tiddler field hashmaps:
Given a block of text and a MIME type or file extension, returns an array of hashmaps of tiddler fields. One or more source fields can be provided to pre-populate the tiddler before the text is parsed.
If the type is not recognised then the raw text is assigned to the `text` field.
Text processors are components that know how to parse and render tiddlers of particular types. The core of TiddlyWiki is the WikiText processor, which can parse TiddlyWiki wikitext into a JavaScript object tree representation, and then render the tree into HTML or plain text. Other text processors planned include:
*`JSONText` to allow JSON objects to display nicely, and make their content available with TiddlyWiki section/slice notation
*`CSSText` to parse CSS, and process extensions such as transclusion, theming and variables
*`JavaScriptText` to parse JavaScript tiddlers for clearer display, and allow sandboxed execution through compilation
Note that text processors encapsulate two operations: parsing into a tree, and rendering that tree into text representations. Parsing doesn't need a context, but rendering needs to have access to a context consisting of a WikiStore to use to retrieve any referenced tiddlers, and the title of the tiddler that is being rendered.
Chooses a text processor based on the MIME type of the content and calls the `parse` method to parse the text into a parse tree. Returns null if the type was not recognised by a registered parser.
If the MIME type is unrecognised or unknown, it defaults to "text/x-tiddlywiki".
Creates a new instance of the wiki text processor with the specified options. The options are a hashmap of optional members and are planned as follows:
A collection of uniquely titled tiddlers. Although the tiddlers themselves are immutable, new tiddlers can be stored under an existing title, replacing the previous tiddler.
Each wiki store is connected to a shadow store that is also a WikiStore() object. Under usual circumstances, when an attempt is made to retrieve a tiddler that doesn't exist in the store, the search continues into its shadow store (and so on, if the shadow store itself has a shadow store).
Creates a new wiki store. The options are a hashmap of optional members as follows:
* **textProcessors:** A reference to the TextProcessors() instance to be used to resolve parsing and rendering requests
* **shadowStore:** An optional reference to an existing WikiStore to use as the source of shadow tiddlers. Pass null to disable shadow tiddlers for the new store
Invokes a callback for each tiddler in the store, optionally sorting by a specified field and excluding tiddlers with a specified tag. The callback is called with the title of the tiddler and a reference to the tiddler itself. For example:
The Recipe() class loads a TiddlyWiki recipe file, resolving references to subrecipe files. Tiddlers referenced by the recipe are loaded into a WikiStore. A fully loaded recipe can then be cooked to produce an HTML or RSS TiddlyWiki representation of the recipe.
Creates a new Recipe object by loading the specified recipe file. On completion the callback is invoked with a single parameter `err` that is null if the recipe loading was successful, or an Error() object otherwise.
var recipe = new Recipe({
filepath: "recent.recipe",
tiddlerConverters: tiddlerConverters,
store: store
},function callback(err) {
if(err) {
throw err;
} else {
console.log(recipe.cook())
}
}
Options is a hashmap with the following mandatory fields:
* **filepath:** The filepath to the recipe file to load
* **tiddlerConverters:** The TiddlerConverters() object to use to serialize and deserialize tiddlers
* **textProcessors:** The TextProcessors() object to use to parse and render tiddler text
* **store:** The WikiStore object to use to store the tiddlers in the recipe
The options can also contain these optional fields:
Wiki text is parsed into a simple object tree format. The parser doesn't need access to a WikiStore, nor does it need to know which tiddler, if any, the text it is parsing came from. This means that the parse tree can be cached within a tiddler when needed, and that parse trees can exist independently of tiddlers and WikiStores.
Rendering converts the parse tree into a textual format. The process requires access to a WikiStore so that it can resolve references to other tiddlers. It also needs to know which tiddler to use as the context for rendering the parse tree - this is the tiddler that is referenced by the `<<view>>` macro, for example.
The parse tree described here is defined as part of the WikiTextParser; other text processors could choose to represent their parse trees differently. However, it is hoped that this format is a sufficiently generic, transparent representation of HTML (and SVG) that it could be used by many other parsers, allowing the rendering engine, and macro processing, to be shared across parsers.
* the MIME type of the target format (currently `text/plain` or `text/html`)
* the parse tree to be rendered
* the WikiStore object to be used for resolving references to other tiddlers
* the title of the tiddler to use as the rendering context
The rendering process is to recursively scan the parse tree and stitch together the fragments making up the HTML elements corresponding to the nodes of the tree.
Macros are executed during rendering, which involves invoking the named macro handler with the macro parameter string, and a reference to the parse tree node for the macro. The handler then inserts the output parse tree into the `output` member of the macro node.
The `<<tiddler>>` macro doesn't change the tiddler context for its children. This means that when you transclude a tiddler, any `<<view>>` macros within it reference the fields of the tiddler that did the transcluding.
Here, `MyTiddler` specifies the information that should be displayed about each tiddler. In this example, there is a link to the tiddler, a link to the author, and the modification date:
