Content updates for TW5

tiddlywiki5/tiddlers/CommandLineInterface.tid

@@ -0,0 +1,36 @@
+title: CommandLineInterface
+modifier: JeremyRuston
+
+TiddlyWiki5 can be used on the command line to perform an extensive set of operations based on RecipeFiles, TiddlerFiles and TiddlyWikiFiles.
+
+Usage:
+{{{
+node tiddlywiki.js <options>
+}}}
+The command line options are processed sequentially from left to right. Processing pauses during long operations, like loading a [[recipe file|RecipeFiles]] and all the subrecipes and [[tiddlers|TiddlerFiles]] that it references. The following options are available:
+
+|{{{--recipe <filepath>}}} |Loads a specfied `.recipe` file |
+|{{{--load <filepath>}}} |Load additional tiddlers from TiddlyWiki files ({{{.html}}}), {{{.tiddler}}}, {{{.tid}}}, {{{.json}}} or other files |
+|{{{--savewiki <dirpath>}}} |Saves all the loaded tiddlers as a single file TiddlyWiki called {{{index.html}}} and an RSS feed called {{{index.xml}}} |
+|{{{--savetiddlers <outdir>}}} |Saves all the loaded tiddlers as {{{.tid}}} files in the specified directory |
+|{{{--servewiki <port>}}} |Serve the cooked TiddlyWiki over HTTP at {{{/}}} |
+|{{{--servetiddlers <port>}}} |Serve individual tiddlers over HTTP at {{{/tiddlertitle}}} |
+|{{{--dumpstore}}} |Dump the TiddlyWiki store in JSON format |
+|{{{--dumprecipe}}} |Dump the current recipe in JSON format |
+|{{{--verbose}}} |verbose output, useful for debugging |
+
+!! Examples
+This example loads the tiddlers from a TiddlyWiki HTML file and makes them available over HTTP:
+{{{
+node tiddlywiki.js --load mywiki.html --servewiki 127.0.0.1:8000
+}}}
+This example cooks a TiddlyWiki from a recipe:
+{{{
+node tiddlywiki.js --recipe tiddlywiki.com/index.recipe --savewiki tmp/
+}}}
+This example ginsus a TiddlyWiki into its constituent tiddlers:
+{{{
+node tiddlywiki.js --load mywiki.html --savetiddlers tmp/tiddlers
+}}}
+!! Notes
+{{{--servewiki}}} and {{{--servertiddlers}}} are for different purposes and should not be used together. The former is for TiddlyWiki core developers who want to be able to edit the TiddlyWiki source files in a text editor and view the results in the browser by clicking refresh; it is slow because it reloads all the TiddlyWiki JavaScript files each time the page is loaded. The latter is for experimenting with the new wikification engine.

tiddlywiki5/tiddlers/HelloThere.tid

@@ -1,12 +1,12 @@
 title: HelloThere
 modifier: JeremyRuston
 
-Welcome to TiddlyWiki5, a [[primordial|ThisIsAlpha]], modernised reusable non-linear personal web notebook.
+Welcome to TiddlyWiki5, an [[in-progress|ThisIsAlpha]] reboot of TiddlyWiki, the reusable non-linear personal web notebook first released in 2004.
 
-TiddlyWiki is a unique [[wiki|WikiWikiWeb]] that people [[love using|Raves]] to keep ideas and information organised. It was originally created by JeremyRuston and is now a thriving [[open source|OpenSourceLicense]] project with a busy [[Community]] of independent developers.
+Over the years, TiddlyWiki has earned an enduring role as a tool that people [[love using|Raves]] for its rich, interactive interface to [[manipulate complex data|TiddlyWikiConcepts]] with structure that doesn't easily fit into conventional tools like spreadsheets and wordprocessors. Because people can use it without needing any complicated server infrastructure, and because it is [[open source|OpenSourceLicense]], it has bought unprecedented freedom to people to keep their precious information under their own control. TiddlyWiki was originally created by JeremyRuston and is now a thriving [[open source|OpenSourceLicense]] project with a busy [[Community]] of independent developers.
 
-TiddlyWiki is written in [[HTML]], [[CSS]] and JavaScript to run on any reasonably modern [[browser|Browsers]] without needing any ServerSide logic. It allows anyone to create personal SelfContained hypertext documents that can be published to a WebServer, sent by email, stored in a DropBox or kept on a USB thumb drive to make a WikiOnAStick. Because it doesn't need to be installed and configured it makes a great GuerillaWiki. This is revision <<version>> of TiddlyWiki, and is published under an OpenSourceLicense.
+TiddlyWiki5 gains new capabilities through a [[completely rebuilt architecture|TiddlyWikiArchitecture]] using the latest features of HTML5 and node.js. It runs natively under node.js, and can also use its own components to construct a version of itself that works entirely within the browser, just as TiddlyWiki has always done.
 
-Unlike most wikis, TiddlyWiki doesn't directly support group collaboration; it is a wiki in the sense of elevating linking be a part of the punctuation of writing. You can easily publish a TiddlyWiki you have created by placing the single file on a web server (for instance the homepage hosting provided by many ISPs). If you need full  group collaboration features, there are several HostedOptions to choose from.
+TiddlyWiki5 also functions as the build system for earlier versions of TiddlyWiki, replacing the elderly Cook and Ginsu tools (see https://github.com/TiddlyWiki/cooker for details). See the CommandLineInterface for details.
 
-<<echo {{2+2 /* something */ - 3 * 17 + title + tiddlywiki + out.length}}>>
+<<list all>>

tiddlywiki5/tiddlers/NewWikiTextFeatures.tid

@@ -0,0 +1,24 @@
+title: NewWikiTextFeatures
+modifier: JeremyRuston
+
+It is proposed to extend the existing TiddlyWiki wikitext syntax with the following extensions
+
+1. Addition of {{{**bold**}}} character formatting
+2. Addition of {{{`backtick for code`}}} character formatting
+4. Addition of WikiCreole-style forced line break, e.g. {{{force\\linebreak}}}
+3. Addition of WikiCreole-style headings, e.g. {{{==Heading}}}
+5. Addition of WikiCreole-style headings in tables, e.g. {{{|=|=table|=header|}}}
+6. Addition of white-listed HTML tags intermixed with wikitext
+7. Addition of WikiCreole-style pretty links, e.g. {{{[[description -> link]]}}}
+8. Addition of multiline macros, e.g.
+{{{
+<<myMacro
+param1: Parameter value
+param2: value
+"unnamed parameter"
+param4: ((
+A multiline parameter that can go on for as long as it likes
+and contain linebreaks.
+))
+>>
+}}}

tiddlywiki5/tiddlers/RecipeFiles.tid

@@ -0,0 +1,43 @@
+title: RecipeFiles
+modifier: JeremyRuston
+
+{{{.recipe}}} files are text files that list the components to be assembled into a TiddlyWiki. They link to a simple template file that contains the basic structure of the TiddlyWiki document with additional markers to identify parts of the file where ingredients are inserted. Recipes determine which tiddlers should be included in the file, and put together the individual JavaScript files making up the TiddlyWiki core code.
+
+Each line of the recipe file lists an ingredient, prefixed with a tag that describes what to do with the ingredient. Tags either identify a marker within the template file or are special tags that initiate an action.
+
+Recipe files contain lines consisting of a marker, a colon and the pathname of an ingredient:
+{{{
+marker: filepath
+}}}
+The filepath is interpreted relative to the directory containing the recipe file.
+
+You can use filepaths or URLs to reference recipe files and tiddlers. For example, this recipe cooks the latest TiddlyWiki components directly from the online repositories:
+{{{
+recipe: https://raw.github.com/TiddlyWiki/tiddlywiki/master/tiddlywikinonoscript.html.recipe
+tiddler: http://tiddlywiki-com.tiddlyspace.com/bags/tiddlywiki-com-ref_public/tiddlers.json?fat=1
+tiddler: http://tiddlywiki-com.tiddlyspace.com/bags/tiddlywiki-com_public/tiddlers.json?fat=1
+}}}
+The special marker {{{recipe}}} is used to load a sub-recipe file.
+
+The special marker {{{template}}} is used to identify the HTML template. The HTML template contains markers in one of two alternative forms:
+{{{
+<!--@@marker@@-->
+&lt;!--@@marker@@--&gt;
+}}}
+The special marker {{{title}}} is automatically filled in with a plain text rendering of the WindowTitle tiddler.
+
+The special marker {{{tiddler}}} is used for the main content tiddlers that will be encoded in TiddlyWiki {{{<DIV>}}} format.
+
+The special marker {{{copy}}} is used to indicate files that should be copied alongside the finished TiddlyWiki file.
+
+Tiddler fields can be overridden by following an ingredient with a set of indented fields:
+{{{
+tiddler: ../js/myscript.js
+	title: MyScript
+	modifier: JoeBloggs
+tiddler: ../js/myoldscript.js
+	title: MyOldScript
+	tags: [[one long one]] Short
+}}}
+Note that if tags are applied in this way they completely replace any previous tags on the tiddler.
+

tiddlywiki5/tiddlers/SiteSubtitle.tid

@@ -0,0 +1,4 @@
+title: SiteSubtitle
+modifier: JeremyRuston
+
+a reusable non-linear personal web notebook

tiddlywiki5/tiddlers/SiteTitle.tid

@@ -0,0 +1,4 @@
+title: SiteTitle
+modifier: JeremyRuston
+
+TiddlyWiki5

tiddlywiki5/tiddlers/TiddlerFiles.tid

@@ -0,0 +1,41 @@
+title: TiddlerFiles
+modified: JeremyRuston
+
+Tiddlers can be stored in text files in several different formats. Files containing single tiddlers can also have an auxiliary {{{.meta}}} file formatted as a sequence of name:value pairs:
+{{{
+title: TheTitle
+modifier: someone
+}}}
+!! ~TiddlyWeb-style .tid files
+
+These files consist of a sequence of lines containing name:value pairs, a blank line and then the text of the tiddler. For example:
+{{{
+title: MyTiddler
+modifier: Jeremy
+
+This is the text of my tiddler.
+}}}
+!! TiddlyWiki {{{<DIV>}}} .tiddler files
+
+Modern {{{*.tiddler}}} files look like this:
+{{{
+<div title="AnotherExampleStyleSheet" modifier="blaine" created="201102111106" modified="201102111310" tags="examples" creator="psd">
+<pre>Note that there is an embedded <pre> tag, and line feeds are not escaped.
+	
+And, weirdly, there is no HTML encoding of the body.</pre>
+</div>
+}}}
+These {{{*.tiddler}}} files are therefore not quite the same as the tiddlers found inside a TiddlyWiki HTML file, where the body is HTML encoded in the expected way.
+
+Older {{{*.tiddler}}} files more closely matched the store format used by TiddlyWiki at the time:
+{{{
+<div tiddler="AnotherExampleStyleSheet" modifier="JeremyRuston" modified="200508181432" created="200508181432" tags="examples">This is an old-school .tiddler file, without an embedded &lt;pre&gt; tag.\nNote how the body is &quot;HTML encoded&quot; and new lines are escaped to \\n</div>
+}}}
+!! ~TiddlyWeb-style JSON files
+
+These files are a straightforward array of hashmaps of name:value fields. Currently only these known fields are processed: {{{title}}}, {{{text}}}, {{{created}}}, {{{creator}}}, {{{modified}}}, {{{modifier}}}, {{{type}}} and {{{tags}}}.
+
+!! TiddlyWiki HTML files
+
+TiddlyWiki HTML files contain a collection of tiddlers encoded in {{{<DIV>}}} format.
+

tiddlywiki5/tiddlers/TiddlyWikiArchitecture.tid

@@ -0,0 +1,149 @@
+title: TiddlyWikiArchitecture
+modifier: JeremyRuston
+
+//This is a broad, conceptual overview of how TiddlyWiki works. See TiddlyWikiInternals for implementation details.//
+
+!! Overview
+
+TiddlyWiki is based on the idea of making information more useful by modelling it in the smallest meaningful semantic units. We call these units "tiddlers".  Structure coming from links, tags, and stories (sequences of tiddlers). TiddlyWiki's role is to provide a rich, useful and enjoyable interactive interface for manipulating tiddlers.
+
+Tiddlers use a special wikitext notation that concisely represents text formatting and hypertext features. A cornerstone of wikitext is the ability to combine and embed tiddlers in various ways.
+
+!! Content Types
+
+TiddlyWiki uses MIME types to indicate how the text of tiddlers should be interpreted. It uses standard MIME types such as {{{text/plain}}} and {{{text/html}}}, as well as these non-standard types for TiddlyWiki-specific formats:
+
+* {{{text/x-tiddlywiki}}}: TiddlyWiki-format wiki text
+* {{{application/x-tiddlywiki}}}: A TiddlyWiki HTML file containing tiddlers
+* {{{application/x-tiddler}}}: A tiddler in TiddlyWeb-style tiddler file format
+* {{{application/x-tiddler-html-div}}}: A tiddler in TiddlyWiki {{{<div>}}} format
+In some situations //parameterised MIME types// are used to allow parameters to be specified for the conversion. For example, a tiddler can be rendered to an image of a particular size:
+{{{
+image/png({width:500})
+}}}
+!! Tiddlers
+
+Tiddlers are a dictionary of name:value pairs called fields.
+
+The only field that is required is the {{{title}}} field, but useful tiddlers also have a {{{text}}} field, and some or all of the standard fields {{{modified}}}, {{{modifier}}}, {{{created}}}, {{{creator}}}, {{{tags}}} and {{{type}}}.
+
+Values can be a string, an array of strings, or a JavaScript {{{Date}}} object. Hardcoded in the system is the knowledge that the 'tags' field is a string array, and that the 'modified' and 'created' fields are dates. All other fields are strings.
+
+The {{{type}}} field identifies the representation of the tiddler text with a raw, unparameterised, MIME type.
+
+!! WikiStore
+
+Groups of uniquely title tiddlers are contained in WikiStore objects.
+
+The WikiStore also manages the plugin modules used for macros, and operations like serializing, deserializing, parsing and rendering tiddlers.
+
+Each WikiStore is connected to another shadow store that is used to provide default content. Under usual circumstances, when an attempt is made to retrieve a tiddler that doesn't exist in the store, the search continues into its shadow store (and so on, if the shadow store itself has a shadow store).
+
+!! WikiStore Events
+
+Clients can register event handlers with the WikiStore object. Event handlers can be registered to be triggered for modifications to any tiddler in the store, or with a filter to only be invoked when a particular tiddler or set of tiddlers changes.
+
+
+
+Whenever a change is made to a tiddler, the wikistore registers a nexttick handler (if it hasn't already done so). The nexttick handler looks back at all the tiddler changes, and dispatches matching event handlers. 
+
+!! Parsing
+
+TiddlyWiki parses the content of tiddlers to build an internal tree representation that is used for several purposes:
+* Rendering a tiddler to other formats (e.g. converting wikitext to HTML)
+* Detecting outgoing links from a tiddler, and from them...
+* ...computing incoming links to a tiddler
+* Detecting tiddlers that are orphans with no incoming links
+* Detecting tiddlers that are referred to but missing
+
+The parse tree is built when needed, and then cached by the WikiStore until the text changes.
+
+TiddlyWiki5 uses multiple parsers:
+* Wikitext ({{{text/x-tiddlywiki}}})
+* JavaScript ({{{text/javascript}}})
+* CSS ({{{text/css}}})
+* JSON ({{{application/json}}})
+* Recipe ({{{text/x-tiddlywiki-recipe}}})
+
+The parsers support several useful conversions:
+* Converting wikitext to HTML
+* Converting Less-style CSS to real CSS, using tiddler references for variable names
+* Whitelisting JavaScript for safe execution of untrusted code
+
+In the future the architecture might also support:
+* Converting JavaScript procedural images to PNGs, using a synthetic graphics context mocking the canvas
+* Converting images from one size or format to another
+* Minifying JavaScript
+
+!! Compiling and Rendering
+
+When the text of a tiddler is requested in a different format than its native type, TiddlyWiki5 compiles a JavaScript function that generates the new format from the text of the tiddler.
+
+So, a simple tiddler in {{{application/x-tiddlywiki}}} format might read:
+{{{
+Hello World
+}}}
+The function to render it to {{{text/html}}} might look like this:
+{{{
+function() {
+	return "<p>Hello World</p>";
+}
+}}}
+
+The function can also include calls to the store to incorporate the values of other tiddlers. Consider this tiddler, called {{{HelloThere}}}:
+{{{
+Hello <<tiddler Who>>
+}}}
+And this one called {{{Who}}}:
+{{{
+World
+}}}
+The function to generate {{{HelloThere}}} in {{{text/html}}} might be:
+{{{
+function() {
+	return ["<p>","Hello ", getTiddlerText("Who","text/html"), "</p>"].join(""); 
+}
+}}}
+
+Now, the return value of this function can be cached until a tiddler in the dependency chain changes. The function itself can be cached until the tiddler itself changes.
+
+The dependency chain is calculated when a tiddler is compiled. Every tiddler that is directly referenced is accumulated (until the point at which it is concluded that it is simpler to mark the tiddler as being dependent on any other tiddler changing).
+
+The render function is invoked with `this` pointing to a tiddler object. Parameters to the render function include:
+* ''getTiddler(title)'' - a function to retrieve a specified tiddler
+* ''renderTiddler(title,type,params)'' - a function to render a tiddler to a specified type
+* ''forEachTiddler(sortField,excludeTag,callback)'' - a function to enumerate all tiddlers
+
+Evaluated macro parameters would be parsed and checked for safeness, and then included in the compiled code. For example,
+{{{
+Hello <<echo {{2+2}}>>
+}}}
+Compiles to:
+{{{
+function() {
+	return ["Hello ",(function(){
+		return 2+2;
+	})().toString()].join("");
+}
+}}}
+
+The compilation process has several steps:
+* First, the parse tree is used to generate a JavaScript tree
+* The JavaScript tree is scanned to determine the tiddlers on which this one depends
+* Finally, executable JavaScript text is generated by walking the JavaScript tree
+
+!! Macros
+
+
+
+Macros should be examples of javascript tiddlers that are invoked with parameters. The familiar ones all generate text/html output. For example:
+
+	function tiddler_macro(args) {
+		return getTiddlerText(args.tiddler,"text/html");
+	}
+
+These tiddlers should also have extractable and translateable docs, and perhaps some metadata about how the parameters should be parsed.
+
+Maybe we can use the compiler to convert this form into the former? Yes, of course, we just lift and shift the parse tree from the macro definition into the point where it is used. Yay. So when we're walking the tree, we don't walk into the macro nodes children so much as into the definition tree.
+
+

tiddlywiki5/tiddlers/TiddlyWikiInternals.tid

@@ -0,0 +1,5 @@
+title: TiddlyWikiInternals
+modifier: JeremyRuston
+
+See TiddlyWikiArchitecture for a broad overview
+

tiddlywiki5/tiddlers/split.recipe

@@ -1,3 +1,13 @@
 tiddler: HelloThere.tid
 tiddler: ThisIsAlpha.tid
+tiddler: CommandLineInterface.tid
+tiddler: RecipeFiles.tid
+tiddler: TiddlerFiles.tid
+tiddler: TiddlyWikiArchitecture.tid
+tiddler: TiddlyWikiInternals.tid
+tiddler: NewWikiTextFeatures.tid
+
+tiddler: SiteTitle.tid
+tiddler: SiteSubtitle.tid
+
 tiddler: SimpleTemplate.tid

tiddlywiki5/tiddlywiki5.recipe

@@ -2,9 +2,6 @@ template: tiddlywiki5.template.html
 copyright: ../copyright.txt
 style: styles.css
 
-recipe: ../test/tiddlywiki.2.6.5/source/tiddlywiki.com/tiddlywiki-com-ref/split.recipe
-recipe: ../test/tiddlywiki.2.6.5/source/tiddlywiki.com/tiddlywiki-com/split.recipe
-
 recipe: tiddlers/split.recipe
 
 jslib: ../test/tiddlywiki.2.6.5/source/tiddlywiki/jquery/jquery.js