From d455072f13dd00955ae82d62d2d6e44581bdba9e Mon Sep 17 00:00:00 2001 From: Jeremy Ruston Date: Wed, 14 Jul 2021 09:15:30 +0100 Subject: [PATCH] Add support for JSON-formatted tiddler store, and make it the default (#5708) * Add support for JSON-formatted tiddler store, and make it the default The change to `getTiddlersAsJson()` is to allow experimentation * Move JSON tiddlers into their own store area, and fix support for encrypted tiddlers Also add a dummy old-style store area for backwards compatibility The current arrangement is that JSON tiddlers will always override old-style tiddlers. * Use the deserialiser mechanism to decode the content * Refactor $:/core/modules/deserializers.js before we start extending it Cleaning up the helper function names and ordering * Drop support for the "systemArea" div It was only used in really old v5.0.x * Update deserializer to support JSON store format and add some tests * Life UI restrictions on characters in fieldnames * Add another test case * Correct mis-merge * Remove toLowerCase() methods applied to fieldnames * Insert line breaks in output of getTiddlersAsJson (#5786) Rather than have the entire store on one line, insert a line break after each tiddler. * Refactor #5786 for backwards compatibility * Only read .tiddlywiki-tiddler-store blocks from script tags Prompted by @simonbaird's comment here: https://github.com/Jermolene/TiddlyWiki5/pull/5708#discussion_r648833367 * Clean up escaping of unsafe script characters It seems that escaping `<` is sufficient * Add docs from @saqimtiaz Thanks @saqimtiaz * Docs tweaks * Remove excess whitespace Thanks @simonbaird * Fix templates for lazy loading * Remove obsolete item from release note * Clean up whitespace * Docs for the jsontiddler widget * Fix whitespace Fixes #5840 * Comments * Fix newlines in JSON store area * Remove obsolete docs change Co-authored-by: Simon Baird --- boot/boot.js | 35 ++-- core/language/en-GB/Misc.multids | 1 - core/modules/deserializers.js | 174 ++++++++++-------- core/modules/filters/contains.js | 2 +- core/modules/filters/field.js | 2 +- core/modules/filters/regexp.js | 2 +- core/modules/utils/utils.js | 5 +- core/modules/widgets/fieldmangler.js | 22 +-- core/modules/widgets/jsontiddler.js | 84 +++++++++ core/templates/html-json-skinny-tiddler.tid | 4 + core/templates/html-json-tiddler.tid | 3 + core/templates/store.area.template.html.tid | 47 +++-- editions/dev/tiddlers/HelloThere.tid | 3 +- .../from Heigele and Jurke/Data-Storage.tid | 10 +- ...Data Storage in Single File TiddlyWiki.tid | 97 ++++++++++ editions/dev/tiddlers/new/Data Storage.tid | 10 + .../tests/data/deserializers/case 1.tid | 4 + .../tests/data/deserializers/case 2.tid | 12 ++ .../tests/data/deserializers/case 3.tid | 18 ++ .../tests/data/deserializers/case 4.tid | 19 ++ .../tests/data/deserializers/case 5.tid | 20 ++ .../test/tiddlers/tests/test-deserializers.js | 39 ++++ .../tiddlers/widgets/JSONTiddlerWidget.tid | 22 +++ languages/ar-PS/Misc.multids | 1 - languages/ca-ES/Misc.multids | 1 - languages/cs-CZ/Misc.multids | 1 - languages/da-DK/Misc.multids | 1 - languages/el-GR/Misc.multids | 1 - languages/es-ES/Misc.multids | 1 - languages/fa-IR/Misc.multids | 1 - languages/fr-FR/Misc.multids | 1 - languages/he-IL/Misc.multids | 1 - languages/hi-IN/Misc.multids | 1 - languages/ia-IA/Misc.multids | 1 - languages/it-IT/Misc.multids | 1 - languages/ja-JP/Misc.multids | 1 - languages/ko-KR/Misc.multids | 1 - languages/nl-NL/Misc.multids | 1 - languages/pa-IN/Misc.multids | 1 - languages/pt-BR/Misc.multids | 1 - languages/pt-PT/Misc.multids | 1 - languages/ru-RU/Misc.multids | 1 - languages/sk-SK/Misc.multids | 1 - languages/sl-SI/Misc.multids | 1 - languages/sv-SE/Misc.multids | 1 - languages/zh-Hans/Misc.multids | 1 - languages/zh-Hant/Misc.multids | 1 - .../tiddlyweb/html-json-skinny-tiddler.tid | 4 + .../tiddlyweb/html-json-tiddler.tid | 4 + 49 files changed, 506 insertions(+), 161 deletions(-) create mode 100644 core/modules/widgets/jsontiddler.js create mode 100644 core/templates/html-json-skinny-tiddler.tid create mode 100644 core/templates/html-json-tiddler.tid create mode 100644 editions/dev/tiddlers/new/Data Storage in Single File TiddlyWiki.tid create mode 100644 editions/dev/tiddlers/new/Data Storage.tid create mode 100644 editions/test/tiddlers/tests/data/deserializers/case 1.tid create mode 100644 editions/test/tiddlers/tests/data/deserializers/case 2.tid create mode 100644 editions/test/tiddlers/tests/data/deserializers/case 3.tid create mode 100644 editions/test/tiddlers/tests/data/deserializers/case 4.tid create mode 100644 editions/test/tiddlers/tests/data/deserializers/case 5.tid create mode 100644 editions/test/tiddlers/tests/test-deserializers.js create mode 100644 editions/tw5.com/tiddlers/widgets/JSONTiddlerWidget.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/html-json-skinny-tiddler.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/html-json-tiddler.tid diff --git a/boot/boot.js b/boot/boot.js index 1288cc22d..d1dd9b952 100644 --- a/boot/boot.js +++ b/boot/boot.js @@ -1724,13 +1724,20 @@ $tw.modules.define("$:/boot/tiddlerdeserializer/dom","tiddlerdeserializer",{ }, t,result = []; if(node) { - for(t = 0; t < node.childNodes.length; t++) { + var type = (node.getAttribute && node.getAttribute("type")) || null; + if(type) { + // A new-style container with an explicit deserialization type + result = $tw.wiki.deserializeTiddlers(type,node.textContent); + } else { + // An old-style container of classic DIV-based tiddlers + for(t = 0; t < node.childNodes.length; t++) { var childNode = node.childNodes[t], tiddlers = extractTextTiddlers(childNode); tiddlers = tiddlers || extractModuleTiddlers(childNode); if(tiddlers) { result.push.apply(result,tiddlers); } + } } } return result; @@ -1739,17 +1746,23 @@ $tw.modules.define("$:/boot/tiddlerdeserializer/dom","tiddlerdeserializer",{ $tw.loadTiddlersBrowser = function() { // In the browser, we load tiddlers from certain elements - var containerIds = [ - "libraryModules", - "modules", - "bootKernelPrefix", - "bootKernel", - "styleArea", - "storeArea", - "systemArea" + var containerSelectors = [ + // IDs for old-style v5.1.x tiddler stores + "#libraryModules", + "#modules", + "#bootKernelPrefix", + "#bootKernel", + "#styleArea", + "#storeArea", + "#systemArea", + // Classes for new-style v5.2.x JSON tiddler stores + "script.tiddlywiki-tiddler-store" ]; - for(var t=0; t>/>". Fields can only contain lowercase letters, digits and the characters underscore (`_`), hyphen (`-`) and period (`.`) LayoutSwitcher/Description: Open the layout switcher LazyLoadingWarning:

Trying to load external content from ''<$text text={{!!_canonical_uri}}/>''

If this message doesn't disappear, either the tiddler content type doesn't match the type of the external content, or you may be using a browser that doesn't support external content for wikis loaded as standalone files. See https://tiddlywiki.com/#ExternalText

LoginToTiddlySpace: Login to TiddlySpace diff --git a/core/modules/deserializers.js b/core/modules/deserializers.js index 47e891ed7..bd0a969e2 100644 --- a/core/modules/deserializers.js +++ b/core/modules/deserializers.js @@ -12,63 +12,8 @@ Functions to deserialise tiddlers from a block of text /*global $tw: false */ "use strict"; -/* -Utility function to parse an old-style tiddler DIV in a *.tid file. It looks like this: - -
-
The text of the tiddler (without the expected HTML encoding).
-
-
- -Note that the field attributes are HTML encoded, but that the body of the 
 tag is not encoded.
-
-When these tiddler DIVs are encountered within a TiddlyWiki HTML file then the body is encoded in the usual way.
-*/
-var parseTiddlerDiv = function(text /* [,fields] */) {
-	// Slot together the default results
-	var result = {};
-	if(arguments.length > 1) {
-		for(var f=1; f]*)>(\s*
)?/gi,
-		endRegExp,
-		match = startRegExp.exec(text);
-	if(match) {
-		// Old-style DIVs don't have the 
 tag
-		if(match[2]) {
-			endRegExp = /<\/pre>\s*<\/div>\s*$/gi;
-		} else {
-			endRegExp = /<\/div>\s*$/gi;
-		}
-		var endMatch = endRegExp.exec(text);
-		if(endMatch) {
-			// Extract the text
-			result.text = text.substring(match.index + match[0].length,endMatch.index);
-			// Process the attributes
-			var attrRegExp = /\s*([^=\s]+)\s*=\s*(?:"([^"]*)"|'([^']*)')/gi,
-				attrMatch;
-			do {
-				attrMatch = attrRegExp.exec(match[1]);
-				if(attrMatch) {
-					var name = attrMatch[1];
-					var value = attrMatch[2] !== undefined ? attrMatch[2] : attrMatch[3];
-					result[name] = value;
-				}
-			} while(attrMatch);
-			return result;
-		}
-	}
-	return undefined;
-};
-
 exports["application/x-tiddler-html-div"] = function(text,fields) {
-	return [parseTiddlerDiv(text,fields)];
+	return [deserializeTiddlerDiv(text,fields)];
 };
 
 exports["application/json"] = function(text,fields) {
@@ -105,30 +50,34 @@ Parse an HTML file into tiddlers. There are three possibilities:
 # An ordinary HTML file
 */
 exports["text/html"] = function(text,fields) {
-	// Check if we've got a store area
+	var results = [];
+	// Check if we've got an old-style store area
 	var storeAreaMarkerRegExp = /
/gi,
-		match = storeAreaMarkerRegExp.exec(text);
-	if(match) {
-		// If so, it's either a classic TiddlyWiki file or an unencrypted TW5 file
-		// First read the normal tiddlers
-		var results = deserializeTiddlyWikiFile(text,storeAreaMarkerRegExp.lastIndex,!!match[1],fields);
-		// Then any system tiddlers
-		var systemAreaMarkerRegExp = /
/gi,
-			sysMatch = systemAreaMarkerRegExp.exec(text);
-		if(sysMatch) {
-			results.push.apply(results,deserializeTiddlyWikiFile(text,systemAreaMarkerRegExp.lastIndex,!!sysMatch[1],fields));
-		}
+		storeAreaMatch = storeAreaMarkerRegExp.exec(text);
+	if(storeAreaMatch) {
+		// If so, we've got tiddlers in classic TiddlyWiki format or unencrypted old-style TW5 format
+		results.push.apply(results,deserializeStoreArea(text,storeAreaMarkerRegExp.lastIndex,!!storeAreaMatch[1],fields));
+	}
+	// Check for new-style store areas
+	var newStoreAreaMarkerRegExp = /`
+        ``
+    
+    <$list filter="[[storeAreaFormat]is[variable]getvariable[]else[json]match[div]]">
+        
+        <$reveal type="nomatch" state="$:/isEncrypted" text="yes">
+            ``
+        
+    
+
+
+<$list filter="[[$:/isEncrypted]get[text]else[no]match[yes]]">
+    ``
+    ``
+
\ No newline at end of file
diff --git a/editions/dev/tiddlers/HelloThere.tid b/editions/dev/tiddlers/HelloThere.tid
index 8afc1b22c..50052f2c8 100644
--- a/editions/dev/tiddlers/HelloThere.tid
+++ b/editions/dev/tiddlers/HelloThere.tid
@@ -8,6 +8,7 @@ Welcome to the developer documentation for TiddlyWiki (https://tiddlywiki.com/).
 
 * An assignment by Christian Jurke and Christian Heigele, two students working on their Master's degree in Information Technology at the Gießen University of Applied Sciences (Technische Hochschule Mittelhessen). Their work can be seen in the [[Introduction]] and the tiddlers that link from it.
 * New developer documentation
+** [[Data Storage in Single File TiddlyWiki]]
 ** [[Continuous Deployment]]
 ** [[GitHub Branches]]
 ** HookMechanism
@@ -29,4 +30,4 @@ Welcome to the developer documentation for TiddlyWiki (https://tiddlywiki.com/).
 ** [[Scripts for building tiddlywiki.com]]
 ** SyncAdaptorModules
 ** WidgetModules
-** WikiRuleModules
\ No newline at end of file
+** WikiRuleModules
diff --git a/editions/dev/tiddlers/from Heigele and Jurke/Data-Storage.tid b/editions/dev/tiddlers/from Heigele and Jurke/Data-Storage.tid
index c26af2aa8..6beff984a 100644
--- a/editions/dev/tiddlers/from Heigele and Jurke/Data-Storage.tid	
+++ b/editions/dev/tiddlers/from Heigele and Jurke/Data-Storage.tid	
@@ -3,12 +3,4 @@ modified: 20140710081051087
 tags: doc
 title: Data-Storage
 
-TW has two approaches to save the user data. These approaches depends on way you use TW. either you use node.js as a server for TW its saves the tiddlers as plain text in different files or you use TW as standalone in a browser it persists the data within the HTML-File in two Div-Areas depending on whether the encryption of the TiddlyWiki is activated or not. If the TiddlyWiki is not encrypted the data is stored in the Div-Area called "~StoreArea". Every created Tiddler is stored in a own Div-area with a few custom values. An example of a saved Tiddler is shown below (\prettyref{lst:data-div}). 
-
-```html
-

-	
testText

-

-```
-The Div-Area has the same attributes like the standard tillder fields, listed in  [[TiddlerFields|https://tiddlywiki.com/#TiddlerFields]], all attributes which are not in this list are parsed as a custom field. The only required attribute is the name attribute, all other attributes are optional.\\
-With a activated encryption the data is stored in a special Div-Area called "encryptedStoreArea". TiddlyWiki uses the Standford [[JavaScript Crypto Libary|http://bitwiseshiftleft.github.io/sjcl/]]. The encrypted Tiddlers are saved in a JSON string within this Div-Area.
+{{Data Storage}}
diff --git a/editions/dev/tiddlers/new/Data Storage in Single File TiddlyWiki.tid b/editions/dev/tiddlers/new/Data Storage in Single File TiddlyWiki.tid
new file mode 100644
index 000000000..7ca85be31
--- /dev/null
+++ b/editions/dev/tiddlers/new/Data Storage in Single File TiddlyWiki.tid	
@@ -0,0 +1,97 @@
+created: 20210525165258247
+modified: 20210614134705320
+tags: Data-Storage doc [[Data Storage]]
+title: Data Storage in Single File TiddlyWiki
+
+The single file version of ~TiddlyWiki saves the tiddlers within the HTML file. 
+
+Version 5.2.0 of ~TiddlyWiki introduced a new format for how tiddlers are saved within the HTML file.
+
+
+---
+
+!! Up to and including ~TiddlyWiki v5.1.23
+
+Tiddlers are saved within the HTML file in one of two `
` tags depending on whether the TiddlyWiki is configured to encrypt its content or not. 
+
+!!! Without encryption
+If the ~TiddlyWiki is not encrypted the data is stored in a `
` tag with an `id` attribute of "storeArea".
+
+```html
+