From b95723a022669371cd139b332cf3a64bc438b73e Mon Sep 17 00:00:00 2001 From: Jeremy Ruston Date: Mon, 30 Mar 2020 15:24:05 +0100 Subject: [PATCH] Fix syncer to handler errors properly (#4373) * First commit * Add throttling of saves Now we refuse to save a tiddler more often than once per second. * Wait for a timeout before trying again after an error * Modest optimisations of isDirty() method * Synchronise system tiddlers and deletions from the server Fixes two long-standing issues: * Changes to system tiddlers are not synchronised from the server to the browser * Deletions of tiddlers on the server are not propagated to browser clients * Make sure we update the dirty status even if there isn't a task to perform * Replace save-wiki button with popup sync menu * Remove the "Server" control panel tab We don't need it with the enhanced sync dropdown * Add indentation to the save-wiki button * Fix spacing in dropdown menu items * Switch between cloud icons according to dirty status * Add a menu item to copy syncer logs to the clipboard * Improve animated icon * Remove indentation from save-wiki button @pmario the annoying thing is that using `\trim whitespace` trims significant whitespace too, so it means we have to use <$text text=" "/> when we need a space that won't be trimmed. For the moment, I've removed the indentation but will keep thinking about it. * Further icon, UI and copy text tweaks Move the icons and styles from the core into the TiddlyWeb plugin * Clean up PR diff * Tweak animation durations * Break the actions from the syncer dropdown into separate tiddlers @pmario I think this makes things a bit easier to follow * Refactor syncadaptor creation and logging The goal is for the syncadaptor to be able to log to the same logger as the syncer, so that the "copy syncer logs to clipboard" data is more useful. * Don't transition the dirty indicator container colour, just the SVG's colour * Only trigger a sync for changes to tiddlers we're interested in Otherwise it is triggered by the creation of the alert tiddlers used to display errors. * Restore deleting local tiddlers removed from the server (I had commented it out for some testing and accidentally commited it). * Guard against missing adaptor info * We still need to trigger a timeout when there was no task to process * Avoid repeatedly polling for changes Instead we only trigger a timeout call at if there is a pending task (ie a tiddler that has changed but isn't yet old enough to save). * Lazy loading: include skinny versions of lazily loaded tiddlers in the index.html * Introduce _is_skinny field for indicating that a tiddler is subject to lazy loading * Remove savetrail plugin from prerelease It doesn't yet work with the new syncer * Make the savetrail plugin work again * Clear outstanding alerts when synchronisation is restored * Logger: only remove alerts from the same component Missed off 9f5c0de07 * Make the saving throttle interval configurable (#4385) After switching Bob to use the core syncer the throttle interval makes saving feel very sluggish compared to the message queue setup that I had before. The editing lock that I use to prevent conflicts with multiple users doesn't go away until the save is completed, and with the 1 second delay it means that if you edit a tiddler and save it than you have to wait one second before you can edit it again. * Tweaks to appearance of alerts * Exclude temp tiddlers from offline snapshots Otherwise alerts will persist * Tweak appearance of status line in dropdown * Update release note * Web server: Don't include full path in error messages Fixes #3724 * In change event handler check for deletions * Disable the official plugin library when the tiddlyweb plugin is loaded * Hide error details from browser for /files/ route See https://github.com/Jermolene/TiddlyWiki5/issues/3724#issuecomment-565702492 -- thanks @pmario * Revert all the changes to the relationship between the syncer and the syncadaptor Previously we had some major rearrangements to make it possible for the syncadaptor to route it's logging to the logger used by the syncer. The motivation is so that the "copy logs to clipboard" button is more useful. On reflection, changing the interface this drastically is undesirable from a backwards compatibility perspective, so I'm going to investigate other ways to achieve the logger sharing * Make the tiddlyweb adaptor use the syncer's logger So that both are availavble when copying the syncer logs to the clipboard * Update release note * Support setting port=0 to get an OS assigned port Quite useful * Update code comment * UI: Use "Get latest changes from server" instead of "Refresh" * Add getUpdatedTiddlers() method to syncadaptor API See https://github.com/Jermolene/TiddlyWiki5/pull/4373#issuecomment-573579495 * Refactor revision handling within the syncer Thanks @pmario * Fix typo in tiddlywebadaptor * Improve presentation of errors See https://github.com/Jermolene/TiddlyWiki5/pull/4373#issuecomment-573695267 * Add docs for getTiddlerRevision() * Remove unused error animation * Update comment for GET /recipes/default/tiddlers/tiddlers.json * Optimise SVG cloud image * Add optional list of allowed filters for get all tiddlers route An attempt to address @Arlen22's concern here: https://github.com/Jermolene/TiddlyWiki5/pull/4373#pullrequestreview-342146190 * Fix network error alert text translatability * Fix error code and logging for GET /recipes/default/tiddlers/tiddlers.json Thanks @Arlen22 * Flip GET /recipes/default/tiddlers/tiddlers.json allowed filter handling to be secure by default * Validate updates received from getUpdatedTiddlers() * Add syncer method to force loading of a tiddler from the server * Remove the release note update to remove the merge conflict * Fix crash when there's no config section in the tiddlywiki.info file * Use config tiddler title to check filter query (merge into fix-syncer) (#4478) * Use config tiddler title to check filter query * Create config-tiddlers-filter.tid * Add config switch to enable all filters on GET /recipes/default/tiddlers/tiddlers.json And update docs * Fix bug when deleting a tiddler with a shadow Reported by @kookma at https://github.com/Jermolene/TiddlyWiki5/pull/4373#issuecomment-604027528 Co-authored-by: jed Co-authored-by: Arlen22 --- core/language/en-GB/Fields.multids | 1 + core/language/en-GB/Misc.multids | 1 + core/modules/server/routes/get-file.js | 18 +- .../server/routes/get-tiddlers-json.js | 33 +- core/modules/server/server.js | 16 +- core/modules/startup/startup.js | 2 +- core/modules/syncer.js | 584 ++++++++++-------- core/modules/utils/dom/http.js | 9 +- core/modules/utils/logger.js | 50 +- core/modules/utils/utils.js | 18 + core/modules/wiki.js | 4 +- core/templates/html-div-skinny-tiddler.tid | 9 + core/templates/save-lazy-all.tid | 3 + core/templates/save-lazy-images.tid | 3 + core/templates/store.area.template.html.tid | 1 + core/ui/AlertTemplate.tid | 6 +- .../config/ServerExternalFiltersDefault.tid | 2 + .../moduletypes/SyncAdaptorModules.tid | 47 +- .../tiddlers/concepts/TiddlerFields.tid | 4 +- .../WebServer API_ Get All Tiddlers.tid | 18 +- .../webserver/WebServer Parameter_ port.tid | 3 +- .../filesystem/filesystemadaptor.js | 2 + .../tiddlyweb/ServerControlPanel.tid | 20 - .../tiddlyweb/config-tiddlers-filter.tid | 2 + .../tiddlyweb/configOfficialPluginLibrary.tid | 3 + plugins/tiddlywiki/tiddlyweb/css-tiddler.tid | 7 + .../tiddlyweb/html-div-skinny-tiddler.tid | 9 + plugins/tiddlywiki/tiddlyweb/icon-cloud.tid | 4 + .../tiddlyweb/javascript-tiddler.tid | 7 + plugins/tiddlywiki/tiddlyweb/save-offline.tid | 2 +- .../tiddlywiki/tiddlyweb/save-wiki-button.tid | 25 + plugins/tiddlywiki/tiddlyweb/styles.tid | 40 ++ .../tiddlyweb/syncer-actions-copy-logs.tid | 6 + .../tiddlyweb/syncer-actions-login-status.tid | 9 + .../tiddlyweb/syncer-actions-login.tid | 8 + .../tiddlyweb/syncer-actions-logout.tid | 8 + .../tiddlyweb/syncer-actions-refresh.tid | 9 + .../syncer-actions-save-snapshot.tid | 9 + .../tiddlyweb/tags-syncerdropdown.tid | 2 + .../tiddlywiki/tiddlyweb/tiddlywebadaptor.js | 16 +- themes/tiddlywiki/vanilla/base.tid | 26 +- 41 files changed, 722 insertions(+), 324 deletions(-) create mode 100644 core/templates/html-div-skinny-tiddler.tid create mode 100644 core/wiki/config/ServerExternalFiltersDefault.tid delete mode 100644 plugins/tiddlywiki/tiddlyweb/ServerControlPanel.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/config-tiddlers-filter.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/configOfficialPluginLibrary.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/css-tiddler.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/html-div-skinny-tiddler.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/icon-cloud.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/javascript-tiddler.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/save-wiki-button.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/styles.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/syncer-actions-copy-logs.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/syncer-actions-login-status.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/syncer-actions-login.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/syncer-actions-logout.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/syncer-actions-refresh.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/syncer-actions-save-snapshot.tid create mode 100644 plugins/tiddlywiki/tiddlyweb/tags-syncerdropdown.tid diff --git a/core/language/en-GB/Fields.multids b/core/language/en-GB/Fields.multids index 6b4823c02..d1eeabd42 100644 --- a/core/language/en-GB/Fields.multids +++ b/core/language/en-GB/Fields.multids @@ -35,3 +35,4 @@ title: The unique name of a tiddler toc-link: Suppresses the tiddler's link in a Table of Contents tree if set to: ''no'' type: The content type of a tiddler version: Version information for a plugin +_is_skinny: If present, indicates that the tiddler text field must be loaded from the server diff --git a/core/language/en-GB/Misc.multids b/core/language/en-GB/Misc.multids index ee1d0320e..861bb6899 100644 --- a/core/language/en-GB/Misc.multids +++ b/core/language/en-GB/Misc.multids @@ -28,6 +28,7 @@ Error/Filter: Filter error Error/FilterSyntax: Syntax error in filter expression Error/IsFilterOperator: Filter Error: Unknown operand for the 'is' filter operator Error/LoadingPluginLibrary: Error loading plugin library +Error/NetworkErrorAlert: `

''Network Error''

It looks like the connection to the server has been lost. This may indicate a problem with your network connection. Please attempt to restore network connectivity before continuing.

''Any unsaved changes will be automatically synchronised when connectivity is restored''.` Error/RecursiveTransclusion: Recursive transclusion error in transclude widget Error/RetrievingSkinny: Error retrieving skinny tiddler list Error/SavingToTWEdit: Error saving to TWEdit diff --git a/core/modules/server/routes/get-file.js b/core/modules/server/routes/get-file.js index 3429c4cd2..2a0ef647a 100644 --- a/core/modules/server/routes/get-file.js +++ b/core/modules/server/routes/get-file.js @@ -19,22 +19,16 @@ exports.path = /^\/files\/(.+)$/; exports.handler = function(request,response,state) { var path = require("path"), fs = require("fs"), - util = require("util"); - var filename = path.resolve($tw.boot.wikiPath,"files",decodeURIComponent(state.params[0])), + util = require("util"), + suppliedFilename = decodeURIComponent(state.params[0]), + filename = path.resolve($tw.boot.wikiPath,"files",suppliedFilename), extension = path.extname(filename); fs.readFile(filename,function(err,content) { var status,content,type = "text/plain"; if(err) { - if(err.code === "ENOENT") { - status = 404; - content = "File '" + filename + "' not found"; - } else if(err.code === "EACCES") { - status = 403; - content = "You do not have permission to access the file '" + filename + "'"; - } else { - status = 500; - content = err.toString(); - } + console.log("Error accessing file " + filename + ": " + err.toString()); + status = 404; + content = "File '" + suppliedFilename + "' not found"; } else { status = 200; content = content; diff --git a/core/modules/server/routes/get-tiddlers-json.js b/core/modules/server/routes/get-tiddlers-json.js index 3ece35ce1..8e93733e4 100644 --- a/core/modules/server/routes/get-tiddlers-json.js +++ b/core/modules/server/routes/get-tiddlers-json.js @@ -3,7 +3,7 @@ title: $:/core/modules/server/routes/get-tiddlers-json.js type: application/javascript module-type: route -GET /recipes/default/tiddlers/tiddlers.json +GET /recipes/default/tiddlers/tiddlers.json?filter= \*/ (function() { @@ -12,23 +12,34 @@ GET /recipes/default/tiddlers/tiddlers.json /*global $tw: false */ "use strict"; +var DEFAULT_FILTER = "[all[tiddlers]!is[system]sort[title]]"; + exports.method = "GET"; exports.path = /^\/recipes\/default\/tiddlers.json$/; exports.handler = function(request,response,state) { + var filter = state.queryParameters.filter || DEFAULT_FILTER; + if($tw.wiki.getTiddlerText("$:/config/Server/AllowAllExternalFilters") !== "yes") { + if($tw.wiki.getTiddlerText("$:/config/Server/ExternalFilters/" + filter) !== "yes") { + console.log("Blocked attempt to GET /recipes/default/tiddlers/tiddlers.json with filter: " + filter); + response.writeHead(403); + response.end(); + return; + } + } + var excludeFields = (state.queryParameters.exclude || "text").split(","), + titles = state.wiki.filterTiddlers(filter); response.writeHead(200, {"Content-Type": "application/json"}); var tiddlers = []; - state.wiki.forEachTiddler({sortField: "title"},function(title,tiddler) { - var tiddlerFields = {}; - $tw.utils.each(tiddler.fields,function(field,name) { - if(name !== "text") { - tiddlerFields[name] = tiddler.getFieldString(name); - } - }); - tiddlerFields.revision = state.wiki.getChangeCount(title); - tiddlerFields.type = tiddlerFields.type || "text/vnd.tiddlywiki"; - tiddlers.push(tiddlerFields); + $tw.utils.each(titles,function(title) { + var tiddler = state.wiki.getTiddler(title); + if(tiddler) { + var tiddlerFields = tiddler.getFieldStrings({exclude: excludeFields}); + tiddlerFields.revision = state.wiki.getChangeCount(title); + tiddlerFields.type = tiddlerFields.type || "text/vnd.tiddlywiki"; + tiddlers.push(tiddlerFields); + } }); var text = JSON.stringify(tiddlers); response.end(text,"utf8"); diff --git a/core/modules/server/server.js b/core/modules/server/server.js index 3225b95f3..3226cacd7 100644 --- a/core/modules/server/server.js +++ b/core/modules/server/server.js @@ -16,7 +16,8 @@ if($tw.node) { var util = require("util"), fs = require("fs"), url = require("url"), - path = require("path"); + path = require("path"), + querystring = require("querystring"); } /* @@ -162,6 +163,7 @@ Server.prototype.requestHandler = function(request,response) { state.wiki = self.wiki; state.server = self; state.urlInfo = url.parse(request.url); + state.queryParameters = querystring.parse(state.urlInfo.query); // Get the principals authorized to access this resource var authorizationType = this.methodMappings[request.method] || "readers"; // Check for the CSRF header if this is a write @@ -236,6 +238,7 @@ host: optional host address (falls back to value of "host" variable) prefix: optional prefix (falls back to value of "path-prefix" variable) */ Server.prototype.listen = function(port,host,prefix) { + var self = this; // Handle defaults for port and host port = port || this.get("port"); host = host || this.get("host"); @@ -244,19 +247,24 @@ Server.prototype.listen = function(port,host,prefix) { if(parseInt(port,10).toString() !== port) { port = process.env[port] || 8080; } - $tw.utils.log("Serving on " + this.protocol + "://" + host + ":" + port + prefix,"brown/orange"); - $tw.utils.log("(press ctrl-C to exit)","red"); // Warn if required plugins are missing if(!$tw.wiki.getTiddler("$:/plugins/tiddlywiki/tiddlyweb") || !$tw.wiki.getTiddler("$:/plugins/tiddlywiki/filesystem")) { $tw.utils.warning("Warning: Plugins required for client-server operation (\"tiddlywiki/filesystem\" and \"tiddlywiki/tiddlyweb\") are missing from tiddlywiki.info file"); } - // Listen + // Create the server var server; if(this.listenOptions) { server = this.transport.createServer(this.listenOptions,this.requestHandler.bind(this)); } else { server = this.transport.createServer(this.requestHandler.bind(this)); } + // Display the port number after we've started listening (the port number might have been specified as zero, in which case we will get an assigned port) + server.on("listening",function() { + var address = server.address(); + $tw.utils.log("Serving on " + self.protocol + "://" + address.address + ":" + address.port + prefix,"brown/orange"); + $tw.utils.log("(press ctrl-C to exit)","red"); + }); + // Listen return server.listen(port,host); }; diff --git a/core/modules/startup/startup.js b/core/modules/startup/startup.js index 4cd53dfcd..ad1416bf3 100755 --- a/core/modules/startup/startup.js +++ b/core/modules/startup/startup.js @@ -128,7 +128,7 @@ exports.startup = function() { // Set up the syncer object if we've got a syncadaptor if($tw.syncadaptor) { $tw.syncer = new $tw.Syncer({wiki: $tw.wiki, syncadaptor: $tw.syncadaptor}); - } + } // Setup the saver handler $tw.saverHandler = new $tw.SaverHandler({ wiki: $tw.wiki, diff --git a/core/modules/syncer.js b/core/modules/syncer.js index 0b84be750..f39646eac 100644 --- a/core/modules/syncer.js +++ b/core/modules/syncer.js @@ -3,7 +3,7 @@ title: $:/core/modules/syncer.js type: application/javascript module-type: global -The syncer tracks changes to the store. If a syncadaptor is used then individual tiddlers are synchronised through it. If there is no syncadaptor then the entire wiki is saved via saver modules. +The syncer tracks changes to the store and synchronises them to a remote data store represented as a "sync adaptor" \*/ (function(){ @@ -23,8 +23,10 @@ Syncer.prototype.titleSyncFilter = "$:/config/SyncFilter"; Syncer.prototype.titleSyncPollingInterval = "$:/config/SyncPollingInterval"; Syncer.prototype.titleSyncDisableLazyLoading = "$:/config/SyncDisableLazyLoading"; Syncer.prototype.titleSavedNotification = "$:/language/Notifications/Save/Done"; +Syncer.prototype.titleSyncThrottleInterval = "$:/config/SyncThrottleInterval"; Syncer.prototype.taskTimerInterval = 1 * 1000; // Interval for sync timer Syncer.prototype.throttleInterval = 1 * 1000; // Defer saving tiddlers if they've changed in the last 1s... +Syncer.prototype.errorRetryInterval = 5 * 1000; // Interval to retry after an error Syncer.prototype.fallbackInterval = 10 * 1000; // Unless the task is older than 10s Syncer.prototype.pollTimerInterval = 60 * 1000; // Interval for polling for changes from the adaptor @@ -36,6 +38,7 @@ wiki: wiki to be synced function Syncer(options) { var self = this; this.wiki = options.wiki; + // Save parameters this.syncadaptor = options.syncadaptor; this.disableUI = !!options.disableUI; this.titleIsLoggedIn = options.titleIsLoggedIn || this.titleIsLoggedIn; @@ -43,27 +46,54 @@ function Syncer(options) { this.titleSyncFilter = options.titleSyncFilter || this.titleSyncFilter; this.titleSavedNotification = options.titleSavedNotification || this.titleSavedNotification; this.taskTimerInterval = options.taskTimerInterval || this.taskTimerInterval; - this.throttleInterval = options.throttleInterval || this.throttleInterval; + this.throttleInterval = options.throttleInterval || parseInt(this.wiki.getTiddlerText(this.titleSyncThrottleInterval,""),10) || this.throttleInterval; + this.errorRetryInterval = options.errorRetryInterval || this.errorRetryInterval; this.fallbackInterval = options.fallbackInterval || this.fallbackInterval; this.pollTimerInterval = options.pollTimerInterval || parseInt(this.wiki.getTiddlerText(this.titleSyncPollingInterval,""),10) || this.pollTimerInterval; this.logging = "logging" in options ? options.logging : true; // Make a logger this.logger = new $tw.utils.Logger("syncer" + ($tw.browser ? "-browser" : "") + ($tw.node ? "-server" : "") + (this.syncadaptor.name ? ("-" + this.syncadaptor.name) : ""),{ - colour: "cyan", - enable: this.logging - }); + colour: "cyan", + enable: this.logging, + saveHistory: true + }); + if(this.syncadaptor.setLoggerSaveBuffer) { + this.syncadaptor.setLoggerSaveBuffer(this.logger); + } // Compile the dirty tiddler filter this.filterFn = this.wiki.compileFilter(this.wiki.getTiddlerText(this.titleSyncFilter)); // Record information for known tiddlers this.readTiddlerInfo(); - // Tasks are {type: "load"/"save"/"delete", title:, queueTime:, lastModificationTime:} - this.taskQueue = {}; // Hashmap of tasks yet to be performed - this.taskInProgress = {}; // Hash of tasks in progress + this.titlesToBeLoaded = {}; // Hashmap of titles of tiddlers that need loading from the server + this.titlesHaveBeenLazyLoaded = {}; // Hashmap of titles of tiddlers that have already been lazily loaded from the server + // Timers this.taskTimerId = null; // Timer for task dispatch this.pollTimerId = null; // Timer for polling server + // Number of outstanding requests + this.numTasksInProgress = 0; // Listen out for changes to tiddlers this.wiki.addEventListener("change",function(changes) { - self.syncToServer(changes); + // Filter the changes to just include ones that are being synced + var filteredChanges = self.getSyncedTiddlers(function(callback) { + $tw.utils.each(changes,function(change,title) { + var tiddler = self.wiki.tiddlerExists(title) && self.wiki.getTiddler(title); + callback(tiddler,title); + }); + }); + if(filteredChanges.length > 0) { + self.processTaskQueue(); + } else { + // Look for deletions of tiddlers we're already syncing + var outstandingDeletion = false + $tw.utils.each(changes,function(change,title,object) { + if(change.deleted && $tw.utils.hop(self.tiddlerInfo,title)) { + outstandingDeletion = true; + } + }); + if(outstandingDeletion) { + self.processTaskQueue(); + } + } }); // Browser event handlers if($tw.browser && !this.disableUI) { @@ -86,6 +116,9 @@ function Syncer(options) { $tw.rootWidget.addEventListener("tm-server-refresh",function() { self.handleRefreshEvent(); }); + $tw.rootWidget.addEventListener("tm-copy-syncer-logs-to-clipboard",function() { + $tw.utils.copyToClipboard($tw.utils.getSystemInfo() + "\n\nLog:\n" + self.logger.getBuffer()); + }); } // Listen out for lazyLoad events if(!this.disableUI && $tw.wiki.getTiddlerText(this.titleSyncDisableLazyLoading) !== "yes") { @@ -100,45 +133,83 @@ function Syncer(options) { }); } +/* +Show a generic network error alert +*/ +Syncer.prototype.showErrorAlert = function() { +console.log($tw.language.getString("Error/NetworkErrorAlert")) + this.logger.alert($tw.language.getString("Error/NetworkErrorAlert")); +}; + +/* +Return an array of the tiddler titles that are subjected to syncing +*/ +Syncer.prototype.getSyncedTiddlers = function(source) { + return this.filterFn.call(this.wiki,source); +}; + +/* +Return an array of the tiddler titles that are subjected to syncing +*/ +Syncer.prototype.getTiddlerRevision = function(title) { + if(this.syncadaptor && this.syncadaptor.getTiddlerRevision) { + return this.syncadaptor.getTiddlerRevision(title); + } else { + return this.wiki.getTiddler(title).fields.revision; + } +}; + /* Read (or re-read) the latest tiddler info from the store */ Syncer.prototype.readTiddlerInfo = function() { // Hashmap by title of {revision:,changeCount:,adaptorInfo:} + // "revision" is the revision of the tiddler last seen on the server, and "changecount" is the corresponding local changecount this.tiddlerInfo = {}; // Record information for known tiddlers var self = this, - tiddlers = this.filterFn.call(this.wiki); + tiddlers = this.getSyncedTiddlers(); $tw.utils.each(tiddlers,function(title) { - var tiddler = self.wiki.getTiddler(title); + var tiddler = self.wiki.tiddlerExists(title) && self.wiki.getTiddler(title); self.tiddlerInfo[title] = { - revision: tiddler.fields.revision, + revision: self.getTiddlerRevision(title), adaptorInfo: self.syncadaptor && self.syncadaptor.getTiddlerInfo(tiddler), - changeCount: self.wiki.getChangeCount(title), - hasBeenLazyLoaded: false + changeCount: self.wiki.getChangeCount(title) }; }); }; -/* -Create an tiddlerInfo structure if it doesn't already exist -*/ -Syncer.prototype.createTiddlerInfo = function(title) { - if(!$tw.utils.hop(this.tiddlerInfo,title)) { - this.tiddlerInfo[title] = { - revision: null, - adaptorInfo: {}, - changeCount: -1, - hasBeenLazyLoaded: false - }; - } -}; - /* Checks whether the wiki is dirty (ie the window shouldn't be closed) */ Syncer.prototype.isDirty = function() { - return (this.numTasksInQueue() > 0) || (this.numTasksInProgress() > 0); + this.logger.log("Checking dirty status"); + // Check tiddlers that are in the store and included in the filter function + var titles = this.getSyncedTiddlers(); + for(var index=0; index tiddlerInfo.changeCount) { + return true; + } + } else { + // If the tiddler isn't known on the server then it needs to be saved to the server + return true; + } + } + } + // Check tiddlers that are known from the server but not currently in the store + titles = Object.keys(this.tiddlerInfo); + for(index=0; index 0 || updates.deletions.length > 0) { + self.processTaskQueue(); + } + } + }); + } else if(this.syncadaptor && this.syncadaptor.getSkinnyTiddlers) { + this.logger.log("Retrieving skinny tiddler list"); + cancelNextSync(); + this.syncadaptor.getSkinnyTiddlers(function(err,tiddlers) { + triggerNextSync(); + // Check for errors + if(err) { + self.showErrorAlert(); + self.logger.log($tw.language.getString("Error/RetrievingSkinny") + ":",err); + return; + } + // Keep track of which tiddlers we already know about have been reported this time + var previousTitles = Object.keys(self.tiddlerInfo); // Process each incoming tiddler for(var t=0; t 0) { - this.triggerTimeout(); - } - } - } -}; +Choose the next sync task. We prioritise saves, then deletes, then loads from the server -/* -Choose the next applicable task +Returns either a task object, null if there's no upcoming tasks, or the boolean true if there are pending tasks that aren't yet due */ Syncer.prototype.chooseNextTask = function() { - var self = this, - candidateTask = null, - now = Date.now(); - // Select the best candidate task - $tw.utils.each(this.taskQueue,function(task,title) { - // Exclude the task if there's one of the same name in progress - if($tw.utils.hop(self.taskInProgress,title)) { - return; + var thresholdLastSaved = (new Date()) - this.throttleInterval, + havePending = null; + // First we look for tiddlers that have been modified locally and need saving back to the server + var titles = this.getSyncedTiddlers(); + for(var index=0; index tiddlerInfo.changeCount, + isReadyToSave = !tiddlerInfo || !tiddlerInfo.timestampLastSaved || tiddlerInfo.timestampLastSaved < thresholdLastSaved; + if(hasChanged) { + if(isReadyToSave) { + return new SaveTiddlerTask(this,title); + } else { + havePending = true; + } + } } - // Exclude the task if it is a save and the tiddler has been modified recently, but not hit the fallback time - if(task.type === "save" && (now - task.lastModificationTime) < self.throttleInterval && - (now - task.queueTime) < self.fallbackInterval) { - return; + } + // Second, we check tiddlers that are known from the server but not currently in the store, and so need deleting on the server + titles = Object.keys(this.tiddlerInfo); + for(index=0; index 1) { @@ -87,6 +115,22 @@ Logger.prototype.alert = function(/* args */) { } }; +/* +Clear outstanding alerts +*/ +Logger.prototype.clearAlerts = function() { + var self = this; + if($tw.browser && this.alertCount > 0) { + $tw.utils.each($tw.wiki.getTiddlersWithTag(ALERT_TAG),function(title) { + var tiddler = $tw.wiki.getTiddler(title); + if(tiddler.fields.component === self.componentName) { + $tw.wiki.deleteTiddler(title); + } + }); + this.alertCount = 0; + } +}; + exports.Logger = Logger; })(); diff --git a/core/modules/utils/utils.js b/core/modules/utils/utils.js index 26fed0f3c..8d8c294d9 100644 --- a/core/modules/utils/utils.js +++ b/core/modules/utils/utils.js @@ -783,4 +783,22 @@ exports.strEndsWith = function(str,ending,position) { } }; +/* +Return system information useful for debugging +*/ +exports.getSystemInfo = function(str,ending,position) { + var results = [], + save = function(desc,value) { + results.push(desc + ": " + value); + }; + if($tw.browser) { + save("User Agent",navigator.userAgent); + save("Online Status",window.navigator.onLine); + } + if($tw.node) { + save("Node Version",process.version); + } + return results.join("\n"); +}; + })(); diff --git a/core/modules/wiki.js b/core/modules/wiki.js index d44940cb6..f734650de 100755 --- a/core/modules/wiki.js +++ b/core/modules/wiki.js @@ -1246,9 +1246,9 @@ exports.getTiddlerText = function(title,defaultText) { if(!tiddler) { return defaultText; } - if(tiddler.fields.text !== undefined) { + if(!tiddler.hasField("_is_skinny")) { // Just return the text if we've got it - return tiddler.fields.text; + return tiddler.fields.text || ""; } else { // Tell any listeners about the need to lazily load this tiddler this.dispatchEvent("lazyLoad",title); diff --git a/core/templates/html-div-skinny-tiddler.tid b/core/templates/html-div-skinny-tiddler.tid new file mode 100644 index 000000000..eaf388605 --- /dev/null +++ b/core/templates/html-div-skinny-tiddler.tid @@ -0,0 +1,9 @@ +title: $:/core/templates/html-div-skinny-tiddler + +``> +

+`
diff --git a/core/templates/save-lazy-all.tid b/core/templates/save-lazy-all.tid
index 5f9220e19..bf7f9f606 100644
--- a/core/templates/save-lazy-all.tid
+++ b/core/templates/save-lazy-all.tid
@@ -3,4 +3,7 @@ title: $:/core/save/lazy-all
 \define saveTiddlerFilter()
 [is[system]] -[prefix[$:/state/popup/]] -[[$:/HistoryList]] -[[$:/boot/boot.css]] -[type[application/javascript]library[yes]] -[[$:/boot/boot.js]] -[[$:/boot/bootprefix.js]] +[sort[title]] 
 \end
+\define skinnySaveTiddlerFilter()
+[!is[system]]
+\end
 {{$:/core/templates/tiddlywiki5.html}}
diff --git a/core/templates/save-lazy-images.tid b/core/templates/save-lazy-images.tid
index ff3204729..62334f0db 100644
--- a/core/templates/save-lazy-images.tid
+++ b/core/templates/save-lazy-images.tid
@@ -3,4 +3,7 @@ title: $:/core/save/lazy-images
 \define saveTiddlerFilter()
 [is[tiddler]] -[prefix[$:/state/popup/]] -[[$:/HistoryList]] -[[$:/boot/boot.css]] -[type[application/javascript]library[yes]] -[[$:/boot/boot.js]] -[[$:/boot/bootprefix.js]] -[!is[system]is[image]] +[sort[title]] 
 \end
+\define skinnySaveTiddlerFilter()
+[is[image]]
+\end
 {{$:/core/templates/tiddlywiki5.html}}
diff --git a/core/templates/store.area.template.html.tid b/core/templates/store.area.template.html.tid
index 3563da318..bd9232be3 100644
--- a/core/templates/store.area.template.html.tid
+++ b/core/templates/store.area.template.html.tid
@@ -3,6 +3,7 @@ title: $:/core/templates/store.area.template.html
 <$reveal type="nomatch" state="$:/isEncrypted" text="yes">
 ``
 
 <$reveal type="match" state="$:/isEncrypted" text="yes">
diff --git a/core/ui/AlertTemplate.tid b/core/ui/AlertTemplate.tid
index bcfc3c3fa..84b9632f7 100644
--- a/core/ui/AlertTemplate.tid
+++ b/core/ui/AlertTemplate.tid
@@ -2,10 +2,12 @@ title: $:/core/ui/AlertTemplate
 
 

 

-<$button class="tc-btn-invisible"><$action-deletetiddler $tiddler=<>/>{{$:/core/images/delete-button}}
+<$button class="tc-btn-invisible"><$action-deletetiddler $tiddler=<>/>{{$:/core/images/cancel-button}}
 

 

-<$view field="component"/> - <$view field="modified" format="date" template="0hh:0mm:0ss DD MM YYYY"/> <$reveal type="nomatch" state="!!count" text="">({{$:/language/Count}}: <$view field="count"/>)
+<$wikify name="format" text=<>>
+<$view field="component"/> - <$view field="modified" format="date" template=<>/> <$reveal type="nomatch" state="!!count" text="">({{$:/language/Count}}: <$view field="count"/>)
+
 

 

 
diff --git a/core/wiki/config/ServerExternalFiltersDefault.tid b/core/wiki/config/ServerExternalFiltersDefault.tid
new file mode 100644
index 000000000..7ef93ecf3
--- /dev/null
+++ b/core/wiki/config/ServerExternalFiltersDefault.tid
@@ -0,0 +1,2 @@
+title: $:/config/Server/ExternalFilters/[all[tiddlers]!is[system]sort[title]]
+text: yes
diff --git a/editions/dev/tiddlers/from tw5.com/moduletypes/SyncAdaptorModules.tid b/editions/dev/tiddlers/from tw5.com/moduletypes/SyncAdaptorModules.tid
index 6fdc6af3c..6f88fb9a5 100644
--- a/editions/dev/tiddlers/from tw5.com/moduletypes/SyncAdaptorModules.tid	
+++ b/editions/dev/tiddlers/from tw5.com/moduletypes/SyncAdaptorModules.tid	
@@ -1,5 +1,5 @@
 created: 20130825162100000
-modified: 20140814094907624
+modified: 20200113094126878
 tags: dev moduletypes
 title: SyncAdaptorModules
 type: text/vnd.tiddlywiki
@@ -14,6 +14,8 @@ SyncAdaptorModules encapsulate storage mechanisms that can be used by the SyncMe
 
 SyncAdaptorModules are represented as JavaScript tiddlers with the field `module-type` set to `syncadaptor`.
 
+See [[this pull request|https://github.com/Jermolene/TiddlyWiki5/pull/4373]] for background on the evolution of this API.
+
 ! Exports
 
 The following properties should be exposed via the `exports` object:
@@ -47,12 +49,21 @@ Gets the supplemental information that the adaptor needs to keep track of for a
 
 Returns an object storing any additional information required by the adaptor.
 
+!! `getTiddlerRevision(title)`
+
+Gets the revision ID associated with the specified tiddler title.
+
+|!Parameter |!Description |
+|title |Tiddler title |
+
+Returns a revision ID.
+
 !! `getStatus(callback)`
 
 Retrieves status information from the server. This method is optional.
 
 |!Parameter |!Description |
-|callback |Callback function invoked with parameters `err,isLoggedIn,username` |
+|callback |Callback function invoked with parameters `err,isLoggedIn,username,isReadOnly` |
 
 !! `login(username,password,callback)`
 
@@ -70,16 +81,39 @@ Attempts to logout of the server. This method is optional.
 |!Parameter |!Description |
 |callback |Callback function invoked with parameter `err` |
 
+!! `getUpdatedTiddlers(syncer,callback)`
+
+Retrieves the titles of tiddlers that need to be updated from the server.
+
+This method is optional. If an adaptor doesn't implement it then synchronisation will be unidirectional from the TiddlyWiki store to the adaptor, but not the other way.
+
+The syncer will use the `getUpdatedTiddlers()` method in preference to the `getSkinnyTiddlers()` method.
+
+|!Parameter |!Description |
+|syncer |Reference to the syncer object making the call |
+|callback |Callback function invoked with parameter `err,data` -- see below |
+
+The data provided by the callback is as follows:
+
+```
+{
+modifications: [],
+deletions: [],
+}
+```
+
 !! `getSkinnyTiddlers(callback)`
 
 Retrieves a list of skinny tiddlers from the server.
 
 This method is optional. If an adaptor doesn't implement it then synchronisation will be unidirectional from the TiddlyWiki store to the adaptor, but not the other way.
 
+The syncer will use the `getUpdatedTiddlers()` method in preference to the `getSkinnyTiddlers()` method.
+
 |!Parameter |!Description |
 |callback |Callback function invoked with parameter `err,tiddlers`, where `tiddlers` is an array of tiddler field objects |
 
-!! `saveTiddler(tiddler,callback,tiddlerInfo)`
+!! `saveTiddler(tiddler,callback)`
 
 Saves a tiddler to the server.
 
@@ -96,11 +130,16 @@ Loads a tiddler from the server.
 |title |Title of tiddler to be retrieved |
 |callback |Callback function invoked with parameter `err,tiddlerFields` |
 
-!! `deleteTiddler(title,callback,tiddlerInfo)`
+!! `deleteTiddler(title,callback,options)`
 
 Delete a tiddler from the server.
 
 |!Parameter |!Description |
 |title |Title of tiddler to be deleted |
 |callback |Callback function invoked with parameter `err` |
+|options |See below |
+
+The options parameter contains the following properties:
+
+|!Property |!Description |
 |tiddlerInfo |The tiddlerInfo maintained by the syncer for this tiddler |
diff --git a/editions/tw5.com/tiddlers/concepts/TiddlerFields.tid b/editions/tw5.com/tiddlers/concepts/TiddlerFields.tid
index 0a0c07c7e..ced343454 100644
--- a/editions/tw5.com/tiddlers/concepts/TiddlerFields.tid
+++ b/editions/tw5.com/tiddlers/concepts/TiddlerFields.tid
@@ -1,5 +1,5 @@
 created: 20130825213300000
-modified: 20191013093910961
+modified: 20191206152031468
 tags: Concepts
 title: TiddlerFields
 type: text/vnd.tiddlywiki
@@ -42,11 +42,13 @@ Other fields used by the core are:
 |`subtitle` |<> |
 |`throttle.refresh` |<> |
 |`toc-link`|<>|
+|`_canonical_uri`|<>|
 
 The TiddlyWebAdaptor uses a few more fields:
 
 |!Field Name |!Description |
 |`bag` |<> |
 |`revision` |<> |
+|`_is_skinny` |<> |
 
 Details of the fields used in this ~TiddlyWiki are shown in the [[control panel|$:/ControlPanel]] {{$:/core/ui/Buttons/control-panel}} under the <<.controlpanel-tab Info>> tab >> <<.info-tab Advanced>> sub-tab >> Tiddler Fields
diff --git a/editions/tw5.com/tiddlers/webserver/WebServer API_ Get All Tiddlers.tid b/editions/tw5.com/tiddlers/webserver/WebServer API_ Get All Tiddlers.tid
index 98119804c..52ca8964f 100644
--- a/editions/tw5.com/tiddlers/webserver/WebServer API_ Get All Tiddlers.tid	
+++ b/editions/tw5.com/tiddlers/webserver/WebServer API_ Get All Tiddlers.tid	
@@ -1,5 +1,5 @@
 created: 20181002131215403
-modified: 20190903094711346
+modified: 2020031109590546
 tags: [[WebServer API]]
 title: WebServer API: Get All Tiddlers
 type: text/vnd.tiddlywiki
@@ -12,11 +12,23 @@ GET /recipes/default/tiddlers.json
 
 Parameters:
 
-* none
+* ''filter'' - filter identifying tiddlers to be returned (optional, defaults to "[all[tiddlers]!is[system]sort[title]]")
+* ''exclude'' - comma delimited list of fields to excluded from the returned tiddlers (optional, defaults to "text")
+
+In order to avoid denial of service attacks with malformed filters in the default configuration the only filter that is accepted is the default filter "[all[tiddlers]!is[system]sort[title]]"; attempts to use any other filter will result in an HTTP 403 error.
+
+To enable a particular filter, create a tiddler with the title "$:/config/Server/ExternalFilters/" concatenated with the filter text, and the text field set to "yes". For example, the TiddlyWeb plugin includes the following shadow tiddler to enable the filter that it requires:
+
+```
+title: $:/config/Server/ExternalFilters/[all[tiddlers]] -[[$:/isEncrypted]] -[prefix[$:/temp/]] -[prefix[$:/status/]]
+text: yes
+```
+
+It is also possible to configure the server to accept any filter by creating a tiddler titled $:/config/Server/AllowAllExternalFilters with the text "yes". This should not be done for public facing servers.
 
 Response:
 
-
 * 200 OK
 *> `Content-Type: application/json`
 *> Body: array of all non-system tiddlers in [[TiddlyWeb JSON tiddler format]] 
+* 403 Forbidden
diff --git a/editions/tw5.com/tiddlers/webserver/WebServer Parameter_ port.tid b/editions/tw5.com/tiddlers/webserver/WebServer Parameter_ port.tid
index f05da08fc..bbfee9906 100644
--- a/editions/tw5.com/tiddlers/webserver/WebServer Parameter_ port.tid	
+++ b/editions/tw5.com/tiddlers/webserver/WebServer Parameter_ port.tid	
@@ -1,6 +1,6 @@
 caption: port
 created: 20180630180552254
-modified: 20180702155017130
+modified: 20191219123751824
 tags: [[WebServer Parameters]]
 title: WebServer Parameter: port
 type: text/vnd.tiddlywiki
@@ -10,6 +10,7 @@ The [[web server configuration parameter|WebServer Parameters]] ''port'' specifi
 The ''port'' parameter accepts two types of value:
 
 * Numerical values are interpreted as a decimal port number
+** The special value 0 (zero) causes the operating system to assign an available port
 * Non-numeric values are interpreted as an environment variable from which the port should be read
 
 This example configures the server to listen on port 8090:
diff --git a/plugins/tiddlywiki/filesystem/filesystemadaptor.js b/plugins/tiddlywiki/filesystem/filesystemadaptor.js
index a346a6606..9e0734814 100644
--- a/plugins/tiddlywiki/filesystem/filesystemadaptor.js
+++ b/plugins/tiddlywiki/filesystem/filesystemadaptor.js
@@ -26,6 +26,8 @@ function FileSystemAdaptor(options) {
 
 FileSystemAdaptor.prototype.name = "filesystem";
 
+FileSystemAdaptor.prototype.supportsLazyLoading = false;
+
 FileSystemAdaptor.prototype.isReady = function() {
 	// The file system adaptor is always ready
 	return true;
diff --git a/plugins/tiddlywiki/tiddlyweb/ServerControlPanel.tid b/plugins/tiddlywiki/tiddlyweb/ServerControlPanel.tid
deleted file mode 100644
index 9ef1e27af..000000000
--- a/plugins/tiddlywiki/tiddlyweb/ServerControlPanel.tid
+++ /dev/null
@@ -1,20 +0,0 @@
-title: $:/plugins/tiddlywiki/tiddlyweb/ServerControlPanel
-caption: Server
-tags: $:/tags/ControlPanel
-
-<$reveal state="$:/status/IsLoggedIn" type="nomatch" text="yes">
-Log in to ~TiddlyWeb: <$button message="tm-login">Login
-
-<$reveal state="$:/status/IsLoggedIn" type="match" text="yes">
-Logged in as {{$:/status/UserName}} <$button message="tm-logout">Logout
-
-
-----
-
-Host configuration: <$edit-text tiddler="$:/config/tiddlyweb/host" tag="input" default=""/>
-
-
//for example, `$protocol$//$host$/folder`, where `$protocol$` is replaced by the protocol (typically `http` or `https`), and `$host$` by the host name//

-
-----
-
-<$button message="tm-server-refresh">Refresh to fetch changes from the server immediately
diff --git a/plugins/tiddlywiki/tiddlyweb/config-tiddlers-filter.tid b/plugins/tiddlywiki/tiddlyweb/config-tiddlers-filter.tid
new file mode 100644
index 000000000..04bf24613
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/config-tiddlers-filter.tid
@@ -0,0 +1,2 @@
+title: $:/config/Server/ExternalFilters/[all[tiddlers]] -[[$:/isEncrypted]] -[prefix[$:/temp/]] -[prefix[$:/status/]]
+text: yes
diff --git a/plugins/tiddlywiki/tiddlyweb/configOfficialPluginLibrary.tid b/plugins/tiddlywiki/tiddlyweb/configOfficialPluginLibrary.tid
new file mode 100644
index 000000000..9f0e164f4
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/configOfficialPluginLibrary.tid
@@ -0,0 +1,3 @@
+title: $:/config/OfficialPluginLibrary
+
+(This core tiddler is overridden by the tiddlyweb plugin to prevent users from installing official plugins via control panel. Instead they should be installed by editing tiddlywiki.info in the root of the wiki folder)
\ No newline at end of file
diff --git a/plugins/tiddlywiki/tiddlyweb/css-tiddler.tid b/plugins/tiddlywiki/tiddlyweb/css-tiddler.tid
new file mode 100644
index 000000000..2d7367fa9
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/css-tiddler.tid
@@ -0,0 +1,7 @@
+title: $:/core/templates/css-tiddler
+
+`` data-tiddler-revision="`<>`" data-tiddler-bag="default" type="text/css">`<$view field="text" format="text" />``
\ No newline at end of file
diff --git a/plugins/tiddlywiki/tiddlyweb/html-div-skinny-tiddler.tid b/plugins/tiddlywiki/tiddlyweb/html-div-skinny-tiddler.tid
new file mode 100644
index 000000000..010a603b5
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/html-div-skinny-tiddler.tid
@@ -0,0 +1,9 @@
+title: $:/core/templates/html-div-skinny-tiddler
+
+`` revision="`<>`" bag="default" _is_skinny="">
+

+
`
diff --git a/plugins/tiddlywiki/tiddlyweb/icon-cloud.tid b/plugins/tiddlywiki/tiddlyweb/icon-cloud.tid
new file mode 100644
index 000000000..08c5127af
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/icon-cloud.tid
@@ -0,0 +1,4 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/icon/cloud
+tags: $:/tags/Image
+
+
\ No newline at end of file
diff --git a/plugins/tiddlywiki/tiddlyweb/javascript-tiddler.tid b/plugins/tiddlywiki/tiddlyweb/javascript-tiddler.tid
new file mode 100644
index 000000000..847873811
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/javascript-tiddler.tid
@@ -0,0 +1,7 @@
+title: $:/core/templates/javascript-tiddler
+
+`` data-tiddler-revision="`<>`" data-tiddler-bag="default" type="text/javascript">`<$view field="text" format="text" />``
\ No newline at end of file
diff --git a/plugins/tiddlywiki/tiddlyweb/save-offline.tid b/plugins/tiddlywiki/tiddlyweb/save-offline.tid
index b2bfdbdd1..76f07fe29 100644
--- a/plugins/tiddlywiki/tiddlyweb/save-offline.tid
+++ b/plugins/tiddlywiki/tiddlyweb/save-offline.tid
@@ -2,6 +2,6 @@ title: $:/plugins/tiddlywiki/tiddlyweb/save/offline
 
 \import [[$:/core/ui/PageMacros]] [all[shadows+tiddlers]tag[$:/tags/Macro]!has[draft.of]]
 \define saveTiddlerFilter()
-[is[tiddler]] -[[$:/boot/boot.css]] -[[$:/HistoryList]] -[type[application/javascript]library[yes]] -[[$:/boot/boot.js]] -[[$:/boot/bootprefix.js]] -[[$:/plugins/tiddlywiki/filesystem]] -[[$:/plugins/tiddlywiki/tiddlyweb]] +[sort[title]] $(publishFilter)$
+[is[tiddler]] -[[$:/boot/boot.css]] -[[$:/HistoryList]] -[type[application/javascript]library[yes]] -[[$:/boot/boot.js]] -[[$:/boot/bootprefix.js]] -[[$:/plugins/tiddlywiki/filesystem]] -[[$:/plugins/tiddlywiki/tiddlyweb]] -[prefix[$:/temp/]] +[sort[title]] $(publishFilter)$
 \end
 {{$:/core/templates/tiddlywiki5.html}}
diff --git a/plugins/tiddlywiki/tiddlyweb/save-wiki-button.tid b/plugins/tiddlywiki/tiddlyweb/save-wiki-button.tid
new file mode 100644
index 000000000..b0f688349
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/save-wiki-button.tid
@@ -0,0 +1,25 @@
+title: $:/core/ui/Buttons/save-wiki
+tags: $:/tags/PageControls
+caption: {{$:/plugins/tiddlywiki/tiddlyweb/icon/cloud}} Server status
+description: Status of synchronisation with server
+
+\define config-title()
+$:/config/PageControlButtons/Visibility/$(listItem)$
+\end
+<$button popup=<> tooltip="Status of synchronisation with server" aria-label="Server status" class=<> selectedClass="tc-selected">
+
+<$list filter="[match[yes]]">
+{{$:/plugins/tiddlywiki/tiddlyweb/icon/cloud}}
+
+<$list filter="[match[yes]]">
+<$text text="Server status"/>
+
+
+
+<$reveal state=<> type="popup" position="below" animate="yes">
+

+<$list filter="[all[shadows+tiddlers]tag[$:/tags/SyncerDropdown]!has[draft.of]]" variable="listItem">
+<$transclude tiddler=<>/>
+
+

+
diff --git a/plugins/tiddlywiki/tiddlyweb/styles.tid b/plugins/tiddlywiki/tiddlyweb/styles.tid
new file mode 100644
index 000000000..5d59fa379
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/styles.tid
@@ -0,0 +1,40 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/styles
+tags: [[$:/tags/Stylesheet]]
+
+\rules only filteredtranscludeinline transcludeinline macrodef macrocallinline macrocallblock
+
+body.tc-dirty span.tc-dirty-indicator svg {
+	transition: fill 250ms ease-in-out;
+}
+
+body .tc-image-cloud-idle {
+	fill: <>;
+	transition: opacity 250ms ease-in-out;
+	opacity: 1;	
+}
+
+body.tc-dirty .tc-image-cloud-idle {
+	opacity: 0;
+}
+
+body .tc-image-cloud-progress {
+	transition: opacity 250ms ease-in-out;
+	transform-origin: 50% 50%;
+	transform: rotate(359deg);
+	animation: animation-rotate-slow 2s infinite linear;
+	fill: <>;
+	opacity: 0;
+}
+
+body.tc-dirty .tc-image-cloud-progress {
+	opacity: 1;
+}
+
+@keyframes animation-rotate-slow {
+	from {
+		transform: rotate(0deg);
+	}
+	to {
+		transform: scale(359deg);
+	}
+}
diff --git a/plugins/tiddlywiki/tiddlyweb/syncer-actions-copy-logs.tid b/plugins/tiddlywiki/tiddlyweb/syncer-actions-copy-logs.tid
new file mode 100644
index 000000000..b141670e6
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/syncer-actions-copy-logs.tid
@@ -0,0 +1,6 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/copy-logs
+tags: $:/tags/SyncerDropdown
+
+<$button message="tm-copy-syncer-logs-to-clipboard" class="tc-btn-invisible">
+{{$:/core/images/copy-clipboard}} Copy syncer logs to clipboard
+
diff --git a/plugins/tiddlywiki/tiddlyweb/syncer-actions-login-status.tid b/plugins/tiddlywiki/tiddlyweb/syncer-actions-login-status.tid
new file mode 100644
index 000000000..11816f1b4
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/syncer-actions-login-status.tid
@@ -0,0 +1,9 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/login-status
+tags: $:/tags/SyncerDropdown
+
+<$reveal state="$:/status/IsLoggedIn" type="match" text="yes">
+

+You are logged in<$reveal state="$:/status/UserName" type="nomatch" text="" default=""> as <$text text={{$:/status/UserName}}/><$reveal state="$:/status/IsReadOnly" type="match" text="yes" default="no"> (read-only)
+

+

+
diff --git a/plugins/tiddlywiki/tiddlyweb/syncer-actions-login.tid b/plugins/tiddlywiki/tiddlyweb/syncer-actions-login.tid
new file mode 100644
index 000000000..cdd95f5a6
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/syncer-actions-login.tid
@@ -0,0 +1,8 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/login
+tags: $:/tags/SyncerDropdown
+
+<$reveal state="$:/status/IsLoggedIn" type="nomatch" text="yes">
+<$button message="tm-login" class="tc-btn-invisible">
+{{$:/core/images/unlocked-padlock}} Login
+
+
diff --git a/plugins/tiddlywiki/tiddlyweb/syncer-actions-logout.tid b/plugins/tiddlywiki/tiddlyweb/syncer-actions-logout.tid
new file mode 100644
index 000000000..358944d1a
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/syncer-actions-logout.tid
@@ -0,0 +1,8 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/logout
+tags: $:/tags/SyncerDropdown
+
+<$reveal state="$:/status/IsLoggedIn" type="match" text="yes">
+<$button message="tm-logout" class="tc-btn-invisible">
+{{$:/core/images/cancel-button}} Logout
+
+
diff --git a/plugins/tiddlywiki/tiddlyweb/syncer-actions-refresh.tid b/plugins/tiddlywiki/tiddlyweb/syncer-actions-refresh.tid
new file mode 100644
index 000000000..eeb0ddba2
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/syncer-actions-refresh.tid
@@ -0,0 +1,9 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/refresh
+tags: $:/tags/SyncerDropdown
+
+<$reveal state="$:/status/IsLoggedIn" type="match" text="yes">
+<$button tooltip="Get latest changes from the server" aria-label="Refresh from server" class="tc-btn-invisible">
+<$action-sendmessage $message="tm-server-refresh"/>
+{{$:/core/images/refresh-button}} <$text text="Get latest changes from the server"/>
+
+
diff --git a/plugins/tiddlywiki/tiddlyweb/syncer-actions-save-snapshot.tid b/plugins/tiddlywiki/tiddlyweb/syncer-actions-save-snapshot.tid
new file mode 100644
index 000000000..23bb4c914
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/syncer-actions-save-snapshot.tid
@@ -0,0 +1,9 @@
+title: $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/save-snapshot
+tags: $:/tags/SyncerDropdown
+
+<$button class="tc-btn-invisible">
+<$wikify name="site-title" text={{$:/config/SaveWikiButton/Filename}}>
+<$action-sendmessage $message="tm-download-file" $param={{$:/config/SaveWikiButton/Template}} filename=<>/>
+
+{{$:/core/images/download-button}} Save snapshot for offline use
+
diff --git a/plugins/tiddlywiki/tiddlyweb/tags-syncerdropdown.tid b/plugins/tiddlywiki/tiddlyweb/tags-syncerdropdown.tid
new file mode 100644
index 000000000..07135a75b
--- /dev/null
+++ b/plugins/tiddlywiki/tiddlyweb/tags-syncerdropdown.tid
@@ -0,0 +1,2 @@
+title: $:/tags/SyncerDropdown
+list: $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/login-status $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/login $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/refresh $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/logout $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/save-snapshot $:/plugins/tiddlywiki/tiddlyweb/syncer-actions/copy-logs
diff --git a/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js b/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js
index 492909324..5bc4e502d 100644
--- a/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js
+++ b/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js
@@ -27,6 +27,12 @@ function TiddlyWebAdaptor(options) {
 
 TiddlyWebAdaptor.prototype.name = "tiddlyweb";
 
+TiddlyWebAdaptor.prototype.supportsLazyLoading = true;
+
+TiddlyWebAdaptor.prototype.setLoggerSaveBuffer = function(loggerForSaving) {
+	this.logger.setSaveBuffer(loggerForSaving);
+};
+
 TiddlyWebAdaptor.prototype.isReady = function() {
 	return this.hasStatus;
 };
@@ -50,6 +56,11 @@ TiddlyWebAdaptor.prototype.getTiddlerInfo = function(tiddler) {
 	};
 };
 
+TiddlyWebAdaptor.prototype.getTiddlerRevision = function(title) {
+	var tiddler = this.wiki.getTiddler(title);
+	return tiddler.fields.revision;
+};
+
 /*
 Get the current status of the TiddlyWeb connection
 */
@@ -147,6 +158,9 @@ TiddlyWebAdaptor.prototype.getSkinnyTiddlers = function(callback) {
 	var self = this;
 	$tw.utils.httpRequest({
 		url: this.host + "recipes/" + this.recipe + "/tiddlers.json",
+		data: {
+			filter: "[all[tiddlers]] -[[$:/isEncrypted]] -[prefix[$:/temp/]] -[prefix[$:/status/]]"
+		},
 		callback: function(err,data) {
 			// Check for errors
 			if(err) {
@@ -220,7 +234,7 @@ TiddlyWebAdaptor.prototype.deleteTiddler = function(title,callback,options) {
 		return callback(null);
 	}
 	// If we don't have a bag it means that the tiddler hasn't been seen by the server, so we don't need to delete it
-	var bag = options.tiddlerInfo.adaptorInfo.bag;
+	var bag = options.tiddlerInfo.adaptorInfo && options.tiddlerInfo.adaptorInfo.bag;
 	if(!bag) {
 		return callback(null);
 	}
diff --git a/themes/tiddlywiki/vanilla/base.tid b/themes/tiddlywiki/vanilla/base.tid
index 91e2e6b61..2541938c1 100644
--- a/themes/tiddlywiki/vanilla/base.tid
+++ b/themes/tiddlywiki/vanilla/base.tid
@@ -1450,6 +1450,10 @@ html body.tc-body.tc-single-tiddler-window {
 	fill: <>;
 }
 
+.tc-drop-down .tc-drop-down-info {
+	padding-left: 14px;
+}
+
 .tc-drop-down p {
 	padding: 0 14px 0 14px;
 }
@@ -1978,24 +1982,26 @@ html body.tc-body.tc-single-tiddler-window {
 
 .tc-alerts {
 	position: fixed;
-	top: 0;
+	top: 28px;
 	left: 0;
-	max-width: 500px;
+	right: 0;
+	max-width: 50%;
 	z-index: 20000;
 }
 
 .tc-alert {
 	position: relative;
-	margin: 28px;
-	padding: 14px 14px 14px 14px;
-	border: 2px solid <>;
+	margin: 14px;
+	padding: 7px;
+	border: 1px solid <>;
 	background-color: <>;
 }
 
 .tc-alert-toolbar {
 	position: absolute;
-	top: 14px;
-	right: 14px;
+	top: 7px;
+	right: 7px;
+    line-height: 0;
 }
 
 .tc-alert-toolbar svg {
@@ -2005,6 +2011,12 @@ html body.tc-body.tc-single-tiddler-window {
 .tc-alert-subtitle {
 	color: <>;
 	font-weight: bold;
+    font-size: 0.8em;
+    margin-bottom: 0.5em;
+}
+
+.tc-alert-body > p {
+	margin: 0;
 }
 
 .tc-alert-highlight {