* Module-ize server routes and add static file support (#2510)
* Refactor server routes to modules
New module type: serverroute
Caveats: Loading order is not deterministic but this would only matter
if two route modules attempted to use the same path regexp (that would
be silly).
* Add static assets plugin
This plugin allows the node server to fetch static assets in the /assets
directory. I felt that this was a feature that goes above the core
functionality. That is why I added it as a plugin. with the modular
route extensions this was a breeze.
* Add serverroute description to ModuleTypes
* Coding standards tweaks
* Fix filename typo
* Move support for attachments from a plugin into the core
* Missing "else"
* Refactor server handling
* Introduce a new named parameter scheme for commands
* Move the SimpleServer class into it's own module
* Deprecate the --server command because of the unwieldy syntax
* Add a new --listen command using the new syntax
For example:
tiddlywiki mywiki --listen host:0.0.0.0 port:8090
* Add check for unknown parameters
* Add support for multiple basic authentication credentials in a CSV file
Beware: Passwords are stored in plain text. If that's a problem, use an authenticating proxy and the trusted header authentication approach.
* Refactor module locations
* Rename "serverroute" module type to "route"
* Remove support for verifying optional named command parameters
The idea was to be able to flag unknown parameter names, but requiring a command to pre-specify all the parameter names makes it harder for (say) the listen command to be extensible so that plugins can add new optional parameters that they handle. (This is particularly in the context of work in progress to encapsulate authenticators into their own modules).
* Refactor the two authenticators into separate modules and add support for authorization
* Correct mistaken path.join vs. path.resolve
See https://stackoverflow.com/a/39836259
* Docs for the named command parameters
I'd be grateful if anyone with sufficient Windows experience could confirm that the note about double quotes in "NamedCommandParameters" is correct.
* Be consistent about lower case parameter names
* Do the right thing when we have a username but no password
With a username parameter but no password parameter we'll attribute edits to that username, but not require authentication.
* Remove obsolete code
* Add support for requiring authentication without restricting the username
* Refactor authorization checks
* Return read_only status in /status response
* Fix two code typos
* Add basic support for detecting readonly status and avoiding write errors
We now have syncadaptors returning readonly status and avoid attempting to write to the server if it's going to fail
* Add readonly-styles
We hide editing-related buttons in read only mode
I've made this part of the tiddlyweb plugin but I think a case could be made for putting it into the core.
* Add custom request header as CSRF mitigation
By default we require the header X-Requested-With to be set to TiddlyWiki. Can be overriden by setting csrfdisable to "yes"
See https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF)_Prevention_Cheat_Sheet#Protecting_REST_Services:_Use_of_Custom_Request_Headers
* Add support for HTTPS
* First pass at a route for serving rendered tiddlers
cc @Drakor
* Tweaks to the single tiddler static view
Adding a simple sidebar
* Switch to "dash" separated parameter names
* Typo
* Docs: Update ServerCommand and ListenCommand
* First pass at docs for the new web server stuff
Writing the docs is turning out to be quite an undertaking, much harder than writing the code!
* Get rid of extraneous paragraphs in static renderings
* Rejig anonymous user handling
Now we can support wikis that are read-only for anonymous access, but allow a user to login for read/write access.
* More docs
Slowly getting there...
* Static tiddler rendering: Fix HTML content in page title
* Docs updates
* Fix server command parameter names
Missed off 30ce7ea
* Docs: Missing quotes
* Avoid inadvertent dependency on Node.js > v9.6.0
The listenOptions parameter of the plain HTTP version of CreateServer was only introduced in v9.6.0
cc @Drakor @pmario
* Typo
* fixed the "0 is not a number bug" in listops and x-listops
* Fixed one comment
* "default" is not a good name for a variable
* Following code styles.
Moving getInt to utils.
* Removing unwanted spaces introduced by me
* Add ability to now macro to return same UTC string used in the modified and created fields.
* Revert "Add ability to now macro to return same UTC string used in the modified and created fields."
This reverts commit 7b6ad7db4a.
* Mods to formatDateString to adjust displayed date to UTC for [UTC]
format. Corresponding doc changes, and optimization for special
case.
These changes allow drag and drop to work with one issue: <a> links are
not draggable; draggable divs, spans, buttons etc. seem to work fine.
There’s some issue with IE11 that I don’t understand.
For testing, you can force links to become spans by changing line 64 of
$:/core/modules/widgets/link.js to:
var domNode = this.document.createElement("span");
The problem was that `this.responseText` crashes for non-text data. We
fix it by letting the client specify which property should be returned.
@ericshulman does this work for you?
We now use highlight.js in raw HTML mode on the server, rather than
trying to use it with the fakedom. This causes problems with fakedoms
inability to get textContent for a node that has been created by
assigning innerHTML. So we extend the fakedom to allow the original
text content to be saved.
See #2778 for discussion.
The code here had got a bit broken by some PRs that I should have
checked more carefully. I’ve done a major refactoring which will
hopefully make it easier to understand, and fixes a number of problems:
* Problem with eg .md tiddlers not being deleted correctly
* Problem with Windows path separators not being usable within
$:/config/FileSystemPaths on Windows
* Problem with filename clashes not being detected correctly when
saving to a different directory via $:/config/FileSystemPaths
* Enables slashes within tiddler titles to be mapped into folders
* Enables plain text files like .md and .css to be saved with .meta
files instead of as .tid files (see #2558)
* No longer replaces spaces with underscores
As this is such a major update, I’d be grateful if Node.js users could
give it a careful run through — in particular, you’ll need to try
creating new tiddlers of various types and ensure that the expected
files are created.
We were using `String.prototype.replace()` without addressing the
wrinkle that dollar signs in the replacement string have special
handling. This caused problems in situations where the replacement
string is derived from user input and contains dollar signs.
Fixes#2517
* Save binary tiddlers with meta file
The filesystemadaptor plugin was a little simplistic in its
understanding of a binary file. It was using the typeInfo dictionary to
choose what tiddler types were binary (and hence needed a meta file when
saving).
I looked as if it was trying to be smart by looking for the hasMetaFile
*OR* had the encoding of base64. Unfortunately the typeInfo only defined
image/jpeg and so any other base64 encoded tiddler was assumed to be of
type text/vnd.tiddlywiki.
The net effect was only JPG images got a meta file and everything else
were saved as .tid files with base64 encoding. It all still worked but
made working with binary data in a Git repo a bit daunting.
There is enough information in the $tw.config.contentTypeInfo to
determine if a tiddler type is encoded with base64 or not. A better list
is available from boot/boot.js who registers all the types thorough the
registerFileType and marks then with base64 were appropriate.
This commit uses the typeInfo dictionary first for any filesystem
specific overrides, then the contentTypeInfo, and finally defaults to
the typeInfo["text/vnd.tiddlywiki"]. It also eliminates the now
unnecessary override for image/jpeg.
I think this might have been the original intent from commit 10b192e7.
From my limited testing all files described in boot/boot.js (lines
1832-1856) with an encoding of base64 now save as the original binary
and a meta file. Meaning that when you start the node server and then
drag-n-drop a binary file (i.e. image/png) it will PUT to the server
and then save it on the filesystem as-is allowing the file to be managed
as a binary file and not a text file. (Binary diffs are better and
GitHub supports them as well).
* Prevent duplicate file extensions
A side effects of using the $tw.config.contentFileInfo in the previous
commit is that it will always append a file extension to the tiddler
title when saving. In most cases this is the correct course of action.
However, sometimes that title is already a proper filename with an
extension (for example importing 'foobar.png' would save a file named
'foobar.png.png') which seemed silly.
This commit simply checks to make sure the title does not already end
with the file extension before appending it to the filename. A little
convenience really.
Since IE apparently doesn't have the String endsWith method I took the
liberty to add a helper method to $tw.utils trying to follow the other
polyfill patterns. I figured this was more generic and readable then
attempting to use a one-off solution inline. I got the polyfill code
from MDN.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/endsWith#Polyfill
Is strEndsWith the best method name?
When saving new tiddlers on node.js, allow the user to override the path of the
generated .tid file. This is done by creating a tiddler
$:/config/FileSystemPaths which contains one or more filter expressions, one
per line. These filters are applied in turn to the tiddler to be saved, and
the first output produced is taken as a logical path relative to the wiki's
tiddlers directory. Any occurences of "/" in the logical path are replaced with
the platform's path separator, the extension ".tid" is appended, illegal
characters are replaced by "_" and the path is disambiguated (if necessary) in
order to arrive at the final tiddler file path. If none of the filters matches,
or the configuration tiddler does not exist, fall back to the previous file
naming scheme (i.e. replacing "/" by "_").
This implies we will now, for tiddlers matching the user-specified filters,
create directory trees below the tiddlers directory. In order to avoid
cluttering it with empty directory trees when renaming or removing tiddlers, any
directories that become empty by deleting a tiddler file are removed
(recursively).
Benefits of this configuration option include the ability to organize git
repositories of TiddlyWikis running on node.js, ability to replace characters
that cause trouble with particular operating systems or workflows (e.g. '$' on
unix) and the ability to replicate tiddler "paths" in the filesystem (by
including a filter like "[!has[draft.of]]") without forcing such a (potentially
problematic) change on all users.
Entities such as `👷` were broken because
`String.fromCharCode()` is not fully Unicode aware. The fix is to use
`String.fromCodePoint()` where available.
Noted by @ericshulman
Now we process the rendered HTML of tiddlers, which allows us to
process HTML generated by MS Word. In fact, the HTML that MS Word
generates is so awful, I’ve instead been using Mammoth to do the
conversion: https://github.com/mwilliamson/mammoth.js
Also some necessary improvements to the fake dom implementation.
They don’t get automatically decoded when the browser reads the
resulting HTML. So, instead, we’ll solve
1e9e1a1fdc by switching to double quotes
for attribute values.
This may cause backwards compatibility problems for people relying on
the block mode parsing, but it’s much better for the rest of us as it
avoids an unsightly paragraph tag
Stop the logger from creating alert tiddlers on the server. They
propagate to the client but are not deletable from the client because
they are in the `$:/temp` namespace.
cc @loleg
The previous fix changed the scrolling behaviour such that it only
scrolled to the top of a tiddler if the tiddler was entirely offscreen.
It wasn’t entirely satisfactory because scrolling was prevented even if
only a few pixels of a tiddler are in view. This commit ensures that
the scroll does occur if less than 50 pixels of the target is in view
@aelocson here’s an alternative fix for #981, as discussed in
https://github.com/Jermolene/TiddlyWiki5/commit/691e5719a4ff74a04d389bd1
26ba2a69e7651a2a#commitcomment-9848682
It does seem a lot nicer. It avoids the problem you raised, and also
avoids scrolling when a permalink is used.
I suspect that we’d need to make the hard-coded 50 pixels be
configurable.
Cecily and Zoomin story views use a document.body that is smaller than
the document.documentElement. We were just clearing the popups on
clicks on the document.body Clicks on the document element (ie, on the
background of the page) were not being trapped, meaning that you
couldn’t dismiss a popup.
@felixhayashi I’m afraid I haven’t used your pull request as there were
a few details that I wanted to do differently. The main change is that
these changes allow both a param string and hashmap to be specified.
With this commit tag pills will now choose for the foreground colour
either the current palette “foreground” or “background” colours,
according to which has the higher contrast. It’s something @gernert has
expressed an interest in in the past, and I’ve tended to agree that it
is a nice piece of polish. It opens up the possibility of paler colours
for tag pills than are currently possible.
The trouble is that in order to implement it I’ve had to bring in a
third party library for parsing CSS colours. It weighs in just over
9KB, making quite a lot of weight for such a small feature. I don’t see
any other immediate uses for the colour parsing library either.
So, I’m undecided at the moment whether this should stay in the core.
There are still some warnings about making functions in a loop, but
I’ll fix those as a separate pull request because the fixes are more
than typographic errors.
Re-introduces the “tw-auto-save-wiki” message. The previous approach of
automatically triggering autosave whenever a tiddler changed meant that
changing configuration changes in control panel was triggering an
autosave. Using the explicit message gives us better control of the
situations in which we’ll autosave.
Now we solve the earlier problem of there being outstanding tiddler
change events at the time that we process the “tw-auto-save-wiki” by
deferring the autosave until the outstanding change event comes in.
Previously we were using a message `tw-auto-save-wiki` to trigger an
autosave. The message was generated by certain UI actions such as
saving a tiddler. The trouble was that the message was being processed
before the wiki change event for the accompanying change had had a
chance to percolate. The end result was that the dirty indicator was
staying lit when using autosave.
The new approach abandons the autosave message and instead triggers the
autosave in the wiki change event when a relevant change occurs.
One happy side effect of these changes is that the dirty indicator now
works as expected with the client server edition - ie, when typing in a
draft tiddler the dirty indicator will flash briefly, and then clear
when the sync mechanism has completed saving the draft.
Importing an encrypted wiki ordinarily doesn’t place the password in
the password vault on the basis that one ought to be able to import
from a file without automatically inheriting its password.
Now there’s a configuration option that can be used by the upgrade
plugin to cause the password vault to be updated with any password
entered by the user. The end result is that the user only needs to
enter their password once.
Previously any refreshing of the content of a modal would cause a
crash. The problem is the way that we steal the root widget for the
render trees used in the modal. The root widget is tied to the
container DOM node for the main content area, which isn’t actually a
parent of the modal DOM nodes, hence the confusion for the refresh
mechanism.
We were parsing the boot tiddlers, making them into a widget and then
refreshing the widget tree. The problem is that subsequent chances to
the boot tiddlers themselves wouldn’t be picked up as part of the
refresh.
Now we indirectly parse those UI boot tiddlers through a transclusion,
which does get refreshed in the desired way.
1. Moved some methods out of boot.js because they are not needed until
after bootup
2. Added alternate message for editing an overridden shadow tiddler
3. Minor style tweaks
This is an experimental module to help us measure the performance of
the refresh cycle and the filter mechanism. Not intended to replace the
performance measurement features in browser developer tools, just to
make it easier to automate performance checks cross-browser.
The immediate purpose is to help in refactoring the filter mechanism.
The recent change to encapsulate the wiki store “tiddlers” object has
hurt the performance of filters, and it’s going to be helpful to have
decent measurements while refactoring that code.
I’m still not convinced that this stuff should be in the core, and may
well end up removing it after the present refactoring cycle.
Now we get decent visual indication of sync errors, for instance. Still
work to do to coalesce alerts so that the screen doesn’t fill up with
them after an interval. And probably we should add a button to clear
all alerts.
Thr trouble is that the transitionEnd event doesn't fire under some
circumstances (eg if the animated element is hidden). So, it's more
reliable to use a timer instead
