This probably fails "responsible disclosure", but it's not an RCE and
frankly the whole bug is utterly hilarious so here we are...
It's possible to open a file on a disk drive and continue to read/write
to them after the disk has been removed:
local disk = peripheral.find("drive")
local input = fs.open(fs.combine(disk.getMountPath(), "stream"), "rb")
local output = fs.open(fs.combine(disk.getMountPath(), "stream"), "wb")
disk.ejectDisk()
-- input/output can still be interacted with.
This is pretty amusing, as now it allows us to move the disk somewhere
else and repeat - we've now got a private tunnel which two computers can
use to communicate.
Fixing this is intuitively quite simple - just close any open files
belonging to this mount. However, this is where things get messy thanks
to the wonderful joy of how CC's streams are handled.
As things stand, the filesystem effectively does the following flow::
- There is a function `open : String -> Channel' (file modes are
irrelevant here).
- Once a file is opened, we transform it into some <T extends
Closeable>. This is, for instance, a BufferedReader.
- We generate a "token" (i.e. FileSystemWrapper<T>), which we generate
a week reference to and map it to a tuple of our Channel and T. If
this token is ever garbage collected (someone forgot to call close()
on a file), then we close our T and Channel.
- This token and T are returned to the calling function, which then
constructs a Lua object.
The problem here is that if we close the underlying Channel+T before the
Lua object calls .close(), then it won't know the underlying channel is
closed, and you get some pretty ugly errors (e.g. "Stream Closed"). So
we've moved the "is open" state into the FileSystemWrapper<T>.
The whole system is incredibly complex at this point, and I'd really
like to clean it up. Ideally we could treat the HandleGeneric as the
token instead - this way we could potentially also clean up
FileSystemWrapperMount.
BBut something to play with in the future, and not when it's 10:30pm.
---
All this wall of text, and this isn't the only bug I've found with disks
today :/.
ForgeGradle (probably sensibly) yells at me about doing this. However:
- There's a reasonable number of mods doing this, which establishes
some optimistic precedent.
- The licence update in Aug 2020 now allows you to use them for
"development purposes". I guess source code counts??
- I'm fairly sure this is also compatible with the CCPL - there's an
exception for Minecraft code.
The main motivation for this is to make the Fabric port a little
easier. Hopefully folks (maybe me in the future, we'll see) will no
longer have to deal with mapping hell when merging - only mod loader
hell.
Means we can now do fs.combine("a", "b", "c"). Of course, one may just
write "a/b/c" in this case, but it's definitely useful elsewhere.
This is /technically/ a breaking change as fs.combine(a, b:gsub(...))
will no longer function (as gsub returns multiple arguments). However,
I've done a quick search through GH and my Pastebin archives and can't
find any programs which would break. Fingers crossed.
When dealing with invalid paths (for instance, ones which are too long
or malformed), Java may throw a FileSystemException. This contains the
absolute path (i.e. C:/Users/Moi/.../.minecraft/...), which is printed
to the user within CC - obviously not ideal!
We simply catch this exception within the MountWrapper and map it back
to the local path. The disadvantage of doing it here is that we can't
map the path in the exception back to the computer - we'd need to catch
it in FileMount for that - so we just assume it referrs to the original
path instead.
Doing it in FileMount ends up being a little uglier, as we already do
all the exception wrangling in FileWrapper, so this'll do for now.
Fixes#495
This moves monitor networking into its own packet, rather than serialising
using NBT. This allows us to be more flexible with how monitors are
serialised.
We now compress terminal data using gzip. This reduces the packet size
of a max-sized-monitor from ~25kb to as little as 100b.
On my test set of images (what I would consider to be the extreme end of
the "reasonable" case), we have packets from 1.4kb bytes up to 12kb,
with a mean of 6kb. Even in the worst case, this is a 2x reduction in
packet size.
While this is a fantastic win for the common case, it is not abuse-proof.
One can create a terminal with high entropy (and so uncompressible). This
will still be close to the original packet size.
In order to prevent any other abuse, we also limit the amount of monitor
data a client can possibly receive to 1MB (configurable).
- Return EPOCH if a zip entry's creation/modification/access time is
missing.
- If a BasicFileAttributes.*Time method returns null, use 0 as our
time. This shouldn't happen, but is a good sanity check.
Fixes#371
This means we are already provided with file attributes, which halfs[^1]
the time taken to scan folders.
[^1]: This dropped the fastest scan time from ~1.3s to ~0.6. It's by no
means a perfect benchmark, but this is still an obvious improvement.
- fs.getCapacity just returns the capacity of the current drive, if
available. This will be nil on rom mounts.
- fs.attributes returns an lfs like table of various file attributes.
Currently, this contains:
- access, modification, created: When this file was last accessed,
modified and created. Time is measured in milliseconds since the
epoch, same as os.epoch("utc") and what is accepted by os.date.
- size: Same as fs.getSize
- isDir: same as fs.isDir
Closes#262
- contains now performs a case-insensitive comparison. While this is a
little dubious, it's required for systems like Windows, where foo and
Foo are the same folder.
- Impose a depth limit on copyRecursive. If there are any other cases
where we may try to copy a folder into itself, this should prevent
the computer entirely crashing.
- Remove *Stream methods on IMount/IWritableMount, and make the channel
ones the primary.
- Fix location of AbstractTurtleUpgrade
- Make IComputerAccess.getAvailablePeripheral and .getMainThreadMonitor
mandatory.
- IComputerAccess throws a specialised NotAttachedException
Before it would remain the same across world reloads, and thus would be
out-of-date after leaving the first world. This architecture technically
allows for running multiple servers at once, though that's not going to
matter that soon.
So very little works, but it compiles and runs.
Things to resolve over the next few days:
- Horrible mappings (should largely be resolved by tomorrow).
- Cannot send extra data over containers - we'll have to see what Forge
does here.
- Turtle models are broken
- No block drops yet - this will largely be cherry-picking whatever I
did on Fabric.
- Weird inventory desyncs (items don't show up initially when
interacting with a CC inventory).
- Probably lots of other things.
- Adds a CheckStyle configuration which is pretty similar to CC's
existing one.
- Add the Gradle license plugin.
- Ensure the existing source code is compatible with these additional
checks.
See #239
Look, I originally had this split into several commits, but lots of
other cleanups got mixed in. I then backported some of the cleanups to
1.12, did other tidy ups there, and eventually the web of merges was
unreadable.
Yes, this is a horrible mess, but it's still nicer than it was. Anyway,
changes:
- Flatten everything. For instance, there are now three instances of
BlockComputer, two BlockTurtle, ItemPocketComputer. There's also no
more BlockPeripheral (thank heavens) - there's separate block classes
for each peripheral type.
- Remove pretty much all legacy code. As we're breaking world
compatibility anyway, we can remove all the code to load worlds from
1.4 days.
- The command system is largely rewriten to take advantage of 1.13's
new system. It's very fancy!
- WidgetTerminal now uses Minecraft's "GUI listener" system.
- BREAKING CHANGE: All the codes in keys.lua are different, due to the
move to LWJGL 3. Hopefully this won't have too much of an impact.
I don't want to map to the old key codes on the Java side, as there
always ends up being small but slight inconsistencies. IMO it's
better to make a clean break - people should be using keys rather
than hard coding the constants anyway.
- commands.list now allows fetching sub-commands. The ROM has already
been updated to allow fancy usage such as commands.time.set("noon").
- Turtles, modems and cables can be waterlogged.
Are most of these changes small and petty? Yes. However, IMO they do
make the code more readable. Anyway, a summary of some of the more
interesting changes:
- Expose Abstract*Upgrade classes in the API
- Fix the spelling of Jonathan in the API docs (*shakes fist*)
- Fix bug with printout not working in the offhand.
- Rename any argments/variables accidentally named "m_*", and add an
inspection to prevent it happening again.
- Remove most of the Block*.Properties classes - just inline them in
the parent class.
- Return super.writeToNBT instead of reassigning at the top.
OK, so let's get this out of the way, there's some actual changes mixed
in here too. I'm really sorry:
- Turtles can now not be renamed with unnamed item tags (previously it
would clear the name, this seemed a little unideal).
- commands.getBlock(s)Data will also include NBT.
Now, onto the horror story which is these inspection changes:
- Make a lot of methods static
- Typo fixes
- Make utility classes final + private constructor
- Lots of reformatting (ifs -> ternary, invert control flow, etc...)
- ???
- Profit!
I'm so going to regret this - can pretty much guarantee this is going to
break something.
When closing a BufferedWriter, we close the underlying writer. As we're
using channels, this is an instance of sun.nio.cs.StreamEncoder. This
will attempt to flush the pending character.
However, if throwing an exception within .write errors, the flush will
fail and so the underlying stream is not closed. This was causing us to
leak file descriptors.
We fix this by introducing ChannelWrappers - this holds the wrapper
object (say, a BufferedWriter) and underlying channel. When closed, we
dispose of the wrapper, and then the channel. You could think of this as
doing a nested try-with-resources, rather than a single one.
Note, this is not related to JDK-6378948 - this occurs in the underlying
stream encoder instead.
- Rename unload -> close to be a little more consistent
- Make pollAndResetChanged be atomic, so we don't need to aquire a lock
- Get the computer queue from the task owner, rather than a separate
argument.
FileSystemMount was originally added to allow using ReadableByteChannels
instead of InputStreams. However, as zip files do not allow seeking,
there is no benefit of using them over the original JarMount (which we
need to preserve for backwards compatibility).
Instead of maintaining two near-identical mounts, we remove the
FileSystemMount and rewrite the JarMount implementation with several
improvements:
- Rewrite the jar scanning algorithm to be closer to 1.13+'s data pack
mount. This means we no longer require the jar file to have
directories before the file (though this was not a problem in
practice).
- Add all JarMounts to a ReferenceQueue, closing up the ZipFile when
they have been garbage collected (fixes#100).
- Cache the contents of all files for 60 seconds (with some constraints
on size). This allows us to seek on ROM files too (assuming they are
small), by reading the whole thing into memory.
The cache is shared across all mounts, and has a 64MiB limit, and
thus should not have an adverse impact on memory.
- Move all HTTP tasks to a unified "MonitoredResource" model. This
provides a uniform way of tracking object's lifetimes and disposing
of them when complete.
- Rewrite HTTP requests to use Netty instead of standard Java. This
offers several advantages:
- We have access to more HTTP verbs (mostly PATCH).
- We can now do http -> https redirects.
- We no longer need to spawn in a new thread for each HTTP request.
While we do need to run some tasks off-thread in order to resolve
IPs, it's generally a much shorter task, and so is less likely to
inflate the thread pool.
- Introduce several limits for the http API:
- There's a limit on how many HTTP requests and websockets may exist
at the same time. If the limit is reached, additional ones will be
queued up until pending requests have finished.
- HTTP requests may upload a maximum of 4Mib and download a maximum
of 16Mib (configurable).
- .getResponseCode now returns the status text, as well as the status
code.
- Remove redundant constructors and super calls
- Standardise naming of texture fields
- Always use postfix notations for loops
- Cleanup several peripheral classes
- Only generate resource pack mounts if the desired directory exists.
- Allow mounting files, as well as directories (fixes#90).
As always, also a wee bit of cleanup to some of the surrounding code.
Some mods (*cough* Computronics *cough*) directly access this class,
rather than using the API. We add this back to ensure they still behave
as expected.
Truth be told, I can't really complain, as Plethora also does dodgy
thing with CC internals.
- Rename openStreamFor* methods to more accurate openChannelFor*
- Fix ArrayByteChannel having an incorrect .position() implementation
Cherry-picked from the PR against dan200/ComputerCraft
This replaces the existing IMount openFor* method with openChannelFor*
ones, which return an appropriate byte channel instead.
As channels are not correctly closed when GCed, we introduce a
FileSystemWrapper. We store a weak reference to this, and when it is
GCed or the file closed, we will remove it from our "open file" set and
ensure any underlying buffers are closed.
While this change may seem a little odd, it does introduce some
benefits:
- We can replace JarMount with a more general FileSystemMount. This
does assume a read-only file system, but could technically be used
for other sources.
- Add support for seekable (binary) handles. We can now look for
instances of SeekableByteChannel and dynamically add it. This works
for all binary filesystem and HTTP streams.
- Rewrite the io library to more accurately emulate PUC Lua's
implementation. We do not correctly implement some elements (most
noticably "*n", but it's a definite improvement.
