New things, documentation
This commit is contained in:
parent
d9933d52c5
commit
c80ea1d0c7
|
|
@ -0,0 +1 @@
|
||||||
|
out.wav
|
||||||
44
README.md
44
README.md
|
|
@ -1,26 +1,42 @@
|
||||||
# random-stuff
|
# random-stuff
|
||||||
|
|
||||||
In the interest of transparency and/or being vaguely useful, I'm releasing random junk from my projects folder publicly.
|
In the interest of transparency and/or being vaguely useful, I'm releasing many random and/or deterministic small things accumulated from various projects folders over the years.
|
||||||
This comes with absolutely no guarantee of support or correct function, although if you need some of this for something I will *try* and answer any queries you might have.
|
This comes with absolutely no guarantee of support or correct function, although if you need some of this for something I will *try* and answer any queries you might have.
|
||||||
|
|
||||||
## Contents (incomplete and rough list)
|
## Contents (incomplete and rough list)
|
||||||
|
|
||||||
* some interpreters for esolangs due to events on the esolangs discord
|
* some interpreters for various esolangs, written for competitions on an esolangs Discord server
|
||||||
* bad assembly hello world for some reason
|
* trivial x86-64 assembly hello world for some reason
|
||||||
* political compass visualizer thing for one Discord server
|
* political compass visualizer thing for one Discord server
|
||||||
* simple un-hexadecimal-izer program (`base64 -d` exists but somehow not `base16 -d` or something)
|
* simple un-hexadecimal-izer program (`base64 -d` exists but somehow not `base16 -d` or some equivalent)
|
||||||
* scripts for packing music + metadata onto Computronics (a Minecraft mod) tape images
|
* `generate-tape-image.py` - scripts for packing music + metadata onto Computronics (a Minecraft mod) tape images - these require LionRay to do the DFPWM conversion as of now. Can be played with something like [this](https://pastebin.com/SPyr8jrh).
|
||||||
* some thing to generate WAV files containing beeping noises
|
* a thing to generate WAV files containing beeping noises
|
||||||
* fairly transferable small bits of an abandoned JS project
|
* fairly transferable small bits of an abandoned JS project
|
||||||
* an extremely bad and/or unfinished cookie clicker thing where you press the enter key instead of clicking
|
* an extremely bad cookie clicker-style incremental thing where you press the enter key instead of clicking
|
||||||
* a very simple web API wrapper for `luamin`
|
* a very simple web API wrapper for `luamin`
|
||||||
* some bodged old version of [it-was-inevitable](https://github.com/BenLubar/it_was_inevitable) which runs a local webserver instead of sending to Mastodon
|
* a tweaked old version of [it-was-inevitable](https://github.com/BenLubar/it_was_inevitable) which runs a local webserver instead of sending to Mastodon. Used to be used by PotatOS but this was discontinued due to RAM use.
|
||||||
* an extremely cursed program which bruteforces regexes or something?
|
* an extremely accursed program which bruteforces regexes or something? Not that this actually does anything beyond using vast amounts of CPU and printing things.
|
||||||
* `realtau.txt`, which seemingly contains 100000 digits of τ. I wonder if there's a faketau somewhere.
|
* `realtau.txt`, which seemingly contains 100000 digits of τ. I wonder if there's a faketau somewhere.
|
||||||
* some weird thing which lets you use synonyms to get attributes on python objects
|
* a strange thing which lets you use synonyms to get attributes on python objects
|
||||||
* something which generates random vaguely human readable names in Elm. Please note that I do NOT endorse the use of Elm and this is provided mostly just to make the languages list at the side weirder.
|
* code for generating random vaguely human readable names in Elm. Please note that I do NOT endorse the use of Elm and this is provided mostly just to make the languages list at the side weirder rather than for any actual uses.
|
||||||
* F# kerbal name generator & very basic stack calculator
|
* F# kerbal name generator & very basic stack calculator
|
||||||
* importer part of an unfinished wikipedia database dump viewer
|
* Wikipedia dump index indexer (I think some of this is just example code for an oddly specific crate which parses the dump XML)
|
||||||
* `ptt.py` - Python-based systemwide push to talk (mutes and unmutes microphone via PulseAudio) with tray icon
|
* `ptt.py` - Python-based systemwide push to talk (mutes and unmutes microphone via PulseAudio) with tray icon
|
||||||
* `list-sort.py` - Made for a competition, it sorts a list by making a somewhat weird Lispy language and implementing quicksort(ish) in it.
|
* `code-guessing` - contains my entries, some test code, and build processes for my submissions to the Esolangs code guessing competition. There are also some things which never made it into an entry, such as my abuse of [Z3](https://github.com/Z3Prover/z3) to solve mazes (it's surprisingly effective).
|
||||||
* `mcc.py` - a chat program. Unlike most chat programs, it runs over IPv6 multicast so you can talk to anyone on your LAN who also happens to have this program somehow. Very flaky, due to trying to autoguess a network interface to use and also limited testing, as well as quite barebones.
|
* `list-sort.py`, which sorts lists of integers by interpreting a simple Lispy language and doing a continuation-passing-style quicksort (to avoid stack issues; it supports tail call optimization so this is "efficient").
|
||||||
|
* `maze2.py`, which does simple depth first search to solve a maze in a pleasantly compact format.
|
||||||
|
* `multiply_matrices.py`, which abuses many Python features and does matrix multiplication in an inefficient recursive way which *looks* like Strassen's algorithm but isn't.
|
||||||
|
* `anagram.c`, which detects whether strings are (case-insensitively, and ignoring spaces) anagrams by uppercasing them, sorting them, and removing spaces and comparing them for equality. It does this by dividing the string into 16-byte chunks which can fit into a 128-bit `xmm` register (this had to run on Sandy Bridge systems, which lack AVX2), uppercasing them using three vector instructions (via the invariant that the input won't contain anything but `[A-Za-z ]`), applying a SIMD-based bubblesort to each chunk which swaps all the necessary pairs at once until it stops changing, and then using a 32-way (sequential; no idea how to parallelize this) merge to output a sorted string and discard spaces. These can then be checked for equality.
|
||||||
|
* `mcc.py` - a chat program. Unlike most chat programs, it runs over IPv6 multicast so you can talk to anyone on your LAN who also happens to have this program somehow. Very flaky, due to trying to autoguess a network interface to use and also limited testing, as well as quite barebones.
|
||||||
|
* `tiscubed.py` - an esolang somewhat like TIS-100, but with a somewhat exotic (almost no immediate operands, no registers, only 256B of memory per node) binary machine code format instead of assembly (there is an assembler available too). It's called "cubed" due to three dimensions, but I haven't actually done this yet.
|
||||||
|
* `iterated-prisoners-dilemma` - some scripts from an iterated prisoners' dilemma competition. Unfortunately, nobody came up with any particularly exciting algorithms for this.
|
||||||
|
* `calibre-indexer` - full text search for Calibre libraries, via SQLite.
|
||||||
|
* SQLite may not have been a great choice for this, as it cannot do concurrent writes. Nevertheless, the code works, if not particularly efficiently, and allows you to build a full text table (using [FTS5](https://sqlite.org/fts5.html)) to rapidly search in your Calibre library.
|
||||||
|
* While searches are near-instant, building the index is very slow (several deciseconds per book) and it takes up large amounts of disk space (though less than the original books, funnily, because those contain images). It's smart enough to not operate again on books it already has which haven't been changed, though.
|
||||||
|
* It only works on EPUBs, because I couldn't be bothered to support other formats (calibre can convert them anyway).
|
||||||
|
* Text (and chapter titles, ish) is extracted using a simple but seemingly fairly reliable state machine and `xml-rs`.
|
||||||
|
* I have not gotten round to releasing a nice-to-use frontend for this. You can use `run-query.py` for a less nice one.
|
||||||
|
* `length_terminated_strings.c` - a revolution in computer science, combining the efficient `strlen` of null-terminated strings with the... inclusion of length? of length-prefixed/fat-pointer strings. A length-terminated string has its length at the *end*, occupying 1 to 8 bytes. To find its length, simply traverse the string until the data at the end matches the length traversed so far. Yes, this implementation might slightly have a bit of undefined behaviour.
|
||||||
|
* `discord-message-dump.py`, which reads a GDPR data dump from Discord and copies all the messages in public channels to a CSV file. I used this for training of a GPT-2 instance on my messages (available on request).
|
||||||
|
* `spudnet-http.py` - connect to the SPUDNET backend underlying [PotatOS](https://git.osmarks.net/osmarks/potatOS/)'s ~~backdoors~~ remote debugging system via the convenient new HTTP long-polling-based API.
|
||||||
|
* `fractalart-rs` - [this](https://github.com/TomSmeets/FractalArt/) in Rust and faster, see its own README for more details.
|
||||||
|
|
@ -0,0 +1,381 @@
|
||||||
|
# This file is automatically @generated by Cargo.
|
||||||
|
# It is not intended for manual editing.
|
||||||
|
[[package]]
|
||||||
|
name = "adler32"
|
||||||
|
version = "1.2.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "aae1277d39aeec15cb388266ecc24b11c80469deae6067e17a1a7aa9e5c1f234"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "ahash"
|
||||||
|
version = "0.4.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "739f4a8db6605981345c5654f3a85b056ce52f37a39d34da03f25bf2151ea16e"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "aho-corasick"
|
||||||
|
version = "0.7.15"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "7404febffaa47dac81aa44dba71523c9d069b1bdc50a77db41195149e17f68e5"
|
||||||
|
dependencies = [
|
||||||
|
"memchr",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "anyhow"
|
||||||
|
version = "1.0.39"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "81cddc5f91628367664cc7c69714ff08deee8a3efc54623011c772544d7b2767"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "bitflags"
|
||||||
|
version = "1.2.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "byteorder"
|
||||||
|
version = "1.4.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "bzip2"
|
||||||
|
version = "0.3.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "42b7c3cbf0fa9c1b82308d57191728ca0256cb821220f4e2fd410a72ade26e3b"
|
||||||
|
dependencies = [
|
||||||
|
"bzip2-sys",
|
||||||
|
"libc",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "bzip2-sys"
|
||||||
|
version = "0.1.10+1.0.8"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "17fa3d1ac1ca21c5c4e36a97f3c3eb25084576f6fc47bf0139c1123434216c6c"
|
||||||
|
dependencies = [
|
||||||
|
"cc",
|
||||||
|
"libc",
|
||||||
|
"pkg-config",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "calibre-indexer"
|
||||||
|
version = "0.1.0"
|
||||||
|
dependencies = [
|
||||||
|
"anyhow",
|
||||||
|
"epub",
|
||||||
|
"lazy_static",
|
||||||
|
"num_cpus",
|
||||||
|
"rusqlite",
|
||||||
|
"xml-rs",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "cc"
|
||||||
|
version = "1.0.67"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e3c69b077ad434294d3ce9f1f6143a2a4b89a8a2d54ef813d85003a4fd1137fd"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "cfg-if"
|
||||||
|
version = "0.1.10"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "cfg-if"
|
||||||
|
version = "1.0.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "crc32fast"
|
||||||
|
version = "1.2.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "81156fece84ab6a9f2afdb109ce3ae577e42b1228441eded99bd77f627953b1a"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if 1.0.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "epub"
|
||||||
|
version = "1.2.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1f4086fc0bc91524e0a88bc13fa622e3b9fce38d5a91454e0667db97a4f39dc3"
|
||||||
|
dependencies = [
|
||||||
|
"anyhow",
|
||||||
|
"percent-encoding",
|
||||||
|
"regex",
|
||||||
|
"xml-rs",
|
||||||
|
"zip",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "fallible-iterator"
|
||||||
|
version = "0.2.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "4443176a9f2c162692bd3d352d745ef9413eec5782a80d8fd6f8a1ac692a07f7"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "fallible-streaming-iterator"
|
||||||
|
version = "0.1.9"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "flate2"
|
||||||
|
version = "1.0.14"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "2cfff41391129e0a856d6d822600b8d71179d46879e310417eb9c762eb178b42"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if 0.1.10",
|
||||||
|
"crc32fast",
|
||||||
|
"libc",
|
||||||
|
"miniz_oxide",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "hashbrown"
|
||||||
|
version = "0.9.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d7afe4a420e3fe79967a00898cc1f4db7c8a49a9333a29f8a4bd76a253d5cd04"
|
||||||
|
dependencies = [
|
||||||
|
"ahash",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "hashlink"
|
||||||
|
version = "0.6.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d99cf782f0dc4372d26846bec3de7804ceb5df083c2d4462c0b8d2330e894fa8"
|
||||||
|
dependencies = [
|
||||||
|
"hashbrown",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "hermit-abi"
|
||||||
|
version = "0.1.18"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "322f4de77956e22ed0e5032c359a0f1273f1f7f0d79bfa3b8ffbc730d7fbcc5c"
|
||||||
|
dependencies = [
|
||||||
|
"libc",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "lazy_static"
|
||||||
|
version = "1.4.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "libc"
|
||||||
|
version = "0.2.91"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8916b1f6ca17130ec6568feccee27c156ad12037880833a3b842a823236502e7"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "libsqlite3-sys"
|
||||||
|
version = "0.20.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "64d31059f22935e6c31830db5249ba2b7ecd54fd73a9909286f0a67aa55c2fbd"
|
||||||
|
dependencies = [
|
||||||
|
"pkg-config",
|
||||||
|
"vcpkg",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "memchr"
|
||||||
|
version = "2.3.4"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "0ee1c47aaa256ecabcaea351eae4a9b01ef39ed810004e298d2511ed284b1525"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "miniz_oxide"
|
||||||
|
version = "0.3.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "791daaae1ed6889560f8c4359194f56648355540573244a5448a83ba1ecc7435"
|
||||||
|
dependencies = [
|
||||||
|
"adler32",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "num_cpus"
|
||||||
|
version = "1.13.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "05499f3756671c15885fee9034446956fff3f243d6077b91e5767df161f766b3"
|
||||||
|
dependencies = [
|
||||||
|
"hermit-abi",
|
||||||
|
"libc",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "percent-encoding"
|
||||||
|
version = "2.1.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d4fd5641d01c8f18a23da7b6fe29298ff4b55afcccdf78973b24cf3175fee32e"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "pkg-config"
|
||||||
|
version = "0.3.19"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "3831453b3449ceb48b6d9c7ad7c96d5ea673e9b470a1dc578c2ce6521230884c"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "proc-macro2"
|
||||||
|
version = "1.0.24"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1e0704ee1a7e00d7bb417d0770ea303c1bccbabf0ef1667dae92b5967f5f8a71"
|
||||||
|
dependencies = [
|
||||||
|
"unicode-xid",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quote"
|
||||||
|
version = "1.0.9"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "c3d0b9745dc2debf507c8422de05d7226cc1f0644216dfdfead988f9b1ab32a7"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "regex"
|
||||||
|
version = "1.4.5"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "957056ecddbeba1b26965114e191d2e8589ce74db242b6ea25fc4062427a5c19"
|
||||||
|
dependencies = [
|
||||||
|
"aho-corasick",
|
||||||
|
"memchr",
|
||||||
|
"regex-syntax",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "regex-syntax"
|
||||||
|
version = "0.6.23"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "24d5f089152e60f62d28b835fbff2cd2e8dc0baf1ac13343bef92ab7eed84548"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "rusqlite"
|
||||||
|
version = "0.24.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "d5f38ee71cbab2c827ec0ac24e76f82eca723cee92c509a65f67dee393c25112"
|
||||||
|
dependencies = [
|
||||||
|
"bitflags",
|
||||||
|
"fallible-iterator",
|
||||||
|
"fallible-streaming-iterator",
|
||||||
|
"hashlink",
|
||||||
|
"libsqlite3-sys",
|
||||||
|
"memchr",
|
||||||
|
"smallvec",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "smallvec"
|
||||||
|
version = "1.6.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "fe0f37c9e8f3c5a4a66ad655a93c74daac4ad00c441533bf5c6e7990bb42604e"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "syn"
|
||||||
|
version = "1.0.65"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f3a1d708c221c5a612956ef9f75b37e454e88d1f7b899fbd3a18d4252012d663"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"unicode-xid",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "thiserror"
|
||||||
|
version = "1.0.24"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "e0f4a65597094d4483ddaed134f409b2cb7c1beccf25201a9f73c719254fa98e"
|
||||||
|
dependencies = [
|
||||||
|
"thiserror-impl",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "thiserror-impl"
|
||||||
|
version = "1.0.24"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "7765189610d8241a44529806d6fd1f2e0a08734313a35d5b3a556f92b381f3c0"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"syn",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "time"
|
||||||
|
version = "0.1.44"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "6db9e6914ab8b1ae1c260a4ae7a49b6c5611b40328a735b21862567685e73255"
|
||||||
|
dependencies = [
|
||||||
|
"libc",
|
||||||
|
"wasi",
|
||||||
|
"winapi",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "unicode-xid"
|
||||||
|
version = "0.2.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f7fe0bb3479651439c9112f72b6c505038574c9fbb575ed1bf3b797fa39dd564"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "vcpkg"
|
||||||
|
version = "0.2.11"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "b00bca6106a5e23f3eee943593759b7fcddb00554332e856d990c893966879fb"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "wasi"
|
||||||
|
version = "0.10.0+wasi-snapshot-preview1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1a143597ca7c7793eff794def352d41792a93c481eb1042423ff7ff72ba2c31f"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "winapi"
|
||||||
|
version = "0.3.9"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
|
||||||
|
dependencies = [
|
||||||
|
"winapi-i686-pc-windows-gnu",
|
||||||
|
"winapi-x86_64-pc-windows-gnu",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "winapi-i686-pc-windows-gnu"
|
||||||
|
version = "0.4.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "winapi-x86_64-pc-windows-gnu"
|
||||||
|
version = "0.4.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "xml-rs"
|
||||||
|
version = "0.8.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "b07db065a5cf61a7e4ba64f29e67db906fb1787316516c4e6e5ff0fea1efcd8a"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "zip"
|
||||||
|
version = "0.5.11"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8264fcea9b7a036a4a5103d7153e988dbc2ebbafb34f68a3c2d404b6b82d74b6"
|
||||||
|
dependencies = [
|
||||||
|
"byteorder",
|
||||||
|
"bzip2",
|
||||||
|
"crc32fast",
|
||||||
|
"flate2",
|
||||||
|
"thiserror",
|
||||||
|
"time",
|
||||||
|
]
|
||||||
|
|
@ -0,0 +1,15 @@
|
||||||
|
[package]
|
||||||
|
name = "calibre-indexer"
|
||||||
|
version = "0.1.0"
|
||||||
|
authors = ["osmarks <osmarks@protonmail.com>"]
|
||||||
|
edition = "2018"
|
||||||
|
|
||||||
|
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
rusqlite = "0.24"
|
||||||
|
anyhow = "1"
|
||||||
|
num_cpus = "1"
|
||||||
|
epub = "1"
|
||||||
|
xml-rs = "0.8" # TODO: faster XML parser
|
||||||
|
lazy_static = "1"
|
||||||
|
|
@ -0,0 +1,231 @@
|
||||||
|
// Earlier version attempting to use quick-xml
|
||||||
|
// Dropped because SQLite appears to be what most of the time is spent in anyway, and because quick-xml had some issues wrt. escaping
|
||||||
|
|
||||||
|
use std::fs;
|
||||||
|
use anyhow::{Result, Context};
|
||||||
|
use crossbeam::channel::{bounded};
|
||||||
|
use crossbeam::thread;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use rusqlite::{params, Connection};
|
||||||
|
use std::fs::File;
|
||||||
|
use xml::reader::{EventReader, XmlEvent, ParserConfig};
|
||||||
|
use quick_xml::{Reader, events::Event};
|
||||||
|
use std::io::BufReader;
|
||||||
|
use epub::doc::EpubDoc;
|
||||||
|
use lazy_static::lazy_static;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
struct BookMeta {
|
||||||
|
title: String,
|
||||||
|
author: String,
|
||||||
|
description: String
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||||
|
enum XMLReadState {
|
||||||
|
None,
|
||||||
|
ReadingTitle,
|
||||||
|
ReadingAuthor,
|
||||||
|
ReadingDescription
|
||||||
|
}
|
||||||
|
|
||||||
|
lazy_static! {
|
||||||
|
static ref ESCAPES: HashMap<Vec<u8>, Vec<u8>> = {
|
||||||
|
let mut m = HashMap::new();
|
||||||
|
m.insert(b"nbsp".to_vec(), b"\xc2\xa0".to_vec());
|
||||||
|
m.insert(b"copy".to_vec(), b"\xc2\xa9".to_vec());
|
||||||
|
m.insert(b"eacute".to_vec(), b"\xc3\x89".to_vec());
|
||||||
|
m.insert(b"shy".to_vec(), b"\xc2\xad".to_vec());
|
||||||
|
m.insert(b"iuml".to_vec(), b"\xc3\x8f".to_vec());
|
||||||
|
m
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract text from an XHTML page in an ebook
|
||||||
|
// Ignores <script>, <style>, etc
|
||||||
|
// Also extracts chapter titles via assuming that any <hN> is part of a chapter title
|
||||||
|
fn extract_text(r: Vec<u8>) -> Result<(String, String)> {
|
||||||
|
//println!("{:?}", String::from_utf8_lossy(r.as_slice()));
|
||||||
|
let mut text = String::new();
|
||||||
|
let conf = ParserConfig::new()
|
||||||
|
.ignore_comments(true)
|
||||||
|
.cdata_to_characters(true);
|
||||||
|
|
||||||
|
let mut ignoring = false;
|
||||||
|
let mut newline_appended_last = false;
|
||||||
|
let mut in_header = None;
|
||||||
|
let mut chapter = String::new();
|
||||||
|
let mut reader = Reader::from_reader(r.as_slice());
|
||||||
|
let mut buf = Vec::new();
|
||||||
|
reader.trim_text_end(true);
|
||||||
|
loop {
|
||||||
|
match reader.read_event(&mut buf)? {
|
||||||
|
Event::Start(ref e) => {
|
||||||
|
match e.name() {
|
||||||
|
b"style" | b"script" | b"nav" | b"iframe" | b"svg" => { ignoring = true },
|
||||||
|
b"h1" | b"h2" | b"h3" | b"h4" | b"h5" | b"h6" => { ignoring = false; in_header = Some(e.name().to_vec()) }
|
||||||
|
_ => { ignoring = false }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
Event::Text(new) => {
|
||||||
|
if !ignoring {
|
||||||
|
text += &new.unescape_and_decode_with_custom_entities(&reader, &*ESCAPES)?;
|
||||||
|
if in_header.is_some() && &*new != b"\xA7" && &*new != b"*" {
|
||||||
|
chapter += &new.unescape_and_decode_with_custom_entities(&reader, &*ESCAPES)?;
|
||||||
|
}
|
||||||
|
newline_appended_last = false;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
Event::Eof => break,
|
||||||
|
Event::End(ref e) => {
|
||||||
|
if let Some(ref h) = in_header {
|
||||||
|
if h == &e.name() {
|
||||||
|
chapter += "\n";
|
||||||
|
in_header = None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ignoring = false;
|
||||||
|
match e.name() {
|
||||||
|
b"span" | b"sub" | b"sup" | b"small" | b"i" | b"b" | b"em" | b"strike" | b"strong" | b"a" | b"link" | b"head" => {}
|
||||||
|
x => {
|
||||||
|
if !newline_appended_last {
|
||||||
|
text += "\n";
|
||||||
|
newline_appended_last = true;
|
||||||
|
if in_header.is_some() && x == b"br" {
|
||||||
|
chapter += "\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok((text, chapter))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_opf(path: PathBuf) -> Result<BookMeta> {
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let file = BufReader::new(file);
|
||||||
|
let conf = ParserConfig::new()
|
||||||
|
.ignore_comments(true)
|
||||||
|
.cdata_to_characters(true);
|
||||||
|
|
||||||
|
let mut meta = BookMeta {
|
||||||
|
title: "".to_string(),
|
||||||
|
author: "".to_string(),
|
||||||
|
description: "".to_string()
|
||||||
|
};
|
||||||
|
let mut buf = String::new();
|
||||||
|
let mut state = XMLReadState::None;
|
||||||
|
for e in EventReader::new_with_config(file, conf) {
|
||||||
|
match e? {
|
||||||
|
XmlEvent::StartElement { name, .. } => {
|
||||||
|
match name.local_name.as_str() {
|
||||||
|
"title" => { state = XMLReadState::ReadingTitle },
|
||||||
|
"creator" => { state = XMLReadState::ReadingAuthor },
|
||||||
|
"description" => { state = XMLReadState::ReadingDescription },
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
XmlEvent::Characters(s) => {
|
||||||
|
if state != XMLReadState::None {
|
||||||
|
buf += &s;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
XmlEvent::EndElement { .. } => {
|
||||||
|
match state {
|
||||||
|
XMLReadState::ReadingTitle => { meta.title = buf.clone() },
|
||||||
|
XMLReadState::ReadingDescription => { meta.description = buf.clone() },
|
||||||
|
XMLReadState::ReadingAuthor => { meta.author = buf.clone() },
|
||||||
|
XMLReadState::None => {}
|
||||||
|
}
|
||||||
|
state = XMLReadState::None;
|
||||||
|
buf.clear();
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(meta)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn path_append(p: &PathBuf, c: &str) -> PathBuf {
|
||||||
|
let mut o = p.clone();
|
||||||
|
o.push(c);
|
||||||
|
o
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() -> Result<()> {
|
||||||
|
let (tx, rx) = bounded::<PathBuf>(16);
|
||||||
|
let res: Result<()> = thread::scope(|sc| {
|
||||||
|
let db = Connection::open("./data.sqlite3")?;
|
||||||
|
db.execute_batch("
|
||||||
|
BEGIN;
|
||||||
|
CREATE TABLE IF NOT EXISTS files (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
path BLOB NOT NULL,
|
||||||
|
last_modified INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS data USING fts5 (
|
||||||
|
author,
|
||||||
|
book,
|
||||||
|
chapter,
|
||||||
|
content,
|
||||||
|
file
|
||||||
|
);
|
||||||
|
COMMIT;
|
||||||
|
").with_context(|| "database initialization")?;
|
||||||
|
|
||||||
|
let mut threads: Vec<thread::ScopedJoinHandle<()>> = vec![];
|
||||||
|
for i in 0..num_cpus::get() {
|
||||||
|
let rx = rx.clone();
|
||||||
|
let go = move || -> Result<()> {
|
||||||
|
for book_dir in rx.iter() {
|
||||||
|
println!("{} begin handling {:?}", i, book_dir);
|
||||||
|
let meta = read_opf(path_append(&book_dir, "metadata.opf")).with_context(|| format!("OPF metadata parsing for {:?}", book_dir))?;
|
||||||
|
let epub_path = path_append(&book_dir, &format!("{} - {}.epub", meta.title, meta.author));
|
||||||
|
if epub_path.exists() {
|
||||||
|
let mut doc = EpubDoc::new(epub_path).with_context(|| format!("reading {:?}", book_dir))?;
|
||||||
|
let spine = doc.spine.clone();
|
||||||
|
for resource in spine {
|
||||||
|
let content = doc.get_resource(&resource).with_context(|| format!("reading {:?} in {:?}", &resource, book_dir))?;
|
||||||
|
let (mut content, chapter) = extract_text(content).with_context(|| format!("parsing {:?} in {:?}", &resource, book_dir))?;
|
||||||
|
let chapter = chapter.trim();
|
||||||
|
// in place trim of newlines - avoid allocating new string
|
||||||
|
while content.ends_with("\n") {
|
||||||
|
content.truncate(content.len() - 1);
|
||||||
|
}
|
||||||
|
if content != "" {
|
||||||
|
//println!("{}: {}: {}", meta.title, resource, chapter);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
//println!("{} - {}", meta.author, meta.title);
|
||||||
|
}
|
||||||
|
println!("{} end handling {:?}", i, book_dir);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
};
|
||||||
|
threads.push(sc.spawn(move |_| { go().unwrap() }));
|
||||||
|
}
|
||||||
|
|
||||||
|
for author_dir in fs::read_dir("/data/calibre").with_context(|| "reading library location")? {
|
||||||
|
let author_dir = author_dir?;
|
||||||
|
if author_dir.file_type()?.is_dir() {
|
||||||
|
for book_dir in fs::read_dir(author_dir.path())? {
|
||||||
|
let path = book_dir?.path();
|
||||||
|
//println!("{:?}", path);
|
||||||
|
tx.send(path)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
std::mem::drop(tx);
|
||||||
|
|
||||||
|
for thread in threads {
|
||||||
|
thread.join().unwrap();
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}).unwrap();
|
||||||
|
res?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,219 @@
|
||||||
|
use std::fs;
|
||||||
|
use anyhow::{Result, Context};
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use rusqlite::{params, Connection, OptionalExtension};
|
||||||
|
use std::fs::File;
|
||||||
|
use xml::reader::{EventReader, XmlEvent, ParserConfig};
|
||||||
|
use std::io::BufReader;
|
||||||
|
use epub::doc::EpubDoc;
|
||||||
|
use std::time::SystemTime;
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
struct BookMeta {
|
||||||
|
title: String,
|
||||||
|
author: String,
|
||||||
|
description: String
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||||
|
enum XMLReadState {
|
||||||
|
None,
|
||||||
|
ReadingTitle,
|
||||||
|
ReadingAuthor,
|
||||||
|
ReadingDescription
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract text from an XHTML page in an ebook
|
||||||
|
// Ignores <script>, <style>, etc
|
||||||
|
// Also extracts chapter titles via assuming that any <hN> is part of a chapter title
|
||||||
|
fn extract_text<R: std::io::Read>(r: R) -> Result<(String, String)> {
|
||||||
|
let mut text = String::new();
|
||||||
|
let conf = ParserConfig::new()
|
||||||
|
.ignore_comments(true)
|
||||||
|
.cdata_to_characters(true)
|
||||||
|
.add_entity("nbsp", "\u{A0}");
|
||||||
|
|
||||||
|
let mut ignoring = false;
|
||||||
|
let mut newline_appended_last = false;
|
||||||
|
let mut in_header = None;
|
||||||
|
let mut chapter = String::new();
|
||||||
|
for e in EventReader::new_with_config(r, conf) {
|
||||||
|
match e? {
|
||||||
|
XmlEvent::StartElement { name, .. } => {
|
||||||
|
match name.local_name.as_str() {
|
||||||
|
"style" | "script" | "nav" | "iframe" | "svg" => { ignoring = true },
|
||||||
|
"h1" | "h2" | "h3" | "h4" | "h5" | "h6" => { ignoring = false; in_header = Some(name.local_name) }
|
||||||
|
_ => { ignoring = false }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
XmlEvent::Characters(new) => {
|
||||||
|
if !ignoring {
|
||||||
|
text += &new;
|
||||||
|
if in_header.is_some() && new != "§" && new != "*" {
|
||||||
|
chapter += &new;
|
||||||
|
}
|
||||||
|
newline_appended_last = false;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
XmlEvent::EndElement { name, .. } => {
|
||||||
|
if let Some(ref h) = in_header {
|
||||||
|
if h == &name.local_name {
|
||||||
|
chapter += "\n";
|
||||||
|
in_header = None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ignoring = false;
|
||||||
|
match name.local_name.as_str() {
|
||||||
|
"span" | "sub" | "sup" | "small" | "i" | "b" | "em" | "strike" | "strong" | "a" | "link" | "head" => {}
|
||||||
|
x => {
|
||||||
|
if !newline_appended_last {
|
||||||
|
text += "\n";
|
||||||
|
newline_appended_last = true;
|
||||||
|
if in_header.is_some() && x == "br" {
|
||||||
|
chapter += "\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok((text, chapter))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_opf(path: PathBuf) -> Result<BookMeta> {
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let file = BufReader::new(file);
|
||||||
|
let conf = ParserConfig::new()
|
||||||
|
.ignore_comments(true)
|
||||||
|
.cdata_to_characters(true);
|
||||||
|
|
||||||
|
let mut meta = BookMeta {
|
||||||
|
title: "".to_string(),
|
||||||
|
author: "".to_string(),
|
||||||
|
description: "".to_string()
|
||||||
|
};
|
||||||
|
let mut buf = String::new();
|
||||||
|
let mut state = XMLReadState::None;
|
||||||
|
for e in EventReader::new_with_config(file, conf) {
|
||||||
|
match e? {
|
||||||
|
XmlEvent::StartElement { name, .. } => {
|
||||||
|
match name.local_name.as_str() {
|
||||||
|
"title" => { state = XMLReadState::ReadingTitle },
|
||||||
|
"creator" => { state = XMLReadState::ReadingAuthor },
|
||||||
|
"description" => { state = XMLReadState::ReadingDescription },
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
XmlEvent::Characters(s) => {
|
||||||
|
if state != XMLReadState::None {
|
||||||
|
buf += &s;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
XmlEvent::EndElement { .. } => {
|
||||||
|
match state {
|
||||||
|
XMLReadState::ReadingTitle => { meta.title = buf.clone() },
|
||||||
|
XMLReadState::ReadingDescription => { meta.description = buf.clone() },
|
||||||
|
XMLReadState::ReadingAuthor => { meta.author = buf.clone() },
|
||||||
|
XMLReadState::None => {}
|
||||||
|
}
|
||||||
|
state = XMLReadState::None;
|
||||||
|
buf.clear();
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(meta)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn path_append(p: &PathBuf, c: &str) -> PathBuf {
|
||||||
|
let mut o = p.clone();
|
||||||
|
o.push(c);
|
||||||
|
o
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run(db: &mut Connection, book_dir: PathBuf) -> Result<()> {
|
||||||
|
let meta = read_opf(path_append(&book_dir, "metadata.opf")).with_context(|| format!("OPF metadata parsing for {:?}", book_dir))?;
|
||||||
|
|
||||||
|
let dirent = book_dir.read_dir()?.collect::<std::io::Result<Vec<fs::DirEntry>>>()?.into_iter().filter(|ent| ent.file_name().to_str().unwrap().ends_with(".epub")).next();
|
||||||
|
if let Some(dirent) = dirent {
|
||||||
|
let epub_path = dirent.path();
|
||||||
|
let path_str = epub_path.to_str().unwrap().to_string();
|
||||||
|
let row: Option<i64> = db.query_row("SELECT last_modified FROM files WHERE path = ?", params![path_str], |row| row.get(0)).optional()?;
|
||||||
|
|
||||||
|
let timestamp = epub_path.metadata()?.modified()?;
|
||||||
|
let timestamp = timestamp.duration_since(SystemTime::UNIX_EPOCH)?.as_secs() as i64;
|
||||||
|
match row {
|
||||||
|
Some(orig_last_modified) if orig_last_modified == timestamp => {
|
||||||
|
println!("Already have {} - {}", meta.title, meta.author);
|
||||||
|
return Ok(())
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
|
||||||
|
println!("Processing {} - {}", meta.title, meta.author);
|
||||||
|
let tx = db.transaction()?;
|
||||||
|
|
||||||
|
tx.execute("INSERT OR REPLACE INTO files (path, last_modified) VALUES (?, ?)", params![path_str, timestamp])?;
|
||||||
|
let fid = tx.last_insert_rowid();
|
||||||
|
tx.execute("DELETE FROM data WHERE file = ?", params![fid])?;
|
||||||
|
|
||||||
|
let mut doc = EpubDoc::new(epub_path).with_context(|| format!("reading {:?}", book_dir))?;
|
||||||
|
let spine = doc.spine.clone();
|
||||||
|
for resource in spine {
|
||||||
|
let content = doc.get_resource(&resource).with_context(|| format!("reading {:?} in {:?}", &resource, book_dir))?;
|
||||||
|
let (mut content, chapter) = extract_text(content.as_slice()).with_context(|| format!("parsing {:?} in {:?}", &resource, book_dir))?;
|
||||||
|
let chapter = chapter.trim();
|
||||||
|
// in place trim of newlines - avoid allocating new string
|
||||||
|
while content.ends_with("\n") {
|
||||||
|
content.truncate(content.len() - 1);
|
||||||
|
}
|
||||||
|
if content != "" {
|
||||||
|
tx.execute("INSERT INTO data VALUES (?, ?, ?, ?, ?)", params![meta.author, meta.title, chapter, content, fid])?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
tx.commit()?;
|
||||||
|
println!("Done {} - {}", meta.title, meta.author);
|
||||||
|
} else {
|
||||||
|
println!("No EPUB for {} - {}", meta.title, meta.author);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
// TODO: Make this work concurrently again, by finding a more performant backend
|
||||||
|
fn main() -> Result<()> {
|
||||||
|
let argv: Vec<String> = std::env::args().collect();
|
||||||
|
|
||||||
|
let mut db = Connection::open(argv[1].clone())?;
|
||||||
|
db.execute_batch("
|
||||||
|
BEGIN;
|
||||||
|
CREATE TABLE IF NOT EXISTS files (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
path BLOB NOT NULL UNIQUE,
|
||||||
|
last_modified INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS data USING fts5 (
|
||||||
|
author,
|
||||||
|
book,
|
||||||
|
chapter,
|
||||||
|
content,
|
||||||
|
file UNINDEXED
|
||||||
|
);
|
||||||
|
COMMIT;
|
||||||
|
PRAGMA journal_mode = WAL;
|
||||||
|
").with_context(|| "database initialization")?;
|
||||||
|
|
||||||
|
for author_dir in fs::read_dir(argv[2].clone()).with_context(|| "reading library location")? {
|
||||||
|
let author_dir = author_dir?;
|
||||||
|
if author_dir.file_type()?.is_dir() {
|
||||||
|
for book_dir in fs::read_dir(author_dir.path())? {
|
||||||
|
let path = book_dir?.path();
|
||||||
|
run(&mut db, path)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,12 @@
|
||||||
|
import sqlite3
|
||||||
|
import re
|
||||||
|
|
||||||
|
conn = sqlite3.connect("./data.sqlite3")
|
||||||
|
#conn.row_factory = sqlite3.Row
|
||||||
|
csr = conn.execute("""SELECT author, book, chapter, snippet(data, -1, '[!]', '[!]', ' ... ', 32)
|
||||||
|
FROM data WHERE data MATCH ? AND rank MATCH 'bm25(10.0, 10.0, 5.0, 1.0)' ORDER BY rank LIMIT 100""", (input("query:"),))
|
||||||
|
while x := csr.fetchone():
|
||||||
|
author, book, chapter, snippet = x
|
||||||
|
snippet = re.sub("\n+", "\n", snippet)
|
||||||
|
chapter = chapter.replace("\n", " ")
|
||||||
|
print(f"[{author}: {book}] {'<' + chapter + '> ' if chapter != '' else ''}{snippet}")
|
||||||
|
|
@ -0,0 +1,44 @@
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <immintrin.h>
|
||||||
|
#include <memory.h>
|
||||||
|
#include <time.h>
|
||||||
|
|
||||||
|
#define MAXLEN 512
|
||||||
|
#define M128SIZE 16
|
||||||
|
#define CHUNKS (MAXLEN / M128SIZE)
|
||||||
|
|
||||||
|
int entry(char *s1, char *s2) {
|
||||||
|
char *m1 = aligned_alloc(M128SIZE, MAXLEN);
|
||||||
|
char *m2 = aligned_alloc(M128SIZE, MAXLEN);
|
||||||
|
strncpy(m1, s1, MAXLEN);
|
||||||
|
strncpy(m2, s2, MAXLEN);
|
||||||
|
|
||||||
|
for (int i = 0; i < MAXLEN; i += M128SIZE) {
|
||||||
|
__m128i *x = (__m128i*)(&m1[i]);
|
||||||
|
__m128i input = *x;
|
||||||
|
__m128i perm1 = _mm_set_epi8(1, 0, 3, 2, 5, 4, 7, 6, 9, 8, 11, 10, 13, 12, 15, 14);
|
||||||
|
__m128i pair_swapped = _mm_shuffle_epi8(input, perm1);
|
||||||
|
__m128i comp = _mm_cmplt_epi8(input, pair_swapped);
|
||||||
|
__m128i mask = _mm_xor_si128(comp, _mm_set_epi8(0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1));
|
||||||
|
__m128i blendeded = _mm_blendv_epi8(input, pair_swapped, mask);
|
||||||
|
__m128i perm2 = _mm_set_epi8(0, 2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 12, 11, 14, 13, 15);
|
||||||
|
__m128i opair_swapped = _mm_shuffle_epi8(input, perm2);
|
||||||
|
__m128i comp2 = _mm_cmplt_epi8(input, opair_swapped);
|
||||||
|
|
||||||
|
//*x = _mm_shuffle_epi8(input, s);
|
||||||
|
for (int i = 0; i < 16; i++) {
|
||||||
|
//printf("%d %c→%c / ", ((char*)&comp2)[i], ((char*)input)[i], ((char*)&opair_swapped)[i]);
|
||||||
|
//printf("%d %c-%c", ((char*)&comp)[i], ((char*)input)[i], ((char*)&pair_swapped)[i]);
|
||||||
|
printf("%c", ((char*)&blendeded)[i]);
|
||||||
|
}
|
||||||
|
printf(" %d\n", i);
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int main() {
|
||||||
|
srand(time(NULL));
|
||||||
|
entry("aopfmffooapfariproompoafimfafppfiopoiformrmmafoiioiommiooraomoamppoiorfammapparamoarpmpoarpoampfmrarorfirfrpoafariiroipripoooofaioairampiooopoppopoimaamroooofamrprororoirfmorormmoaooiopoooooaoaopfmiiaoaaoffofioraorffrmfoioofriraiioappipooofoaomfifopmofmafforippfamoaopiopmiafopmfmpifmaroomiopoapppforfffrmiioaapafoaorfpmffofrporoaaaopmffimomroarifimmrpfrpofofaopoapiormmopriimrmrifroofrirmmoaipmrrofoorprmpoprofapiopommopoomoariapiooraoiiampmmprpmiporoiaofrariaorppifoomarfoirfmimffofmiioooriaammiiafiooioipamofm", "fiaaafamrofafprooparomaprioifiomfofaoproooooiooorooioamfipofroommoafopaopaioppiiproirfimpiorfmaroamopmiipmoapofoaoaofiioofrormaimrpmooafiiorifimoaprrrraoiamoorpfapofffpoorppfmmffmifimofrofoaopoiorrpaaaoioipofpmimrmoooiiafproimpprifrrrrrrriromfirmrpmaompiaooommpmrriromioppoirorapoaoiiaoaioaoaofiaaoaafamraffioooiiaopofooioommoooiropopmiapfpimpoappmmfpmapofmfamrppaaomfamfpopraiamfpofrrmomaofmiaaorriofaimaoomripfpfoomoaafmfiimrfaroofpaffopromarmomopfiafoopopifimopmoroimrrmrrfmooporroioofpfomomomrorfppipffofaipp");
|
||||||
|
printf("ALL is now bee\n");
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,75 @@
|
||||||
|
#include <immintrin.h>
|
||||||
|
#include <memory.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#define MAXLEN 512
|
||||||
|
#define M128SIZE 16
|
||||||
|
#define M128SIZEBITS 4
|
||||||
|
#define CHUNKS (MAXLEN >> M128SIZEBITS)
|
||||||
|
#define BROADCAST_EPI8(x) _mm_set_epi8(x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x)
|
||||||
|
|
||||||
|
char* run(char *m) {
|
||||||
|
for (uint16_t i = 0; i < MAXLEN; i += M128SIZE) {
|
||||||
|
__m128i *x = (__m128i*)(&m[i]);
|
||||||
|
__m128i curr = *x;
|
||||||
|
curr = _mm_add_epi8(curr, _mm_and_si128(_mm_cmpgt_epi8(curr, BROADCAST_EPI8(96)), BROADCAST_EPI8(-32)));
|
||||||
|
int32_t match = 0;
|
||||||
|
while (match != 0xFFFF) {
|
||||||
|
__m128i ps = _mm_shuffle_epi8(curr, _mm_set_epi8(14, 15, 12, 13, 10, 11, 8, 9, 6, 7, 4, 5, 2, 3, 0, 1));
|
||||||
|
__m128i sw1 = _mm_blendv_epi8(curr, ps, _mm_xor_si128(_mm_cmplt_epi8(curr, ps), _mm_set_epi8(0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1)));
|
||||||
|
__m128i ops = _mm_shuffle_epi8(sw1, _mm_set_epi8(15, 13, 14, 11, 12, 9, 10, 7, 8, 5, 6, 3, 4, 1, 2, 0));
|
||||||
|
__m128i sw2 = _mm_blendv_epi8(sw1, ops, _mm_xor_si128(_mm_cmplt_epi8(sw1, ops), _mm_set_epi8(0, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0)));
|
||||||
|
match = _mm_movemask_epi8(_mm_cmpeq_epi8(sw2, curr));
|
||||||
|
curr = sw2;
|
||||||
|
}
|
||||||
|
*x = curr;
|
||||||
|
}
|
||||||
|
|
||||||
|
char *buf = aligned_alloc(M128SIZE, MAXLEN);
|
||||||
|
if (!buf) exit(1);
|
||||||
|
memset(buf, 0, MAXLEN);
|
||||||
|
uint8_t pos[CHUNKS] = {0};
|
||||||
|
uint16_t opos = 0;
|
||||||
|
while (1) {
|
||||||
|
uint8_t max = 255;
|
||||||
|
uint8_t bc = 255;
|
||||||
|
for (uint16_t i = 0; i < MAXLEN; i += M128SIZE) {
|
||||||
|
uint8_t chunk = i >> M128SIZEBITS;
|
||||||
|
uint8_t icpos = pos[chunk];
|
||||||
|
char v = m[icpos + i];
|
||||||
|
if (v < max && icpos < M128SIZE) {
|
||||||
|
max = v;
|
||||||
|
bc = chunk;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (bc == 255) break;
|
||||||
|
pos[bc]++;
|
||||||
|
if (max > ' ') {
|
||||||
|
buf[opos] = max;
|
||||||
|
opos++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return buf;
|
||||||
|
}
|
||||||
|
|
||||||
|
uint8_t entry(char *s1, char *s2) {
|
||||||
|
char *m1 = aligned_alloc(M128SIZE, MAXLEN);
|
||||||
|
if (!m1) exit(1);
|
||||||
|
char *m2 = aligned_alloc(M128SIZE, MAXLEN);
|
||||||
|
if (!m2) exit(1);
|
||||||
|
memset(m1, 0, MAXLEN);
|
||||||
|
memset(m2, 0, MAXLEN);
|
||||||
|
strncpy(m1, s1, MAXLEN);
|
||||||
|
strncpy(m2, s2, MAXLEN);
|
||||||
|
|
||||||
|
char *x1 = run(m1);
|
||||||
|
char *x2 = run(m2);
|
||||||
|
free(m1);
|
||||||
|
free(m2);
|
||||||
|
|
||||||
|
uint8_t result = !strncmp(x1, x2, MAXLEN);
|
||||||
|
free(x1);
|
||||||
|
free(x2);
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
target
|
||||||
|
out.png
|
||||||
|
|
@ -0,0 +1,216 @@
|
||||||
|
# This file is automatically @generated by Cargo.
|
||||||
|
# It is not intended for manual editing.
|
||||||
|
[[package]]
|
||||||
|
name = "adler32"
|
||||||
|
version = "1.1.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "567b077b825e468cc974f0020d4082ee6e03132512f207ef1a02fd5d00d1f32d"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "autocfg"
|
||||||
|
version = "1.0.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "bitflags"
|
||||||
|
version = "1.2.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "bytemuck"
|
||||||
|
version = "1.2.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "37fa13df2292ecb479ec23aa06f4507928bef07839be9ef15281411076629431"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "byteorder"
|
||||||
|
version = "1.3.4"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "cfg-if"
|
||||||
|
version = "0.1.10"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "crc32fast"
|
||||||
|
version = "1.2.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ba125de2af0df55319f41944744ad91c71113bf74a4646efff39afe1f6842db1"
|
||||||
|
dependencies = [
|
||||||
|
"cfg-if",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "deflate"
|
||||||
|
version = "0.8.6"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "73770f8e1fe7d64df17ca66ad28994a0a623ea497fa69486e14984e715c5d174"
|
||||||
|
dependencies = [
|
||||||
|
"adler32",
|
||||||
|
"byteorder",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "fractalart"
|
||||||
|
version = "0.1.0"
|
||||||
|
dependencies = [
|
||||||
|
"gumdrop",
|
||||||
|
"hsl",
|
||||||
|
"image",
|
||||||
|
"nanorand",
|
||||||
|
"smallvec",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "gumdrop"
|
||||||
|
version = "0.8.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "46571f5d540478cf70d2a42dd0d6d8e9f4b9cc7531544b93311e657b86568a0b"
|
||||||
|
dependencies = [
|
||||||
|
"gumdrop_derive",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "gumdrop_derive"
|
||||||
|
version = "0.8.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "915ef07c710d84733522461de2a734d4d62a3fd39a4d4f404c2f385ef8618d05"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"syn",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "hsl"
|
||||||
|
version = "0.1.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "575fb7f1167f3b88ed825e90eb14918ac460461fdeaa3965c6a50951dee1c970"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "image"
|
||||||
|
version = "0.23.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "a2397fc43bd5648b7117aabb3c5e62d0e62c194826ec77b0b4d0c41e62744635"
|
||||||
|
dependencies = [
|
||||||
|
"bytemuck",
|
||||||
|
"byteorder",
|
||||||
|
"num-iter",
|
||||||
|
"num-rational",
|
||||||
|
"num-traits",
|
||||||
|
"png",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "miniz_oxide"
|
||||||
|
version = "0.3.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "791daaae1ed6889560f8c4359194f56648355540573244a5448a83ba1ecc7435"
|
||||||
|
dependencies = [
|
||||||
|
"adler32",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "nanorand"
|
||||||
|
version = "0.5.2"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ac1378b66f7c93a1c0f8464a19bf47df8795083842e5090f4b7305973d5a22d0"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "num-integer"
|
||||||
|
version = "0.1.43"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "8d59457e662d541ba17869cf51cf177c0b5f0cbf476c66bdc90bf1edac4f875b"
|
||||||
|
dependencies = [
|
||||||
|
"autocfg",
|
||||||
|
"num-traits",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "num-iter"
|
||||||
|
version = "0.1.41"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "7a6e6b7c748f995c4c29c5f5ae0248536e04a5739927c74ec0fa564805094b9f"
|
||||||
|
dependencies = [
|
||||||
|
"autocfg",
|
||||||
|
"num-integer",
|
||||||
|
"num-traits",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "num-rational"
|
||||||
|
version = "0.3.0"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "a5b4d7360f362cfb50dde8143501e6940b22f644be75a4cc90b2d81968908138"
|
||||||
|
dependencies = [
|
||||||
|
"autocfg",
|
||||||
|
"num-integer",
|
||||||
|
"num-traits",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "num-traits"
|
||||||
|
version = "0.2.12"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "ac267bcc07f48ee5f8935ab0d24f316fb722d7a1292e2913f0cc196b29ffd611"
|
||||||
|
dependencies = [
|
||||||
|
"autocfg",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "png"
|
||||||
|
version = "0.16.6"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "c150bf7479fafe3dd8740dbe48cc33b2a3efb7b0fe3483aced8bbc39f6d0238d"
|
||||||
|
dependencies = [
|
||||||
|
"bitflags",
|
||||||
|
"crc32fast",
|
||||||
|
"deflate",
|
||||||
|
"miniz_oxide",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "proc-macro2"
|
||||||
|
version = "1.0.18"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "beae6331a816b1f65d04c45b078fd8e6c93e8071771f41b8163255bbd8d7c8fa"
|
||||||
|
dependencies = [
|
||||||
|
"unicode-xid",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "quote"
|
||||||
|
version = "1.0.7"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "aa563d17ecb180e500da1cfd2b028310ac758de548efdd203e18f283af693f37"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "smallvec"
|
||||||
|
version = "1.4.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "3757cb9d89161a2f24e1cf78efa0c1fcff485d18e3f55e0aa3480824ddaa0f3f"
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "syn"
|
||||||
|
version = "1.0.34"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "936cae2873c940d92e697597c5eee105fb570cd5689c695806f672883653349b"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"unicode-xid",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "unicode-xid"
|
||||||
|
version = "0.2.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "f7fe0bb3479651439c9112f72b6c505038574c9fbb575ed1bf3b797fa39dd564"
|
||||||
|
|
@ -0,0 +1,12 @@
|
||||||
|
[package]
|
||||||
|
name = "fractalart"
|
||||||
|
version = "0.1.0"
|
||||||
|
authors = ["osmarks <osmarks@protonmail.com>"]
|
||||||
|
edition = "2018"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
image = { version = "0.23", features = ["png"], default-features = false }
|
||||||
|
nanorand = { version = "0.5", features = ["std", "rdseed", "wyrand", "tls"], default-features = false }
|
||||||
|
hsl = "0.1.1"
|
||||||
|
gumdrop = "0.8"
|
||||||
|
smallvec = "1.4.1"
|
||||||
|
|
@ -0,0 +1,11 @@
|
||||||
|
# FractalArt-rs
|
||||||
|
|
||||||
|
This is [this](https://github.com/TomSmeets/FractalArt) "fractal" wallpaper thing, ported to Rust and with significant performance improvements and more configuration.
|
||||||
|
It is lacking the set-as-wallpaper feature of the original, but it should be possible to put this in a shellscript or something with whatever works on your platform.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Compile via `cargo build --release`.
|
||||||
|
The resulting binary is in `target/release/fractalart`. With no other arguments, it will generate a 1000x1000 image and save it to `out.png`. You can use `-W`/`-H`/`-o` to change that.
|
||||||
|
The initial seed color can also be configured with `-u` (hue; this will default to a randomly generated one), `-s` and `-l`. The random seed is set with `-r`; if the other parameters are the same, then the same seed will give the same image.
|
||||||
|
Images generated by this are slightly different to the original program's because of the use of 16-bit integers instead of floats (it might actually be possible to go down to `u8`s without quality problems, but I haven't tried), and also the different default variance (`-v`); the original program hardcodes this to the equivalent of 1573, but I picked 2048, as a rounder number.
|
||||||
|
|
@ -0,0 +1,140 @@
|
||||||
|
use image::{Rgb, ImageBuffer, RgbImage};
|
||||||
|
use std::cmp::{max, min};
|
||||||
|
use nanorand::{WyRand, RNG};
|
||||||
|
use gumdrop::Options;
|
||||||
|
use smallvec::{smallvec, SmallVec};
|
||||||
|
|
||||||
|
type R16Image = ImageBuffer<Rgb<u16>, Vec<u16>>;
|
||||||
|
|
||||||
|
// convert HSL to RGB, wrapping the hsl crate
|
||||||
|
fn hsl_to_rgb(h: f64, s: f64, l: f64) -> Rgb<u8> {
|
||||||
|
let (r, g, b) = (hsl::HSL { h, s, l }).to_rgb();
|
||||||
|
Rgb::from([r, g, b])
|
||||||
|
}
|
||||||
|
|
||||||
|
type IPos = (i32, i32);
|
||||||
|
|
||||||
|
// checks if a position is inside the given width/height
|
||||||
|
fn is_inside((w, h): IPos, (x, y): IPos) -> bool {
|
||||||
|
x < w && y < h && x >= 0 && y >= 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// generate a square "ring" around a point with supplied distance
|
||||||
|
// drops all values outside of grid
|
||||||
|
fn ring_at(dim: IPos, (x, y): IPos, l: i32) -> impl Iterator<Item = IPos> {
|
||||||
|
// use smallvec for higher efficiency if generating small ring for next_color
|
||||||
|
// it might be better to just have a separate codepath entirely for size-1 rings really
|
||||||
|
// or make it work as an iterator properly
|
||||||
|
let mut out: SmallVec<[_; 8]> = smallvec![];
|
||||||
|
// top and bottom of ring
|
||||||
|
for n in -l..=l {
|
||||||
|
out.push((n + x, l + y));
|
||||||
|
out.push((n + x, -l + y));
|
||||||
|
}
|
||||||
|
// sides of ring
|
||||||
|
for n in (1 - l)..l {
|
||||||
|
out.push((l + x, n + y));
|
||||||
|
out.push((-l + x, n + y));
|
||||||
|
}
|
||||||
|
out.into_iter().filter(move |c| is_inside(dim, *c))
|
||||||
|
}
|
||||||
|
|
||||||
|
// randomly increase/decrease one of the channels in a color by `range`
|
||||||
|
fn mod_channel(rng: &mut WyRand, range: u16, n: u16) -> u16 {
|
||||||
|
// to avoid conversion to signed integers here, do things of some sort
|
||||||
|
let rand: u16 = rng.generate::<u16>() % (range * 2 + 2);
|
||||||
|
let o = ((n as u32) + (rand as u32)).saturating_sub(range as u32);
|
||||||
|
// avoid weird artifacts - just directly using `as` truncates it, i.e. drops the high bytes, which leads to integer-overflow-like issues
|
||||||
|
min(o, 65535) as u16
|
||||||
|
}
|
||||||
|
|
||||||
|
// randomly adjust all the channels in a color by `range`
|
||||||
|
fn mod_color(rng: &mut WyRand, range: u16, col: Rgb<u16>) -> Rgb<u16> {
|
||||||
|
Rgb([mod_channel(rng, range, col[0]), mod_channel(rng, range, col[1]), mod_channel(rng, range, col[2])])
|
||||||
|
}
|
||||||
|
|
||||||
|
// the original Haskell implementation has a grid of booleans and colors for this
|
||||||
|
// doing that in this would probably introduce more complexity and reduce efficiency significantly, so just reserve black for uninitialized pixels
|
||||||
|
const BLANK: Rgb<u16> = Rgb([0, 0, 0]);
|
||||||
|
|
||||||
|
// get the next color to use by randomly selecting an adjacent nonblank pixel and randomizing it a bit
|
||||||
|
fn next_color(opts: &CLIOptions, rng: &mut WyRand, im: &R16Image, pos: IPos) -> Rgb<u16> {
|
||||||
|
let (w, h) = im.dimensions();
|
||||||
|
let dim = (w as i32, h as i32);
|
||||||
|
let mut colors: SmallVec<[_; 8]> = smallvec![];
|
||||||
|
for (x, y) in ring_at(dim, pos, 1) {
|
||||||
|
let px = *im.get_pixel(x as u32, y as u32);
|
||||||
|
if px != BLANK {
|
||||||
|
colors.push(px);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let chosen = colors[rng.generate_range(0, colors.len())];
|
||||||
|
mod_color(rng, opts.variance / colors.len() as u16, chosen)
|
||||||
|
}
|
||||||
|
|
||||||
|
// run an iteration by filling in all the pixels in a ring around the start position
|
||||||
|
fn iter(opts: &CLIOptions, rng: &mut WyRand, im: &mut R16Image, pos: IPos, i: i32) {
|
||||||
|
let (w, h) = im.dimensions();
|
||||||
|
let dim = (w as i32, h as i32);
|
||||||
|
|
||||||
|
for (x, y) in ring_at(dim, pos, i) {
|
||||||
|
im.put_pixel(x as u32, y as u32, next_color(opts, rng, im, (x, y)));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Options, Debug)]
|
||||||
|
struct CLIOptions {
|
||||||
|
#[options(help = "width of generated image", short = "H", default = "1000")]
|
||||||
|
width: u32,
|
||||||
|
#[options(help = "height of generated image", short = "W", default = "1000")]
|
||||||
|
height: u32,
|
||||||
|
#[options(help = "filename to save generated image to", short = "o", default = "./out.png")]
|
||||||
|
filename: String,
|
||||||
|
// required for gumdrop to provide help text
|
||||||
|
#[options(help = "display this help", short = "h")]
|
||||||
|
help: bool,
|
||||||
|
#[options(help = "max color channel difference from previous pixel", short = "v", default = "2048")]
|
||||||
|
variance: u16,
|
||||||
|
#[options(help = "base color saturation", short = "s", default = "1.0")]
|
||||||
|
saturation: f64,
|
||||||
|
#[options(help = "base color lightness", short = "l", default = "0.6")]
|
||||||
|
lightness: f64,
|
||||||
|
#[options(help = "base color hue (default: random)", short = "u")]
|
||||||
|
hue: Option<f64>,
|
||||||
|
#[options(help = "random seed (default: random)", short="r")]
|
||||||
|
seed: Option<u64>
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() {
|
||||||
|
let opts = CLIOptions::parse_args_default_or_exit();
|
||||||
|
println!("{:?}", opts);
|
||||||
|
|
||||||
|
let mut rng = match opts.seed {
|
||||||
|
Some(seed) => nanorand::WyRand::new_seed(seed),
|
||||||
|
None => nanorand::WyRand::new()
|
||||||
|
};
|
||||||
|
let w = opts.width;
|
||||||
|
let h = opts.height;
|
||||||
|
let start_x = rng.generate_range(0, w - 1);
|
||||||
|
let start_y = rng.generate_range(0, h - 1);
|
||||||
|
|
||||||
|
// generate a starting color
|
||||||
|
let hue = opts.hue.unwrap_or_else(|| rng.generate_range(0u64, 360 * 65536) as f64 / 65536.);
|
||||||
|
let start_color = hsl_to_rgb(hue, opts.saturation, opts.lightness);
|
||||||
|
let start_color = Rgb([start_color[0] as u16 * 256, start_color[1] as u16 * 256, start_color[2] as u16 * 256]);
|
||||||
|
|
||||||
|
let iterations = max(max(start_x, w - 1 - start_x), max(start_y, h - 1 - start_y)) as i32;
|
||||||
|
let mut im = R16Image::from_pixel(w, h, BLANK);
|
||||||
|
im.put_pixel(start_x, start_y, start_color);
|
||||||
|
for i in 1..=iterations {
|
||||||
|
iter(&opts, &mut rng, &mut im, (start_x as i32, start_y as i32), i);
|
||||||
|
}
|
||||||
|
|
||||||
|
// discard low bits of image pixels before saving, as monitors mostly can't render these and it wastes space
|
||||||
|
let low_color_depth_image = RgbImage::from_fn(w, h, |x, y| {
|
||||||
|
let Rgb([r, g, b]) = im.get_pixel(x, y);
|
||||||
|
Rgb([(r >> 8) as u8, (g >> 8) as u8, (b >> 8) as u8])
|
||||||
|
});
|
||||||
|
|
||||||
|
low_color_depth_image.save(&opts.filename).expect("failed to save image");
|
||||||
|
}
|
||||||
Loading…
Reference in New Issue