copyedits, tag system, new post(s)

blog/autocrafting-algorithms.md

@@ -0,0 +1,144 @@
+---
+title: Autocrafting algorithms
+description: How it's made (automatically) (in computer games with Minecraft-like recipe systems).
+created: 18/05/2025
+slug: craft
+highlightCode: true
+tags: ["fiction", "games", "maths"]
+---
+I have previously [written](/mcpower/#logistics) briefly about on-demand autocrafting systems in Minecraft, which radically restructured ingame industry by allowing flexible timeshared allocation of machines. Rather than every item being made and stockpiled in large quantity on a dedicated production line, as in Factorio, or produced manually, they are made in response to user requests. An autocrafting-capable base can get away with a much smaller set of machines - one or a few of each kind, regardless of the number of item types being produced, as well as a few extra systems where autocrafting doesn't work or make sense. Controlling such a system is a harder problem than it first appears, however.
+
+## Basic definition
+
+Given:
+
+- a set of valid items
+- a "multiset/bag" (a set where items can be repeated, or an unsorted list) of items which are currently in storage
+- a partial function mapping (some) items to a multiset (the input) and natural number (the quantity), i.e. crafting recipes - the inputs are consumed, producing the given quantity of the specified item
+- an item and natural number (target item and quantity to produce)
+
+we want to produce an autocrafting plan which will make the target item in the target quantity. Specifically, we want a sequence of recipes (identified by output items) and quantities (to run the recipes in)[^1] such that, when we start from the initial contents of storage and sequentially substitute each recipe's inputs for its outputs, each substitution is valid (we always have nonnegative quantities of items) and the final state of storage contains the target item in the target quantity.
+
+If solutions exist, it is quite easy to find one, using backward-chaining/depth first search. [Dragon](https://github.com/osmarks/dragon), my old storage system, has a simple, bad implementation which, in retrospect, probably should not actually work. I wrote a [cleaner Python implementation](https://github.com/osmarks/misc/blob/master/autocrafter.py)[^3]:
+
+```python
+def solve(item: str, quantity: int, inventory: Inventory, use_callback, craft_callback) -> Inventory:
+    directly_available = min(inventory[item], quantity) # Consume items from storage if available
+    if directly_available > 0:
+        use_callback(item, directly_available)
+    inventory = inventory.take(item, directly_available)
+    quantity -= directly_available
+
+    if quantity > 0:
+        if recipe := recipes.get(item):
+            recipe_runs = math.ceil(quantity / recipe.quantity)
+            for citem, cquantity in Counter(recipe.slots).items():
+                if citem is not None:
+                    inventory = solve(citem, recipe_runs * cquantity, inventory, use_callback, craft_callback) # Recurse into subrecipe
+            craft_callback(item, recipe_runs)
+            inventory = inventory.add(item, recipe_runs * recipe.quantity - quantity) # Add spare items to tracked inventory
+        else:
+            raise NoRecipe(item, quantity) # We need to make this and can't
+
+    return inventory
+```
+
+::: captioned src=/assets/images/basic_autocraft.png
+A simplified diagram of the ways to make ComputerCraft "turtle" robots. While there are multiple inputs at several steps, they all have to be used together.
+:::
+
+When it needs an item, it takes it from storage if possible, and otherwise it recursively crafts as many as it needs. Simple, clean, reasonably efficient in practice[^2] - what more could you want? As it turns out, several things.
+
+## Extending the problem
+
+We made a very large simplification in the previous definition: each item has *at most one* recipe to produce it. The algorithm has almost no choices to make - in principle, it could redundantly craft things it already has or do things in different orders, but these are not very interesting differences. But this isn't true in a real modded playthrough: at the very least, [OreDictionary](https://docs.minecraftforge.net/en/1.12.x/utilities/oredictionary/) (and now the [tag system](https://minecraft.wiki/w/Item_tag_(Java_Edition))) means that you can substitute different varieties of "the same" material for each other - for instance, almost everything which needs "a chest" can be made with oak, acacia, spruce, birch, jungle, etc. This seems trivial, but in larger modpacks you can often substitute equivalent tiers of separate mods' intermediate crafting components for each other despite wildly different recipes.
+
+Considering multiple alternatives at each step, and the stateful inventory tracking preventing (obvious methods of) caching branches, means that [autocrafting is NP-hard](https://squiddev.github.io/ae-sat/) (wrt. recipe count, I think)[^4] if we permit multiple recipes per item, through reduction of [SAT](https://en.wikipedia.org/wiki/Boolean_satisfiability_problem) to autocrafting, though real-world systems (<span class="hoverdefn" title="Applied Energistics 2">AE2</span> and <span class="hoverdefn" title="Refined Storage">RS</span>) don't have general enough solvers for all instances. As far as I can tell, the [AE2 crafting solver](https://github.com/AppliedEnergistics/Applied-Energistics-2/blob/70838d74f81917ece64335db2aca3cb3a604b780/src/main/java/appeng/crafting/CraftingTreeNode.java#L167) uses a greedy algorithm - this could return technically-avoidable failures if a branch which is tried earlier consumes resources a later branch could have used more efficiently. [Refined Storage](https://github.com/refinedmods/refinedstorage2/blob/accdcf28c52752c1512c16b97efad7b59ab3b340/refinedstorage-autocrafting-api/src/main/java/com/refinedmods/refinedstorage/api/autocrafting/calculation/CraftingTree.java) appears to work the same way.
+
+::: captioned src=/assets/images/complex_autocraft.png
+If we also introduce the ability to make chests from birch wood, add smelting recipes and add the turtle to advanced turtle upgrade recipe, we have this more complex example.
+:::
+
+The greedy algorithm is, in practice, fine (in terms of returning solutions where they exist). Game progression is such that by the time it might become a problem, you have far more basic resources than you need, so the naive solver will work[^6]. We could stop here. But that isn't fun. Instead, we can make a general solution by accepting the problem's NP-hardness and running the problem through a general integer linear programming solver. I considered using an [SMT](https://en.wikipedia.org/wiki/Satisfiability_modulo_theories) solver as [previously proposed](https://borretti.me/fiction/eog581/the-lunar-surface), but with the preprocessing necessary for it to be tractable this is unnecessarily general. By performing a topological sort on the relevant section of the recipe graph[^5] to generate an ordered list of items to (possibly) make, substituting in all the possible recipes and solving for the number of invocations of each recipe under the constraint that we have positive quantities of every item and a sufficient quantity of the target item, we can find an autocrafting plan:
+
+```python
+def solve_ilp(item: str, quantity: int, inventory: Inventory, use_callback, craft_callback):
+    sequence = list(topo_sort_inputs(item))
+
+    recipe_steps = []
+
+    # Rewrite (involved) recipes as production/consumption numbers.
+    items = { sitem: [] for sitem in sequence }
+
+    for item in sequence:
+        for recipe in recipes_general[item]:
+            step_changes = { sitem: 0 for sitem in sequence }
+            for citem, cquantity in Counter(recipe.slots).items():
+                if citem is not None:
+                    step_changes[citem] = -cquantity
+            step_changes[item] = recipe.quantity
+
+            recipe_steps.append((item, recipe))
+
+            for sitem, coef in step_changes.items():
+                items[sitem].append(coef)
+
+    objective = np.ones(len(recipe_steps))
+    # The amount of each item we produce/consume is linearly dependent on how many times each recipe is executed.
+    # This matrix is that linear transform.
+    # Solver wants upper bounds so flip signs.
+    production_matrix = -np.stack([np.array(coefs) for item, coefs in items.items()])
+    # production_matrix @ x is the vector of item consumption, so we upper-bound that with inventory item counts
+    # and require that we produce the required output (negative net consumption)
+    item_constraint_vector = np.array([ -quantity + inventory[i] if i == item else inventory[i] for i in sequence ])
+
+    soln = opt.linprog(objective, integrality=np.ones_like(objective), A_ub=production_matrix, b_ub=item_constraint_vector)
+
+    match soln.status:
+        case 0:
+            print("OK")
+            # soln.x is now the number of times to execute each recipe_step
+            item_consumption = production_matrix @ soln.x
+            for item_name, consumption in zip(sequence, item_consumption):
+                consumption = int(consumption)
+                if consumption > 0:
+                    use_callback(item_name, consumption)
+                inventory = inventory.take(item_name, consumption)
+            for (recipe_output, recipe_spec), execution_count in zip(recipe_steps, soln.x):
+                execution_count = int(execution_count)
+                if execution_count > 0:
+                    craft_callback(recipe_spec, recipe_output, execution_count)
+            return inventory
+        case 1:
+            print("iteration limit reached")
+            raise NoRecipe
+        case 2:
+            print("infeasible")
+            raise NoRecipe
+```
+
+This is not very efficient - it globally optimizes over the whole relevant subtree at once - but it can find solutions greedy search cannot, and is more easily adapted to multi-output recipes.
+
+## Subjectivity and other limitations
+
+The problem is still not over. For one thing, the solver ignores cycles, where recipes (indirectly) contain themselves, because it can't topologically sort them, though you could likely adapt it quite easily since the ILP side doesn't care about order. More importantly, as an [AE2 developer describes](https://www.reddit.com/r/feedthebeast/comments/7t8v0o/autocrafting_is_npcomplete/dtbgkz9/), when there are multiple ways to make something there are subjective tradeoffs to make: is it better to consume slightly fewer items but use more machine time? What if avoiding extra machine time consumes an expensive resource? What if a large job is running and, without consuming this item, the user will have to wait tens of minutes? Since the algorithm I describe is ILP-based, it admits and indeed requires an objective function (where a SAT-based one wouldn't) - in my code, it minimizes crafting steps, but it could be replaced with one which cares about time and resource consumption. However, while it could use this information, it doesn't have it.
+
+We can easily imagine tracking the time taken to run a recipe on a machine automatically, and perhaps having a "priority" switch to bump lower-priority jobs and switch to faster operations, but it can't easily know how valued an input is (though several heuristics are plausible), and many, such as power draw (or sometimes fluid inputs), are not known as part of crafting patterns (they could in principle be added, but this would be inconvenient). AE2 "solves" this by having users manually configure every recipe they want to use, but this is a large time sink and does not solve the problem in cases where users do actually want to use multiple recipes in different circumstances.
+
+There is a further issue: randomness. Sometimes - mostly with "raw material" operations such as pulverizing ores or growing crops - outputs are produced nondeterministically some fraction of the time. This makes exact forward planning of recipe execution completely impossible, though in principle you could assume, say, the 95th percentile cost and rerun the random part until it works. In practice, most people run these as part of "always-on" systems which build up stockpiles (e.g. [this](https://guide.appliedenergistics.org/1.20.4/example-setups/recursive-crafting-setup)), or assume that the recipe will run enough as part of deterministic queries.
+
+Does this have any other practical applications? Not that I know of - I am not sure this even turns up in games outside of modded Minecraft[^7]. This is apparently related to [Petri](https://en.wikipedia.org/wiki/Petri_net) [nets](https://isr.umd.edu/Labs/CIM/miscs/wmsor97.pdf), though I haven't paid enough attention to them to see why, as well as (much more obviously) to [vector addition systems](https://en.wikipedia.org/wiki/Vector_addition_system) and [commutative](https://www.sciencedirect.com/science/article/pii/S0019995883800229) [grammars](/assets/misc/commutative_grammars.pdf).
+
+[^1]: We could equivalently repeat steps rather than giving them a quantity, but real implementations don't usually do that, for efficiency.
+
+[^2]: You can imagine pathological cases in which it does much more computation than necessary because the same item is used repeatedly in different parts of the tree and the code makes no attempt to deduplicate here. In practice I don't think the trees get wide enough that this is a significant issue, and I don't see an obvious algorithm to coalesce repeats to fix it without already having done most of the traversal.
+
+[^3]: I wanted to make it a generator rather than using the callbacks, but there wasn't an obvious way to do that and pass the inventories around.
+
+[^4]: Less-technical discussions of this usually imply that if something is NP-hard it's intractable in general. NP-hardness means, loosely, that very difficult instances of a problem can be constructed, not that all instances are difficult. See [Wikipedia](https://en.wikipedia.org/wiki/P_versus_NP_problem).
+
+[^5]: Technically, it's not a graph, because recipes don't map to a single item to item edge. The topological sort procedure uses a graph such that if any recipe for A contains B, there is an edge from B to A.
+
+[^6]: And recipes tend not to hit corner cases anyway.
+
+[^7]: Nothing else that I know of has the same range of items to make, general-purpose crafting machines and automation.

blog/number-distribution.md

@@ -0,0 +1,101 @@
+---
+title: On the empirical distribution of numbers
+description: At last, data-driven numerology.
+created: 02/05/2025
+slug: number
+katex: true
+tags: ["maths", "own tech"]
+---
+::: epigraph link=https://x.com/1thousandfaces_/status/1828105487411544170 attribution=@1thousandfaces_
+What are some good numbers for someone just starting to get into math?
+:::
+
+You've probably heard of [Benford's Law](https://en.wikipedia.org/wiki/Benford%27s_law) - the nonuniform empirical distribution of the first digits of "real-world numbers". While thinking about [other things](https://gwern.net/oen), I realized that it would be very easy to investigate this more generally by counting all instances of "numbers" within [a mid-sized LLM training dataset](https://pile.eleuther.ai/).
+
+I did this, using a somewhat complex [state-machine parser](https://github.com/osmarks/misc/blob/master/empirical-number-distribution/src/main.rs#L18) for "numbers" which attempts to correctly interpret numbers containing commas and dots and percentages, or separated by dashes, but which does *not* handle scientific notation or exponents[^1]. You can get the resulting number data [here](https://datasets.osmarks.net/counts.csv); I have run some analyses myself, described below. The lack of scientific notation parsing likely cuts off lots of the top end, and the URLs in the dataset probably skew it too.
+
+Run over the entire Pile training set, my code extracts 60633452 unique numbers (6.1e7)[^3], and 12598942216 (1.2e10) total instances. Firstly, here are the 30 most common numbers. You can see that 1 makes up about 10% of all numbers, with 2 very close behind[^2]. Only one negative number makes it into these rarefied heights, and we can see that "round numbers" are more salient than merely small numbers.
+
+<div class="numeric-table">
+
+| number | count      |
+|--------|------------|
+| 1      | 1285091487 |
+| 2      | 1245931860 |
+| 0      | 759996917  |
+| 3      | 658078393  |
+| 4      | 442882667  |
+| 5      | 342835723  |
+| 6      | 246669458  |
+| 8      | 207732046  |
+| 10     | 204988418  |
+| 7      | 195558391  |
+| 9      | 161918480  |
+| 12     | 129697054  |
+| 11     | 111310452  |
+| 20     | 101514897  |
+| 15     | 91434782   |
+| 16     | 87265477   |
+| 13     | 79321181   |
+| 14     | 75491719   |
+| 30     | 72532266   |
+| -1     | 70044908   |
+| 18     | 69886735   |
+| 17     | 62675170   |
+| 32     | 59691749   |
+| 24     | 58510498   |
+| 19     | 57800365   |
+| 25     | 55035188   |
+| 21     | 54834586   |
+| 100    | 50744706   |
+| 22     | 47568552   |
+| 50     | 47307684   |
+
+</div>
+
+We might expect the distribution to be Zipfian, i.e. $\mathrm{frequency} \propto \frac{1}{\mathrm{rank}}$, although this doesn't hold for the top two numbers. I find that it's not: on a log/log plot, the best-fit line has a substantially steeper gradient than Zipf's law predicts. I don't know how to fit a Zipf-Mandelbrot law $\mathrm{frequency} \propto \frac{1}{(\mathrm{rank}+b)^a}$ with the $b$ parameter, so I haven't.
+
+::: captioned src=/assets/images/top_1000_numbers.png
+It's closer to Zipfian for the top thousand numbers.
+:::
+::: captioned src=/assets/images/top_10000_numbers.png
+It intuitively feels like the line should be shallower here, but the points are much denser toward the right.
+:::
+::: captioned src=/assets/images/top_100000_numbers.png
+No significant change here.
+:::
+
+I also analyzed the number of numbers with numbers of occurrences falling into exponentially distributed bins:
+
+::: captioned src=/assets/images/number_freq_freq.png
+The majority of seen numbers are seen very few times. I think the straight trend here implies number frequency is [Pareto-distributed](https://en.wikipedia.org/wiki/Pareto_distribution).
+:::
+
+We can also look at properties of the numbers themselves, rather than just their frequencies, since they're less opaque than words. The most obvious one is their size (absolute value). Below $10^0$ (1), the results are somewhat unreliable, because percentages are parsed but not other units or fractions or scientific notation. Regardless:
+
+::: captioned src=/assets/images/number_size_histogram.png
+I am not sure what causes the spikiness - possibly numerical issues.
+:::
+
+By sorting the numbers, I also determined that the median number is 7, plus or minus some roundoff error (conversion to 64-bit floating points loses some precision over arbitrarily long decimal strings). I also have the frequency of small integers (0 to 100) and some plausible year numbers.
+
+::: captioned src=/assets/images/small_numbers_frequency.png
+Spikes mostly correspond to "round numbers" in base 10 or 2 (marked with x-axis ticks).
+:::
+::: captioned src=/assets/images/years_frequency.png
+The dataset was compiled in 2020, and I suppose it contains less forward-looking content than backward-looking.
+:::
+
+Finally, first digits. I get results quite close to Benford's law across this dataset, though not a perfect fit. This is discarding anything which begins with 0, though.
+
+::: captioned src=/assets/images/benford.png
+"Noninteger" means I excluded every number without a `.` indicating fractional part.
+:::
+
+In the future, it might be "useful" to see how well you can predict number popularity with a linear model based on (some transforms of) magnitude, sign, number of trailing zeroes and first digit.
+
+[^1]: These have very nonstandard formats. I don't know how to do this without writing hundreds of separate regexes. Already the parser is convoluted enough due to trying to normalize numbers.
+
+[^2]: In some sense this is undercounting because "a" and "the" refer to one thing as well.
+
+[^3]: 1% is counted as separate from 0.01 for reasons, so this is a slight overestimate.

src/dark-theme-highlight.css

@@ -0,0 +1,164 @@
+pre code.hljs {
+  display: block;
+  overflow-x: auto;
+  padding: 1em
+}
+code.hljs {
+  padding: 3px 5px
+}
+/*!
+  Theme: PaperColor Dark
+  Author: Jon Leopard (http://github.com/jonleopard) based on PaperColor Theme (https://github.com/NLKNguyen/papercolor-theme)
+  License: ~ MIT (or more permissive) [via base16-schemes-source]
+  Maintainer: @highlightjs/core-team
+  Version: 2021.09.0
+*/
+/*
+  WARNING: DO NOT EDIT THIS FILE DIRECTLY.
+
+  This theme file was auto-generated from the Base16 scheme papercolor-dark
+  by the Highlight.js Base16 template builder.
+
+  - https://github.com/highlightjs/base16-highlightjs
+*/
+/*
+base00  #111     Default Background
+base01  #af005f  Lighter Background (Used for status bars, line number and folding marks)
+base02  #5faf00  Selection Background
+base03  #d7af5f  Comments, Invisibles, Line Highlighting
+base04  #5fafd7  Dark Foreground (Used for status bars)
+base05  #ccc     Default Foreground, Caret, Delimiters, Operators
+base06  #d7875f  Light Foreground (Not often used)
+base07  #d0d0d0  Light Background (Not often used)
+base08  #585858  Variables, XML Tags, Markup Link Text, Markup Lists, Diff Deleted
+base09  #5faf5f  Integers, Boolean, Constants, XML Attributes, Markup Link Url
+base0A  #afd700  Classes, Markup Bold, Search Text Background
+base0B  #af87d7  Strings, Inherited Class, Markup Code, Diff Inserted
+base0C  #ffaf00  Support, Regular Expressions, Escape Characters, Markup Quotes
+base0D  #ff5faf  Functions, Methods, Attribute IDs, Headings
+base0E  #00afaf  Keywords, Storage, Selector, Markup Italic, Diff Changed
+base0F  #5f8787  Deprecated, Opening/Closing Embedded Language Tags, e.g. <?php ?>
+*/
+pre code.hljs {
+  display: block;
+  overflow-x: auto;
+  padding: 1em;
+  border: 1px solid var(--box-border);
+}
+code.hljs {
+  padding: 3px 5px
+}
+.hljs {
+  color: #ccc;
+  background: #111
+}
+.hljs::selection,
+.hljs ::selection {
+  background-color: #5faf00;
+  color: #ccc
+}
+/* purposely do not highlight these things */
+.hljs-formula,
+.hljs-params,
+.hljs-property {
+
+}
+/* base03 - #d7af5f -  Comments, Invisibles, Line Highlighting */
+.hljs-comment {
+  color: #d7af5f
+}
+/* base04 - #5fafd7 -  Dark Foreground (Used for status bars) */
+.hljs-tag {
+  color: #5fafd7
+}
+/* base05 - #ccc -  Default Foreground, Caret, Delimiters, Operators */
+.hljs-subst,
+.hljs-punctuation,
+.hljs-operator {
+  color: #ccc
+}
+.hljs-operator {
+  opacity: 0.7
+}
+/* base08 - Variables, XML Tags, Markup Link Text, Markup Lists, Diff Deleted */
+.hljs-bullet,
+.hljs-variable,
+.hljs-template-variable,
+.hljs-selector-tag,
+.hljs-name,
+.hljs-deletion {
+  color: #585858
+}
+/* base09 - Integers, Boolean, Constants, XML Attributes, Markup Link Url */
+.hljs-symbol,
+.hljs-number,
+.hljs-link,
+.hljs-attr,
+.hljs-variable.constant_,
+.hljs-literal {
+  color: #5faf5f
+}
+/* base0A - Classes, Markup Bold, Search Text Background */
+.hljs-title,
+.hljs-class .hljs-title,
+.hljs-title.class_ {
+  color: #afd700
+}
+.hljs-strong {
+  font-weight: bold;
+  color: #afd700
+}
+/* base0B - Strings, Inherited Class, Markup Code, Diff Inserted */
+.hljs-code,
+.hljs-addition,
+.hljs-title.class_.inherited__,
+.hljs-string {
+  color: #af87d7
+}
+/* base0C - Support, Regular Expressions, Escape Characters, Markup Quotes */
+/* guessing */
+.hljs-built_in,
+.hljs-doctag,
+.hljs-quote,
+.hljs-keyword.hljs-atrule,
+.hljs-regexp {
+  color: #ffaf00
+}
+/* base0D - Functions, Methods, Attribute IDs, Headings */
+.hljs-function .hljs-title,
+.hljs-attribute,
+.ruby .hljs-property,
+.hljs-title.function_,
+.hljs-section {
+  color: #ff5faf
+}
+/* base0E - Keywords, Storage, Selector, Markup Italic, Diff Changed */
+/* .hljs-selector-id, */
+/* .hljs-selector-class, */
+/* .hljs-selector-attr, */
+/* .hljs-selector-pseudo, */
+.hljs-type,
+.hljs-template-tag,
+.diff .hljs-meta,
+.hljs-keyword {
+  color: #00afaf
+}
+.hljs-emphasis {
+  color: #00afaf;
+  font-style: italic
+}
+/* base0F - Deprecated, Opening/Closing Embedded Language Tags, e.g. <?php ?> */
+/*
+  prevent top level .keyword and .string scopes
+  from leaking into meta by accident
+*/
+.hljs-meta,
+.hljs-meta .hljs-keyword,
+.hljs-meta .hljs-string {
+  color: #5f8787
+}
+/* for v10 compatible themes */
+.hljs-meta .hljs-keyword,
+.hljs-meta-keyword {
+  font-weight: bold
+}

src/global.json

@@ -43,7 +43,7 @@
         "Overcoming Bias": "https://www.overcomingbias.com/feed",
         "Construction Physics": "https://www.construction-physics.com/feed",
         "Factorio": "https://www.factorio.com/blog/rss",
-        "The Eldraeverse": "https://eldraeverse.com/feed/",
+        "The Eldraeverse": "https://eldraeverse.com/rss/",
         "ServeTheHome": "https://www.servethehome.com/feed/",
         "Graphcore Research": "https://graphcore-research.github.io/feed.xml",
         "ToughSF": "https://toughsf.blogspot.com/rss.xml",
@@ -58,7 +58,10 @@
         "Mythic Beasts": "https://www.mythic-beasts.com/blog/feed/",
         "Tales from the Void": "https://randomsprint.substack.com/feed",
         "Max Barry": "https://maxbarry.com/index.rss",
-        "Real World Tech": "https://www.realworldtech.com/feed/"
+        "Real World Tech": "https://www.realworldtech.com/feed/",
+        "Entropic Thoughts": "https://entropicthoughts.com/feed",
+        "phils web site": "https://phils-web-site.net/rss.xml",
+        "Sixty Degrees North": "https://sixtydegreesnorth.substack.com/feed.xml"
     },
     "dateFormat": "YYYY-MM-DD",
     "microblogSource": "https://b.osmarks.net/outbox",
@@ -92,5 +95,14 @@
         ["rhombic_dodecahedron.gif", "https://en.wikipedia.org/wiki/Rhombic_dodecahedron"],
         ["zeroptr.gif", "https://zptr.cc/88x31/"]
     ],
-    "mycorrhiza": "https://docs.osmarks.net"
+    "mycorrhiza": "https://docs.osmarks.net",
+    "tagColors": {
+        "ai": "#703be7",
+        "hardware": "#5e819d",
+        "fiction": "#ff028d",
+        "economics": "#15b01a",
+        "opinion": "#fdaa48",
+        "own tech": "#04d8b2",
+        "maths": "#f8481c"
+    }
 }

src/index.js

@@ -32,6 +32,7 @@ const pLimit = require("p-limit")
 const json5 = require("json5")
 const readability = require("@mozilla/readability")
 const { JSDOM } = require("jsdom")
+const hljs = require("highlight.js")
 
 const fts = require("./fts.mjs")
 
@@ -164,12 +165,13 @@ const renderContainer = (tokens, idx) => {
             options[k] += " " + arg
         } else {
             [k, ...vs] = arg.split("=")
+            let vEmpty = vs.length == 0
             v = vs.join("=")
             if (v && v[0] == '"') {
                 inQuotes = true
                 v = v.slice(1)
             }
-            options[k] = v ?? true
+            options[k] = vEmpty ? true : v ?? true
         }
     }
 
@@ -215,7 +217,13 @@ const md = new MarkdownIt({
         html: true,
         highlight: (code, language) => {
             if (language === "squiggle") {
-                return `<textarea class="squiggle" rows=${code.split("\n").length}>${md.utils.escapeHtml(code.trim())}</textarea>`
+                return `<textarea class=squiggle rows=${code.split("\n").length}>${md.utils.escapeHtml(code.trim())}</textarea>`
+            }
+            if (language && hljs.getLanguage(language)) {
+                try {
+                    const hl = hljs.highlight(code, { language }).value
+                    return `<pre class=wider><code class="hljs">${hl}</code></pre>`
+                } catch(e) {}
             }
             return "" // default escaping
         }
@@ -320,6 +328,16 @@ const applyTemplate = async (template, input, getOutput, options = {}) => {
 
 const addGuids = R.map(x => ({ ...x, guid: uuid.v5(`${x.lastUpdate}:${x.slug}`, "9111a1fc-59c3-46f0-9ab4-47c607a958f2") }))
 
+const processTags = meta => {
+    meta.accentColor = null
+    for (const tag of meta.tags ?? []) {
+        if (globalData.tagColors[tag]) {
+            meta.accentColor = globalData.tagColors[tag]
+            break
+        }
+    }
+}
+
 const processExperiments = async () => {
     const templates = globalData.templates
     const experiments = await loadDir(experimentDir, (subdirectory, basename) => {
@@ -336,6 +354,7 @@ const processExperiments = async () => {
                     }
                 }))
                 const indexPath = path.join(out, "index.html")
+                processTags(page.data)
                 fts.pushEntry("experiment", {
                     url: "/" + page.data.slug,
                     title: page.data.title,
@@ -360,6 +379,7 @@ const processBlog = async () => {
         meta.slug = meta.slug || removeExtension(basename)
         meta.wordCount = page.content.split(/\s+/).map(x => x.trim()).filter(x => x).length
         meta.haveSidenotes = true
+        processTags(meta)
         const [html, urls] = renderMarkdown(page.content)
         meta.content = html
         meta.references = []
@@ -593,6 +613,24 @@ const compileCSS = async () => {
     css += "\n"
     css += await fsp.readFile(path.join(srcDir, "comments.css"))
     globalData.css = css
+
+    const lightThemeHighlight = await fsp.readFile(path.join(srcDir, "light-theme-highlight.css"))
+    const darkThemeHighlight = await fsp.readFile(path.join(srcDir, "dark-theme-highlight.css"))
+
+    // abuse SASS compiler to do minification
+    const highlightCSS = sass.compileString(`
+${lightThemeHighlight}
+
+@media (prefers-color-scheme: dark) {
+    ${darkThemeHighlight}
+}
+    `, {
+        style: "compressed",
+        indentedSyntax: false,
+        charset: false
+    })
+
+    await fsp.writeFile(path.join(outAssets, "highlight.min.css"), highlightCSS.css)
 }
 
 const loadTemplates = async () => {

src/light-theme-highlight.css

@@ -0,0 +1,164 @@
+pre code.hljs {
+  display: block;
+  overflow-x: auto;
+  padding: 1em
+}
+code.hljs {
+  padding: 3px 5px
+}
+/*!
+  Theme: PaperColor Light
+  Author: Jon Leopard (http://github.com/jonleopard) based on PaperColor Theme (https://github.com/NLKNguyen/papercolor-theme)
+  License: ~ MIT (or more permissive) [via base16-schemes-source]
+  Maintainer: @highlightjs/core-team
+  Version: 2021.09.0
+*/
+/*
+  WARNING: DO NOT EDIT THIS FILE DIRECTLY.
+
+  This theme file was auto-generated from the Base16 scheme papercolor-light
+  by the Highlight.js Base16 template builder.
+
+  - https://github.com/highlightjs/base16-highlightjs
+*/
+/*
+base00  #eeeeee  Default Background
+base01  #af0000  Lighter Background (Used for status bars, line number and folding marks)
+base02  #00ee00  Selection Background
+base03  #5f8700  Comments, Invisibles, Line Highlighting
+base04  #0087af  Dark Foreground (Used for status bars)
+base05  #000000  Default Foreground, Caret, Delimiters, Operators
+base06  #005f87  Light Foreground (Not often used)
+base07  #878787  Light Background (Not often used)
+base08  #bcbcbc  Variables, XML Tags, Markup Link Text, Markup Lists, Diff Deleted
+base09  #d70000  Integers, Boolean, Constants, XML Attributes, Markup Link Url
+base0A  #d70087  Classes, Markup Bold, Search Text Background
+base0B  #8700af  Strings, Inherited Class, Markup Code, Diff Inserted
+base0C  #a83c09  Support, Regular Expressions, Escape Characters, Markup Quotes
+base0D  #a83c09  Functions, Methods, Attribute IDs, Headings
+base0E  #005faf  Keywords, Storage, Selector, Markup Italic, Diff Changed
+base0F  #005f87  Deprecated, Opening/Closing Embedded Language Tags, e.g. <?php ?>
+*/
+pre code.hljs {
+  display: block;
+  overflow-x: auto;
+  padding: 1em;
+  border: 1px solid var(--box-border);
+}
+code.hljs {
+  padding: 3px 5px
+}
+.hljs {
+  color: #000000;
+  background: #eeeeee
+}
+.hljs::selection,
+.hljs ::selection {
+  background-color: #00ee00;
+  color: #000000
+}
+/* purposely do not highlight these things */
+.hljs-formula,
+.hljs-params,
+.hljs-property {
+
+}
+/* base03 - #5f8700 -  Comments, Invisibles, Line Highlighting */
+.hljs-comment {
+  color: #5f8700
+}
+/* base04 - #0087af -  Dark Foreground (Used for status bars) */
+.hljs-tag {
+  color: #0087af
+}
+/* base05 - #000000 -  Default Foreground, Caret, Delimiters, Operators */
+.hljs-subst,
+.hljs-punctuation,
+.hljs-operator {
+  color: #000000
+}
+.hljs-operator {
+  opacity: 0.7
+}
+/* base08 - Variables, XML Tags, Markup Link Text, Markup Lists, Diff Deleted */
+.hljs-bullet,
+.hljs-variable,
+.hljs-template-variable,
+.hljs-selector-tag,
+.hljs-name,
+.hljs-deletion {
+  color: #bcbcbc
+}
+/* base09 - Integers, Boolean, Constants, XML Attributes, Markup Link Url */
+.hljs-symbol,
+.hljs-number,
+.hljs-link,
+.hljs-attr,
+.hljs-variable.constant_,
+.hljs-literal {
+  color: #d70000
+}
+/* base0A - Classes, Markup Bold, Search Text Background */
+.hljs-title,
+.hljs-class .hljs-title,
+.hljs-title.class_ {
+  color: #d70087
+}
+.hljs-strong {
+  font-weight: bold;
+  color: #d70087
+}
+/* base0B - Strings, Inherited Class, Markup Code, Diff Inserted */
+.hljs-code,
+.hljs-addition,
+.hljs-title.class_.inherited__,
+.hljs-string {
+  color: #8700af
+}
+/* base0C - Support, Regular Expressions, Escape Characters, Markup Quotes */
+/* guessing */
+.hljs-built_in,
+.hljs-doctag,
+.hljs-quote,
+.hljs-keyword.hljs-atrule,
+.hljs-regexp {
+  color: #a83c09
+}
+/* base0D - Functions, Methods, Attribute IDs, Headings */
+.hljs-function .hljs-title,
+.hljs-attribute,
+.ruby .hljs-property,
+.hljs-title.function_,
+.hljs-section {
+  color: #a83c09
+}
+/* base0E - Keywords, Storage, Selector, Markup Italic, Diff Changed */
+/* .hljs-selector-id, */
+/* .hljs-selector-class, */
+/* .hljs-selector-attr, */
+/* .hljs-selector-pseudo, */
+.hljs-type,
+.hljs-template-tag,
+.diff .hljs-meta,
+.hljs-keyword {
+  color: #005faf
+}
+.hljs-emphasis {
+  color: #005faf;
+  font-style: italic
+}
+/* base0F - Deprecated, Opening/Closing Embedded Language Tags, e.g. <?php ?> */
+/*
+  prevent top level .keyword and .string scopes
+  from leaking into meta by accident
+*/
+.hljs-meta,
+.hljs-meta .hljs-keyword,
+.hljs-meta .hljs-string {
+  color: #005f87
+}
+/* for v10 compatible themes */
+.hljs-meta .hljs-keyword,
+.hljs-meta-keyword {
+  font-weight: bold
+}

src/links.json

@@ -606,5 +606,24 @@
             "accel": ""
        },
        excerpt: "The software brained answer is that for precision insensitive workloads, you effectively double your bandwidth from caches/device mem/host mem. But there's a lot more to it."
-    }
+    },
+    "https://www.hc33.hotchips.org/assets/program/conference/day2/HC2021.DESRES.AdamButts.v03.pdf": {
+        title: "The ANTON 3 ASIC: a Fire-Breathing Monster for Molecular Dynamics Simulation",
+        author: "DE Shaw Research",
+        date: "2021-08-24"
+    },
+    "https://datasets.osmarks.net/counts.csv": {
+        title: "Pile Number Frequency Data",
+        website: "osmarks datasets"
+    },
+    "https://isr.umd.edu/Labs/CIM/miscs/wmsor97.pdf": {
+        title: "Petri Nets: Tutorial and Applications",
+        author: ["Jeffrey W. Herrmann", "Edward Lin"],
+        date: "1997-11-05"
+    },
+    "/assets/misc/commutative_grammars.pdf": {
+        title: "Commutative Grammars",
+        author: ["S. Crespi-Reghizzi", "D. Mandrioli"],
+        date: "1975-05-05"
+    },
 }

src/style.sass

@@ -7,12 +7,12 @@ $navbar-width: 20rem
     box-sizing: border-box
 
 :root
+    // worth considering: #d9facc?
     --box-bg: #d0faab
     --box-border: black
     // xkcd bluegreen
     --link: #017a79
-    // xkcd dark blue green
+    --visited-link: #013c7a
-    --visited-link: #005249
     --block-lightness: 85%
     --overlay-opacity: 0.8
 
@@ -330,8 +330,7 @@ $hl-border: 3px
         --box-border: #004020
         // bright teal
         --link: #01f9c6
-        // aqua green
+        --visited-link: #01f9f9
-        --visited-link: #12e193
         --block-lightness: 20%
         --overlay-opacity: 0.5
 
@@ -447,11 +446,21 @@ table
     background: #b1d1fc
     background: var(--box-bg)
     padding: 12px
+    position: relative
     .title
         font-size: 1.1em
         font-family: $headers
         font-weight: 600
 
+.box::after
+    top: 0
+    right: 0
+    height: 100%
+    width: 6px
+    position: absolute
+    content: ""
+    background: var(--stripe, transparent)
+
 #citebox
     width: 100%
 
@@ -493,3 +502,7 @@ textarea.squiggle
     .excerpt
         font-size: 0.9em
         font-style: italic
+
+
+.numeric-table
+    text-align: right