new blog post and redo typography

blog/about.md

@@ -16,7 +16,7 @@ Friend me on [Project Euler](https://projecteuler.net/friends)! My code is `1997
 
 ## My work
 
-* [Meme Search Engine](https://github.com/osmarks/meme-search-engine), probably still the world's most powerful meme search engine due to lack of credible competitors. Now with automatic harvesting pipeline and [scoring model](/memethresher/)!
+* [Meme Search Engine](https://github.com/osmarks/meme-search-engine)/[Nooscope](https://nooscope.osmarks.net), state-of-the art meme (and other image) search technology. [Automatic quality scoring](/memethresher/) and [230-million-item index](/memescale/)!
 * [4D Tic-Tac-Toe](/tictactoe4/), one of the somewhat difficult games in the world.
 * [Untitled Objection Series](https://www.youtube.com/playlist?list=PLIoFMnkvRA5PggXaYGQ2QJPEH8LPDm5Dd), a metafictional work about decision theory, Solomonoff induction and `#off-topic`.
 * [osmarkscalculator](https://esolangs.org/wiki/Osmarkscalculator), an esolang and/or computer algebra system based on string rewriting.

blog/barracuda-f380b.md

@@ -124,7 +124,7 @@ The four SATA ports here and the extra SATA power available from the PSU mean th
 
 I do not have a VGA or serial console adapter, but it has an Ethernet port and USB and boots USB sticks by default, so it was a simple matter to boot an [Arch installer image with SSH configured](https://wiki.archlinux.org/title/Install_Arch_Linux_via_SSH#Using_a_single_USB_flash_drive) and then write OpenWRT to the internal storage. Unfortunately, this did not actually work: it imaged fine but the network ports failed to come up. Fortunately, it turned out that the problem was simply OpenWRT only enabling some of the ports, and in a weird order - it wanted 5 as LAN and 6 as WAN, in the numbering on the front panel.
 
-It is very loud by default - the fan curves are set very aggressively (8000 RPM!), presumably for datacentre use. If you install the drivers for the NCT6791 chip, Linux enumerates the fans through sysfs; I have a hacky script which adjusts the PWM operating points down. There also is an LCD panel with some buttons on the front, which I wanted to use. It turns out that the code for this is all very 1990s; I eventually managed to get everything to work by extracting the `lcd4linux` binaries and configuration from the official firmware[^3] [ISO](https://dlportal.barracudanetworks.com/download/6120/GWAY-9.0.4-0097.iso). It might be possible to use `lcdproc`, but I wasn't able to port the configuration properly.
+It is very loud by default - the fan curves are set aggressively (8000 RPM!), presumably for datacentre use. If you install the drivers for the NCT6791 chip, Linux enumerates the fans through sysfs; I have a hacky script which adjusts the PWM operating points down. There also is an LCD panel with some buttons on the front, which I wanted to use. It turns out that the code for this is all very 1990s; I eventually managed to get everything to work by extracting the `lcd4linux` binaries and configuration from the official firmware[^3] [ISO](https://dlportal.barracudanetworks.com/download/6120/GWAY-9.0.4-0097.iso). It might be possible to use `lcdproc`, but I wasn't able to port the configuration properly.
 
 It's currently in service as the core router for my home network, running horribly underutilized because I only have gigabit internet service and [SQM](https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm) isn't particularly heavy for a "real" processor.
 

blog/maghammer2.md

@@ -1,6 +1,7 @@
 ---
 title: Lessons learned from Maghammer
 created: 06/07/2024
+updated: 13/02/2025
 description: I got annoyed and rewrote everything.
 slug: maghammer2
 series: maghammer
@@ -32,7 +33,9 @@ One significant problem with the last version is that the context window used fo
 
 This has the additional advantage of producing fewer sentence embedding vectors, which makes the index more efficient.
 
-As a minor consequence of picking from models released about a year later, I also replaced the [e5-large-v2](https://huggingface.co/intfloat/e5-large-v2) model previously used for embeddings with the same-sized-and-apparently-better [snowflake-arctic-embed-l](https://huggingface.co/Snowflake/snowflake-arctic-embed-l), since it claims better retrieval performance and is more trustworthy, to me, than the Chinese models also claiming that.
+~~As a minor consequence of picking from models released about a year later, I also replaced the [e5-large-v2](https://huggingface.co/intfloat/e5-large-v2) model previously used for embeddings with the same-sized-and-apparently-better [snowflake-arctic-embed-l](https://huggingface.co/Snowflake/snowflake-arctic-embed-l), since it claims better retrieval performance and is more trustworthy, to me, than the Chinese models also claiming that.~~
+
+This has now been replaced again with [ModernBERT-Embed-Large](https://huggingface.co/lightonai/modernbert-embed-large) for the greater context length, maybe better runtime and better retrieval performance. The long context leads to some VRAM issues with large batches, which I have not yet been able to resolve cleanly.
 
 Both models use a prefix to indicate whether an input is a query or a passage to match against, but the newer one seems to be more sensitive to them (or it could simply be the longer inputs), so I've also split columns into "short" and "long" to determine whether this prefixing mechanism is used for queries or not - without this, short passages are privileged, especially ones containing, for some ridiculous reason[^5], the literal text `passage`. This has its own problems, so I might need an alternative solution.
 

blog/other-stuff.md

@@ -5,8 +5,8 @@ created: 11/06/2020
 updated: 17/09/2024
 slug: otherstuff
 ---
-I'm excluding music from this because music preferences seem to be even more varied between the people I interact with than other stuff.
-Obviously this is just stuff *I* like; you might not like it, which isn't really my concern - this list is primarily made to bring to people's attention stuff they might like but have not heard of.
+I'm excluding music from this because music preferences seem to be even more varied between the people I interact with than other preferences.
+Obviously this is just material *I* like; you might not like it, which isn't really my concern - this list is primarily made to bring to people's attention media they might like but have not heard of.
 Enjoy the newly reformatted version of this list, with my slightly opaque organizational scheme and possibly incorrectly sorted lists.
 
 ## Writing
@@ -157,6 +157,10 @@ Well, I mean, yes idealism, yes the dignity of pure research, yes the pursuit of
 
 [Nice Dragons Finish Last](https://www.goodreads.com/series/128485-heartstrikers) is some reasonably enjoyable and nonstandard urban fantasy.
 
+### Orion's Arm
+
+[Orion's Arm](https://www.orionsarm.com/xcms.php?r=oaeg-front) is the most credible attempt so far to construct a detailed far-future universe. Despite the obvious challenges, the project does a good job conveying the bigness and weirdness (by our standards) of a society with ten thousand years of technological and social development behind it.
+
 ### A Practical Guide to Sorcery
 
 [A Practical Guide to Sorcery](https://forums.spacebattles.com/threads/a-practical-guide-to-sorcery-mage-punk-progression-fantasy.866065/#post-68660854) was read on the recommendation of [this comment](/magic/#com2081). It has a smart and competent (they make silly mistakes sometimes but in reasonable ways they notice later) protagonist, flexible but sufficiently-constrained magic system, and a well-realized setting with some mysteries to explore.
@@ -283,6 +287,7 @@ The author, Zachary Mason, also wrote [The Lost Books of the Odyssey](https://ww
 * [Sekhmet Hunts The Dying Gnosis: A Computation](http://www.beneath-ceaseless-skies.com/stories/sekhmet-hunts-the-dying-gnosis-a-computation/).
 * [The Finale of the Ultimate Meta Mega Crossover](https://m.fanfiction.net/s/5389450/1/).
 * [Shannon's Law](https://web.archive.org/web/20140628065855/http://www.tor.com/stories/2011/05/shannons-law).
+* [Dave Scum](https://docs.google.com/document/d/1SddGHeVfcVa5SCDHHTOA4RlKwnef-Q6IMw_Jqw9I0Mw/mobilebasic).
 * More to be added when I feel like it.
 
 ## Games

blog/superintelligence-lower-bounds.md

@@ -0,0 +1,131 @@
+---
+title: Against some assumed limits on superintelligence
+description: The TAM for God is very large.
+created: 02/03/2025
+slug: asi
+draft: yes
+---
+::: epigraph attribution="Void Star" link=/otherstuff/#void-star
+It’s not a trick. You’ll die if you go on, but it’s not a trick. I know something of your nature. Do you really want to go back to the decay of your biology and days like an endless reshuffling of a fixed set of forms? What the world has, you’ve seen. This is the only other way.
+:::
+
+AI labs are now racing to construct artificial superintelligence, in the rough sense of an AI system more capable than the best humans across a wide range of economically relevant tasks - while existing systems beat humans in many areas, they still have some poorly understood gaps ("agency") preventing them from doing much of what humans can. Many insist that this is fine because of alignment-by-default-style arguments, that it is [generally impossible or impractical](https://arxiv.org/abs/1703.10987), or that it's not on-trend for current research, but another class of objections denies the relevance of superintelligence at all, claiming that it would be practically limited by (variously) hard limits on the effectiveness of intelligence; the requirement to gather new data and conduct experiments to do anything; slow processes in the physical world. I think this is completely wrong. [Starwink](https://alicorn.elcenia.com/stories/starwink.shtml) and [That Alien Message](https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message) provide a good visceral sense of why I think so, but in this post I will be directly arguing for this.
+
+## Path(s) to victory
+
+As a [gwernpost](https://gwern.net/forking-path) describes, technology forecasting often makes the mistake of assuming a goal can only be achieved in one way, finding that that way is impractical, and then declaring the entire area unworkable. Occasionally something is found to be forbidden by hard physical law, but with any barrier short of this and sufficient incentive, workarounds are often found:
+
+* Two months before the first successful heavier-than-air flight, the [New York Times claimed](https://en.wikipedia.org/wiki/Flying_Machines_Which_Do_Not_Fly) that "one to ten million years" of research would be needed to achieve this, because they assumed that aircraft would have to work much like birds and that human designs would have to be very slowly adapted to work around human materials' shortcomings relative to birds. Ultimately, planes work because of mechanical designs and high-energy systems not accessible to biology.
+* Leading-edge semiconductor fabrication uses [extreme ultraviolet lithography](https://en.wikipedia.org/wiki/Extreme_ultraviolet_lithography), machines for which are produced by one company (ASML) and which cost several hundred million dollars each, to print detailed patterns onto wafers. Export controls have prevented China from accessing EUV machines, and replicating the technology directly domestically is considered very difficult since it took decades of expensive Western R&D and a large supply chain. Chinese manufacturers are [fairly](https://www.techinsights.com/blog/techinsights-finds-smic-7nm-n2-huawei-mate-60-pro) [competitive](https://semianalysis.substack.com/p/chinas-smic-is-shipping-7nm-foundry) anyway through pushing older DUV machines harder (with some tradeoffs[^10]).
+    * Also, [nanoimprint lithography](https://global.canon/en/technology/nil-2023.html), an alternative patterning approach using physical imprinting rather than light, is claimed to be similarly accurate with lower running costs.
+    * Chinese researchers are working on [free electron lasers](https://conference-indico.kek.jp/event/160/contributions/2876/attachments/2148/2699/Zhilong_Pan.pdf), which may leapfrog ASML technology.
+* On-demand video streaming with good quality was thought impractical for home users due to high bandwidth requirements and the difficulty of upgrading network infrastructure, but faster computers allowed for good enough video compression to make the existing lines work.
+* The limited energy density of batteries meant electric cars couldn't match <span class="hoverdefn" title="internal combustion engine">ICE</span> cars' range. Tesla made a successful [electric car](https://en.wikipedia.org/wiki/Tesla_Roadster_(first_generation)#Reviews) despite this, by using the better characteristics of electric motors to make a powerful sports car where range was less of a concern, and then achieved wider adoption by setting up many fast charging stations and focusing on home charging rather than trying to match ICE refuelling.
+
+There is an obvious problem of sampling bias here, since I'm not likely to know about something which was abandoned because it can only be done in one way which doesn't work. Please tell me if you can think of anything.
+
+To change the world, a superintelligence doesn't have to be massively better at every possible task, or route through a particular technology you feel is impractical. Perhaps some fields are "irreducibly complex", or humans are already very close to the theoretical limits, or ASI ends up bad at them for contingent reasons. It doesn't matter: being significantly superhuman at a few things - say, computer hacking, persuasion and bioengineering - would be sufficient for power, and danger.
+
+## Hard caps and diminishing returns
+
+For many tasks, there is a hard limit on how good anything can get, and dimishing returns on compute/resources as that limit is approached. For instance, [bin packing](https://en.wikipedia.org/wiki/Bin_packing_problem) is NP-hard, but very simple and cheap algorithms can get within a small constant factor of the optimal solution. Chess engine developers believe that the best engines cannot get significantly better at play from the normal chess starting position, because the game can be drawn too easily, and some think that they're about able to [draw God](https://en.chessbase.com/post/how-god-plays-chess) from there (frontier chess engines are instead tested in asymmetric starting positions such that one side should always be able to win or draw). In tic-tac-toe, perfect play is obviously simple enough that no ASI can, within the constraints of the game, do anything beyond forcing a draw against a competent opponent. In benchmarks with known correct answers, nothing can beat a 100% score[^16]. This is often used to argue for diminishing returns on intelligence in general, which I don't think is correct.
+
+There are several important problems with this: mere "diminishing returns" tells you nothing about how rapidly they diminish, whether they asymptote or just grow more slowly, or how far beyond humans you can go; usefulness isn't always linear in whatever metric shows diminishing returns; and more intelligence unlocks qualitatively new abilities which preexisting benchmarks won't catch.
+
+As I address in the next section, capabilities can frequently go far beyond humans', since ([Moravec's paradox](https://en.wikipedia.org/wiki/Moravec's_paradox)) we are strongly optimized for the kind of task our ancestors regularly faced and rely on less robust general-purpose capability for anything more recent[^17].
+
+In open-ended adversarial and/or somewhat winner-takes-all domains like trading, there is no upper limit, only a succession of ever-escalating counterstrategies. You could reasonably argue that it wouldn't mean much for society as a whole if all humans were moved out of these, though. More compelling are situations where relatively minor-sounding improvements make something work as a practical product rather than a research curiosity, like [image recognition](https://paperswithcode.com/sota/image-classification-on-imagenet). The relatively minor-sounding 10-percentage-point improvement in ImageNet accuracy from 2015 to 2022 masks massive production deployment as image recognizers became smarter, cheaper, more robust and more practical. Protein modelling tools have crept up in capability over the last few years and now we can [design proteins](https://www.nature.com/articles/s41586-024-08393-x) tailored to particular tasks.
+
+This is most relevant when several areas advance at once, as you would expect from superintelligence. Quadcopter drones used to be impractical and/or expensive, but in the 2000s, mildly better batteries, cheaper MEMS IMUs (possibly because of smartphones) and smaller and better control/driver electronics (such as [specialized low-latency brushless motor drivers](/assets/misc/fascination_quadcopter.pdf#page=25)) brought them to hobbyists and then the general commercial market.
+
+But focus on concrete tasks I can think of myself is rather missing the point. Doing things humans are currently doing but somewhat better is a waste of superintelligence. GPT-3 [was exciting](https://arxiv.org/abs/2005.14165) not because it pushed benchmark scores slightly further, and perplexity slightly lower, than GPT-2, but because it had the previously unconsidered ability to learn new tasks in context. The most important technological developments have been unexpected step changes opening up entirely new fields rather than "faster horses" incremental changes, even if they can be described in retrospect that way. The smartest humans are able to usefully reframe problems as easier ones rather than brute-force-execute a more obvious solution, and I expect this to continue.
+
+## Humans are not nearly optimal
+
+Due to limited working memory and the necessity of distributing subtasks in an organization, humans design and model systems based on abstraction - rounding off low-level detail to produce a homogeneous overview with fewer free parameters. [Seeing Like a State](https://en.wikipedia.org/wiki/Seeing_Like_a_State)[^1] describes how this has gone wrong historically - states, wanting the world to be easier to manage, bulldoze fine-tuned local knowledge and install simple rules and neat rectangles which produce worse outcomes. I think this case is somewhat overstated, because abstraction does often work better than the alternatives. People can't simultaneously attend to the high-level requirements of their problem and every low-level point, so myopic focus on the low-level detracts from the overall quality of the result[^2] - given the limitations of humans.
+
+Abstraction amortises intellect, taking good solutions to simpler and more general problems and applying them on any close-enough substrate. This has brought us many successes like industrial farming, digital computers and assembly lines. But an end-to-end design not as concerned with modularity and legibility will usually outperform one based on generalities, if you can afford the intellectual labour, through better addressing cross-cutting concerns, precise tailoring to small quirks and making simplifications across layers of the stack. [This book](https://www.construction-physics.com/p/book-review-building-an-affordable) describes some object-level examples in house construction.
+
+We see the abstractions still even when they have gaps, and this is usually a security threat. A hacker doesn't care that you think your code "parses XML" or "checks authentication" - they care about [what you actually wrote down](https://gwern.net/unseeing), and what the computer will do with it[^3], which is quite possibly [not what you intended](https://blog.siguza.net/psychicpaper/). Your nice "secure" cryptographic code is [running on hardware](http://wiki.newae.com/Correlation_Power_Analysis) which reveals correlates of what it's doing. Your "air-gapped" computer is able to emit [sounds](https://arxiv.org/abs/2409.04930v1) and [radio signals](https://arxiv.org/abs/2207.07413) and [is connected to power cables](https://pushstack.wordpress.com/2017/07/24/data-exfiltration-from-air-gapped-systems-using-power-line-communication/). A "blank wall" [leaks information](https://www.cs.princeton.edu/~fheide/steadystatenlos) through diffuse reflections. Commodity "communication" hardware can [sense people](https://www.usenix.org/system/files/nsdi24-yi.pdf), because the signals travel through the same physical medium as everything else. Strange side channels are everywhere and systematically underestimated. These are the examples we *have* found, but new security vulnerabilities are detected continually and I am confident that essentially all complex software is hopelessly broken in at least one way.
+
+Because of the ability to avoid unhelpful human assumptions, from-scratch <span class="hoverdefn" title="reinforcement learning">RL</span>, gradient descent and evolutionary approaches have been successful in designing [silicon photonics](https://x.company/blog/posts/ai-siph-design/), [antennas](https://en.wikipedia.org/wiki/Evolved_antenna), [chip floorplans](https://deepmind.google/discover/blog/how-alphachip-transformed-computer-chip-design/) and [FPGA](/assets/misc/evolved-fpga-modern.pdf) [designs](/assets/misc/evolved-circuit.pdf) in impressive ways humans can't, or at least don't typically[^5]. Similarly, since [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo)'s "move 37", self-play game solvers running without human preconceptions have been able to discover strange strategies and nowadays utterly dominate all human players in chess and go, to the point that human/engine teams are not better than engines. [A paper](https://arxiv.org/abs/2310.16410) evaluating concept transfer from AlphaZero in chess finds that grandmasters can learn some of AlphaZero's strategies, but they are considered unnatural and not always understood. It also optimizes for win probability directly rather than caring about proxies like margin of victory, leading to further alienation from human play and greater success.
+
+RL is also disliked by researchers for its [tendency to](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml) [reward-hack](https://arxiv.org/abs/1803.03453), i.e. find solutions to the specification of a problem which don't satisfy the "spirit" of it. This is inconvenient for anyone using it for design optimization, but highlights how human reliance on prior knowledge misses things.
+
+In many areas we have a concrete example of what something highly optimized by a process which doesn't care about whether it's legible to humans can do - biology[^6]. Despite the somewhat poor high-level designs generated by evolution's blind local search, low-level functionality (DNA replication, bacterial cell division, etc) apparently [runs close to](https://pubs.aip.org/aip/jcp/article/139/12/121923/74793/Statistical-physics-of-self-replication) physical limits. Generally, biological systems are built with more readily available materials and much less energy than human systems, though they lag in some parameters - biological solar to chemical energy conversion is much lossier than the best human systems, and silicon logic switches roughly eight orders of magnitude faster, if possibly less efficiently[^4], than neurons.
+
+## Data is no barrier
+
+Superhuman AI/optimization systems are usually trained on much more data than a human will ever see - state-of-the-art image classifiers and language models see billions of webpages in pretraining, AlphaGo Zero played[^7] 5 million games against itself, and a recent [self-driving car RL paper](https://arxiv.org/abs/2502.03349) used 1 trillion steps/10000 simulated driver-years to achieve state-of-the-art performance. This suggests that a superintelligence could not rapidly outperform humans without conducting experiments in the areas humans haven't looked at hard and aren't good at. I think this is not true, and that it wouldn't be a binding limit if it was.
+
+The biggest discovery in deep learning in the past decade[^8], which many still don't understand, is scaling laws: the performance of a particular model architecture scales predictably with compute. Often the compute-optimal forms are quoted without nuance - many take the [Chinchilla paper](https://arxiv.org/abs/2203.15556) to mean "you must train LLMs on ~20 tokens per parameter" - but it's also possible to use more data and fewer parameters, or less data and more parameters, for roughly the same downstream performance, with one data/parameter count setting minimizing compute iso-performance[^12]. RL [also has](https://arxiv.org/abs/2502.04327) similar tradeoffs, though parameter count scaling is trickier[^9] so updates-per-data is varied instead. This makes sample efficiency an economic question rather than a theoretical one.
+
+::: captioned src=/assets/images/chinchilla_isoloss_isoflops.png
+This somewhat confusing image from the Chinchilla paper shows the tradeoffs in training an LLM. Holding training FLOPS constant, model size is inversely proportional to training data size. Scaling model size while holding training data size constant means going up and right with a steeper slope than the efficient frontier (which uses more data). [Another paper](https://proceedings.neurips.cc/paper_files/paper/2023/file/9d89448b63ce1e2e8dc7af72c984c196-Paper-Conference.pdf) describes multi-epoch training, which provides further improvements (up to a point).
+:::
+
+Combining this with [clever](https://arxiv.org/abs/2111.00210) [algorithmic](https://github.com/danijar/dreamerv3) [innovations](https://arxiv.org/abs/2403.00564) has brought *trained-from-scratch* neural nets above human sample efficiency on standard Atari benchmarks.
+
+Training against simulation environments is also sometimes effective. As far as I know, "sim2real" transfer has not worked very well in robotics, and direct numerical simulation of [some physical systems](https://titotal.substack.com/p/bandgaps-brains-and-bioweapons-the) is computationally intractable. However, for maths and softare tasks, simulations and the real thing are (mostly) the same, so this doesn't limit capability at mathematics (important for e.g. cryptography, as well as clever tricks to model physical systems) or software engineering, and I would be very surprised if it weren't possible to design better simulation environments for transfer than we are currently managing to.
+
+We also have very large amounts of experimental and observational data which could be reanalyzed with better methods. For instance, the [AlphaFold](https://pmc.ncbi.nlm.nih.gov/articles/PMC8371605/) models were trained on long-available protein datasets (including known amino acid sequences without known structures), but significantly outperformed every other solution at novel structure prediction through application of smarter deep learning. This did *not* require extremely expensive first-principles simulation.
+
+Finally, human development of theory and explanations sometimes precedes or postdates data significantly. The most obvious examples of this are in mathematics - the concepts of [group theory](https://en.wikipedia.org/wiki/Group_theory) were accessible through all of human history and did not require any special world knowledge, but weren't invented until the 1800s. Classical mechanics (as opposed to blatantly wrong Aristotelian physics) could probably have been invented millenia earlier as the simplest explanation for projectile trajectories and falling masses. General relativity was invented based on thought experiments and notions of elegance before it could be robustly tested, based on maths available significantly beforehand. Maxwell's equations predated their empirical tests by 30 years, and semiconductor switching devices were theorized 15 years before they were built. This does not happen as much in biology and chemistry, which are less theory-driven, but the periodic table made advance predictions of chemical element properties and the [triplet code](https://en.wikipedia.org/wiki/Genetic_code#History) structure of DNA was derived before it could be directly checked.
+
+## Structural advantages
+
+People often compare AGIs and ASIs to corporations. This analogy [is wrong](https://www.youtube.com/watch?v=L5pUA3LsEaw), both because of the [aforementioned](#humans-are-not-nearly-optimal) issues with splitting up tasks (some don't parallelize well, and there are losses to specialization) and because large organizations are hopelessly dysfunctional compared to what would be possible with ideal coordination. Because people in corporations are poorly incentive-aligned, they are often focused on [internal politics](https://thezvi.wordpress.com/2019/05/30/quotes-from-moral-mazes/) rather than actually solving problems, and this is internally selected for - I believe materially everyone who has worked in a large company has some stories about the organization barely working despite vast reserves of talent and resources. Since corporations [don't have easily copied "genes"](https://gwern.net/backstop#artificial-persons) and "generations" are long, this has not been selected out.
+
+Coordination between humans and between organizations also faces substantial transaction costs, the most obvious of which is meetings. I don't know exactly how much time is spent in meetings, but dubiously trustworthy estimates suggest 5-10 hours per week (for "professionals") and more for managers - and these do not do a good job accurately conveying all necessary information. Between firms, vast amounts of effort is spent on sales, inverting the efforts of other people's sales, negotiating and litigating contracts, and bidding for work, such that small transactions aren't worth the effort and work is duplicated, and larger ones suffer overhead. We also generally struggle to collectively make changes which would be good for everyone if fully implemented but which require everyone to change their behaviour at the same time.
+
+This is avoidable in the sense that a top-down planner wouldn't require it, but we don't know how to organize human groups to avoid this.
+
+ASI neatly sidesteps these problems: depending on structure, perhaps one "instance" can manage an entire vertically integrated operation itself, but even separate copies should be capable of superhuman coordination. With matching goals, self-knowledge much more accurate than our lossy human models of each other and potentially the ability to directly share mind-state data, efficiency greater than the nimblest and best-run startups should be achievable at the scale of a whole industry[^11].
+
+While it would still have to interface with human institutions, I'd like to remind readers that persuasion and social modelling are intellectual capabilities, there are many ways around "the proper channels" available to an unaligned/uncorrigible AI, and the usual slow rate of technological diffusion is somewhat mitigated when you can replace entire legacy companies at once.
+
+Robin Hanson has written about a somewhat similar scenario - [Age of Em](https://en.wikipedia.org/wiki/The_Age_of_Em)[^13]. In this, the cost of intellectual labour drops precipitously thanks to brain uploading technology allowing cheap duplication of the smartest and most competent humans (note that this is *weaker* than the assumption of ASI I make here). He has a lot to say about labour economics in this world, and several mechanisms (such as "copy-clans" and "combinatorial auctions") for coordination, but chapter 16 ("Growth") contains some estimates of economic doubling times. Currently, the economy grows in size about 5% per year, thus doubling every ~15 years. With trivially cheap intellectual labour[^14], Hanson predicts doubling times of around a month:
+
+> To generate an empirical estimate of em economy doubling times, we can look at the timescales it takes for machine shops and factories today to make a mass of machines of a quality, quantity, variety, and value similar to that of machines that they themselves contain. Today, that timescale is roughly 1 to 3 months. Also, designs were sketched two to three decades ago for systems that might self-replicate nearly completely in 6 to 12 months (Freitas and Merkle 2004). [...] Together, these estimates suggest that today’s manufacturing technology is capable of self-replicating on a scale of a few weeks to a few months.
+
+> Of course machine shops and 3D printers are fastest when making the simplest and easiest to construct devices and components. Large complex facilities such as chip factories take longer to build. So the estimates above may under-estimate doubling times to the extent that they leave out the replication of important components that take longer to make. Today, humans are such a left-out component; our economy doesn’t grow this fast because we can’t replicate people as quickly as machines.
+
+> However, these machine reproduction estimates also tend to over-estimate doubling times because a faster growing em economy offers stronger rewards for reducing factory-doubling times. [...] Innovation also makes the em economic doubling time shorter than the time it takes em factories to reproduce themselves. A lot of innovation happens because of “learning by doing,” where the rate of technology gains is tied less to absolute clock speeds than to the rate at which people make and use products (Weil 2012)[^15]. [...] As mentioned before, an em economy focuses much more than ours does on computer capital, which has long seen much faster rates of innovation than has other forms of manufactured capital. More generally, machine-based capital has seen faster rates of innovation than have human and land capital.
+
+## Conclusion
+
+Enough of the world is bottlenecked on (availability of) human intelligence, rather than natural law or the availability of data, that a "solution to intelligence" in whatever form that takes would have a transformative impact on the world, and it's not valid to dismiss its relevance. Whether this is on path for current AI development is of course a different question.
+
+[^1]: Reviewed by the [other Scott](https://slatestarcodex.com/2017/03/16/book-review-seeing-like-a-state/).
+
+[^2]: Consider [this comment](https://news.ycombinator.com/item?id=36153989) on how an OS written entirely in assembly had performance problems because nobody understood the overall design.
+
+[^3]: I also talk about this in my post on [programming education](/progedu/).
+
+[^4]: This was heavily debated [on LessWrong](https://www.lesswrong.com/posts/KsKfvLx7nFBZnWtEu/no-human-brains-are-not-much-more-efficient-than-computers#xo8kPxPEfb6QKeynu) and I do not know enough thermodynamics to evaluate the arguments.
+
+[^5]: These systems use many more feedback cycles than humans during training, so it's possible that with lots more time on the task a human would truly generalize, rather than memorizing higher-level solutions.
+
+[^6]: It [has been argued](https://www.lesswrong.com/posts/bNXdnRTpSXk9p4zmi/book-review-design-principles-of-biological-circuits) that biology is less haphazard than it seems, and convergently adopts certain patterns as the near-unique solutions to problems it faces, but it is messy enough that the function of [>20% of genes](https://www.pnas.org/doi/10.1073/pnas.0510013103) in minimal organisms remains unknown, entire [organelles](https://en.wikipedia.org/wiki/Vault_(organelle)#Function) are unexplained and people will occasionally stumble on [things](https://en.wikipedia.org/wiki/Obelisk_(biology)) nobody had previously noticed.
+
+[^7]: This is different from AlphaZero and AlphaGo because Google cannot name things, but it is the only one I found this figure for.
+
+[^8]: First documented [by Baidu in 2017](https://arxiv.org/abs/1712.00409).
+
+[^9]: This is probably just an algorithmic issue: see e.g. [this paper](https://arxiv.org/abs/2405.16158).
+
+[^10]: Defect rates are much higher so it's somewhat uneconomical.
+
+[^11]: [Acausal negotiation](https://joecarlsmith.com/2021/08/27/can-you-control-the-past/) should also be possible, but for these purposes it doesn't matter much.
+
+[^12]: The form of scaling law fit in Chinchilla assumes that a model's loss comprises the irreducible loss of language (i.e. inherent uncertainty - for example, if you see the string `Today's date is ` as a perfect text-predictor then you have uncertainty about when the text was written), a minimum loss component related to the parameter count, and a minimum loss component related to the dataset size. Under this model, there *is* still a minimum loss achievable with a fixed amount of data, but it's much lower than the ones reached by contemporary systems, and varies with architecture and training process.
+
+[^13]: Hanson seems to think that current AI is essentially "business as usual" and won't have similar effects to his Age of Em, though.
+
+[^14]: I am assuming that once some kind of superintelligence is achieved it will at some point become cheap (not necessarily widely available, but low-marginal-cost), as has happened with other computing technologies.
+
+[^15]: This reduces the relevance of ASI for accelerating things, but see [Data is no barrier](#data-is-no-barrier) for why I don't think it matters much.
+
+[^16]: Most benchmarks for LLMs and image models contain many wrong answers, so nothing can *achieve* a 100% score, except by perfectly modelling all the highly contingent human mistakes.
+
+[^17]: This may centrally be an I/O limitation.

src/index.js

@@ -557,7 +557,7 @@ const compileServiceWorkerJSTask = async () => {
 }
 
 const doImages = async () => {
-    await Promise.all(["images", "titillium-web.woff2", "titillium-web-semibold.woff2", "iosevka.woff2", "misc"].map(subpath => fse.copy(path.join(assetsDir, subpath), path.join(outAssets, subpath))))
+    await Promise.all(["images", "osmarks-primary.woff2", "osmarks-primary-semibold.woff2", "osmarks-primary-italic.woff2", "osmarks-mono.woff2", "misc"].map(subpath => fse.copy(path.join(assetsDir, subpath), path.join(outAssets, subpath))))
 
     await fse.copy(path.join(nodeModules, "katex", "dist", "fonts"), path.join(outAssets, "fonts"))
     await fse.copy(path.join(nodeModules, "katex", "dist", "katex.min.css"), path.join(outAssets, "katex.min.css"))

src/style.sass

@@ -12,32 +12,40 @@ $navbar-width: 20rem
     --autocol-border: 0%
 
 @font-face
-    font-family: 'Titillium Web'
+    font-family: 'osmarks-primary'
     font-style: normal
     font-weight: 400
     font-display: swap
-    src: url(/assets/titillium-web.woff2) format('woff2')
-    unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD
+    src: url(/assets/osmarks-primary.woff2) format('woff2')
+    unicode-range: U+0000-00FF,U+FFFD
 
 @font-face
-    font-family: 'Titillium Web'
+    font-family: 'osmarks-primary'
     font-style: normal
     font-weight: 600
     font-display: swap
-    src: url(/assets/titillium-web-semibold.woff2) format('woff2')
-    unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+2074, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD
+    src: url(/assets/osmarks-primary-semibold.woff2) format('woff2')
+    unicode-range: U+0000-00FF,U+FFFD
 
 @font-face
-    font-family: 'Iosevka'
+    font-family: 'osmarks-primary'
+    font-style: italic
+    font-weight: 400
+    font-display: swap
+    src: url(/assets/osmarks-primary-italic.woff2) format('woff2')
+    unicode-range: U+0000-00FF,U+FFFD
+
+@font-face
+    font-family: 'osmarks-mono'
     font-style: normal
     font-weight: 400
     font-display: swap
-    src: url(/assets/iosevka.woff2) format('woff2')
+    src: url(/assets/osmarks-mono.woff2) format('woff2')
     unicode-range: U+0000-00C0, U+A2-A9, U+AC-AE, U+00D7, U+00F7, U+FEFF, U+FFFD
 
-$headers: "Iosevka", "Titillium Web", "Fira Sans", sans-serif
-$monospace: "Iosevka", "Fira Code", monospace
-$body: "Titillium Web", "Fira Sans", sans-serif
+$headers: "osmarks-mono", "osmarks-primary", "Fira Sans", sans-serif
+$monospace: "osmarks-mono", "Fira Code", monospace
+$body: "osmarks-primary", "Fira Sans", sans-serif
 
 body
     margin: 0