actually add the tags, oops

blog/superintelligence-lower-bounds.md

@@ -3,6 +3,7 @@ title: Against some assumed limits on superintelligence
 description: The TAM for God is very large.
 created: 02/03/2025
 slug: asi
+tags: ["ai"]
 ---
 ::: epigraph attribution="Void Star" link=/otherstuff/#void-star
 It’s not a trick. You’ll die if you go on, but it’s not a trick. I know something of your nature. Do you really want to go back to the decay of your biology and days like an endless reshuffling of a fixed set of forms? What the world has, you’ve seen. This is the only other way.
@@ -43,7 +44,7 @@ But focus on concrete tasks I can think of myself is rather missing the point. D
 
 Due to limited working memory and the necessity of distributing subtasks in an organization, humans design and model systems based on abstraction - rounding off low-level detail to produce a homogeneous overview with fewer free parameters. [Seeing Like a State](https://en.wikipedia.org/wiki/Seeing_Like_a_State)[^1] describes how this has gone wrong historically - states, wanting the world to be easier to manage, bulldoze fine-tuned local knowledge and install simple rules and neat rectangles which produce worse outcomes. I think this case is somewhat overstated, because abstraction does often work better than the alternatives. People can't simultaneously attend to the high-level requirements of their problem and every low-level point, so myopic focus on the low-level detracts from the overall quality of the result[^2] - given the limitations of humans.
 
-Abstraction amortises intellect, taking good solutions to simpler and more general problems and applying them on any close-enough substrate. This has brought us many successes like industrial farming, digital computers and assembly lines. But an end-to-end design not as concerned with modularity and legibility will usually outperform one based on generalities, if you can afford the intellectual labour, through better addressing cross-cutting concerns, precise tailoring to small quirks and making simplifications across layers of the stack. Due to organizational issues, the cost of human intelligence, and working memory limitations, this frequently doesn't happen. [This book](https://www.construction-physics.com/p/book-review-building-an-affordable) describes some object-level examples in house construction and [this blog post](https://yosefk.com/blog/my-history-with-forth-stack-machines.html) suggests that Forth is this for computing.
+Abstraction amortises intellect, taking good solutions to simpler and more general problems and applying them on any close-enough substrate. This has brought us many successes like industrial farming, digital computers and assembly lines. But an end-to-end design not as concerned with modularity and legibility will usually outperform one based on generalities, if you can afford the intellectual labour, through better addressing cross-cutting concerns, precise tailoring to small quirks and making simplifications across layers of the stack. Due to organizational issues, the cost of human intelligence, and working memory limitations, this frequently doesn't happen. [This book](https://www.construction-physics.com/p/book-review-building-an-affordable) describes some object-level examples in house construction and [this blog post](https://yosefk.com/blog/my-history-with-forth-stack-machines.html) suggests that Forth is this for computing. See also [Anton](https://www.hc33.hotchips.org/assets/program/conference/day2/HC2021.DESRES.AdamButts.v03.pdf): by designing computers end-to-end for molecular dynamics, D E Shaw Research managed 100x speedups over standard supercomputers.
 
 We see the abstractions still even when they have gaps, and this is usually a security threat. A hacker doesn't care that you think your code "parses XML" or "checks authentication" - they care about [what you actually wrote down](https://gwern.net/unseeing), and what the computer will do with it[^3], which is quite possibly [not what you intended](https://blog.siguza.net/psychicpaper/). Your nice "secure" cryptographic code is [running on hardware](http://wiki.newae.com/Correlation_Power_Analysis) which reveals correlates of what it's doing. Your "air-gapped" computer is able to emit [sounds](https://arxiv.org/abs/2409.04930v1) and [radio signals](https://arxiv.org/abs/2207.07413) and [is connected to power cables](https://pushstack.wordpress.com/2017/07/24/data-exfiltration-from-air-gapped-systems-using-power-line-communication/). A "blank wall" [leaks information](https://www.cs.princeton.edu/~fheide/steadystatenlos) through diffuse reflections. Commodity "communication" hardware can [sense people](https://www.usenix.org/system/files/nsdi24-yi.pdf), because the signals travel through the same physical medium as everything else. Strange side channels are everywhere and systematically underestimated. These are the examples we *have* found, but new security vulnerabilities are detected continually and I am confident that essentially all complex software is hopelessly broken in at least one way.
 
@@ -55,7 +56,7 @@ In many areas we have a concrete example of what something highly optimized by a
 
 ## Data is no barrier
 
-Superhuman AI/optimization systems are usually trained on much more data than a human will ever see - state-of-the-art image classifiers and language models see billions of webpages in pretraining, AlphaGo Zero played[^7] 5 million games against itself, and a recent [self-driving car RL paper](https://arxiv.org/abs/2502.03349) used 1 trillion steps/10000 simulated driver-years to achieve state-of-the-art performance. This suggests that a superintelligence could not rapidly outperform humans without conducting experiments in the areas humans haven't looked at hard and aren't good at. I think this is not true, and that it wouldn't be a binding limit if it was.
+Superhuman AI/optimization systems are usually trained on much more data than a human will ever see - state-of-the-art image classifiers and language models see billions of webpages in pretraining, AlphaGo Zero played[^7] 5 million games against itself, and a recent [self-driving car RL paper](https://arxiv.org/abs/2502.03349) used 1 trillion steps/10000 simulated driver-years to achieve state-of-the-art performance. This suggests that a superintelligence could not rapidly outperform humans without conducting experiments in the areas humans haven't looked at hard and aren't good at. I think this is not true, and that it wouldn't be a binding limit if it was[^18].
 
 The biggest discovery in deep learning in the past decade[^8], which many still don't understand, is scaling laws: the performance of a particular model architecture scales predictably with compute. Often the compute-optimal forms are quoted without nuance - many take the [Chinchilla paper](https://arxiv.org/abs/2203.15556) to mean "you must train LLMs on ~20 tokens per parameter" - but it's also possible to use more data and fewer parameters, or less data and more parameters, for roughly the same downstream performance, with one data/parameter count setting minimizing compute iso-performance[^12]. RL [also has](https://arxiv.org/abs/2502.04327) similar tradeoffs, though parameter count scaling is trickier[^9] so updates-per-data is varied instead. This makes sample efficiency an economic question rather than a theoretical one.
 
@@ -130,3 +131,5 @@ Enough of the world is bottlenecked on (availability of) human intelligence, rat
 [^16]: Most benchmarks for LLMs and image models contain many wrong answers, so nothing can *achieve* a 100% score, except by perfectly modelling all the highly contingent human mistakes.
 
 [^17]: This may centrally be an I/O limitation.
+
+[^18]: This section uses mostly evidence from deep learning. It is possible that ASI won't be built this way (though I personally expect it to be), but whatever is used should be "at least as good" in these ways.