copyedit

blog/superintelligence-lower-bounds.md

@@ -43,7 +43,7 @@ But focus on concrete tasks I can think of myself is rather missing the point. D
 
 Due to limited working memory and the necessity of distributing subtasks in an organization, humans design and model systems based on abstraction - rounding off low-level detail to produce a homogeneous overview with fewer free parameters. [Seeing Like a State](https://en.wikipedia.org/wiki/Seeing_Like_a_State)[^1] describes how this has gone wrong historically - states, wanting the world to be easier to manage, bulldoze fine-tuned local knowledge and install simple rules and neat rectangles which produce worse outcomes. I think this case is somewhat overstated, because abstraction does often work better than the alternatives. People can't simultaneously attend to the high-level requirements of their problem and every low-level point, so myopic focus on the low-level detracts from the overall quality of the result[^2] - given the limitations of humans.
 
-Abstraction amortises intellect, taking good solutions to simpler and more general problems and applying them on any close-enough substrate. This has brought us many successes like industrial farming, digital computers and assembly lines. But an end-to-end design not as concerned with modularity and legibility will usually outperform one based on generalities, if you can afford the intellectual labour, through better addressing cross-cutting concerns, precise tailoring to small quirks and making simplifications across layers of the stack. Due to organizational issues, the cost of human intelligence, and working memory limitations, this frequently doesn't happen. [This book](https://www.construction-physics.com/p/book-review-building-an-affordable) describes some object-level examples in house construction.
+Abstraction amortises intellect, taking good solutions to simpler and more general problems and applying them on any close-enough substrate. This has brought us many successes like industrial farming, digital computers and assembly lines. But an end-to-end design not as concerned with modularity and legibility will usually outperform one based on generalities, if you can afford the intellectual labour, through better addressing cross-cutting concerns, precise tailoring to small quirks and making simplifications across layers of the stack. Due to organizational issues, the cost of human intelligence, and working memory limitations, this frequently doesn't happen. [This book](https://www.construction-physics.com/p/book-review-building-an-affordable) describes some object-level examples in house construction and [this blog post](https://yosefk.com/blog/my-history-with-forth-stack-machines.html) suggests that Forth is this for computing.
 
 We see the abstractions still even when they have gaps, and this is usually a security threat. A hacker doesn't care that you think your code "parses XML" or "checks authentication" - they care about [what you actually wrote down](https://gwern.net/unseeing), and what the computer will do with it[^3], which is quite possibly [not what you intended](https://blog.siguza.net/psychicpaper/). Your nice "secure" cryptographic code is [running on hardware](http://wiki.newae.com/Correlation_Power_Analysis) which reveals correlates of what it's doing. Your "air-gapped" computer is able to emit [sounds](https://arxiv.org/abs/2409.04930v1) and [radio signals](https://arxiv.org/abs/2207.07413) and [is connected to power cables](https://pushstack.wordpress.com/2017/07/24/data-exfiltration-from-air-gapped-systems-using-power-line-communication/). A "blank wall" [leaks information](https://www.cs.princeton.edu/~fheide/steadystatenlos) through diffuse reflections. Commodity "communication" hardware can [sense people](https://www.usenix.org/system/files/nsdi24-yi.pdf), because the signals travel through the same physical medium as everything else. Strange side channels are everywhere and systematically underestimated. These are the examples we *have* found, but new security vulnerabilities are detected continually and I am confident that essentially all complex software is hopelessly broken in at least one way.