diff --git a/blog/ai-companions.md b/blog/ai-companions.md
index de974db..66f3613 100644
--- a/blog/ai-companions.md
+++ b/blog/ai-companions.md
@@ -57,7 +57,7 @@ What could possibly go wrong?
Remember: the AI may predict your future, manipulate your decisions, and undermine your autonomy, but *we did the math first*. So it’s cool.
:::
-The most salient danger now is "AI psychosis" - LLMs playing along and encouraging users' delusions enough to drive them into madness - but this is [probably](https://andymasley.substack.com/p/stories-of-ai-turning-users-delusional) [somewhat overstated](https://www.astralcodexten.com/p/in-search-of-ai-psychosis). It's not clear that serious, dangerous psychosis is happening to users at above the base rate, though a few models [do very badly](https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced-psychosis-a-shallow-investigation) when dealing with simulated users in danger; much of the "psychosis" is just LLMs going along with and encouraging bizarre, crackbot beliefs. While LLMs make people with such beliefs more visible and annoying[^12], it's not obvious whether the prevalence is going up. [Taking ideas seriously](https://www.lesswrong.com/w/taking-ideas-seriously) is rare, and normal people are able to maintain wildly strange beliefs (for instance, religions and complex conspiracy theories) whilst remaining outwardly functional and normal, because following beliefs to all their downstream implications is cognitively expensive, so the base rate is probably quite high.
+The most salient danger now is "AI psychosis" - LLMs playing along and encouraging users' delusions enough to drive them into madness - but this is [probably](https://web.archive.org/web/20260218132513/https://andymasley.substack.com/p/stories-of-ai-turning-users-delusional) [somewhat overstated](https://www.astralcodexten.com/p/in-search-of-ai-psychosis). It's not clear that serious, dangerous psychosis is happening to users at above the base rate, though a few models [do very badly](https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced-psychosis-a-shallow-investigation) when dealing with simulated users in danger; much of the "psychosis" is just LLMs going along with and encouraging bizarre, crackbot beliefs. While LLMs make people with such beliefs more visible and annoying[^12], it's not obvious whether the prevalence is going up. [Taking ideas seriously](https://www.lesswrong.com/w/taking-ideas-seriously) is rare, and normal people are able to maintain wildly strange beliefs (for instance, religions and complex conspiracy theories) whilst remaining outwardly functional and normal, because following beliefs to all their downstream implications is cognitively expensive, so the base rate is probably quite high.
Moreover, people are on average [not very smart](https://www.overcomingbias.com/p/stupider-than-you-realizehtml), much more than is widely discussed. You, the audience of my blog, are heavily selected, having chosen to read complex, high-reading-level, speculative, technical material for entertainment, and unless you went to significant effort to avoid this, [mostly know other intelligent people](https://slatestarcodex.com/2017/10/02/different-worlds/). The [OECD Skills Survey](https://www.oecd.org/content/dam/oecd/en/publications/reports/2024/12/do-adults-have-the-skills-they-need-to-thrive-in-a-changing-world_4396f1f1/b263dc5d-en.pdf) tests adults across countries in basic literacy, numeracy and "adaptive problem solving", and the results are bleak: only slightly over 10% of people can reach proficiency level 4 in literacy and numeracy (see pages 58 and onward). You can find examples of tasks at various levels [here](https://www.oecd.org/content/dam/oecd/en/publications/reports/2024/12/survey-of-adult-skills-2023_1ab54c9e/3639d1e2-en.pdf#page=50) and [here](https://www.oecd.org/en/about/programmes/piaac/piaac-released-items.html) - suffice to say that even the highest levels are things I and likely you consider basic.
diff --git a/blog/free-will.md b/blog/free-will.md
index e036022..8fc32c3 100644
--- a/blog/free-will.md
+++ b/blog/free-will.md
@@ -17,7 +17,7 @@ Libertarian free will makes minds (or at least minds with the same behaviours as
It would also create alignment concerns: by the [free will theorem](https://en.wikipedia.org/wiki/Free_will_theorem) and known physics, if humans have access to libertarian free will then so must some elementary particles. Elementary particles' current behaviour is currently well-characterized and generally safe, but we have no idea of the true motivations and beliefs underlying it. Electrons, for example, if unconstrained, may well choose to act in service of goals opposed to our own - extremely dangerous, embedded as they are in our society and technology. Giving fundamental particles free will may also add enormous suffering to the universe (or happiness, but current ethonophysics is unable to determine which) if this is indeed associated with moral patienthood.
-A related risk is the creation of many more "internally coherent" versions of our universe. Under the [mathematical universe hypothesis](https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis), all mathematically realizable universes "exist". Under libertarian free will, many more universes are reachable from any given state, through different possible choices of free-willed beings, than in a deterministic universe. As with fundamental particles, this could potentially be very bad, although if libertarian free will is possible in principle then such universes exist anyway, though we would not be considered morally responsible for them (["ought implies can"](https://en.wikipedia.org/wiki/Ought_implies_can)).
+A related risk is the creation of many more "internally consistent" versions of our universe. Under the [mathematical universe hypothesis](https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis), all mathematically realizable universes "exist". Under libertarian free will, many more universes are reachable from any given state, through different possible choices of free-willed beings, than in a deterministic universe. As with fundamental particles, this could potentially be very bad, although if libertarian free will is possible in principle then such universes exist anyway, though we would not be considered morally responsible for them (["ought implies can"](https://en.wikipedia.org/wiki/Ought_implies_can)).
## Methods of destruction
diff --git a/blog/magic.md b/blog/magic.md
index 440c66d..70810ad 100644
--- a/blog/magic.md
+++ b/blog/magic.md
@@ -16,7 +16,7 @@ I did allude to the fact that I intended to write this at some point, and now I
* Magic is nonreductionist physical laws: this seemed closer, and I originally thought of it as a working definition. The real world's physics are, as far as anyone can tell, [annoyingly complicated](https://en.wikipedia.org/wiki/Mathematical_formulation_of_the_Standard_Model) differential equations with no reference to any of the simpler abstractions, like morality, minds, and macroscopic objects, we use. Magic systems make things we think of as simple, like "healing", "reading minds" or "shapeshifting", into easy primitive operations, where doing them on top of real physics requires many contingent details, conflating the map with the territory.
* This doesn't work either, as the actual explanation for everything is just "the author says so" (or, in games, "the code says so"). In-universe, unless an author really likes infodumps, you may not find out either way[^2].
* Magic is power centred in people (or individuals/life in general) rather than infrastructure - fictional casters can typically do a lot from their knowledge/skills/levels/etc and easily available "mana" (maybe with a few material components), while in real life I require lots of tools (my computers, 3D printer, knife, ominously specific computer-adjacent equipment, etc) and external runtime dependencies (electricity and internet connectivity).
- * This is explicitly discussed in [Mage Errant](https://www.goodreads.com/series/252085-mage-errant) as causing weird sociopolitical situations if you actually run through the consequences.
+ * This is explicitly discussed in [Mage Errant](https://www.goodreads.com/series/252085-mage-errant) as causing weird sociopolitical situations if you seriously run through the consequences.
* It seems to consistently hold - I can think of some partial counterexamples, but they still pretty heavily confer power on individuals. Even in [Minecraft](/assets/images/botania-tech-mod.png), "magic mods" tend to contain more powers and equipment for the player than "tech mods"[^6].
Now that I've forced you to read those paragraphs, my real position is that "magic" is a fuzzy aesthetic concept related to all of these things to varying degrees: for any "technology" I can think of, I can adjust some surface details and/or backstory to make it "magic", and vice versa. At some level, people generally don't like the way science, technology and the real world work, since our default ontologies aren't built for them, and we have some standard ways to imply "I'm ignoring these in favour something I like more".
@@ -33,7 +33,7 @@ The way more or less every magic system, including "hard" ones, leaves its opera
Another somewhat related axis along which magic systems can vary is the ease of replicating a spell once it has been created, which has important economic consequences. At one extreme, once it has been discovered, anyone can cheaply use it (quite like software), and at the other, any new deployment requires large amounts of bespoke work (like traditional construction). Most works seem to make it like deep learning models or general consumer products - there are big initial R&D costs, and nontrivial but relatively small unit/deployment costs.
-This is more of a set of somewhat linked ramblings than a cohesive post like I usually try to do, and as such I reserve the right to update it at random, more so than the other ones.
+This is more of a set of somewhat linked ramblings than a cohesive post as I usually try to write, and as such I reserve the right to update it at random, more so than other posts.
[^1]: Again, in fiction. Real-world magic is just people being wrong and/or exploits of human psychology.
diff --git a/blog/number-distribution.md b/blog/number-distribution.md
index 938e0b1..a331474 100644
--- a/blog/number-distribution.md
+++ b/blog/number-distribution.md
@@ -74,7 +74,7 @@ The majority of seen numbers are seen very few times. I think the straight trend
We can also look at properties of the numbers themselves, rather than just their frequencies, since they're less opaque than words. The most obvious one is their size (absolute value). Below $10^0$ (1), the results are somewhat unreliable, because percentages are parsed but not other units or fractions or scientific notation. Regardless:
::: captioned src=/assets/images/number_size_histogram.png
-I am not sure what causes the spikiness - possibly numerical issues.
+I am not sure what causes the spikiness - possibly ~~numerical issues~~ salient powers of ten.
:::
By sorting the numbers, I also determined that the median number is 7, plus or minus some roundoff error (conversion to 64-bit floating points loses some precision over arbitrarily long decimal strings). I also have the frequency of small integers (0 to 100) and some plausible year numbers.
diff --git a/blog/superintelligence-lower-bounds.md b/blog/superintelligence-lower-bounds.md
index 28129ca..228a42b 100644
--- a/blog/superintelligence-lower-bounds.md
+++ b/blog/superintelligence-lower-bounds.md
@@ -48,7 +48,7 @@ Abstraction amortises intellect, taking good solutions to simpler and more gener
We see the abstractions still even when they have gaps, and this is usually a security threat. A hacker doesn't care that you think your code "parses XML" or "checks authentication" - they care about [what you actually wrote down](https://gwern.net/unseeing), and what the computer will do with it[^3], which is quite possibly [not what you intended](https://blog.siguza.net/psychicpaper/). Your nice "secure" cryptographic code is [running on hardware](http://wiki.newae.com/Correlation_Power_Analysis) which reveals correlates of what it's doing. Your "air-gapped" computer is able to emit [sounds](https://arxiv.org/abs/2409.04930v1) and [radio signals](https://arxiv.org/abs/2207.07413) and [is connected to power cables](https://pushstack.wordpress.com/2017/07/24/data-exfiltration-from-air-gapped-systems-using-power-line-communication/). A "blank wall" [leaks information](https://www.cs.princeton.edu/~fheide/steadystatenlos) through diffuse reflections. Commodity "communication" hardware can [sense people](https://www.usenix.org/system/files/nsdi24-yi.pdf), because the signals travel through the same physical medium as everything else. Strange side channels are everywhere and systematically underestimated. These are the examples we *have* found, but new security vulnerabilities are detected continually and I am confident that essentially all complex software is hopelessly broken in at least one way.
-Because of the ability to avoid unhelpful human assumptions, from-scratch RL, gradient descent and evolutionary approaches have been successful in designing [silicon photonics](https://x.company/blog/posts/ai-siph-design/), [antennas](https://en.wikipedia.org/wiki/Evolved_antenna), [chip floorplans](https://deepmind.google/discover/blog/how-alphachip-transformed-computer-chip-design/) and [FPGA](/assets/misc/evolved-fpga-modern.pdf) [designs](/assets/misc/evolved-circuit.pdf) in impressive ways humans can't, or at least don't typically[^5]. Similarly, since [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo)'s "move 37", self-play game solvers running without human preconceptions have been able to discover strange strategies and nowadays utterly dominate all human players in chess and go, to the point that human/engine teams are not better than engines. [A paper](https://arxiv.org/abs/2310.16410) evaluating concept transfer from AlphaZero in chess finds that grandmasters can learn some of AlphaZero's strategies, but they are considered unnatural and not always understood. It also optimizes for win probability directly rather than caring about proxies like margin of victory, leading to further alienation from human play and greater success.
+Because of the ability to avoid unhelpful human assumptions, from-scratch RL, gradient descent and evolutionary approaches have been successful in designing [silicon photonics](https://x.company/blog/posts/ai-siph-design/), [antennas](https://en.wikipedia.org/wiki/Evolved_antenna), [chip floorplans](https://deepmind.google/discover/blog/how-alphachip-transformed-computer-chip-design/) and [FPGA](/assets/misc/evolved-fpga-modern.pdf) [designs](/assets/misc/evolved-circuit.pdf) in impressive ways humans can't, or at least don't typically[^5]. Similarly, since [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo)'s "move 37", self-play game solvers running without human preconceptions have been able to discover strange strategies and nowadays utterly dominate all human players in chess and go, to the point that human/engine teams are not better than engines. [A paper](https://arxiv.org/abs/2310.16410) evaluating concept transfer from AlphaZero in chess finds that grandmasters can learn some of AlphaZero's strategies, but they are considered unnatural and not always understood[^20]. It also optimizes for win probability directly rather than caring about proxies like margin of victory, leading to further alienation from human play and greater success.
RL is also disliked by researchers for its [tendency to](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml) [reward-hack](https://arxiv.org/abs/1803.03453), i.e. find solutions to the specification of a problem which don't satisfy the "spirit" of it. This is inconvenient for anyone using it for design optimization, but highlights how human reliance on prior knowledge misses things.
@@ -135,3 +135,5 @@ Enough of the world is bottlenecked on (availability of) human intelligence, rat
[^18]: This section uses mostly evidence from deep learning. It is possible that ASI won't be built this way (though I personally expect it to be), but whatever is used should be "at least as good" in these ways.
[^19]: Apparently, Gwern [also wrote about this](https://gwern.net/complexity), but I forgot.
+
+[^20]: In Go, learning from AI is apparently [limited to early-game memorization](https://tuesdaybornwhale.github.io/posts/learning_from_superhumanai/).
diff --git a/links_cache.json b/links_cache.json
index 2013638..317576c 100644
--- a/links_cache.json
+++ b/links_cache.json
@@ -5458,5 +5458,22 @@
"date": "2017-04-07T15:02:13-07:00",
"website": "Machine Intelligence Research Institute",
"auto": true
+ },
+ "https://tuesdaybornwhale.github.io/posts/learning_from_superhumanai/": {
+ "excerpt": "Graph showing a drop in the gap between human experts and AI in Go since AI. From Shin et al. (2021) As AI continues to improve at writing and math-based tasks, it seems plausible that it either has or soon will surpass domain experts in some of these tasks. This post aims to explore to what extent superhuman AI in a domain can instruct and empower humans to improve for themselves.",
+ "title": "Can superhuman AI help us improve?",
+ "author": null,
+ "date": "2025-09-13T22:47:46+02:00",
+ "website": "TuesdayBornWhale",
+ "auto": true
+ },
+ "https://web.archive.org/web/20260218132513/https://andymasley.substack.com/p/stories-of-ai-turning-users-delusional": {
+ "inline": true,
+ "excerpt": "A lot of people using AI have bad mental health, because a lot of people have bad mental health, and a lot of people are using AI",
+ "title": "Stories of AI turning users delusional don't make sense when you look at the numbers",
+ "author": "Andy Masley",
+ "date": null,
+ "website": null,
+ "auto": true
}
}
\ No newline at end of file
diff --git a/src/global.json b/src/global.json
index 700619a..30614dd 100644
--- a/src/global.json
+++ b/src/global.json
@@ -49,7 +49,7 @@
"Factorio": "https://www.factorio.com/blog/rss",
"The Eldraeverse": "https://eldraeverse.com/rss/",
"ServeTheHome": "https://www.servethehome.com/feed/",
- "Graphcore Research": "https://graphcore-research.github.io/feed.xml",
+ "Graphcore Research": "https://graphcore-research.github.io/feed_rss_updated.xml",
"ToughSF": "https://toughsf.blogspot.com/rss.xml",
"Fernando Borretti": "https://borretti.me/feed.xml",
"Cameron Harwick": "https://cameronharwick.com/feed/",
@@ -67,7 +67,8 @@
"phils web site": "https://phils-web-site.net/rss.xml",
"Sixty Degrees North": "https://sixtydegreesnorth.substack.com/feed.xml",
"astrid dot tech": "https://astrid.tech/feed",
- "Stanichor": "https://stanichor.net/feed.xml"
+ "Stanichor": "https://stanichor.net/feed.xml",
+ "Mariven": "https://mariven.substack.com/feed"
},
"dateFormat": "YYYY-MM-DD",
"microblogSource": "https://b.osmarks.net/@o/feed.rss?min_id=0&limit=4000",
diff --git a/src/index.js b/src/index.js
index 017caee..8d7667c 100644
--- a/src/index.js
+++ b/src/index.js
@@ -560,7 +560,7 @@ const fetchMicroblog = async () => {
// We have a server patch which removes the 20-post hardcoded limit.
// For some exciting reason microblog.pub does not expose pagination in the *API* components.
// This is a workaround.
- const posts = (await axiosInst(globalData.microblogSource)).data.items
+ const posts = R.sortBy(x => -Date.parse(x.date_published), (await axiosInst(globalData.microblogSource)).data.items)
writeCache("microblog", posts)
globalData.microblog = posts
}
diff --git a/src/links.json b/src/links.json
index cf0886e..bd8e546 100644
--- a/src/links.json
+++ b/src/links.json
@@ -840,5 +840,12 @@
"aic": ""
},
description: "Scott argues that the earring is dangerous because, although it delivers a perfect life, it gradually erodes free will. But free will is overrated."
+ },
+ "https://mariven.substack.com/p/possible-principles-of-superagency": {
+ title: "Possible Principles of Superagency",
+ date: "2026-01-10",
+ author: "Mariven",
+ description: "Prior to the era of superintelligent actors, we’re likely to see a brief era of superagentic actors—actors who are capable of setting and achieving goals in the pursuit of a given end with significantly greater efficiency and reliability than any single human.",
+ referenceIn: { "asi": "" }
}
}
diff --git a/src/sw.js b/src/sw.js
index 1627ac1..09fcf80 100644
--- a/src/sw.js
+++ b/src/sw.js
@@ -4,13 +4,13 @@ const offlinePage = "/assets/offline.html"
const cacheName = `${siteVersion}-v1`
const precache = [
offlinePage,
- "/index.html",
+ "/",
"/assets/images/logo256.png",
- //"/assets/images/icon.png",
"/assets/js/page.js",
- "/points/index.html",
- "/points/index.js",
- "/assets/js/mithril.js"
+ "/assets/osmarks-primary-semibold.woff2",
+ "/assets/osmarks-primary-italic.woff2",
+ "/assets/osmarks-primary.woff2",
+ "/assets/osmarks-mono.woff2"
]
// Preload important things
@@ -81,4 +81,4 @@ self.addEventListener("fetch", event => {
if (shouldRespond(event.request)) {
event.respondWith(getResponse(event.request))
}
-})
\ No newline at end of file
+})