Minor changes and add the image I apparently forgot

blog/ai-companions.md

@@ -75,7 +75,7 @@ LLMs can't competently do the long-horizon planning sides of this, which is why
 
 This suggests that the role of AI chat systems in many people's[^19] lives could go far beyond "boyfriend/girlfriend in a computer" - without substantial deep technical changes ("just" extensive work on interface design and probably finetuning), the result can be something like "superhumanly compelling omniscient-feeling life coach"[^16] (which may also be your boyfriend/girlfriend)[^13]. If its advice is generally better than what people think of on their own, they will generally defer to the LLM (on increasingly specific decisions, if the technology can keep up).
 
-This sounds like a horrific dystopian nightmare, but in many ways it could be an improvement over the status quo. Almost everyone is continually subject to the opaque whims of economic forces, [unpleasantly accurate modelling](https://x.com/wanyeburkett/status/1927413667173159142) by other people's algorithms (linear regressions) and recommender systems anyway: being managed by a more humanized system more aware of your interests is a step up. There are wider advantages to offloading decisionmaking: [making choices is](https://thezvi.wordpress.com/2017/07/22/choices-are-bad/) [often unpleasant](https://thezvi.wordpress.com/2017/08/12/choices-are-really-bad/), and having them made for you conveniently absolves you from blame in folk morality. It's also plausible to me that most people don't have explore/exploit tradeoffs correctly set for the modern world/big cities and e.g. don't try enough restaurants, hobbies or variety in general.
+This sounds like a horrific dystopian nightmare, but in many ways it could be an improvement over the status quo. Almost everyone is continually subject to the opaque whims of economic forces, [unpleasantly accurate modelling](https://x.com/wanyeburkett/status/1927413667173159142) by other people's algorithms (linear regressions) and recommender systems anyway: being managed by a more humanized system more aware of your interests is a step up. There are wider advantages to offloading decisionmaking: [making choices is](https://thezvi.wordpress.com/2017/07/22/choices-are-bad/) [often unpleasant](https://thezvi.wordpress.com/2017/08/12/choices-are-really-bad/), and having them made for you conveniently absolves you from blame in folk morality. It's also plausible to me that most people don't have explore/exploit tradeoffs correctly set for the modern world/big cities and e.g. don't try enough restaurants, hobbies or variety in general[^21].
 
 However, the incentives of the providers here are very bad: if a user is supported well by your system and becomes better off mentally/financially/etc, you cannot capture that value very easily, whereas it's relatively easy to charge for extra interaction with your product[^17][^20]. Thus, as users enjoy having "takes" and being agreed with, AIs will still be built for sycophancy and not contradict users as much as they should, and will probably aim to capture attention at the expense of some user interests. On the other hand, AI companies are constrained by PR, at least inasmuch as they fear regulation, so nothing obviously or photogenically bad for users, or anything which looks like that, can be shipped[^15]. On the third hand, much user behaviour is "ill-formed and coercible" - if someone hasn't thought deeply about something, they could form several different opinions depending on framing and context, so there are enough degrees of freedom that influence on them and sycophancy don't trade off too badly. I think the result is an unsatisfying compromise in which:
 
@@ -87,7 +87,7 @@ However, the incentives of the providers here are very bad: if a user is support
 
 <details><summary>Aside: local-first AI.</summary>
 
-When this sort of topic, or data privacy issues, are brought up, people often suggest running AI systems locally on your own hardware so it is under your control and bound to you. This will not work. Self-hosting anything is weird and niche enough that very few people do it even with the cost being a few dollars per month for a VPS and some time spent reading the manuals for things which will autoconfigure it for you. LLMs as they currently exist benefit massively from economies of scale (being essentially [memory-bandwidth-bound](/accel/)): without being able to serve multiple users on the same system and batch execution, and to keep hardware busy all the time, it's necessary to accept awful performance or massively underutilize very expensive hardware. Future architecture developments will probably aim to be more compute-bound, but retain high minimum useful deployment sizes. Also, unlike with normal software, where self-hosted replacements can do mostly the same things if more jankily, the best open-ish AI generally lags commercial AI by about a year in general capabilities and longer in product (there's still no real equivalent to ChatGPT Advanced Voice Mode available openly).
+When this sort of topic, or data privacy issues, are brought up, people often suggest running AI systems locally on your own hardware so they are under your control and bound to you. This will not work. Self-hosting anything is weird and niche enough that very few people do it even with the cost being a few dollars per month for a VPS and some time spent reading the manuals for things which will autoconfigure it for you. LLMs as they currently exist benefit massively from economies of scale (being essentially [memory-bandwidth-bound](/accel/)): without being able to serve multiple users on the same system and batch execution, and to keep hardware busy all the time, it's necessary to accept awful performance or massively underutilize very expensive hardware. Future architecture developments will probably aim to be more compute-bound, but retain high minimum useful deployment sizes. Also, unlike with normal software, where self-hosted replacements can do mostly the same things if more jankily, the best open-ish AI generally lags commercial AI by about a year in general capabilities and longer in product (there's still no real equivalent to ChatGPT Advanced Voice Mode available openly).
 
 There's a further problem: training (pre- and post-) is much trickier, both in necessary expertise to resolve all the details and in compute cost, than inference, so selfhosted systems will still be constrained by what large organizations release (although I expect better personalization technology to be developed to make the rest of this work, it will likely only allow shallower changes to model behaviour than a full finetune). Incidental presence of chatbot outputs in pretraining data, and synthetic data pipelines, also bakes in strange and hard-to-remove behaviours.
 
@@ -140,3 +140,5 @@ There is, of course, science fiction about this:
 [^19]: I'm being somewhat loose here: this is going to be differ massively by demographic. Many older people have not tried ChatGPT at all, and many young people are already quite close (continuous use of earphones outside and offloading as much schoolwork as possible to LLMs).
 
 [^20]: I expect someone to experiment with an ad-supported/affiliate-links model (more integrated than Bing's), which will have similar issues, and likely exciting scandals about missold products.
+
+[^21]: A while ago, some sort of "quests as a service" service solving this could have been a reasonable standalone product; in fact, it was the original impetus for this blog post before it was expanded and reconsolidated. It is possible that it existed and I didn't hear about it, or it was poorly pitched. If you tried to build it now, you would run into the issue that it could easily be replaced with a few scripts and a short prompt.