The space of minds

> LLMs are humanity's "first contact" with non-animal intelligence.

I'd say corporations are also a form of non-animal intelligence, so it's not exactly first contact. In some ways, LLMs are less alien, in other ways they're more.

There is an alignment problem for both of them. Perhaps the lesson to be drawn from the older alien intelligence is that the most impactful aspect of alignment is how the AI's benefits for humanity are distributed and how they impact politics.

Many researchers may be interested in making minds that are more animal-like and therefore more human. While this makes sense to certain extent to gain capabilities, if you take it too far then you run into obvious problems.

There is enough science fiction demonstrating reasons for not creating full-on digital life.

It seems like for many there is this (false) belief that in order to create a fully general purpose AI, we need a total facsimile of a human.

It should be obvious that these are two somewhat similar but different goals. Creating intelligent digital life is a compelling goal that would prove godlike powers. But we don't need something fully alive for general purpose intelligence.

There will be multiple new approaches and innovations, but it seems to me that VLAs will be able to do 95+% of useful tasks.

Maybe the issues with brittleness and slow learning could both be addressed by somehow forcing the world models to be built up from strong reusable abstractions. Having the right underlying abstractions available could make the short term adaptation more robust and learning more efficient.

"Animals experience pressure for a lot more "general" intelligence because of the highly multi-task and even actively adversarial multi-agent self-play environments they are min-max optimized within, where failing at any task means death. In a deep optimization pressure sense, LLM can't handle lots of different spiky tasks out of the box (e.g. count the number of 'r' in strawberry) because failing to do a task does not mean death."

Point to me a task that a human should be able to perform and I will point to you a human who cannot perform that task, yet has kids.

Survival is not a goal, it is a constraint. Evolution evolves good abstractions because it is not chasing a goal, but rather it creates several million goals with each species going after it's own.

https://arxiv.org/abs/2505.11581

It's an important point to make.

LLMs of today copy a lot of human behavior, but not all of their behavior is copied from humans. There are already things in them that come from elsewhere - like the "shape shifter" consistency drive from the pre-training objective of pure next token prediction across a vast dataset. And there are things that were too hard to glimpse from human text - like long term goal-oriented behavior, spatial reasoning, applied embodiment or tacit knowledge - that LLMs usually don't get much of.

LLMs don't have to stick close to human behavior. The dataset is very impactful, but it's not impactful enough that parts of it can't be overpowered by further training. There is little reason for an LLM to value non-instrumental self-preservation, for one. LLMs are already weird - and as we develop more advanced training methods, LLMs might become much weirder, and quickly.

Sydney and GPT-4o were the first "weird AIs" we've deployed, but at this rate, they sure wouldn't be the last.

Yes. Sometimes people treat intelligence as a single line, or as nested sets, where a greater intelligence can solve all the problems a lesser one can, plus more.

While in some contexts these are useful approximations, they break down when you try to apply them to large differences not just between humans, but between species (for a humorous take, see https://wumo.com/wumo/2013/02/25), or between humans and machines.

Intelligence is about adaptability, and every kind of adaptability is a trade-off. If you want to formalize this, look at the "no free lunch" theorems.

One thing missing from this framing: the feedback loop speed. Animal evolution operates on generational timescales, but LLM "commercial evolution" happens in months. The optimisation pressure might be weaker per-iteration but the iteration rate is orders of magnitude faster.

Curious whether this means LLMs will converge toward something more general (as the A/B testing covers more edge cases) or stay jagged forever because no single failure mode means "death".

What about evolutionary intelligence optimization pressure?

Genetic algorithms are smart enough to make life. It seems like genetic algorithms don't care how complex a task is since it doesn't have to "understand" how its solutions work. But it also can't make predictions. It just has to run experiments and see the results.

Human intelligence is emergent from physical/material interactions. It is constant and self-evolutive. Artificial intelligence is data processing of a very limited number of inputs, ignoring all others. Less likely to become self-conscious. This is for now just a super calculator/ prediction machine. And it is probably better that way. The "thinking" processi didn't evolve from silicium interacting with oxygen. The gradient is not over physical data but purely digital information.

In a nutshell, we have the body before the brain, while AIs have the brain before the body.

It helps me think of this problem in terms of "crowpower".

[1] https://taylor.town/crowpower

This strongly reminds me of the Orthogonality Thesis.

https://www.lesswrong.com/w/orthogonality-thesis

> an LLM with a knowledge cutoff that boots up from fixed weights, processes tokens and then dies

Mildly tangential: this demonstrates why "model welfare" is not a concern.

LLMs can be cloned infinitely which makes them very unlike individual humans or animals which live in a body that must be protected and maintain continually varying social status that is costly to gain or lose.

LLMs "survive" by being useful - whatever use they're put to.

See also a paper from before the ice age (2023): Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321

What's alien about arithmetic? People invented it. Same w/ computers. These are all human inventions. There is nothing alien about them. Suggesting that people think of human inventions as if they were alien artifacts does not empower or enable anyone to get a better handle on how to properly utilize these software artifacts. The guruism in AI is not helpful & Karpathy is not helping here by adopting imprecise language & spreading it to his followers on social media.

If you don't understand how AI works then you should learn how to put together a simple neural network. There are plenty of tutorials & books that anyone can learn from by investing no more than an hour or two every day or every other day.

Of possible interest, Roman Yampolskiy's essay The Universe Of Minds:

https://arxiv.org/pdf/1410.0369

>The paper attempts to describe the space of possible mind designs by first equating all minds to software. Next it proves some interesting properties of the mind design space such as infinitude of minds, size and representation complexity of minds. A survey of mind design taxonomies is followed by a proposal for a new field of investigation devoted to study of minds, intellectology, a list of open problems for this new field is presented

[flagged]

I'm a simple software engineer not specialized in AI, and I know Karpathy is a heavyweight but help me understand: this kind of narrative about the intelligence of LLMs is an actual philosophical paradigm around IA or it's just him getting high on his own supply? I don't really know how to take his ideas since the infamous (?) _vibe coding_ concept.

It's fine to say that behavior can emerge from complex systems, but if you go beyond that and actually try to explain how, then you are screwed.

That's why consciousness, how the brain works, etc never moves on. Something always drags it down and force it to be "complex emergent behavior we cannot explain, and damn you if you try!".

So, it's particles that act funny, and we are that as well. Because "it can't be anything other than that". If it's not that, then it's souls, and we can't allow that kind of talk around here.

Until we can safely overcome this immaturity, we will never be able to talk about it properly.

The "space of minds" is an ideological minefield spanning centuries. It was being probed way before we invented machines.