> By biological immortality I mean your body doesn't fail, or is fixable - you don't die of old age or random kidney failure. You still die if someone cuts off your head.

That sounds entirely possible then. I'd wager the limits we'd hit would be memories - it's not known if our brains, in their current form, would be able to cope with hundreds of years of memories.

>> self-replicating robots are quite literally what all of biology is

> By the same argument AGI is just same as human brain.

Yes, but the main difference is that biology is merely too big for us to keep track of - it just has too many moving pieces that have been, from our perspective, heavily overfitted. Still, we've reached the level where we can reprogram some of the nanobots. In contrast, the topic of intelligence still has a lot of big mysteries.

Another way of looking at it is, we already have a well-mapped way of building nanotechnology: reprogramming the one that is around us, and is us. Plenty of little self-replicating programmable bots to pick from. But we're not at similar stage with poking in brains just yet.

> Many people think ChatGPT is so ompressive, that we are on the cust of AGI. I am convinced otherwise - chat gpt makes limitations of current AI approached very clear.

I agree. Though to me, it also revealed limitations of human cognition (or rather, it was already quite apparent at GPT-2 level; ChatGPT is only rubbing it in everyones' faces).

Have you ever felt that your own thinking, in many situations, is mostly cache lookups? That your speech and inner monologue (if you have one) both resemble a Markov chain, and your "self" mostly just observes and censors the output? I certainly did, quite a lot over the decades. I know others did, many obviously rejecting it as silly association. But what the recent LLMs show us is that maybe, just maybe, it's not silly at all. Which invites the question, if a lot of our cognition works this way, then just how much more complicated are the bits that don't?

(Another thing ChatGPT and the like are making apparent, is that AI risk isn't tied to reaching AGI. I think people kind of assumed it would take AGI taking off to end human civilization, but at this point I believe it's pretty clear that LLMs as a class of models are up to the task already - all they need is to have more memory, access to APIs that let them affect the real world and observe the results, and being run continuously; Reddit corpus already supplies the token association patterns that would let them end the world if given the means...)

> I think people kind of assumed it would take AGI taking off to end human civilization, but at this point I believe it's pretty clear that LLMs as a class of models are up to the task already

Absolutely, and in addition to that, people with zero grasp of the risks make decisions to trust AI with dangerous things -> money, vehicles, markets, justice, etc.

> Which invites the question, if a lot of our cognition works this way, then just how much more complicated are the bits that don't?

I this we could be mistaking quantity for the important.

Maybe other aspects of thinking, the ones we perceive as mundane, are actually impossible for a computer.

Consider mechanical tasks - a hundred years ago we had huge breakthroughs with engines - oh look, we can make crazy strong machines, we will soon automate all manual labour and blue collar jobs will be a thing of the past!

Well, turns out power was the easy part. A robot arm can lift a car, but getting precision took decades. So now we have power and precision, but try to get a robot arm (under computer control) to undo buttons on your T-shirt without ripping the T-shirt or your face. If a robot arm touches a piece of fabric, or you skin, it has no idea. They routinely kill people.

If you need non-repetitive dexterous movement with fine sensory input, that's basically beyond the cutting edge for today's robots. We can't automate sewing of clothes, cleaning of delicate items, etc.

I agree the stuff any five year old can do is well beyond current AI/robotic tech.

Let's focus on actual intelligence, though. What, exactly, would a 100-mind AGI be able to do that a 1-mind human can't? And what can a 1-mind human do that the 100-mind AGI can't?

My guess is that the 100-mind AGI is capable of building nearly perfectly accurate internal models of the 1-mind humans. So the AGI runs the model, and figures out the exact stimulations needed to get the 1-mind human to do what the AGI wants. You will not be able to resist.

The 1-mind human might be able to come up with physical theories that the AGI has no way to deduce. Maybe. (These physical theories may need experimental corroboration. If the AGI doesn't have access to the inner details of the latest ultra-LHC experiments, it may not be able to properly theorize.)