Unthinking meat

“You’re saying they have an exquisitely sensitive and accurate sensory apparatus, and an unbounded memory capacity, and fully general problem-solving faculties?”

“Well, sort of:

“When they’re not focussing, which is 95% of the time, they can’t really be said to be intelligent at all. Much of what they say and do is hollow reflex motion.

“They also fill most of their bandwidth up with information which is worthless at best and usually actively misleading. They find fabrications more convincing than data. They rarely do what they think is most important.

“They also keep their current sense data, memories, moral evaluations, aesthetic evaluations, and political evaluations - their lust, fear, and avarice - all in the same chamber. This makes them confuse fact with value, rights with wishes, and desire with everything.

“Most of their lives are spent on coalition maintenance, social grooming, and monitoring and enforcing hierarchy.

“They have no access to much of the most action-relevant parts of their processor, which has developed backdoors to systematically delude the narrator about the system’s goals and motives. They are in effect incapable of honesty.

“While the processor is capable of running formal logic, very very slowly, in practice they use a series of appallingly non-Bayesian evolutionary algorithms to do almost all of their reasoning, including about the central concerns of their lives, mates, careers, and finance.”

“… not what you’d call a threat then.”

“Well, not to us.”

See also

• The melancholy of pareidolia
• Why is quality rare?
• Pieties
• Where does reason end?
• Heuristics, cognitive miser, attribute substitution.
• Simler, Elephant in the Brain
• Constantin, Humans Who Are Not Concentrating Are Not General Intelligences
• Crichton, Gell-Mann Amnesia
• Taleb, Against News