“I know he can get the job, but can he do the job?”

— Mr. Waturi, Joe Versus the Volcano

The human body is a hot topic. After months of an AI-fueled hype hurricane, it seems as though some clear lines are emerging. A ceiling exists on the returns we can expect from our current approaches. Their intelligence is limited by the very aspect of their nature that infuriates us the most — their disembodiment. Just this week, Jacob Browning and Yann LeCun published an article detailing this perspective. Without sensory experience and its systemic feedback, accurate models of the universe and the intelligence they enable seem impossible to build.

This is profound. It is profound, not merely as an interesting problem for AI researchers, but because of what it implies as a model of intelligence — for us. What exactly is the body’s role in intelligence? If we can answer that question, we might be able to build tools that capitalize on the answer, taking advantage of our mind’s relationship with our body to help us become better problem solvers. What might these tools look like?

Embodied Intelligence

It’s fairly clear that the state of our peripheral nerves and organs influences what goes on upstairs. For example, take memory, the basis of intelligence. When we talk about memory, we often break it into its constituent pieces, such as declarative (associated with recall, reasoning, and “intelligent” language use) and procedural (associated with bike riding and other “muscle memories”). How we use the rest of our body affects these systems. Anecdotally, many studies have found that aerobic exercise can improve the encoding and later retrieval of both of these kinds of memory.

What about cognition? One of my favorite examples of this comes from Shumita Roy and Norman Park, of York University, Toronto. Their approach is interesting because they focus on a facet of our intelligence that we share with only a handful of other creatures: tool use. In one study, they attempted to deal with both declarative and procedural memory systems in isolation, but found that effective tool use required their collaboration.

However, that’s all happening downstairs … what about more abstract reasoning? In his 2017 book, The Reading Mind, Daniel Willingham makes a strong case for the primitive nature of reading, and all of the activities it enables. He explains a simple progression, whereby our visual, spacial, tactile, and auditory senses — highly developed for survival — were coopted for the purposes of communicating bits of information between individuals. This shows up in obvious ways, such as pictography. However, it also seems to have influenced the angles and symbols we used when we developed alphabets, choosing geometries most similar to our natural environment. It seems very likely, though there is still debate, that this same process was at work in the development of all forms of symbolic reasoning.

Learning, in the sense of forming accurate, useful mental models of the way our universe or arbitrary systems work, requires sensory experience and the grounding it provides.

I Prefer to Text

Let’s be honest, though … this is common sense. From apprenticeships to medical residencies, almost every profession has a notion of “hands-on” training. In high-risk professions, it’s a requirement for practice. This leads to the universally recognized “paradox of experience,” featured in Tom Hanks and Meg Ryan’s 1990 classic, Joe vs. the Volcano. We know that someone can “get the job,” but don’t know that they can “do the job” until they’ve actually done it. Because sensory, experiential feedback is so important to cognition, we’re naturally hesitant to put people in high-risk scenarios where they haven’t already demonstrated a capacity for intelligent behavior. More philosophically, what we’re talking about is the root of empiricism.

But if this is really so intuitively obvious … why is our world the way it is? We live in a world where text is king. In the last several decades, this has shifted slightly as media creation tools have become more powerful and accessible. However, their products are still passive. Images and videos may provide a wider funnel, but they don’t involve systematic feedback.

The problem is particularly pronounced where learning is most vital: the education sector. Why do students still prepare themselves for linguistic assessments of their most vital learning outcomes? Why on earth do we go on living with bone-dry textbooks? Why do we rely on language alone to communicate truths that can only be learned if they are discovered empirically? It’s almost as if the human race, as a whole, were echoing Benedict Cumberbatch’s interpretation of Sherlock Holmes on the BBC when he said…

In their article this week, Yann LeCun and Jacob Browning posit an interesting perspective on this. Language, they write, is a compression of information. Decompressing or interpreting it requires (1) shared symbols, (2) shared rules for those symbols, and (3) a shared body of information that those symbols map, or relate to. When these all hold, it’s just as well that we “prefer to text.” It’s very efficient! Textual learning and communication can fail us at any one of these junctures, though. Most often, it’s the final one. What if there were a way to bake shared experience into our communications, though?

Tools for a Different Kind of Thought

From a communications perspective, this could mean providing some kind of reference. We need some way of saying “This is the set of steps that I followed to acquire this concept, so if you don’t already have it, you can go get it.” In hypermedia systems, such as the Web, we do this by hyperlinks. If you don’t know the meaning of a word or the details surrounding a concept, all you have to do is click through to it. What would it look like if these links, instead of leading someone to more text, lead them to a progression of puzzles, problems, or exercises? By interacting with them and solving them, we could more effectively internalize the concepts we lacked.

Going further, what if we had a shared language for describing the connections between experiences and the concepts they build? What would it look like to sit down and creatively play with this language, drafting with it as one would an essay? How might we define and rearrange these associations? Imagine a group of teachers creating a lesson plan, as easily as a design team might collaborate over a Figma document.

What would this look like for memories that are almost entirely procedural? Instead of simply communicating declarative concepts, we could communicate entire skill sets. More than just watching a video on YouTube that demonstrates an action, we could encapsulate and send the steps required to develop a given skill, in the form of a layered progression of experiences. Multiple different pathways could be compared, contrasting the experiences, problems, or benchmarks at which they climax.