Kind: captions
Language: en
the following is a conversation with
Jeff Hawkins he's the founder of the
redwood centre for theoretical
neuroscience in 2002 and Numenta in 2005
in this 2004 book titled on intelligence
and in the research before and after he
and his team have worked to
reverse-engineer the neocortex and
proposed artificial intelligence
architectures approaches and ideas that
are inspired by the human brain these
ideas include hierarchical temporal
memory
htm' from 2004 and new work the
thousands brains theory of intelligence
from 2000 17 18 and 19
Jeff's ideas have been an inspiration to
many who have looked for progress beyond
the current machine learning approaches
but they have also received criticism
for lacking a body of empirical evidence
supporting the models this is always a
challenge when seeking more than small
incremental steps forward in AI Jeff was
a brilliant mind and many of the ideas
he has developed and aggregated from
your science are worth understanding and
thinking about there are limits to deep
learning as it is currently defined
forward progress in AI is shrouded in
mystery my hope is that conversations
like this can help provide an inspiring
spark for new ideas this is the
artificial intelligence podcast if you
enjoy it subscribe on youtube itunes or
simply connect with me on twitter at lux
friedman spelled fri d and now here's my
conversation with Jeff Hawkins
are you more interested in understanding
the human brain or in creating
artificial systems that have many of the
same qualities but don't necessarily
require that you actually understand the
underpinning workings of our mind so
there's a clear answer to that question
my primary interest is understanding the
human brain no question about it but I
also firmly believe that we will not be
able to create fully intelligent
machines until we understand how the
human brain works so I don't see those
as separate problems
I think there's limits so what can be
done with machine intelligence if you
don't understand the principles by which
the brain works and so I actually
believe that studying the brain is
actually the fast the fastest way to get
to machine intelligence and within that
let me ask the impossible question how
do you not define but at least think
about what it means to be intelligent so
I didn't try to answer that question
first we said let's just talk about how
the brain works let's figure out how
certain parts of the brain mostly the
new your cortex but some other parts to
the parts of the very most associated
intelligence and let's discover the
principles about how they work because
intelligence isn't just like some
mechanism and it's not just some
capabilities it's like okay we don't
even have know where to begin on this
stuff and so now that we've made a lot
of progress on this after we've made a
lot of progress on how the neocortex
works and we can talk about that I now
have a very good idea what's going to be
required to make intelligent machines I
can tell you today you know some of the
things are gonna be necessary I believe
to create intelligent machines well so
we'll get there we'll get to the
neocortex and some of the theories of
how the whole thing works and you're
saying as we understand more and more
about the neocortex about our own human
mind we'll be able to start to more
specifically define what it means to be
intelligent it's not useful to really
talk about that until I don't know if
it's not useful
look there's a long history of AI as you
know right and there's been different
approaches taken to it and who knows
maybe they're all useful right so you
know the good old fashioned AI the
expert systems current convolution
neural networks they all have their
utility
they all have a value in the world but I
would think almost everyone agree that
none of them are really intelligent in a
set of a deep way that that humans are
and so it's it's just the question is
how do you get from where those systems
were or are today to where a lot of
people think we're going to go and just
big big gap there a huge gap and I think
the quickest way of bridging that gap is
to figure out how the brain does that
and then we can sit back and look and
say oh what do these principles that the
brain works on are necessary and which
ones or not
kula we don't have to build this in and
telogen machines aren't going to be
built out of you know organic living
cells but there's a lot of stuff that
goes on the brain it's going to be
necessary so let me ask me B before we
get into the fun details
let me ask me to get depressing or a
difficult question do you think it's
possible that we will never be able to
understand how our brain works that
maybe there's aspects to the human mind
like we ourselves cannot introspectively
get to the core that there's a wall you
eventually hit yeah I don't believe
that's the case
I have never believed that's the case
there's not have been a single thing
we've ever humans have ever put their
minds to so we've said oh we reached the
wall we can't go any further it just
people keep saying that people used to
believe that about life you know Ilan's
Vittal right there's like what's the
difference in living matter and
nonliving matter something special you
never understand we no longer think that
so there's there's no historical
evidence to suggest is the case and I
just never even considered that's a
possibility I would also say today we
understand so much about the neocortex
we've made tremendous progress in the
last few years that I no longer think of
it as an open question the answers are
very clear to me and the pieces we know
we don't know I are clearly me but the
framework is all there and it's like oh
okay we're gonna be able to do this this
is not a problem anymore it just takes
time and effort but there's no mystery a
big mystery anymore so then let's get it
into it for people like myself we're not
very well versed in the human brain
except my own can you describe to me at
the highest level what are the different
parts of the human brain and then
zooming in on the neocortex the parts of
the neocortex and so on a quick overview
yeah sure human brain we can divide it
roughly into two parts there's the old
parts lots of pieces and then there's a
new part the new part is the neocortex
it's new because it didn't exist before
mammals the only mammals have a
neocortex and in humans it's in primates
it's very large in the human brain the
neocortex occupies about seventy to
seventy-five percent of the volume of
the brain it's huge and the old parts of
the brain are there's lots of pieces
there there's a spinal cord and there's
the brain stem and the cerebellum and
the different parts of the basal ganglia
and so on in the old parts of the brain
you have the autonomic regulation like
breathing and heart rate you have basic
behaviors so like walking and running or
controlled by the old parts of the brain
all the emotional centers of the brain
are in the old part of the brains when
you feel anger or hungry lust with
things like that those are all in the
old parts of the brain
and and we associate with the neocortex
all the things we think about as sort of
high-level perception and cognitive
functions anything from seeing and
hearing and touching things to language
to mathematics and engineering and
science and so on those are all
associative the neocortex and they're
certainly correlated our abilities in
those regards are correlated with the
relative size of our neocortex compared
to other mammals so that's like the
rough division and you obviously can't
understand the new your cortex is
completely isolated but you can
understand a lot of it with just a few
interfaces so the all parts of the brain
and so it it gives you a system to study
the other remarkable thing about the
neocortex compared to the old parts of
the brain is the neocortex it's
extremely uniform it's not visually or
anatomically or it's very sucky
I always like to say it's like the size
of a dinner napkin about two and a half
millimeters thick and it looks
remarkably the same everywhere
everywhere you look and that children
have millimeters is this detailed
architecture and it looks remarkably the
same everywhere and that's a cross
species the mouse versus a cat and a dog
and a human or if you look at the old
parts of the brain there's lots of
little pieces do specific things so it's
like the old parts of a brain evolved
look this is the part that controls
heart rate and this is the part that
controls this and this is this the kind
of thing and that's this kind of thing
and he's evolved for eons a long long
time and they have their specific
functions and all sudden mammals come
along and they got this thing called the
neocortex and it got large by just
replicating the same thing over and over
and over again this is like wow this is
incredible so all the evidence we have
and this is an idea that was first
articulated in a very cogent and
beautiful argument by a guy named Vernon
mal Castle in 1978 was that the
neocortex all works on the same
principle so language hearing touch
vision engineering all these things are
basically underlying or all built in the
same computational substrate they're
really all the same problem
all over the building blocks all look
similar yeah and they're not even that
low-level we're not talking about like
like neurons we're talking about this
very complex circuit that exists
throughout the neocortex is remarkably
similar it is it's like yes did you see
variations of it here and there more of
the cell uh so that's not all so on but
what now encruster argued was it says
you know if you take a section on your
cortex why is one a visual area and one
is a auditory area or why 'since and his
answer was it's because one is connected
to eyes and one is connected ears
literally you mean just its most closest
in terms of number of connections to
listen sir literally if you took the
optic nerve and it attached it to a
different part of the neocortex that
part would become a visual region this
actually this experiment was actually
done by Mercosur oh boy
and uh in in developing I think it was
lemurs I can't remember there was some
animal and and there's a lot of evidence
to this you know if you take a blind
person the person is born blind at Birth
they they're born with a visual
neocortex it doesn't may not get any
input from the eyes because of some
congenital defect or something and that
region become does something else it
picks up another task so and it's it's
so it's just it's this very complex
thing it's not like oh they're all built
on neurons no they're all built in this
very complex circuit and and somehow
that circuit underlies everything and so
this is the it it's called the common
cortical algorithm if you will some
scientists just find it hard to believe
and they decide can't really that's true
but the evidence is overwhelming in this
case and so a large part of what it
means to figure out how the brain
creates intelligence and what is
intelligence in the brain is to
understand what that circuit does if you
can figure out what that circuit does as
amazing as it is then you can then you
then you understand what all these other
cognitive functions are so a few words
to sort of put neural cortex outside of
your book on intelligence you look if
you wrote a giant tome a textbook on the
neocortex and you look maybe a couple
centuries from now how much of what we
know now would still be
two centuries from now so how close are
we in terms of understand I have to
speak from my own particular experience
here so I run a small research lab here
it's like yeah it's like I need other
research lab I'm the sort of the
principal investigator there was
actually two of us and there's a bunch
of other people and this is what we do
we started the neocortex and we
published our results and so on so about
three years ago we had a real
breakthrough in this in this film just
tremendous spectrum we started we've now
published I think three papers on it and
so I have I have a pretty good
understanding of all the pieces and what
we're missing I would say that almost
all the empirical data we've collected
about the brain which is enormous if you
don't know the neuroscience literature
it's just incredibly big and it's it's
the most part all correct its facts and
and experimental results and
measurements and all kinds of stuff but
it none none of that has been really
assimilated into a theoretical framework
it's it's data without it's in the
language of Thomas Kuhns a historian it
would be a sort of a pre paradigm
science lots of data but no way to fit
in together I think almost all of that's
correct it's gonna be some mistakes in
there and for the most part there aren't
really good cogent theories about it how
to put it together it's not like we have
two or three competing good theories
which ones are right and which ones are
wrong it's like yeah people just like
scratching their heads wrong things you
know some people given up on trying to
like figure out what the whole thing
does in fact is very very few labs that
we that we do that focus really on
theory and all this unassimilated data
and trying to explain it so it's not
like we have we've got it wrong it's
just that we haven't got it at all so
it's really I would say pretty early
days in terms of understanding the
fundamental theories forces of the way
our mind works I don't think so that
what I would have said that's true five
years ago so I
we have some really big breakthroughs on
this recently and we started publishing
papers on this so look it but so I don't
think it's I you know I'm an optimist
and from where I sit today most people
would disagree with this but from where
I sit city from what I know uh it's not
super early days anymore we are it's
it's you know the way these things go is
it's not a linear path right you don't
just start accumulating and get better
and better better no you okay all the
stuff you've collected none of it makes
sense all these different things we just
turn around and then you're gonna have
some breaking points or all sudden oh my
god now we got it right so that's how it
goes and science and I feel like we
passed that little thing about a couple
years ago all that big thing a couple
years ago so we can talk about that time
will tell if I'm right but I feel very
confident about it that's my moment to
say it on tape like this at least very
optimistic so let's before those few
years ago
let's take take step back to HTM the
hierarchical temporal memory theory
which you first proposed on intelligence
and went through a few different
generations can you describe what it is
how it evolved through the three
generations yes you first put it on
paper yeah so one of the things that
neuroscientists just sort of missed for
many many years and ice and especially
people were thinking about theory was
the nature of time in the brain brains
process information through time the
information coming into the brain is
constantly changing the patterns from my
speech right now if you're listening to
it at normal speed
we'd be changing on IRA's about every 10
milliseconds or so you'd have it change
this constant flow when you look at the
world your eyes are moving constantly
three to five times a second and the
inputs complete completely if I were to
touch something like a coffee cup as I
move my fingers that input changes so
this idea that the brain works on time
changing patterns is almost completely
or was almost completely missing from a
lot of the basic theories like fears of
vision and so it's like oh no we're
going to put this image in front of you
and flash it and say what is it a
convolutional neural networks work that
way today right you know classify this
picture but that's not what visions like
vision is this sort of crazy time-based
pattern that's going all over the place
and
was touched and so is hearing so the
first part of a hierarchal temporal
memory was the temporal part it's it's
the same you you won't understand the
brain orally understand intelligent
machines unless you're dealing with
time-based patterns the second thing was
the memory component of it was is to say
that we aren't just processing input we
learn a model of the world that's the
memory stands for that model we have to
the point of the brain part of the New
York white chest it learns a model of
the world we have to store things that
our experience is in a form that leads
to a model the world so we can move
around the world we can pick things up
and do things and navigate know how it's
going on so that's that's what the
memory referred to and many people just
they were thinking about like certain
processes without memory at all it just
like processing things and finally the
hierarchical component was reflection to
that the New York or check so though
it's just uniform sheet of cells
different parts of it project to other
parts which project to other parts and
there is this sort of rough hierarchy in
terms of them so the hyperbole temporal
memory is just saying look we should be
thinking about the brain as time-based
you know model memory based and
hierarchical processing and and that was
a placeholder for a bunch of components
that we would then plug into that we
still believe all those things I just
said but we now know so much more that
I'm stopping to use the word hierarchal
thumper memory yeah because it's it's
insufficient to capture the stuff we
know so again it's not incorrect but
it's I now know more and I would rather
describe it more accurately yeah so
you're basically we can think of HTM as
emphasizing that there's three aspects
of intelligence that important to think
about whatever the whatever the eventual
theory it converges to yeah so in terms
of time how do you think of nature of
time across different time scales so you
mentioned things changing a sensory
inputs changing every 10 being myself
what about it every few minutes every
few yeah Montse well if you think about
a neuroscience problem the brain problem
neurons themselves can stay active for
certain perks of time they parts of the
brain with this doctor 4-minute
you know so you could hold up a certain
perception or an activity for a certain
period of time but not most of them
don't last that long
and so if you think about your thoughts
are the activity neurons if you're going
to want to involve something that
happened a long time ago I'm even just
this morning for example the neurons
haven't been active throughout that time
so you have to store that so if I asked
you what did you have for breakfast
today that is memory that is you've
built into your model of the world now
you remember that and that memory is in
the in the synapses it's basically in
the formation of synapses and so it's it
you're sliding into what you know is two
different time scales there's time
scales of which we are like
understanding my language and moving
about and seeing things rapidly and over
time that's the time scales of
activities of neurons but if you want to
get longer time scales then it's more
memory and we have to invoke those
memories to say oh yes well now I can
remember what I had for breakfast
because I stored that someplace I may
forget it tomorrow but I'd stored for
afor now so this is memory also need to
have so the hierarchical aspect of
reality is not just about concepts it's
also about time do you think of it that
way yeah time is infused in everything
it's like you really can't separate it
out if I ask you what is the what is
your you know how's the brain learning a
model of this coffee cup here I have a
coffee cup and I'm at the coffee cup I
said well time is not an inherent
property of this of this of the model I
have of this cup whether it's a visual
model or attack the model I can sense it
through time but if the model self
doesn't really much time if I asked you
if I said well what is the model of my
cell phone my brain has learned a model
of the cell phone so if you have a smart
phone like this and I said well this has
time aspects to it I have expectations
when I turn it on what's gonna happen
what water how long it's going to take
to do certain things if I bring up an
app what sequences and so I have instant
it's all like melodies in the world you
know yeah melody has a sense of time so
many things in the world move and act
and there's a sense of time related to
them some don't but most things do
really so it's it's sort of infused
throughout the models of the world you
build a model of the world you're
learning the structure of the objects in
the world and you're also learning how
those things change through time okay so
it's it's it really is just a fourth
dimension that's infused deeply and they
have to make sure that your models have
been intelligence incorporated so like
you mentioned the state of neuroscience
is deeply empirical a lot of data
collection it's uh you know that's
that's where it is using meshing Thomas
Kuhn right yeah and then you're
proposing a theory of intelligence and
which is really the next step the really
important stuff to take but why why is
HTM or what we'll talk about soon
the right theory so is it more in this
it what is it backed by intuition is it
backed by evidence is it backed by a
mixture of both is it kind of closer to
or string theories in physics where this
mathematical components would show that
you know what it seems that this it fits
together too well for not to be true
which is what we're string theory is is
that where your fix of all those things
although definitely where we are right
now it's definitely much more on the
empirical side than let's say string
theory the way this goes about we're
theorists right so we look at all this
data and we're trying to come up with
some sort of model that explains it
basically and there's yeah unlike string
theory there's this vast more amounts of
empirical data here that I think than
most physicists deal with
and so our challenge is to sort through
that and figure out what kind of
constructs would explain this and when
we have an idea you come up with a
theory of some sort you have lots of
ways of testing it first of all I am you
know there are hundred years of
assimilated unassimilated empirical data
from neuroscience so we go back and read
papers we said oh did someone find this
already with you we can predict x y&z
and maybe no one's even talked about it
since 1972 or something but we go back
and find out we say Oh either it can
support the theory or it can invalidate
the theory and we said okay we have to
start over again oh no it's the poor
let's keep going with that one so the
way I kind of view it when we do our
work we come up we we look at all this
empirical data and it's it's what I call
is a set of constraints we're not
interested in something that's
biologically inspired we're trying to
figure out how the actual brain works so
every piece of empirical data is a
constraint on a theory in theory if you
have the correct theory it needs to
explain every pin right so we have this
huge number of constraints on the
problem which initially makes it very
very difficult if you don't have any
constraints you can make up stuff all
the day you know here's an answer how
you can do this you can do that you can
do this but if you consider all biology
as a set of constraints all neuroscience
instead of constraints and even if
you're working on one little part of the
neocortex for example there are hundreds
and hundreds of constraints these are
empirical constraints that it's very
very difficult initially to come up with
a radical framework for that but when
you do and it solves all those
constraints at once you have a high
confidence that you got something close
to correct it's just in mathematically
almost impossible not to be so it that's
the the curse and the advantage of what
we have the curse is we have to solve we
have to meet all these constraints which
is really hard but when you do meet them
then you have a great confidence that
you discover something in addition then
we work with scientific labs so we'll
say oh there's something we can't find
we can predict something but we can't
find it anywhere in the literature so we
will then we have people we collaborated
with
say that sometimes they'll say you know
I have some collected data which I
didn't publish but we can go back and
look in it and see if we can find that
which is much easier than designing in
your experiment you know new
neuroscience experiments take a long
time years so although some people are
doing that now too so but between all of
these things I think it's reasonable
it's actually a very very good approach
we we are blessed with the fact that we
can test our theories out the ying-yang
here because there's so much on a
similar data and we can also falsify our
theories very easily which we do often
it's kind of reminiscent to whenever
whenever that was with Copernicus you
know when you figure out that the sun's
at the center of the the solar system as
opposed to earth the pieces just fall
into place yeah I think that's the
general nature of aha moments is in
history Copernicus it could be you could
say the same thing about Darwin you
could say same thing about you know
about the double helix that that people
have been working on a problem for so
long and I have all this data and they
can't make sense of it they can't make
sense of it but when the answer comes to
you and everything falls into place it's
like oh my gosh that's it
that's got to be right I asked both Jim
Watson and Francis Crick about this I
asked him you know when you were working
on trying to discover the structure of
the double helix and when you came up
with the the sort of the structure that
ended up being correct but it was sort
of a guess you know I wasn't really
verified yeah I said did you know that
it was right and they both said
absolutely so we absolutely knew it was
right and it doesn't matter if other
people didn't believe it or not we knew
it was right they get around the thing
agree with it eventually anyway
and that's the kind of thing you hear a
lot with scientists who who really are
studying a difficult problem and I feel
that way too about our work if you talk
to Kirk or Watson about the the problem
you're trying to solve the of finding
the DNA of the brain yeah in fact
Francis Crick was very interested in
this in the latter part of his
and in fact I got interested in brains
by reading an essay he wrote in 1979
called thinking about the brain and that
is when I decided I'm gonna leave my
profession of computers and engineering
and become a neuroscientist just reading
that one essay from Francis Crick I got
to meet him later in life
I got I spoke at the Salk Institute and
he was in the audience and then I had a
tea with him afterwards you know he was
interested in a different problem and he
was he was focused on consciousness yeah
and the easy problem right well I I
think it's the red herring and and so we
weren't really overlapping a lot there
Jim Watson who's still alive is is also
interested in this problem and he was
when he was director of the coast of
Harbor laboratories he was really sort
of behind moving in the direction of
neuroscience there and so he had a
personal interest in this field and I
have met with him numerous times and in
fact the last time was a little bit over
a year ago I gave a talk close to me
Harbor labs about the progress we were
making in in our work and it was a lot
of fun because he said well you you
wouldn't be coming here unless you had
something important to say so I'm gonna
go change our talk so he sat in the very
front row next to most next to him was
the director of the lab was Stillman so
these guys are in the front row of this
auditorium right so nobody else in the
auditorium wants to sit in the front row
because Jim Watson is detective and and
I gave a talk and I had dinner with Jim
afterwards but it's I there's a great
picture of my colleague sue Battaglia
mahad took where I'm up there sort of
like screaming the basics of this new
framework we have and Jim Watson is on
the edge of his chair he's literally on
the edge of his chair like intently
staring up at the screen and when he
discovered the structure of DNA the
first public talk he gave was that Cold
Spring Harbor labs so and there's a
picture those famous picture Jim Watson
standing at the whiteboard was where the
overrated thing pointing at something
was holding a double helix at this point
it actually looks a lot like the picture
of me so there was funny I got talking
about the brain and there's Jim Watson
staring intently I didn't course there
was you know whatever sixty years
earlier he was standing you know
pointing at the double helix and it's
one of the great discoveries and and all
of you know whatever by all the science
all science yeah yeah hey so this is the
funny that there's echoes of that in
your presentation do you think in terms
of evolutionary timeline in history the
development of the neocortex was a big
leap or is it just a small step so like
if we ran the whole thing over again
from the from the birth of life on Earth
how likely develop the mechanism and you
okay well those are two separate
questions one it was it a big leap and
one was how like it is okay they're not
necessarily related maybe correlated we
don't really have enough data to make a
judgment about that I would say
definitely was a big league and leap and
I can tell you why I think I don't think
it was just another incremental step at
that moment I don't really have any idea
how likely it is if we look at evolution
we have one data point which is earth
right life formed on earth billions of
years ago whether it was introduced here
or it created it here or someone
introduced it we don't really know but
it was here early it took a long long
time to get to multicellular life and
then from multi to other started life it
took a long long time to get his
neocortex and we've only had the New
York Texas for a few hundred thousand
years so that's like nothing okay so is
it likely well certainly isn't something
that happened right away on earth and
there were multiple steps to get there
so I would say it's probably not get
something what happened instantaneous on
other planets that might have life it
might take several billion years on
average um is it likely I don't know but
you'd have to survive for several
billion years to find out probably is it
a big leap yeah I think it's it is a
qualitative difference than all other
evolutionary steps I can try to describe
that if you'd like sure you know which
way uh yeah I can tell you how pretty
much I'll start a little press
many of the things that humans are able
to do do not have obvious survival
advantages precedent yeah you know we
create music is that is there a really
survival advantage to that maybe maybe
not
what about mathematics is there a real
survival advantage to mathematics it's
stretchy you can try to figure these
things out right but up but mostly
evolutionary history everything had
immediate survival advantages too right
so I'll tell you a story which I like me
may not be true but the story goes as
follows organisms have been evolving
first since the beginning of life here
on earth anything this sort of
complexity on to that just sort of
complexity and the brain itself is
evolved this way in fact there's an old
parts and older parts and older older
parts of the brain that kind of just
keeps calling on new things and we keep
adding capabilities and we got for the
neocortex initially it had a very clear
survival advantage and that it produced
better vision and better hearing and
better thoughts and maybe a new place so
on but what what I think happens is that
evolution just kept it took it took a
mechanism and this is in our recent
theories but it took a mechanism evolved
a long time ago for navigating in the
world for knowing who you are these are
the so called grid cells and place cells
of an old part of the brain and it took
that mechanism for building maps of the
world and knowing we are in those maps
and how to navigate those maps and turns
it into a sort of a slimmed-down
idealized version of it mm-hmm and that
ideally this version could now apply to
building maps of other things maps of
coffee cups and maps the phone's maps of
these concepts yes and not just almost
exactly and and so you and it just
started replicating this stuff right you
just think more and more more bits so we
went from being sort of dedicated
purpose neural hardware to solve certain
problems that are important to survival
to a general purpose neural hardware
that could be applied to all problems
and now it's just it's
the orbit of survival it's we are now
able to apply it to things which we find
enjoyment you know but aren't really
clearly survival characteristics and
that it seems to only have happened in
humans to the large extent and so that's
what's going on where we sort of have
we've sort of escape the gravity of
evolutionary pressure in some sense in
the neocortex and it now does things
which but not that are really
interesting discovery models of the
universe which may not really help us
doesn't matter how is it help of
surviving knowing that there might be
multiple no there might be you know the
age of the universe or what how do you
know various stellar things occur it
doesn't really help us survive at all
but we enjoy it and that's what happened
or at least not in the obvious way
perhaps it is required if you look at
the entire universe in an evolutionary
way it's required for us to do
interplanetary travel and therefore
survive past our own Sun but you know
let's not get too but you know evolution
works at one time frame it's it's
survival if you think of a survival of
the phenotype survival of the individual
it is that what you're talking about
there is spans well beyond that so
there's no genetic I'm not transferring
any genetic traits to my children that
are gonna help them survive better on
Mars right it's totally different
mechanism let's yeah so let's get into
the the new as you've mentioned the idea
that I don't know if you have a nice
name thousand you call it a thousand
brain theory often told I like it so can
you talk about the this idea of spatial
view of concepts and so on yeah so can I
just describe sort of the there's an
underlying core discovery which then
everything comes from that that's a very
simple this is really what happened we
were deep into problems about
understanding how we build models of
stuff in the world and how we make
predictions about things and I was
holding a coffee cup just like this in
my hand and I had my finger was touching
the side my index finger and I moved it
to the top and I was going to feel the
the rim at the top of the cover and I
asked myself a very simple question I
said well first of all I have to say I
know that my brain predicts what its
gonna feel before it touches it you can
just think about it and imagine it and
so we know that the brain is making
predictions all the time so the question
is what does it take to predict that
right and there's a very interesting
answer that first of all it says the
brain has to know it's touching a coffee
cup and I said a model or a coffee cup
and needs to know where the finger
currently is on the cup relative to the
cup because when I make a movement and
used to know where it's going to be on
the cup after the movement is completed
relative to the cup and then it can make
a prediction about what's going to sense
so this told me that Dean your cortex
which is making this prediction needs to
know that it's sensing it's touching a
cup and it needs to know the location of
my finger relative to that cup in a
reference frame of the cup it doesn't
matter where the cup is relative my body
it doesn't matter its orientation none
of that matters it's where my finger is
relative to the cup which tells me then
that the neocortex is has a reference
frame that's anchored to the cup because
otherwise I wouldn't be able to say the
location and I wouldn't be able to
predict my new location and then we
quickly vary installation instantly you
can say well every part of my skin could
touch this cup and therefore every part
of my skin is making predictions and
every part my skin must have a reference
frame that it's using to make
predictions so the the big idea is that
throughout the neocortex
there are everything as being is being
stored and referenced in reference
frames you can think of them like XYZ
reference things but they're not like
that we know a lot about the neural
mechanisms for this but the brain thinks
in reference frames and it's an engineer
if you're an engineer this is not
surprising you'd say if I wanted to
build a a CAD model of the coffee cup
well I would bring it up in some CAD
software and I would assign some
reference frame and say this features at
this locations and so on but the fact
that this the idea that this is
occurring through out in your cortex
everywhere it was a novel idea and and
then zillion things fell into place
after that it's doing so now we think
about the neocortex as processing
information quite differently than we
used to do it we used to think about the
neural cortex is processing sensory data
and extracting features from that
sensory data and then extracting
features from the features very much
like a deep Learning Network does today
but that's not how the brain works at
all the brain works by assigning
everything every input everything to
reference frames and there are thousands
hundreds and thousands of them active at
once in your neocortex it's a surprising
thing the thing about but once you sort
of internalize this you understand that
it explains almost every all the almost
all the mysteries we've had about this
it's about this structure so one of the
consequences of that is that every small
part of the neocortex so you have a
millimeter square and there's a hundred
and fifty thousand of those so it's
about 150,000 square millimeters if you
take every little square millimeter of
the cortex it's got some input coming
into it and it's going to have reference
frames which assign that input to and
each square millimeter can learn
complete models of objects so what do I
mean by that if I'm touching the coffee
cup well if I just touch it in one place
I can't learn what this coffee cup is
because I'm just feeling one part but if
I move it around the cup it touched you
to different areas I can build up a
complete model the cup because I'm now
filling in that three dimensional map
which is the coffee cup I can say oh
what am I feeling in all these different
locations that's the basic idea it's
more complicated than that but so
through time and we talked about time
earlier through time even a single
column which is only looking at or a
single part of the cortex it's only
looking at a small part of the world can
build up a complete model of an object
and so if you think about the part of
the brain which is getting input from
all my fingers
so there's they're spread across the top
and here this is the somatosensory
cortex there's columns associated all
these from areas of my skin
and what we believe is happening is that
all of them are building models of this
cup every one of them or things not do
not all building all not every column
every part of the cortex builds models
of everything but they're all building
models of something and and so you have
it so when I when I touch this cup with
my hand there are multiple models of the
cup being invoked if I look at it with
my eyes there again many models of the
cup being invoked because each part of
the visual system and the brain doesn't
process an image that's mr. that's a
misleading idea it's just like your
fingers touching
so different parts of my Radnor of
looking at different parts of the cup
and thousands and thousands of models of
the cup are being invoked at once and
they're all voting with each other
trying to figure out what's going on so
that's why we call it the thousand
brains theory of intelligence because
there isn't one model of a cop there are
thousands of models to this Cup there
are thousands of models for your cell
phone and about cameras and microphones
and so on it's a distributed modeling
system which is very different than what
people have thought about it so this is
a really compelling and interesting idea
of f2 first questions - one on the
ensemble part of everything coming
together you have these thousand brains
how do you know which one has done the
best job of forming the great question
let me try Spain there there's a problem
that's known in neuroscience called the
sensor fusion problem yes and so is the
idea of something like oh the image
comes from the eye there's a picture on
the retina and it gets projected to than
your cortex no by now it's all spread
out all over the place and it's kind of
squirrely and distorted and pieces are
all over this you know it doesn't look
like a picture anymore
when does it all come back together
again right or you might say well yes
but I also I also have sounds or touches
associated with a couple so I'm seeing
the cup and touching the cup how do they
get combined together again so this it's
called the sensor fusion problem is if
all these disparate parts have to be
brought together into one model
someplace that's the wrong idea the
right idea is that you get all these
guys voting there's auditory models of
the cup there's visual models the cup
those tactile models of the cup there's
one the individual system there might be
ones that are more focused on black and
white ones fortunate on color it doesn't
really matter there's just thousands and
thousands of models of this Cup and they
vote they don't actually come together
in one spot it just literally think of
it this way I imagine you have these
columns or like about the size of a
little piece of spaghetti okay like a
two and a half millimeters tall and
about a millimeter in mind they're not
physical like but you could think of
them that way and each one's trying to
guess what this thing is they're
touching now they can they can do a
pretty good job if they're allowed to
move over to us so I could reach my hand
into a black box and move my finger
around an object and if I touch enough
spaces like oh okay I don't know what it
is but often we don't do that often I
can just reach and grab something with
my hand all the once and I get it or if
I had to look through the world through
a straw so long
invoking one little column I can only
see part of some things I have to move
the straw around but if I open my eyes
to see the whole thing at once so what
we think is going on it's all these
little pieces of spaghetti if you know
all these little columns in the cortex
or all trying to guess what it is that
they're sensing they'll do a better
guess if they have time and can move
over time so if I move my eyes and with
my fingers but if they don't they have a
they have a poor guest it's a it's a
probabilistic s of what they might be
touching now imagine they can post their
probability at the top of a little piece
of spaghetti each one of them says I
think and it's not really a probability
decision it's more like a set of
possibilities in the brain it doesn't
work as a probability distribution it
works is more like what we call the
Union so you could say and one column
says I think it could be a coffee cup
sort of can or a water bottle and the
other column says I think it could be a
coffee cup or you know telephone or
camera whatever right and and all these
guys are saying what they think might be
and there's these long range connections
in certain layers in the cortex so
there's been some layers in some cell
types in each column send their
projections across the brain and that's
the voting occurs and so there's a
simple associative memory mechanism
we've described this in a recent paper
and we've modeled this that says they
can all quickly settle on the only or
the one best answer for all of them if
there is a single best answer they all
vote and say yeah it's got to be the
coffee cup and at that point they all
know it's a coffee go and at that point
everyone acts as if it's the coffee cup
they yeah we know it's a coffee even
though I've only seen one little piece
of this world
I know it's coffee cup I'm touching or
I'm seeing or whatever and so you can
think of all these columns are looking
at different parts in different places
different sensory and put different
locations they're all different but this
layer that's doing the voting that's
it's solidifies it's just like it
crystallizes and says oh we all know
what we're doing and so you don't bring
these models together in one model you
just vote and there's a crystallization
of the vote great that's a at least a
compelling way to think about about the
way you form a model of the world now
you talk about a coffee cup do you see
this as far as I understand you're
proposing this as well that this extends
to much more than coffee cups
it does or at least the physical world
it expands to the world of concepts yeah
it does and well first the primary face
every evidence for that is that the
regions of the neocortex that are
associated with language or high-level
thought or mathematics or things like
that they look like the regions of the
new your cortex that process vision
hearing and touch there they don't look
any different or they look only
marginally different and so one would
say well if Vernon now Castle who
proposed it all that come all the parts
of New York or trees doing the same
thing if he's right then the parts that
during language or mathematics or
physics are working on the same
principle they must be working on the
principle of reference frames so that's
a little odd flawed hmm but of course we
had no eye
we had no prior idea how these things
happen so that's let's go with that
and we in our recent paper we talked a
little bit about that I've been working
on it more since I have better ideas
about it now I'm sitting here very
confident that that's what's happening
and I can give you some examples to help
you think about that
it's not we understand it completely but
I understand it better than I've
described it in any paper so far so but
we did put that idea out there says okay
this is it's it's it's it's a good place
to start you know and the evidence would
suggest this how it's happening and then
we can start tackling that problem one
piece at a time like what does it mean
to do high-level thought what it means a
new language how would that fit into a
reference frame framework yes so there's
a if you could tell me if there's a
connection but there's an app called
Anki that helps you remember different
concepts and they they talk about like a
memory palace that helps you remember a
completely random concepts by so trying
to put them in a physical space in your
mind yeah and putting them next to each
other the method of loci okay yeah for
some reason that seems to work really
well yeah no that's a very narrow kind
of application of just remembering some
facts but that's a very very telling one
yes exactly so it seems like you're
describing a mechanism why this seems
yeah so so basically the way what we
think is going on is all things you know
all concepts all ideas words everything
you know are stored in reference frames
and so if you want to remember something
you have to basically navigate through a
reference frame the same way a rat
navigates to a Maeve in the same way my
finger rat navigates to this coffee cup
you are moving through some space and so
what you if you have a random list of
things you were asked to remember by
assigning him to a reference frame
you've already know very well to see
your house right an idea the method of
loci is you can say okay in my lobby I'm
going to put this thing and then and
then the bedroom I put this one I go
down the hall I put this thing and then
you want to recall those facts so we
call this things you just walk mentally
you walk through your house you're
mentally moving through a reference
frame that you already had and that
tells you there's two things are really
important about it tells us the brain
prefers to store things in reference
frames and that the method of recalling
things or thinking if you will is to
move mentally through those reference
frames you could move physically through
some reference frames like I could
physically move through the reference
name of this coffee cup I can also
mentally move to the reference time the
coffee cup imagining me touching it but
I can also mentally move my house and
and so now we can ask yourself or are
all concepts toward this way there's
some recent research using human
subjects in fMRI and I'm gonna apologize
for not knowing the name of the
scientist that did this but what they
did is they they put humans in this fMRI
machine which was one of these imaging
machines and they they gave the humans
tasks to think about Birds so they had
different types of birds and beverage it
looked big and small and long necks and
long legs things like that and what they
could tell from the fMRI it was a very
clever experiment get to tell when
humans were thinking about the birds
that the birds that the knowledge of
birds was arranged in a reference frame
similar to the ones that are used when
you navigate in a room that these are
called grid cells and there are grid
cell like patterns of activity in the
new your cortex when they do this so
that it's a very clever experiment you
know and what it basically says that
even when you're thinking about
something abstract and you're not really
thinking about it as a reference frame
it tells us the brain is actually using
a reference frame and it's using the
same neural mechanisms these grid cells
are the basic same neural mechanism that
we we propose that grid cells