Kind: captions
Language: en
the following is a conversation with
David silver who leads the reinforcement
learning research group a deep mind and
was the lead researcher on alphago alpha
0 and co led the Alpha star and Museum
efforts and a lot of important work in
reinforcement learning in general I
believe alpha zero is one of the most
important accomplishments in the history
of artificial intelligence and David is
one of the key humans who brought alpha
zero to life together with a lot of
other great researchers at deep mind
he's humble kind and brilliant we were
both jet lagged but didn't care and made
it happen
it was a pleasure and truly an honor to
talk with David this conversation was
recorded before the outbreak of the
pandemic for everyone feeling the
medical psychological and financial
burden of this crisis I'm sending love
your way stay strong or in this together
we'll beat this thing this is the
artificial intelligence podcast if you
enjoy it subscribe on youtube review it
with five stars an apple podcast support
on patreon or simply connect with me on
Twitter Alex Friedman spelled Fri DM aen
as usual I'll do a few minutes of as now
and never any ads in the middle they can
break the flow of the conversation I
hope that works for you and doesn't hurt
the listening experience quick summary
of the ads to sponsors masterclass and
cash app please consider supporting the
podcast by signing up to master class
and master class comm slash flex and
downloading cash app and using code and
Lex podcast this show is presented by
cash app the number one finance app in
the App Store when you get it use code
Lex podcast cash app lets you send money
to friends buy Bitcoin and invest in the
stock market with as little as one
dollar since cash app allows you to buy
Bitcoin let me mention that
cryptocurrency in the context of the
history of money it's fascinating
I recommend a cent of money as a great
book on this history debits and credits
and Ledger's started around 30,000 years
ago the US dollar created over two
hundred years ago and Bitcoin the first
decentralized cryptocurrency at least
just over ten years ago so given that
history cryptocurrency is still very
much in its early days of development
but it's still aiming to and just might
redefine the nature of money so again if
you get cash out from the App Store or
Google Play
and use the code let's podcast you get
ten dollars and cash wrap will also
donate ten dollars the first an
organization that is helping to advance
robotics and stem education for young
people around the world
this show is sponsored by masterclass
set up a masterclass complex to get a
discount and to support this podcast in
fact for a limited time now if you sign
up for an all-access pass for a year you
get to get another all-access pass to
share with a friend buy one get one free
when I first heard about masterclass I
thought it was too good to be true
for one hundred eighty dollars a year
you get an all-access pass to watch
courses from to list some of my
favorites Chris Hadfield on space
exploration Neil deGrasse Tyson on
scientific thinking communication will
write the creator of SimCity and Sims on
game design jane goodall on conservation
Carlos Santana on guitar his song Europa
could be the most beautiful guitar song
ever written
garry kasparov on chess daniel negreanu
on poker and many many more Chris
Hadfield explaining how Rockets work and
the experience of being launched into
space alone is worth the money for me
the keys to not be overwhelmed by the
abundance of choice pick three courses
you want to complete watch each of them
all the way through it's not that long
but it's an experience that will stick
with you for a long time I promise it's
easily worth the money you can watch it
on basically any device once again sign
up a master class complex to get a
discount and to support this podcast and
now here's my conversation with David
silver
what was the first program you've ever
written and what programming language
do you remember I remember very clearly
he have my my parents brought home this
BBC modeled B microcomputer it was just
this fascinating thing to me I was about
seven years old and couldn't resist just
playing around with it so I think first
program ever was writing my name out in
different colors and getting it to loop
and repeat that and there was something
magical about that which just led to
more and more how did you think about
computers back then like the magical
aspect of it that you can write a
program and there's this thing that you
just gave birth to it's able to creative
visual elements and live in its own or
did you not think of it in those
romantic notions was it more like oh
that's cool
I can I can solve some puzzles it was
always more than solving puzzles it was
something where you know there was this
limitless possibilities once you have a
computer in front of you you can do
anything with it that's um I used to
play with Lego with the same feeling you
can make anything you want out of Lego
but even more so with a computer you
know you don't you're not constrained by
the amount of kit you've got and so I
was fascinated by it and started pulling
out there you know the user guide and
the advanced user guide and then
learning so I started in basic and then
you know later 6502 my father was also
became interested in there in this
machine and gave up his career to go
back to school and study for an a
master's degree in in artificial
intelligence funnily enough Essex
University when I was when I was seven
so I was exposed to those things at an
early age he showed me how to program in
Prolog and do things like querying your
family tree and those are some of my
earlier earliest memories of trying to
trying to figure things out on a
computer those are the early steps in
computer science programming but when
did you first fall in love with
artificial intelligence or were the
ideas the dreams of AI I think it was
really when I when I went to study at
university so I was an undergrad at
Cambridge and studying computer science
and and I really started to question you
know what what really are the goals what
what's the goal where do we want to go
with with computer science and it seemed
to me that the the only step of major
significance to take was to try and
recreate something akin to human
intelligence if we could do that that
would be a major leap forward and that
idea certainly wasn't the first to have
it but it you know nestled within me
somewhere and and became like a bug you
know I really wanted to to crack that
problem so you thought it was like you
had a notion that this is something that
human beings can do it is possible to
create an intelligent machine well I
mean unless you believe in something
metaphysical then what are our brains
doing well at some level their
information processing systems which are
able to take whatever information is in
there transform it through some form of
program
and produce some kind of output which
enables that that human being to do all
the amazing things that they can do in
this incredible world so so then do you
remember the first time you've written a
program that because you also had an
interesting games do you remember the
first time you were in the program that
beat you in a game said I won't beat you
at anything sort of achieved Super David
silver level performance so I used to
work in the games industry so for five
years I programmed games for my first
job so it was a amazing opportunity to
get involved in a startup company and so
I I was involved in in building AI at
that time and so for sure there was a
sense of building handcrafted what
people used to call AI in the games
industry which i think is not really
what we might think of as AI and its
fullest sense but something which is
able to to take actions and in a way
which which makes things interesting and
challenging for their for the for the
human player and at that time I was able
to build you know these handcrafted
agents which in certain limited cases
could do things which which were able to
do better than me but mostly in these
kind of twitch like scenarios where
where they were able to do things faster
or because they had some pattern which
was able to exploit repeatedly I think
if we're talking about real AI the first
experience for me came after that when I
I realized that this path I was on
wasn't taking me towards it wasn't it
wasn't dealing with that bug which I
still had inside me to really understand
intelligence and try and and try and
solve it everything people were doing in
games was you know short-term fixes
rather than long-term vision and so I
went back to study for my PhD which was
fairly enough trying to apply
reinforcement learning to the game of go
and I built my first go program using
reinforcement learning a system which
would by trial and error play against
itself and was able to learn
which patterns were actually helpful to
predict whether it's going to win or
lose the game and then choose the moves
that led to the combination of patterns
that would mean that you're more likely
to win in that system that system beat
me how did that make you feel make me
feel good I was there as sort of the
yeah then is the it's a mix of a sort of
excitement and was there a tinge of sort
of like almost like a fearful aw you
know it's like in space 2001 Space
Odyssey kind of realizing that you've
created something that there's you know
that is that's achieved human level
intelligence in this one particular
little task and in that case I suppose a
neural networks weren't involved there
were no neural networks in those days
this was pre deep learning revolution
but it was a principled self learning
system based on a lot of the principles
which which people are still using in
deep reinforcement learning how did I
feel
I I think I found it immensely
satisfying that a system which was able
to learn from first principles for
itself was able to reach the point that
it was understanding this domain better
than better than I could and able to
outwit me I don't think it was a sense
of or it was a sense that satisfaction
that this that's something I felt should
work had worked so to me alphago and I
don't know how else to put it but to me
alphago and alpha a girl zero mastery in
the game of girl is again to me the most
profound and inspiring moment in the
history of artificial intelligence so
you're one of the key people behind this
achievement and I'm Russian so I really
felt the first sort of seminal
achievement one deep blue beat garry
kasparov in 1987 so as far as I know the
AI community at that point largely saw
the game of Go was unbeatable in AI
using the the sort of the state of the
art to brute force methods search
methods even if you consider at least
the way I saw it
even if you consider arbitrary
exponential ski scaling of compute go
would still not be solvable hence why it
was thought to be impossible so given
that the game of go was impossible to to
master one was the dream for you you
just mentioned your PG thesis of
building the system that plays go what
was the dream for you that you could
actually build a computer program that
achieves world-class not necessarily
beat the world champion but I cheesed
that kind of level of playing go first
of all thank you that's very kind West
and funnily enough I just came from a
panel where I was actually in a
conversation with Garry Kasparov and
Marie Campbell who was the author of
deep blue and it was their first meeting
together since the since the match
yesterday so I'm literally fresh from
that experience so these are amazing
moments when they happen but where did
it all start well for me it started when
I became fascinated in the game of go so
go for me I've grown up playing games
I've always had a fascination in in in
board games I played chess as a kid I
played Scrabble as a kid when I was at
university I discovered the game of go
and and to me it just blew all of those
other games out of the water it was just
so deep and profound in its in its
complexity with endless levels to it
what I discovered was that I could
devote endless hours to this game and I
knew in my heart of hearts that no
matter how many hours I would devote to
it I would never become a you know a
grandmaster or there was another path
and the other path was to try and
understand how you could get some other
intelligence to play this this game
better than I would be able to and so
even in those days I had this idea that
you know what if what if it was possible
to build a program that could crack this
and as I started to explore the domain I
discovered that you know this was really
the domain where people felt deeply that
if progress could be made and go it
really mean a giant leap forward for a
I it was the the challenge where all
other approaches had failed you know
this is coming out of the area you
mentioned which was in some sense their
the golden era for further classical
methods of a I like heuristic search in
the 90s you know they all they all fell
one after another not just chess with
deep blue but checkers backgammon
Othello
there were numerous cases where where
systems built on top of heuristic search
methods with you know his
high-performance systems have been able
to defeat the human world champion in
each of those domains and yet in that
same time period there was a million
dollar prize available for the game of
go for the first system to be a human
professional player and at the end of
that time period in year 2000 when the
prize expired the strongest go program
in the world was defeated by a
nine-year-old child when that nine year
old child was giving 9 free moves to the
computer at the start of the game and to
try and even things up yeah and computer
go X but beat that strongest same
strongest program with 29 handicaps
tones 29 free moves so that's what the
state of affairs was when I became
interested in this problem in around
2000 and 2003 when I I start started
working computer go there was nothing
they were there was just there was very
very little in the way of progress
towards meaningful performance again
anything approaching human level and so
people they it wasn't through lack of
effort people have tried many many
things and so there was a strong sense
that that something different would be
required for go than then had been
needed for all of these other domains
where I had a I had been successful and
maybe the single clearest example is
that that go unlike those other domains
had this kind of intuitive property that
a go player would look at a position and
say hey you know here's this mess of
black and white stones but from this
mess oh I can I can predict that that's
this part of the board has become my
territory this part of the boards become
your territory and I've got this overall
sense
I'm going to win and this is about the
right move to play and that intuitive
sense of judgment of being able to
evaluate what's going on in a position
it was pivotal to humans being able to
play this game and something that people
had no idea how to put into computers so
this question of how to evaluate in a
position how to come up with these
intuitive judgments was the key reason
why go was so hard in addition to its
enormous search space and the reason why
methods which had succeeded so well
elsewhere
failed and go and so people really felt
deep down that that you know in order to
crack go we would need to get something
akin to human intuition and if we got
something akin to human intuition we'd
be able to self you know much many many
more problems in AI so to me that was
the moment where it's like okay this is
not just about playing the game of Go
this is about something profound and it
was back to that bug which had been
itching me all those years now this is
the opportunity to do something
meaningful and and transformative and
and I guess a dream was born that's a
really interesting way to put it almost
this realization that you need to find
formulate girls are kind of a prediction
problem versus a search problem was the
intuition I mean I maybe that's the
wrong crude term but the to give it us
the ability to kind of Intuit things
about positional structure of the board
well okay but what about the learning
part of it did you have a sense that you
have to that learning has to be part of
the system again something that hasn't
really as as far as I think except with
TD Guerin and in the 90s was RL a little
bit hasn't been part of those
state-of-the-art game playing systems so
I strongly felt that learning would be
necessary and that's why my my PhD topic
back then was trying to apply
reinforcement learning to the game of CO
and not just learning of any type but I
felt that the only way to really have a
system to progress beyond human levels
of performance wouldn't just be to mimic
how humans do it but to understand for
themselves
and how else can a machine hope to
understand what's going on except
through learning if you're not learning
what else are you doing while you're
putting all the knowledge into the
system and that just feels like a
something which decades of AI have told
us is is maybe not a dead end but
certainly has a ceiling to the
capabilities it's known as the you know
knowledge acquisition bottleneck that
there the more you try to put into
something the more brittle the system
becomes and and so you just have to have
learning you have to have learning
that's the only way you're going to be
able to get a system which has
sufficient knowledge in it you know
millions and millions of pieces of
knowledge billions trillions of a form
that it can actually apply for itself
and understand how those billions and
trillions of pieces of knowledge can be
leveraged in a way which will actually
lead it towards its goal without
conflict or or other issues yeah I mean
if I put myself back in there in that
time I just wouldn't think like that
without a good demonstration of RL I
would I would think more in the symbolic
AI like that though it would not
learning but sort of a simulation of
knowledge base like a growing knowledge
base but it would still be sort of
pattern based lot like basically have
little rules that you kind of assemble
together into a large knowledge base
well in a sense that was the state of
the art back then so if you look at the
go programs which had been competing for
this prize I mentioned they were an
assembly of different specialized
systems some of which used huge amounts
of human knowledge to describe how you
should play the opening how you should
all the different patterns that were
required to to play well in the game of
Go endgame Theory combinatorial game
theory and combined with more principled
search based methods which we're trying
to solve for particular sub parts of the
game like life and death
connecting groups together all these
amazing subproblems that just emerged in
the game of Go there were there were
different pieces all put together into
this like collage which together would
try and
play against a human and although not
all of the pieces were handcrafted the
overall effect was nevertheless still
brittle and it was hard to make all
these pieces work well together and so
really what I was pressing for and the
main innovation of the approach they
took was to go back to first principles
and say well let's let's back off that
and try and find a principled approach
where the system can learn for itself it
just from the outcome like you know
learn for itself if you try something
did that did that help or did it not
help and only through that procedure can
you arrive at knowledge which is which
is verified the system has to verify it
for itself not relying on any other
third party to say this is right or this
is wrong so that principle was already
you know very important in those days
but unfortunately we were missing some
important pieces back then so before we
dive into may be discussing the beauty
of reinforcement learning let's think
it's the back who kind of skipped
skipped it a bit but the rules of the
game of go what's the the elements of it
perhaps contrasting to chess that sort
of you really enjoyed as a human being
and also that make it really difficult
as a a I machine learning problem so the
game of CO was has remarkably simple
rules if that's so simple that people
have speculated that if we were to meet
alien life at some point that we
wouldn't be able to communicate with
them but we would be able to play hello
go with that probably have discovered
the same rule set yeah so the game is
played on a on a 19 by 19 grid and you
play on the intersections of the grid
and the players take turns and the aim
of the game is very simple it's to
surround as much territory as you can as
many of these intersections with your
stones and just around more than your
opponent does and the only nuance to the
game is that if you fully surround your
opponent's piece then you get to capture
it and remove it from the board and it
counts as your own territory now from
those very simple rules immense
complexity arises it's kind of profound
strategies in
how to surround territory how to kind of
trade-off between making solid territory
yourself now compared to building up
influence that will help you acquire
territory later in the game how to
connect groups together how to keep your
own groups alive which which patterns of
stones are most useful compared to
others
there's just immense knowledge and human
go players have played this game for it
was discovered thousands of years ago
and human go players have built up its
immense knowledge base over over the
years it's studied very deeply and
played by something like 50 million
players across the world mostly in China
Japan and Korea where it's a important
part of a culture so much so that it's
considered one of the four ancient arts
that was required by Chinese scholars so
there's a deep history there but there's
interesting quality so if I is it a
comparative chess chess is in the same
way as it is in Chinese culture of a
goal in chess in Russia is also
considered one of the secret arts so if
we contrast sort of go with chess as
interesting qualities about go maybe you
can correct me if I'm wrong but the
evaluation of a particular static board
is not as reliable like you can't in
chess you can kind of assign points to
the different units and it's kind of a
pretty good measure of who's one who's
losing it's not so clear yeah so this
game of the HOH you know you find
yourself in a situation where both
players have played the same number of
stones actually captures a strong level
of play happen very rarely which means
that any moment in the game you've got
the same number of white stones and
black stones and the only thing which
differentiates how well you're doing is
this intuitive sense of you know where
are the territories ultimately going to
form on this board and when you if you
look at the complexity of a real go
position you know it's it's mind
boggling that kind of question of what
will happen in in 300 moves from now
when you when you see just a scattering
of twenty white and black stones
intermingled and and so that that
challenge is the reason why position of
value
is so hard in go compared to two other
games in addition to that has an
enormous search space so there's around
ten to one hundred and seventy positions
in the game of go that's an astronomical
number and that search spaces is so
great that traditional heuristic search
methods that were so successful and
things like deep blue and and chess
programs just kind of fall over and go
so a which pointed reinforcement
learning enter your life your research
life your way of thinking we just talked
about learning but reinforcement
learning is very particular kind of
learning one that's both philosophically
sort of profound yeah but also one
that's pretty difficult to get to work
as if we look back in the earth at least
the early days so when did that enter
your life and how did that work progress
so I had just finished working in the
games industry this startup company and
I took I took a year out to discover for
myself exactly which path I wanted to
take I knew I wanted to study
intelligence but I wasn't sure what that
meant at that stage I really didn't feel
had the tools to decide on exactly which
path I wants to follow so during that
year I I read a lot and one of the
things I read was Saturn Umberto the
sort of seminal tech spec are an
introduction to reinforcement learning
and when I read that textbook I I just
had this resonating feeling that this is
what I understood intelligence to be and
this was the path that I felt would be
necessary to go down to make progress in
in AI so I got in touch with rich Saturn
and asked him if he would be interested
in supervising me on a PhD thesis in in
computer go and he he basically said
that if he's still alive he'd be happy
to but unfortunately he'd been you know
struggling with very serious cancer for
some years and he really wasn't
confident at that stage that he'd even
be around to see the end event but
fortunately
that part of the story worked out very
happily and I found myself out there in
Alberta they've got a great games group
out there with a history of fantastic
working in board games as well as rich
that in the father of RL so it was the
the natural place for me to go in some
sense to study this question and the
more I looked into it the more the more
strongly ie
I felt that this wasn't just the path to
progress in computer go but really you
know this this was the thing I'd been
looking for this was really an
opportunity to to frame what
intelligence means like what does what
are the goals of AI in a clear single
clear problem definition such that if
we're able to solve that play a single
problem definition in some sense we've
cracked the problem of AI so to you
reinforcement learning ideas at least
sort of echoes of it would be at the
core of intelligence it is as a core of
intelligence and if we ever create in a
human level intelligence system it would
be at the core of that kind of system
let me say it this way that I think I
think it's helpful to separate out the
problem from the solution so I see the
problem of intelligence I would say it
can be formalized as the reinforcement
learning problem and that that
formalization is enough to capture most
if not all of the things that we mean by
intelligence that that they can all be
brought within this this this framework
and gives us a way to access them in a
meaningful way that allows us as as
scientists to understand intelligence
and us as computer scientists to to
build them and so in that sense I feel
that it gives us a path maybe not the
only path but a path towards AI and so
do I think that any system in the future
that that's you know sold AI would would
have to have RL within it well I think
if you ask that you're asking about the
solution methods I would say that if we
have such a thing it would be a solution
to the RL problem now what particular
methods have been used to get there
well we should keep an open mind about
the best approaches to actually solve
any problem and you know the things we
have right now for reinforcement
learning maybe maybe then maybe I
believe they've got a lot of legs but
maybe we're missing some things maybe
there's gonna be better ideas I think we
should keep her you know let's remain
modest and we're at the early days of
this field and and there are many
amazing discoveries ahead of us for sure
the specifics especially of the
different kinds of our ell approaches
currently there could be other things
there followed is a very large umbrella
of our ell but if it's if it's okay can
we take a step back and kind of ask the
basic question of what is to you
reinforcement learning so reinforcement
learning is the study and the science
and the problem of intelligence in the
form of an agent that interacts with an
environment so the problem is trying to
self is represented by some environment
like the world in which that agent is
situated and the goal of RL is clear
that the agent gets to take actions
those actions have some effects on the
environment and the environment gives
back an observation to the agent saying
you know this is what you see your sense
and one special thing which it gives
back is it's called the raw signal how
well it's doing in the environment and
the reinforcement learning problem is to
simply take actions over time so as to
maximize that reward signal so a couple
of basic questions what types of RL
approaches are there so I don't know if
there's a nice brief in words way to
paint the picture of sort of value based
model based policy based reinforcement
learning yeah so now if we think about
okay so there's this ambitious problem
definition of RL it's really you know
it's truly ambitious it's trying to
capture and encircle all of the things
in which an agent interacts with an
environment and say well how can we
formalize and understand what it means
to to crack that now let's think about
the solution method well how do you
solve a really hard problem like that
well one approach you can take is is to
decompose that that very hard problem
into into pieces that work together to
solve that hard problem
and and so you can kind of look at the
decomposition that's inside the agents
head if you like and ask well what form
does that decomposition take and some of
the most common pieces that people use
when they're kind of putting this system
the solution method together some of the
most common pieces that people use are
whether or not that solution has a value
function that means is it trying to
predict explicitly trying to predict how
much reward it will get in the future
does it have a representation of a
policy that means something which is
deciding how to pick actions is is that
decision-making process explicitly
represented and is there a model in the
system is there something which is
explicitly trying to predict what will
happen in the environment and so those
three pieces are to me some of the most
common building blocks and I understand
the different choices in RL as choices
of whether or not to use those building
blocks when you're trying to decompose
the solution you know should I have a
value function represented so they have
a policy represented should I have a
model represented and there are
combinations of those pieces and of
course other things that you could add
to add into the picture as well but
those those three fundamental choices
give rise to some of the branches of RL
with which we're very familiar and so
those as you mentioned there is the
choice of what's specified or modeled
explicitly and the idea is that all of
these are somehow implicitly learned
within the system so it's almost a
choice of how you approach a problem do
you see those as fundamental differences
or these almost like small specifics
like the details of how you saw the
problem but they're not fundamentally
different from each other I think the
the fundamental idea is is maybe at the
higher level the fundamental idea is the
first step of the decomposition is
really to say well how are we really
going to solve any kind of problem where
you're trying to figure out how to take
actions and just from a stream of
observations you know you've got some
agents situated it's sensory motor
stream and getting all these
observations here and getting to take
these actions and and what should it do
how can even broach that problem you
know me
the complexity of the world is so great
that you can't even imagine how to build
a system that would that would
understand how to deal with that and so
the first step of this decomposition is
to say well you have to learn the system
has to learn for itself and so note that
the reinforcement learning problem
doesn't actually stipulate that you have
to learn but you could maximize your
awards without learning it would just
say wouldn't do a very good job event
yes so learning is required because it's
the only way to achieve good performance
in any sufficiently large and complex
environment so so that's the first step
so that step give commonality to all of
the other pieces because now you might
ask well what should you be learning
what is learning even mean you know in
this sense you know learning might mean
well you're trying to update the
parameters of some system which is then
the thing that actually picks the
actions and and those parameters could
be representing anything they could be
parameterizing a value function or a
model or a policy and so in that sense
there's a lot of commonality in that
whatever is being represented there is
the thing which is being learned and
it's being learned with the ultimate
goal of maximizing rewards but but the
way in which you decompose the problem
is is is really what gives the semantics
to the whole system like are you trying
to learn something to predict well like
a value function or a model are you
learning something to perform well like
a policy and and the form of that
objective like it's kind of giving the
semantics to the system and so it really
is at the next level down a fundamental
choice and we have to make those
fundamental choices a system designers
or enable are our algorithms to be able
to learn how to make those choices for
themselves so then the next step you
mentioned the very for the very first
thing you have to deal with is can you
even take in this huge stream of
observations and do anything with it so
the natural next basic question is what
is the what is deep reinforcement
learning and what is this idea of using
neural networks to deal with this huge
incoming stream so amongst all the
approaches for reinforcement learning
deep reinforcement learning is one
family of solution
feds that tries to utilize powerful
representations that are offered by
neural networks to represent any of
these different components of the
solution of the agent like whether it's
the value function or the model or the
policy the idea of deep learning is to
say well here's a powerful tool kit
that's so powerful that it's Universal
in the sense that it can represent any
function and it can learn any function
and so if we can leverage that
universality that means that whatever
whatever we need to represent for our
policy or offer a value function or for
a model deep learning can do it so that
deep learning is is one approach that
offers us a toolkit that is has no
ceiling to its performance that as we
start to put more resources into the
system or more memory and more
computation and more more data more
experience of more interactions with the
environment that these are systems that
can just get better and better and
better at doing whatever the job is
they've asked them to do whatever we've
asked that function to represent it can
learn a function that does a better and
better job of representing that that
knowledge whether that knowledge be
estimating how well you're going to do
in the world the value function whether
it's going to be choosing what to do in
the world a policy or it's understanding
the world itself what's going to happen
next the model nevertheless the the the
fact that neural networks are able to
learn incredibly complex representations
that allow you to do the policy the
model or the value function is at least
to my mind exceptionally beautiful and
surprising like what was it is it
surprising was it surprising to you can
you still believe it works as well as it
does do you have good intuition about
why it works at all and works as well as
it does I think let me take two parts to
that question I think it's not
surprising to me that the idea of
reinforcement learning works because in
some sense I think it's the I feel it's
the only
which can ultimately and so I feel we
have to we have to address it and there
must be success is possible because we
have examples of intelligence and it
must at some level be able to possible
to acquire experience and use that
experience to to do better in a way
which is meaningful to environments of
the complexity that humans can deal with
it must be am I surprised that our
current systems can do as well as they
can do I think one of the big surprises
for me and a lot of the community it's
really the fact that deep learning can
continue to perform so well despite than
the fact that these neural networks that
they're representing have these
incredibly nonlinear kind of bumpy
surfaces which two are kind of low
dimensional intuitions make it feel like
surely you're just going to get stuck
and learning will get stuck because you
won't be able to make any further
progress and yet the big surprise is
that learning continues and and these
what appear to be local Optima turned
out not to be because in high dimensions
when we make really big neural nets
there's always a way out and there's a
way to go even lower and then he's still
not another local Optima because there's
some other pathway that will take you
out and take you lower still and so no
matter where you are learning can
proceed and do better and better and
breath better without bound and so that
is a surprising and beautiful property
of neural nets which I find elegant and
beautiful and and somewhat shocking that
it turns out to be the case as you said
which I really like to our low
dimensional intuitions that's surprising
yeah yeah we're very we're very tuned to
working within a three-dimensional
environment and so to start to visualize
what a billion dimensional neural
network um surface that you're trying to
optimize over what that even looks like
is very hard for us and so I think that
really if you try to account for
the essentially the AI winter where
where people gave up on Yule networks I
think it's really down to that that lack
of ability to generalize from from low
dimensions to high dimensions because
back then we were in the low dimensional
case people could only build neural nets
with you know 50 nodes in them or
something and to to imagine that it
might be possible to build a billion
dimension on your net and it might have
a completely different qualitatively
different property was very hard to
anticipate and I think even now we're
starting to build the the theory to
support that and and it's incomplete at
the moment but all of the theory seems
to be pointing in the direction that
indeed this is an approach which which
truly is universal both in its
representational capacity which was
known but also in its learning ability
which is which is surprising and it
makes one wonder what else were missing
yes for a low demand intuitions yet
there will seem obvious once it's
discovered I often wonder you know when
we one day do have a eyes which are
superhuman in their abilities to to
understand the world what will they
think of the algorithms that we
developed back now will it be you know
looking back at these these days and you
know and and and thinking that well will
we look back and feel that these
algorithms were were naive faire steps
or will they still be the fundamental
ideas which are used even in 100
thousand 10,000 years yeah Nels and I
they'll they'll watch back to this
conversation and I would the smile maybe
a little bit of a laugh I mean my senses
I think it just like on we used to think
that
the Sun revolved around the earth
they'll see our systems of today in
reinforcement learning as too
complicated that the answer was simple
all along there's something I just just
think you said in a game of Go I mean I
love those systems of like cellular
automata that there's simple rules from
which incredible complexity emerges so
it feels like there might be some very
simple approaches just like where Sutton
says right these simple methods or with
compute over time seem to prove to be
the most effective I 100% agree I think
that if we try to anticipate what will
generalize well into the future I think
it's likely to be the case that it's the
simple clear ideas which will have the
longest legs and walked or carry us
farthest into the future nevertheless
we're in a situation where we need to
make things work day and today and
sometimes that requires putting together
more complex systems where we don't have
the the full answers yet as to what
those minimal ingredients might be so
speaking of which if we could take us
their bag to go what was Mogo and what
was the key idea behind this system so
back during my PhD on computer go around
about that time there was a major new
development in in which actually
happened in the context of computer go
and and it was really a revolution in
the way that heuristic search was was
done and and the idea was essentially
that a position could be evaluated or a
state in general could be evaluated not
by humans saying whether that position
is good or not or even humans providing
rules as to how you might evaluate it
but instead by allowing the system to
randomly play out the game until the end
multiple times and taking the average of
those outcomes as the prediction of what
will happen so for example if you're in
the game of go the intuition is that you
take a position
and you get the system to kind of play
random moves against itself all the way
to the end of the game and you see who
wins and if black ends up winning more
of those random games than white well
you say hey this is a position that
favors white and if white ends up
winning more of those random games than
black then it favors white so that idea
was known as Monte Carlo search and a
particular form of Monte Carlo search
that became very effective and was
developed in computer go first by Remy
Coulomb in 2006 and then taken further
by others was something called Monte
Carlo tree search which basically takes
that same idea and uses that that
insight to evaluate every node of a
search tree is evaluated by the average
of the random play outs from that from
that node onwards and this idea was very
powerful and suddenly led to huge leaps
forward in the strength of computer go
playing programs and among those the the
strongest of the go playing programs in
those days was a program called Mogo
which was the first program to actually
reach human master level on small boards
nine by nine boards and so this was a
program by someone called Sylvan jelly
he was a good colleague of mine but I
worked with him a little bit in those
days of my PhD thesis and Mogo was a a
first step towards the latest successes
we saw and computer go but it was still
missing a key ingredient
Mogo was evaluating purely by random
rollouts against itself and in a way
it's it's truly remarkable that random
play gives you anything at all yeah like
how why why in this perfectly
deterministic game that's very precise
and involves these very exact sequences
why is it that that random randomization
is helpful and so the intuition is that
randomization captures something about
the the nature of the of the search tree
that from a position that you're you're
understanding the nature of the search
tree from that node onwards by by by
using randomization and this was a very
powerful idea
and I've seen this in other spaces talk
to the virtual carpet and so on
randomized algorithms somehow magically
are able to do exceptionally well and
and simplifying the problem somehow
makes you wonder about the fundamental
nature of randomness in our universe it
seems to be a useful thing but so from
that moment can you maybe tell the
origin story in the journey of alphago
yeah so programs based on Monty College
research were a first revolution in the
sense that they led to suddenly programs
that could play the game to any
reasonable level but they they plateaued
it seemed that no matter how much effort
people put into these techniques they
couldn't exceed the level of amateur Dan
level go players so strong players but
not not anywhere near the level of
professionals never mind the world
champion and so that brings us to the
birth of alphago which happened in the
context of a startup company known as
deep mind or where them where a project
was born and the project was really a
scientific investigation where myself
and a jipang and an intern Chris Madison
were exploring a scientific question and
that scientific question was really
is there another fundamentally different
approach to to this key question of Goa
the key challenge of how can you build
that intuition and how can you just have
a system that could look at a position
and understand what moved to play or or
how well you're doing in that position
who's going to win and so the deep
learning Revolution had just begun their
systems like imagenet had suddenly been
won by deep learning techniques back in
2012 and following that it was natural
to ask well you know if if deep learning
is able to scale up so effectively with
images to to understand them enough to
to classify them well why not go why why
not take a the black and white stones of
the NGO board and build some a system
which can understand for itself what
that means in terms of what moved to
pick or who's going to win the game
black or white and so that was our
scientific question which we we were
probing and trying to understand and as
we started to look at it we discovered
that we could build a a system so in
fact our very first paper on alphago was
actually a pure deep learning system
which was trying to answer this question
and we showed that actually a pure deep
learning system with no search at all
was actually able to reach human van
level master level at the full game of
go 19 by 19 boards and so without any
search at all suddenly we had systems
which were playing at the level of the
best Monte Carlo tree search systems the
ones with randomized rollouts so first
I'm sorry to interrupt but there's kind
of a groundbreaking notion let's say
that's like basically a definitive step
away from the a couple of decades of
essentially search dominating AI yeah so
what how do them make you feel would you
that was a surprising from a scientific
perspective in general how to make you
feel I I found this to be profoundly
surprising in fact it was so surprising
that that we had a bet back then and
like many good projects you know bets
are quite motivating and Anna bet was
you know whether it was possible for a
system
purely on on deep learning no search at
all to beat a Dan level human player and
so we had someone who joined our team
who was a damn level player he came in
and and we had this first match against
him and we turned the bit where you want
by the way do you handle losing and they
were in except I tend to be an optimist
with the with the power of of deep
learning and reinforcement learning so
the system won and we were able to beat
this human Dan level player and for me
that was the moment where where it's
like okay something something special is
afoot here we have a system which
without search is able to to already
just look at this position and
understand things as well as a strong
human player and from that point onwards
I really felt that reaching that
reaching the top levels of human play
you know professional level world
champion level I felt it was actually an
inevitability and and if it was an
inevitable outcome
I was rather keen it would be us that
achieve it so we scaled up this was
something where you know so I had lots
of conversations back then with demo so
service that the head of deepmind who
was extremely excited and we we made the
decision to to scale up the project
brought more people on board and and so
alphago became something where where we
we had a clear goal which was to try and
crack this outstanding challenge of AI
to see if we could beat the world's best
players and this led within the space of
not so many months to playing against
the European champion fan way in a match
which became you know memorable in
history is the first time a go program
would ever beated a a professional
player and at that time we had to make a
judgment as to whether when and and
whether we should go and challenge the
world champion and and this was a
difficult
to make again we were basing our
predictions on on our own progress and
had to estimate based on the rapidity of
our own progress when we thought we
would exceeds the level of the human
world champion and and we tried to make
an estimate and set up a match and that
became the the alphago versus Lisa dolls
match in 2016 and we should say spoiler
alert that alphago was able to defeat
Lisa doll that's right yeah so maybe a
could take even a broader view
alphago involves both learning from
expert games and as far as I remember a
self play component - where he learns by
playing guess himself but in your sense
what was the role of learning from
experts there and in terms of your self
evaluation whether you can take on the
world champion what was the thing that
they're trying to do more of sort of
train more on expert games or was
there's now another I'm asking so many
poorly faced questions but did you have
a hope a dream that self play would be
the key component at that moment yet so
in the early days of alphago we we used
human data to explore the science of
what deep learning can achieve and so
when we had our first paper that showed
that it was possible to predict the
winner of the game that it was possible
to suggest moves that was done using
human data of solely human did yes and
and and and so the reason that we did it
that way was at that time we were
exploring
separately the deep learning aspect from
the reinforcement learning aspect that
was the part which was which was new and
unknown to me at that time was how far
could that be stretched once we had that
it then became natural to try and use
that same representation and see if we
could learn for ourselves using that
same representation and so right from
the beginning actually our goal had been
to build a system using self play and to
us the human data right from the
beginning was
an expedient step to help us for
pragmatic reasons to go faster towards
the goals of the project then we might
be able to starting solely from self
play and so in those days we were very
aware that we were choosing to to use
human data and that might not be the
long-term holy grail of AI but that it
was something which was extremely useful
to us it helped us to understand the
system helped us to build deep learning
representations which were clear and
simple and easy to use and so really I
would say it's it served a purpose not
just as part of the algorithm but
something which I continued to use in
our research today which is trying to
break down a very hard challenge into
pieces which are easier to understand
for us as researchers and develop so if
you if you use a component based on
human data it can help you to understand
the system such that then you can build
the more principled version later that
does it for itself so as I said the
alphago victory and I don't think I'm
being sort of romanticizing this notion
I think is one of the greatest moments
in the history of AI
so were you cognizant of this magnitude
of the accomplishment at the time I mean
we are you cognizant of it even now
because to me I feel like it's something
that would we mentioned what the AGI
systems of the future will look back I
think they'll look back at the alphago
tree as like holy crap they figured it
out this is where this is where the
started well thank you again I mean it's
funny because I guess I've been working
on I've been working on computer go for
a long time so I've been working at the
time at the alphago match on computer go
for more than a decade and throughout
that decade I'd had this dream of what
would it be like - what would it be like
really - to actually be able to build a
system that could play against the world
champion and and I imagined that that
would be an interesting moment that
maybe you know some people might care
about that and that this might be you
know a nice achievement but I think when
I arrived in in Seoul and discovered the
legions of
that were following us around and 100
million people that were watching the
match online life I realized that I had
been off in my estimation of how
significant this moment was by several
orders of magnitude and so there was
definitely an adjustment process to to
realize that this this was something
which the world really cared about and
which was a watershed moment and I think
there was that moment of realization it
was also a little bit scary because you
know if you go into something thinking
it's going to be may be of interest and
then discover that 100 million people
are watching it suddenly makes you worry
about whether some of the decisions
you've made where really they're the
best ones or the wisest or we're going
to lead to the best outcome and we knew
for sure that there were still
imperfections in alphago which were
going to be exposed to the whole world
watching and so yeah it was a it was I
think a great experience and I I feel
privileged to have been part of it
privileged to have led that amazing team
I feel privileged to have been in a
moment of history like you say but also
lucky that you know in a sense I was
insulated from from the knowledge of I
think it would have been harder to focus
on the research if the full kind of
reality of what was going to come to
pass her had been known to me and the
team I think it was you know we were we
were in our bubble and we were working
on research and we were trying to answer
the scientific questions and then BAM
you know the public sees it and and I
think it was it was it was better that
way in retrospect were you confident did
I guess what were the chances that you
could get the win so just like you said
I'm a little bit more familiar with
another accomplishment that we may not
even get a chance to talk to I talked to
us about Alpha star which is another
incredible accomplishment but here you
know with alpha star and beating the
Starcraft there was like already a track
record with alphago there this is like
the really first time you get to see
reinforcement learning
face the best humour in the world so
what was your confidence like what was
the odds
well we actually was there a bit but
funnily enough there was so so just
before the match
we weren't betting on anything concrete
but we all held out a hand everyone in
the team held out her hand at beginning
of the match and the number of fingers
that they had out on the hand was
supposed to represent how many games
they thought we would win
I guess Lisa doll and there was an
amazing spread in there in the team's
predictions but I have to say I
predicted four one and and the reason
was based purely on on data so I'm a
scientist first and foremost and one of
the things which we had established was
that
alphago in around 1 in 5 games would
develop something which we called a
delusion which was a kind of inner hole
in its in its knowledge where it wasn't
able to fully understand everything
about the position and that that hole
and its knowledge would persist for tens
of moves throughout the game and we knew
two things we knew that if there were no
delusions that alphago seemed to be
playing at a level that was far beyond
any human capabilities but we also knew
that if there were delusions the office
it was true and and and in fact you know
that's that's what came to pass we saw
we saw all of those outcomes and Lisa
doll in in one of the games played a
really beautiful sequence that that that
alphago just hadn't predicted and after
that it it led it into this situation
where it was unable to really understand
the position fully and and and found
itself in one of these these delusions
so so indeed yeah for one was the
outcome so yeah and can you maybe speak
to it a little bit more what were the
five games like what what happened is
there interesting things that they come
to memory in terms of the play of the
human machine so I remember all of these
games vividly of course you know moments
like these don't come too often in the
lifetime of her of her scientist and the
the first game was was magical because
it was the first time that a computer
program had defeated a world champion in
this Grand Challenge of go and and there
was a moment where where alphago invaded
Lisa dolls
territory towards the end of the game
and and that's quite an audacious thing
to do it's like saying hey you thought
this was gonna be your territory in the
game but I'm going to stick a stone
right in the middle of it and and and
prove to you that I can break it up and
Lisa dolls face just dropped he wasn't
expecting a computer to to do something
that audacious the second game became
famous for a move known as move 37 this
was a move that was played by alphago
that was broke all of the conventions of
go that the go players were so shocked
by this they they they thought that
maybe the operator had made a mistake
they they thought that there's something
crazy going on and and it just broke
every rule that go players are taught
from a very young age they just taught
you know you this kind of move called
the shoulder hit you you you can only
play it on the third line or the fourth
line and alphago played out in the fifth
line and and it turned out to be a
brilliant move and made this beautiful
pattern in the middle of the board that
ended up winning the game and so this
really was a clear instance where we
could say computers exhibited creativity
that this was really a move that was
something humans hadn't known about
hadn't anticipated and computers
discovered this idea they they were the
ones to say actually you know here's a
new idea something new not not in the
domains of human knowledge of the game
and and and now the humans think this is
a reasonable thing to do and and it's
part of go knowledge now the third game
something special happens when you play
against a human world champion which
again I hadn't anticipated before going
there which is you know these these
players are amazing Lisa Dahl was a true
champion eighteen time world champion
and had this amazing ability to to probe
alphago fer for weaknesses of any kind
and in the third game he was losing
and we felt we were sailing comfortably
to victory but he managed to from
nothing stir up this fight and build
what's called a double ko these kind of
repetitive positions and he knew that
historically no no computer go program
had ever been able to deal correctly
with double code positions and he
managed to summon one out of out of
nothing and so for us you know this was
this was a real challenge like would
alphago be able to deal with this or
would it just kind of crumble in the
face of this situation and fortunately
it dealt with it perfectly the force
game was was amazing in that Lisa doll
appeared to be losing this game alphago
thought it was winning and then Lisa
doll did something which I think only a
true world champion can do which is he
found a brilliant sequence in the middle
of the game a brilliant sequence that
led him to really just transform the
position it kind of it it he found it's
just a piece of genius really
and after that alphago
it's it's evaluation just tumbled it
thought it was winning this game and all
of a sudden it tumbled and said oh now
I've got no chance and it starts to
behave rather oddly at that point in the
final game for some reason we as a team
were convinced having seen alphago in
the previous game suffer from delusions
we as a team were convinced that it was
suffering from another delusion we were
convinced that it was miss evaluating
the position and that something was
going terribly wrong and it was only in
the last few moves of the game that we
realized that actually although it had
been predicting it was going to win all
the way through it really was and and so
somehow you know it just taught us yet
again that you have to have faith in in
your systems when they when they exceed
your own level of ability in your own
judgment you have to trust in them too
to know better than the new the designer
once you've you've stowed in them the
ability to to judge better than you can
then trust the system to do so so just
looking in case of deep blue beating
Garry Kasparov
so get garrus is I think the first time
he's ever lost actually to anybody and I
mean there's a similar situation loose
at all it's uh it's a tragic it's a
tragic loss for humans but a beautiful
one I think that's kind of from the
tragedy sort of emerges over time
emerges the kind of inspiring story but
Lisa Dahl recently announced his
retirement I don't know if we can look
too deeply into it but he did say that
even if I become number one there's an
entity that cannot be defeated so what
do you think about these words what do
you think about his retirement from the
game ago well let me take you back first
of all to the first part of your comment
about Garry Kasparov because actually at
the panel yesterday he specifically said
that when he first lost a deep-blue
he he viewed it as a failure he viewed
that this this had been a failure of his
but later on in his career he said he'd
come to realize that actually it was a
success it was a success for everyone
because this marked a transformational
moment for AI and so even for Kip Garry
Kasparov he came to realize at that
moment was was was pivotal and actually
meant something much more than then you
know his personal loss in that moment
Lisa doll I think was a much more
cognizant of that even at the time so in
his closing remarks to the match he
really felt very strongly that what had
happened and the alphago match was not
only meaningful for AI but for humans as
well and he felt as a go player that it
had opened his horizons and meant that
he could start exploring new things it
brought his joy back for the game of go
because it broken all of the conventions
and barriers and meant that you know
suddenly suddenly anything was possible
again and so you know I was sad to hear
that he'd retired but you know he's been
a great a great world champion over many
many years
and I think you know that he'll be he'll
be remembered for that evermore he'll be
remembered as the last person to to beat
alphago I mean after after that we
increased the power of the system and
and the next version of alphago beats
the the other strong human players 60
games to nil so you know what a great
moment for him and something to be
remembered for
it's interestingly you spent time at
triple AI on a panel with Garry Kasparov
what I mean it's almost just curious to
learn the conversations you've had with
Garry and the because he's also now he's
written a book about artificial
intelligence he's thinking about AI he
has kind of a view of it and he talks
about alphago a lot what what's your
sense be arguably I'm not just being
Russian but I think Gary is the greatest
chess player of all time the probably
one of the greatest game players of all
time and you sort of at the center of
creating a system that beats one of the
greatest players of all time
so what's that conversation like is
there anything yeah any interesting digs
any bets and you come and you find new
things and you profound things so Gary
Kasparov has an incredible respect for
what we did with alphago and you know
it's it's an amazing tribute coming from
from him of all people that he really
appreciates and respects what what we've
done and
I think he feels that the progress which
was happened in in computer chess which
later after alphago we we built the
alpha zero system which defeated the the
world's strongest chess programs and to
Garry Kasparov that moment in computer
chess was more profound than than than
deep blue and the reason he believes it
mattered more was because it was done
with with learning and a system which
was able to discover for itself new
principles new ideas which were able to
play the game in a in a in a way which
he hadn't always known about or anyone
and in fact one of the things I
discovered at this panel was that the
current world champion Magnus Carlsen
apparently recently commented on his
improvement in performance and he
attributes it to alpha zero that he's
been studying the games of alpha zero
and he's changed his style play more
like alpha zero and it's led to him
actually increasing his his his rating
to a new peak yeah I guess to me just
like to Gary the inspiring thing is that
and just like you said with
reinforcement learning reinforcement
learning and deep learning machine
learning feels like what intelligence is
yeah and you know you could attribute it
to sort of a bitter viewpoint from
Gary's perspective from us humans
perspective saying that sir pure search
that IBM do Blue was doing is not really
intelligence but somehow it didn't feel
like it and so that's the magical I'm
not sure what it is about learning that
feels like intelligence but it but it
does so I think we should not demean the
achievements of what was done in
previous eras of AI I think that deep
blue was an amazing achievement in
itself and that heuristic search of the
kind that was used by deep blue had some
powerful ideas that were in there but it
also missed some things so so the fact
that the that the evaluation function
the way that the chess position was
understood was created by humans and not
by the machine is a limitation which
means that there's a ceiling on how well
it can do but maybe more importantly it
means
the same idea cannot be applied in other
domains where we don't have access to
the kind of human Grand Master's and
that ability to kind of encode exactly
their knowledge into an evaluation
function and the reality is that the
story of AI is that you know most
domains turn out to be of the second
type where when knowledge is messy it's
hard to extract from experts or it isn't
even available and so so we need to
solve problems in a different way and I
think alphago is a step towards solving
things in a way which which puts
learning as first-class citizen and says
systems need to understand for
themselves how to understand the world
how to judge their the value of any
action that they might take within that
world in any state they might find
themselves in and in order to do that we
we make progress towards AI yeah so one
of the nice things about this about
taking a learning approach to the game
of Go game playing is that the things
you learn the things you figure out are
actually going to be applicable to other
problems there are real-world problems
that's so that's ultimately I mean
there's two really interesting things
about alphago one is the science of it
just the science of learning the science
of intelligence and then the other is
all you're actually learning to figuring
out how to build systems that would be
potentially applicable in in other
applications medical autonomous vehicles
robotics all I mean it's just open the
door to all kinds of applications so the
next incredible step right really the
profound step is probably alphago zero I
mean it's arguable I kind of see them
all as the same place but really in
perhaps you were already thinking that
alphago zeros the natural it was always
going to be the next step
but it's removing the reliance on human
expert games for pre-training as you
mentioned so how big of an intellectual
leap was this that that self play could
achieve superhuman level performance
it's on and maybe could you also say
what is self play we kind of mentioned a
few times but so let me start with self
play so the idea of self play is
something which is really about systems
learning for themselves but in the
situation where there's more than one
agent and so if you're in a game and a
game is a played between two players
then self play is really about
understanding that game just by playing
games against yourself rather than
against any actual real opponent and so
it's a way to kind of um discover
strategies without having to actually
need to go out and play against any
particular human player for example the
main idea of alpha zero was really to
you know try and step back from any of
the knowledge that we'd put into the
system and ask the question is it
possible to come up with a single
elegant principle by which a system can
learn for itself all of the knowledge
which it requires to play to play a game
such as go importantly by taking
knowledge out you not only make the
system less brittle in the sense that
perhaps the knowledge you were putting
in was was just getting in the way and
maybe stopping the system learning for
itself but also you make it more general
the more knowledge you put in the harder
it is for a system to actually be placed
taken out of the system in which it's
kind of been designed and placed in some
other system that maybe would need a
completely different knowledge base to
to understand and perform well and so
the real goal here is to strip out all
of the knowledge that we put in to the
point that we can just plug it into
something totally different and that to
me is really you know the the promise of
AI is that we can have systems such as
that which you know no matter what the
goal is no matter what goal we set to
the system we can come up with we have
an algorithm which can be placed into
that world into that and
and can succeed in achieving that goal
and then that that's to me is almost the
the essence of intelligence if we can
achieve that and so alpha zero is a step
towards that and it's a step that was
taken in the context of two-player
perfect information games like go and
chess we also applied it to Japanese
chess so just to clarify the first step
was alphago zero the first step was to
try and take all of the knowledge out of
alphago in such a way that it could play
in a in a fully self discovered way
purely from self play and to me the the
motivation for that was always that we
could then plug it into other domains
but we saved that bat until later well
in in fact I mean just for fun I could
tell you exactly the moment where where
the idea for alpha zero occurred to me
because I think there's maybe a lesson
there for for researchers who kind of
too deeply embedded in their in their
research and you know working 24/7 to
try and come up with the next idea which
is actually occurred to me on honeymoon
like it's my most fully relaxed state
really enjoying myself and and just
being this like the algorithm for alpha
zero just appeared I come and in in its
full form and this was actually before
we played against Lisa doll but we we
just didn't I think we were so busy
trying to make sure we could beat the
the world champion that it was only
later that we had the the opportunity to
step back and start examining that that
sort of deeper scientific question of
whether this could really work so
nevertheless so soft play is probably
one of the most profound ideas that
represents to me at least artificial
intelligence but the fact that you could
use that kind of mechanism to again be
more
glass players that's very surprising so
we kind of to be it feels like you have
to train in a large number of expert
gamer so was it surprising to you what
was the intuition can you sort of think
not necessarily at that time even now
what's your intuition why this thing
works so well why I was able to learn
from scratch well let me first say why
we tried it so we tried it both because
I feel that it was the deeper scientific
question to to be asking to make
progress towards AI and also because in
general in my research I don't like to
do research on questions for which we
already know the likely outcome I don't
see much value in running an experiment
where you're 95% confident that that you
will succeed and so we could have tried
you know maybe to to take alphago and do
something which we we knew for sure it
would succeed on but much more
interesting to me was to try try it on
the things which we weren't sure about
and one of the big questions on our
minds back then was you know could you
really do this with self play alone how
far could that go would it be as strong
and honestly we weren't sure yeah it was
50/50 I think you know we I really if
you'd asked me I wasn't confident that
it could reach the same level as these
systems but it felt like the right
question to ask and even if even if it
had not achieved the same level I felt
that that was an important direction to
be studying and so then lo and behold it
actually ended up outperforming the
previous version of of alphago and
indeed was able to beat it by 100 games
to zero so what's the intuition as to as
to why I think that the intuition to me
is clear that whenever you have errors
in a in a system as we did in alphago
alphago suffered from these delusions
occasionally it would misunderstand what
was going on in a position and miss
evaluate it how can how can you remove
all of these these errors errors arise
from many sources for us they were
arising both from you know it started
from the human data but also
from there from the nature of the search
and the nature of the algorithm itself
but the only way to address them in any
complex system is to give the system the
ability to correct its own errors it
must be able to correct them it must be
able to learn for itself when it's doing
something wrong and correct for it and
so it seemed to me that the way to
correct delusions was indeed to have
more iterations of reinforcement
learning that you know no matter where
you start you should be able to correct
those errors until it gets to play that
out and understand oh well I thought
that I was going to win in this
situation but then I ended up losing
that suggests that I was miss evaluating
something there's a hole in my knowledge
and now now the system can correct for
itself and and understand how to do
better now if you take that same idea
and trace it back all the way to the
beginning it should be able to take you
from no knowledge from completely random
starting point all the way to the
highest levels of knowledge that you can
achieve in in a domain and the principle
is the same that if you give if you
bestow a system with the ability to
correct its own errors then it can take
you from random to something slightly
better than random because it sees the
stupid things that the random is doing
and it can correct them and then it can
take you from that slightly better
system and understand what what's that
doing wrong and it takes you on to the
next level and the next level and and
this progress it can go on indefinitely
and indeed you know what would have
happened if we'd carried on training
alphago zero for longer we saw no sign
of it slowing down it's in improvements
or at least it was certainly carrying on
to improve and presumably if you had the
computational resources this this could
lead to better and better systems that
discover more and more so your intuition
is fundamentally there's not a ceiling
to this process
the one of the surprising things just
like you said is the process of patching
errors it's intuitively makes sense they
this is a reinforcement learning should
be part of that process but what is
surprising is in the process of patching
your own lack of knowledge you don't
open up other patches you go you keep
sort of cool like there's a monotonic
decrease of your weaknesses well let me
let me back this up you know I think
science always should make falsifiable
hypotheses yes so let me let me back out
this claim with a falsifiable hypothesis
which is that if someone was to in the
future take alpha zero as an algorithm
and run it on with greater computational
resources that we had available today
then I predict that they would be able
to beat the previous system 100 games to
zero and that if they were then to do
the same thing a couple of years later
that that would be that previous system
hundred games to zero and that that
process would continue indefinitely
throughout at least my human lifetime
presumably the game of girl would set
the ceiling I mean the game of go would
set the ceiling but the game of go has
ten to the hundred and seventy states in
it so so the ceiling is unreachable by
any computational device that can be
built out of the you know 10 to the 80
atoms in the universe you asked a really
good question which is you know do you
not open up other errors when you when
you correct your previous ones and the
answer is is yes you do and so so it's a
remarkable fact about about this class
of two-player game and also true of
single agent games that essentially
progress will always lead you to if you
have sufficient representational
resource like imagine you had could
represent every state in a big table of
the game then we we know for sure that a
progress of self-improvement will lead
all the way in the single agent case to
the optimal possible behavior and in the
two-player case to the minimax optimal
behavior and that is that the best way
that I can play knowing that you're
playing perfectly against me and so so
for those cases we know that even if you
do open up some new error that in some
sense you've made progress you've you're
progressing towards the the best that
can be done so alphago was initially
trained
expert games with some self play alphago
zero removed the need to be trained on
expert games and then another incredible
step for me because I just love chess is
to generalize that further to be in
alpha zero to be able to play the game
of go beating alphago zero and alphago
and then also being able to play the
check the game of chess and others so
what was that step like what's the
interesting aspects there that required
to make that happen
I think the remarkable observation which
we saw with alpha zero was that actually
without modifying the algorithm at all
it was able to play and crack some of a
i's greatest previous challenges in
particular we dropped it into the game
of chess and unlike the previous systems
like deep blue which had been worked on
for you know years and years and we were
able to beat the world's strongest
computer chess program convincingly
using a system that was fully discovered
by its own from from scratch with its
own principles and in fact one of the
nice things that that we found was that
in fact we also achieved the same result
in in Japanese chess a variant of chess
where where you get to capture pieces
and then place them back down on your on
your own side as an extra piece so much
more complicated variant of chess and we
also beat the world's strongest programs
and reach superhuman performance in that
game too
and it was the very first time that we'd
ever run the system on that particular
game was the version that we published
in the paper on on alpha zero it just
works out of the box literally no no no
touching it we didn't have to do
anything and and there it was superhuman
performance no tweaking no no twiddling
and so I think there's something
beautiful about that principle that you
can take and algorithm and without
twiddling anything it just it just works
now to go beyond alpha zero what's
required alpha zero is is just a step
and there's a long way to go beyond that
to really crack the deep problems of AI
but one of the important steps is to
acknowledge that the world is a really
messy place you know it's this rich
complex beautiful but messy environment
that we live in and no one gives us the
rules like no one knows the rules of the
world at least maybe we understand that
it operates according to Newtonian or
quantum mechanics at the micro level all
according to relativity at the macro
level but that's not a model that's used
to useful for us as people to to operate
in it somehow the agent needs to
understand the world for itself in a way
where no one tells it the rules of the
game and yet it can still figure out
what to do in that world deal with this
stream of observations coming in rich
sensory input coming in actions going
out in a way that allows it to reason in
the way that alphago or alpha zero can
reason in the way that these go and
chess-playing programs can reason but in
a way that allows it to take actions in
that messy world to to achieve its goals
and so this led us to the most recent
step in the story of alphago
which was a system called mu 0 and mu
zero is a system which learns for itself
even when the rules are not given to it
it actually can be dropped into a system
with messy perceptual inputs we actually
tried it in the in some Atari games the
canonical domains of Atari that have
been used for reinforcement learning and
and this system learned to build a model
of these Atari games they were
sufficiently rich and useful enough for
it to be able to plan successfully and
in fact that system not only went on to
to beat the state of the art in Atari
but the same system without modification
was able to reach the same level of
superhuman performance in go chess and
shogi that we'd seen in alpha zero
showing that even without the rules the
system can learn for itself just by
trial and error just by playing this
game of go and no one tells you what the
rules are but you just get to the end
and and someone says you know win or
loss you play this game
and someone says win or lost so you play
a game of breakout in Atari and someone
just tells you you know your score at
the end and the system for itself
figures out essentially the rules of the
system the dynamics of the world how the
world works and that not in any explicit
way but just implicitly enough
understanding for it to be able to plan
in that in that system in order to
achieve its goals and that's the you
know that's the fundamental process
there to go through when you're facing
any uncertain kind of environment they
would in the real world it's figuring
out the sort of the rules the basic
rules of the game that's right so
there's a lot I mean the ad that that
allows it to be applicable to basically
any domain that could be digitized in
the way that it needs to in order to be
consumable sort of in order for the
reinforcement learning framework to be
able to sense the environment to be able
to act anywhere and so on the full
reinforcement learning problem needs to
deal with with worlds that are unknown
and and complex and and the agent needs
to learn for itself how to deal with
that so museu I was as a step I felt a
step in that direction one of the things
that inspired the general public
interesting conversations I have like
with my parents or something my mom that
just loves what was done is kind of at
least the notion that there was some
display of creativity some new
strategies new behaviors that were
created that that again has echoes of
intelligence so is there something that
stands up do you see it the same way
that there's creativity and there's some
behaviors patterns you saw that alpha
zero was able to display their truly
creative so let me start by I think
saying that I think we should ask what
creativity really means so to me
creativity means discovering something
which wasn't known before something
unexpected something out outside of our
norms and so in that sense the process
of reinforcement learning or
the self play approach that was used by
alpha zero is it's the essence of
creativity it's really saying at every
stage
you're playing according to your current
norms and you try something and if it
works out you say hey here's something
great I'm gonna start using that and
then that process it's like a micro
discovery that happens millions and
millions of times over the course of the
algorithms life where it just discovers
some new idea oh this pattern this
patterns working really well for me I'm
gonna I'm gonna start using that oh now
oh here's this other thing I can do I
can start to to connect these stones
together in this way or I can start to
you know sacrifice stones or give up on
on on pieces or play shoulder hits on
the fifth line or whatever it is the
system is discovering things like this
for itself continually repeatedly all
the time and so it should come as no
surprise to us then when if you leave
these systems going that they discover
things that are not known to humans to
the human norms are considered creative
and we've seen this several times in
fact in alphago zero we saw this
beautiful timeline of discovery where
what we saw was that there are these
opening patterns that humans play called
joseki these are like the patterns that
humans learn to play in the corners and
they've been developed and refined over
over literally thousands of years in the
game of go and what we saw was in the
course of the training
alphago 0 over the course of the 40 days
that we trained this system it's just to
discover exactly these patterns that
human players play and over time we
found that all of the joseki that humans
played were were discovered by the
system through this process of self play
and a sort of essential notion of
creativity well what was really
interesting was that over time it then
started to discard some of these maybe
own joseki that humans didn't know about
yeah and it starts to say oh well you
thought that the Knights move pincer
joseki was a great idea but here's
something you different you can do there
which make some new variation that the
humans didn't know about and actually
now the human go player study the joseki
their alphago played and they become the
new norms that are used in today
um top-level guy competitions that never
gets old even just the first to me maybe
just makes me feel good as a human being
that a self play mechanism knows nothing
about us humans discovers patterns that
we humans do it's just I get an
affirmation that we're doing we're doing
okay as humans yeah in this domain in
other domains we do we figure it out
it's like the Churchill quote about
democracy it's the you know it's the but
it sucks but it's the best song we've
tried so in general taking a step
outside of go and I take a million
accomplishment to have no time to talk
about that with alpha star and so on and
and and the current work but in general
this self play mechanism that you've
inspired the world with by beating the
world champion goal player do you see
that as DC being applied in other
domains do you have sort of dreams and
hopes that is applied in both the
simulated environments in a constrained
environments of games constrained
I mean alpha star really demonstrates
that you can remove a lot of the
constraints but nevertheless it's in a
digital simulated environment do you
have a hope a dream that it starts being
applied in the robotics environment and
maybe even in domains that are a little
safety critical and so on and have you
know have a real impact in the real
world like autonomous vehicles for
example it seems like a very far-out
dream at this point so I absolutely do
hope and and imagine that we will we
will get to the point where ideas just
like these are used in all kinds of
different domains in fact one of the
most satisfying things as a researcher
as when you start to see other people
use your your algorithms in unexpected
ways so in the last couple of years
there have been you know a couple of
nature papers where different teams
unbeknownst to to us took alpha zero and
applied exactly those same algorithms
and ideas to real-world problems of huge
meaning to to society so one of them was
the problem of chemical synthesis and
they were able to beat the
state-of-the-art in finding pathways of
how to actually synthesize chemicals
retro retro chemical synthesis and the
second paper actually actually just came
out a couple of weeks ago in nature
showed that in quantum computation you
know one of the big questions is how to
how to understand the nature of the the
function in quantum computation and a
system based on alpha zero beat the
state of the art by quite some distance
there again so so these are just
examples and I think you know the lesson
which we've seen elsewhere in machine
learning time and time again is that if
you make something general it will be
used in all kinds of ways you know you
provide a really powerful tools to
society and and those tools can be used
in in amazing ways and so I think we're
just at the beginning and and for sure I
hope that we we see all kinds of
outcomes so the the in the the other
side of the question of a reinforcement
learning framework is you know you
usually want to specify a reward
function and an objective function what
do you think about sort of ideas of
intrinsic rewards if we're not really
sure about you know of if we take you
know human beings existence proof that
we don't seem to be operating according
to a single reward do you think that
there's interesting ideas for when you
don't know how to truly specify the
reward you know that there's some
flexibility for discovering it
intrinsically or so on
in the context of reinforcement learning
so I think you know when we think about
intelligence it's really important to be
clear about the problem of intelligence
and I think it's clearest to understand
that problem in terms of some ultimate
goal that we want the system to to try
and solve for and after all if we don't
understand the ultimate purpose of the
system do we really even have a clearly
defined defined problem that we are
solving at all now within that as with
your example for humans the system may
choose to create its own motivations and
sub goals
that helped the system to achieve its
ultimate goal and that may indeed be a
hugely important mechanism to achieve
those altima goals but there is still
some ultimate goal I think the system
needs to be measurable and and evaluated
against and even for humans I mean
humans were incredibly flexible we feel
that we we can you know any goal that
we're given we feel we can we can master
to some degree but if we think of those
goals really you know like the goal of
being able to pick up an object or the
goal of being able to communicate
although influence people to do things
in a particular way or whatever those
goals are really they are that they're
sub goals really that we set ourselves
you know we choose to pick up the object
we choose to communicate we choose to to
influence someone else and we choose
those because we think it will lead us
to something in our in later art and we
think that that's helpful to us to
achieve some ultimate goal now I don't
want to speculate whether or not humans
as a system necessarily have a singular
overall goal of survival or whatever it
is but I think the principle for
understanding and implementing
intelligences has to be that if we're
trying to understand intelligence or
implement our own there has to be a
well-defined problem otherwise if it's
not I think it's it's like an admission
of defeat that forget to be hope for
understanding or implementing
intelligence we have to know what we're
doing we have to know what we're asking
the system to do otherwise if you if you
don't have a clearly defined purpose
you're not going to get a clearly
defined answer the the ridiculous
big question that has to naturally
follow because they have to pin you down
on this on this thing that nevertheless
one of the big silly or big real
questions before humans is the meaning
of life is us trying to figure out our
own reward function yeah and you just
kind of mentioned that if you want to
build the intelligence systems and you
know what you're doing you should be at
least cognizant to some degree of what
the reward function is so the natural
question is what do you think is the
reward function of human life the
meaning of life for us humans
the meaning of our existence I think you
know I'd be speculating beyond my own
expertise but but just for fun let me do
that yes please and say I think that
there are many levels at which you can
understand a system and and you can
understand something as as optimizing
for a goal at many levels and so so you
can understand the the you know let's
start with the universe like um does the
universe have a purpose well it feels
like it's just one level just following
certain mechanical laws of physics and
that that's led to the development of
the universe but at another level you
can view it as actually there's the
second law of thermodynamics that says
that this is increasing in entropy over
time forever and now there's a view
that's been developed by certain people
at MIT that this you can think of this
as as almost like a goal of the universe
that the purpose of the universe is to
maximize entropy so there's multiple
levels at which you can understand a
system the next level down you might say
well if the goal is to is to maximize
entropy well how do how does how can
that be done by a particular system and
maybe evolution is something that the
universe discovered in order in order to
kind of dissipate energy as efficiently
as possible and by the way I'm borrowing
from Max tegmark for some of these
metaphors yes the physicist
but if you can think of evolution as a
mechanism for dispersing energy then
then evolution you you might say as then
becomes a goal which is if if evolution
disperses energy by reproducing as
efficiently as possible
what's evolution then well it's now got
its own goal within that which is to
actually reproduce as effectively as
possible and now how does reproduction
how is that made as effective as
possible well you need entities within
that that can survive and reproduce as
effectively as possible and so it's
natural in order to achieve that high
level goal those individual organisms
discover brains intelligences which
enable them to support the goals of
evolution
and those brains what do they do well
perhaps the early brains maybe they were
controlling things at some direct level
you know maybe they were the equivalent
of pre-programmed systems which were
directly controlling what was going on
and setting certain you know things in
order to achieve these particular
particular goals but that led to a
another level of discovery which was
learning systems you know parts of the
brain which were able to learn from
themselves and learn how to to program
themselves to achieve any goal and
presumably there are parts of the game
of the brain where goals are set to to
parts of that that system and provides
this very flexible notion of
intelligence that we as humans
presumably have which is the ability to
kind of wipe the reason we feel that we
can we can we can achieve any goal so so
it's a very long-winded answer to say
that you know I think there are many
perspectives and many levels at which
intelligence can be understood and and
each of those levels you can take
multiple perspectives that you know you
can view the system as something which
is optimizing for a goal which is
understanding it at a level by which we
can maybe implement it and understand it
as AI researchers or computer scientists
or you can understand it at the level of
the mechanistic thing which is going on
that there are these you know atoms
bouncing around in the brain and they
lead to the the outcome of that system
is not in contradiction with the fact
that it's it's also a a decision-making
system that's optimizing for some goal
and and purpose I've never heard the
description of the meaning of life
structured so beautifully in layers but
you did miss one layer which is the next
step which you're responsible for which
is creating the the artificial
intelligence and data layer on top of
that and I can't wait to see well I may
not be around but they can't wait to see
what the next layer beyond that well we
well let's just take that that argument
you know and pursue it to a central
conclusion so the next level indeed is
for for how can our how can our learning
brain achieve its goals most effectively
well maybe it does so by by us as
learning beings building a system which
is able to solve for those goals more
effectively than we can and so when we
build a system to play the game of go
you know when I said that I wanted to
build a system that can play go better
than I can I've enabled myself to
achieve that goal of playing go better
than I could buy buy directly playing it
and learning it myself and so now a new
layer has been created which is systems
which are able to achieve goals for
themselves
and ultimately there may be layers
beyond that where they set sub goals to
parts of their own system in order to to
achieve those and so forth so incredible
so the story of intelligence I think I
think is is a multi-layered one and a
multi perspective one we live in an
incredible universe David thank you so
much first of all for dreaming of using
learning to solve go and building
intelligent systems and for actually
making it happen and for inspiring
millions of people in the process it's
truly an honor thank you so much for
talking today okay thank you thanks for
listening to this conversation with
David silver and thank you to our
sponsors masterclass and cash app please
consider supporting the podcast by
signing up to master class at
masterclass complex and downloading cash
app and using code lex podcast if you
enjoy this podcast subscribe on youtube
review it with five stars an apple
podcast supported on patreon or simply
connect with me on Twitter at lex
friedman and now let me leave you with
some words from david silver my personal
belief is that we've seen something of a
turning point where we're starting to
understand that many abilities like
intuition and creativity that we've
previously thought or in the domain only
of the human mind are actually
accessible to machine intelligence as
well and I think that's a really
exciting moment in history thank you for
listening and hope to see you next time
you