Kind: captions
Language: en
the following is a conversation with
Petera Beal he's a professor UC Berkeley
and the director of the Berkeley
robotics learning lab he's one of the
top researchers in the world working on
how we make robots understand and
interact with the world around them
especially using imitation and deeper
enforcement learning this conversation
is part of the MIT course and artificial
general intelligence and the artificial
intelligence podcast if you enjoy it
please subscribe on YouTube iTunes where
your podcast provider of choice or
simply connect with me on Twitter at Lex
Friedman spelled Fri D and now here's my
conversation with Peter a Biel you've
mentioned that if there was one person
you could meet you'll be Roger Federer
so let me ask when do you think we will
have a robot that fully autonomously can
beat Roger Federer at tennis Roger
Federer level player at tennis huh well
first if you can make it happen for me
to meet Roger let me know terms of
getting a robot to beat him at tennis
it's kind of an interesting question
because for a lot of the challenges we
think about in AI the software is really
the missing piece but for something like
this the hardware is nowhere near either
like to really have a robot that can
physically run around the Boston
Dynamics robots are starting to get
there but still not really human level
ability to to run around and then swing
a racket that's a hardware problem I
don't think it's a harder problem only I
think it's a hardware and a software
problem I think it's both and I think
they'll they'll have independent
progress so I'd say the the hardware
maybe in 10-15 years I'm just late not
grass I've dressed with a sliding yeah
oh plague I'm not sure what's Carter
grass or clay the clay involves sliding
which might be harder to master actually
yeah but you're not limited to bipedal I
mean I'm sure there's I can build a
machine it's a whole different question
of course you know
you can if you can say okay this robot
can be on wheels they can move around on
wheels and can be designed differently
then I think that that can be done
sooner probably than a full humanoid
type of setup what do you think is swing
a racket so you've worked at basic
manipulation how hard do you think is
the task of swinging or racket would be
able to hit a nice backhand or a
forehand
okay let's say let's say we just set up
stationary a nice robot arm let's say
you know a standard industrial arm and
it can wash the ball come and then swing
the racket it's a good question I'm not
sure it would be super hard to do I mean
I'm sure it would require a lot if we do
it breed with reinforced Maleny would
require a lot of trial and error it's
not gonna swing it right the first time
around but yeah I don't I don't see why
I couldn't see the right way I think
it's learn about I think if you set up a
ball machine let's say on one side and
then a robot with a tennis racket on the
other side
I think it's learn about and maybe a
little bit of pre training and
simulation yeah I think that's I think
that's feasible I think I think the
swinging the racket is feasible I'd be
very interesting to see how much
precision it can get
listen I mean that's that's where I mean
some of the human players can hit it on
the lines which is very high precision
with spin this win is it is an
interesting whether RL can learn to put
a spin on the ball well you got me
interested maybe someday we'll set this
is your answer is basically okay for
this problem it sounds fascinating but
for the general problem of a tennis
player we might be a little bit farther
away what's the most impressive thing
you've seen a robot do in the physical
world so physically for me it's the
Boston Dynamics videos always just ring
home and just super impressed
recently the robot running up the stairs
doing the parkour type thing
I mean yes we don't know what's
underneath they don't really write a lot
of detail but even if it's hard coded
underneath which you might or might not
be just the physical abilities of doing
that parkour that's a very impressive so
a lot right there have you met spot many
or any of those robots in person might
spot mini last hearing in April at the
Mars event that Jeff Bezos organizes
they brought it out there and it was
nicely falling around Jeff when Jeff
left the room they had it
follow him along which is pretty
impressive so I think there's some
confidence to know that there's no
learning going on in those robots the
psychology of it so while knowing that
while knowing there's not if there's any
learning going on it's very limited
I met spot Minnie earlier this year and
knowing everything that's going on
having one-on-one interaction so I got
to spend some time alone and there's a
immediately a deep connection on the
psychological level even though you know
the fundamentals how it works there's
something magical so do you think about
the psychology of interacting with
robots in the physical world even you
just showed me the pr2 the the robot and
and there was a little bit something
like a face head a little bit something
like a face there's something that
immediately draws you to it do you think
about that aspect of
of the robotics problem well it's very
hard with bread here we'll give him a
name Berkeley robot for the elimination
of tedious tasks is very hard to not
think of the robot as a person and it
seems like everybody calls him a he for
whatever reason but that also makes it
more a person than if it was a it and
it's it seems pretty natural to think of
it that way this past weekend really
struck me I've seen pepper many times on
on videos but then I was at an event
organized by this was by fidelity and
they had scripted pepper to help
moderate some sessions and yet scripted
pepper to have the personality of a
child a little bit and it was very hard
to not think of it as its own person in
some sense because it was just kind of
jumping it would just jump into
conversation making it very interactive
moderate will be saying pepper just jump
in hold on how about me can I
participate in this doing it just like I
heard this is like like a person and I
was 100% scripted and even then it was
hard not to have that sense of somehow
there is something there so as we have
robots interact in this physical world
is that a signal that can be used in
reinforcement learning you've you've
worked a little bit in this direction
but do you think that's that psychology
can be somehow pulled in now so that's a
question
I would say a lot a lot of people ask
and I think part of why they ask it is
they're thinking about how unique are we
really still ask people like after they
see some results they see a computer
play go to say computer do this that
they're like ok but can it really have
emotion can it really interact with us
in that way and then once you're around
robots you already start feeling it and
I think that kind of maybe
mythologically the way that I think of
it is if you run something like
reinforce some Linux about optimizing
some objective and there's no reason
that D object couldn't be tied into how
much there's a person like interacting
with this system and why could not the
reinforcement learning system optimized
for their robot being fun to be around
and
why wouldn't it then naturally become
more and more interactive and more and
more maybe like a person or like a pet I
don't know what it would exactly be but
more more have those features and
acquire them automatically as long as
you can formalize an objective of what
it means to like something what how you
exhibit what's the ground truth how do
you how do you get the reward from human
cause you have to somehow collect that
information within you human
but you you're saying if you can
formulate as an objective it can be
learned there is no reason it couldn't
emergent through learning and maybe one
way to formulate has an objective you
wouldn't have to necessarily score it
explicitly so standard rewards are
numbers and numbers are hard to come by
this is a 1.5 or 0.7 on some scale it's
very hard to do for a person but much
easier is for a person to say okay what
you did the last five minutes was much
nicer than we did the previous five
minutes and that now gives a comparison
compare and in fact there have been some
results in that for example Paul
Christiana and collaborators at open e I
had the hopper madoka Hopper one legged
robot the Batman's little back flips
yeah purely from feedback I like this
better than that that's kind of equally
good and after a bunch of interactions
it figured out what it was the person
was asking for it namely a back flip and
so I think the same thing
od wasn't trying to do a back flip it
was just getting a score from the
comparison score from the person based
on hers and having a mind in their own
mind what I wanted to do a back flip but
the robot didn't know what it was
supposed to be doing it just knew that
sometimes the person said this is better
this is worse and then the robot figure
it out what the person was actually
after was a back flip and I'd imagine
the same would be true for things like
more interactive robots that the robot
would figure out over time oh this kind
of thing apparently has appreciated more
than this other kind of thing so when I
first picked up
Sutton's Richard Sutton's reinforcement
learning book before sort of this deep
learning before the re-emergence of
neural networks is a powerful mechanism
for machine learning IRL seemed to me
like magic as a as beautiful so
that seemed like what intelligence is RL
reinforcement learning so how do you
think we can possibly learn anything
about the world when the reward for the
actions is delayed is so sparse like
where is why do you think RL works why
do you think you can learn anything
under such sparse awards whether it's
regular reinforcement learning a deeper
enforcement learning what's your
intuition the kind of part of that is
why is RL why does it need so many
samples so many experiences to learn
from because really what's happening is
when you have a sparse reward you do
something maybe for like I don't know
you take a hundred actions and then you
get a reward and maybe get like a score
of three and I'm like okay three not
sure what that means you go again and
now I get to and now you know that that
sequence of hundred actions that you did
the second time around somehow was worse
than the sequence of hundred actions you
did the first time around but that's
tough to now know which one of those
were better or worse some might have
been good and bad in either one and so
that's why I need so many experience but
once you have enough experiences
effectively rlist easing that apart it's
time to say okay when what is
consistently there when you get a higher
reward and what's consistently there
when you get a lower reward and then
kind of the magic of sums is the policy
grant update is to say
now let's update the neural network to
make the actions that were kind of
present when things are good more likely
and make the actions that are present
when things are not as good less likely
so that's that is the counterpoint but
it seems like you would need to run it a
lot more than you do even though right
now people could say that RL is very
inefficient but it seems to be way more
efficient than one would imagine on
paper that the the simple updates to the
policy the policy gradient that that's
somehow you can learn is exactly users
said what are the common actions that
seem to produce some good results that
that somehow can learn anything it seems
counterintuitive at least did is there
some intuition behind yeah so I think
there's a few ways to think about this
the way I Tennant
about it mostly originally when so when
we started working on deep reinforcement
learning here at Berkeley which was
maybe two thousand eleven twelve
thirteen around that time
challenge Schulman was a PhD student
initially kind of driving it too forward
here and did it the way we thought about
it at the time was if you think about
rectified linear units or kind of break
the fire type neural networks what do
you get you get something that's
piecewise linear feedback control and if
you look at the literature linear
feedback control is extremely successful
can solve many many problems
surprisingly well
I remember for example when we did
helicopter flight if you're in a
stationary flight regime not a non
station by the stationary flight regime
like hover you can use linear feedback
control to stabilize a helicopter a very
complex dynamical system but the
controller is relatively simple and so I
think that's a big part of is that if
you do feedback control even though the
system you control can be very very
complex often relatively simple control
architectures can already do a lot but
then also just linear is not good enough
and so one way you can think of these
neural networks is that in sometimes
they tile the space which people were
already trying to do more by hand or
with finite state machines say this
linear controller here this leaner
controller here you'll network learns
that alva spins a linear controller here
another linear controller here but it's
more subtle than that yeah and so it's
benefiting from this linear control
aspect is benefiting from the tiling but
it's somehow tiling it one dimension at
a time because if let's say you have a
two layer network even the hidden layer
you make a transition from active to
inactive or the other way around that is
essentially one axis but not acts as a
line but one direction that you change
and so you have this kind of very
gradual tiling of the space we have a
lot of sharing between the linear
controllers that tile the space and that
was always my intuition s of why to
expect that this might work pretty well
it's essentially leveraging the fact
that linear feedback control is so good
but of course not enough and this is a
gradual tiling of the space with linear
feedback controls that share a lot of
expertise across them so that that's
that's really nice intuition do you
think that scales to the more and more
general problems of when you start going
up the number of controllers dimensions
when you start going down in terms of
how often you get a clean reward signal
does that intuition carry forward to
those crazy or weird or worlds that we
think of as the real world so I think
where things get really tricky in the
real world compared to the things we've
looked at so far with great success in
reinforcement learning is
the time skills which takes us to an
extreme so when you think about the real
world I mean I don't know maybe some
student decided to do a a PhD here right
okay that's that's the decision that's a
very high-level decision but if you
think about their lives I mean any
person's life it's a sequence of muscle
fiber contractions and relaxations and
that's how you interact with the world
and that's a very high frequency control
thing but it's ultimately what you do
and how you affect the world until I
guess we have brain readings and you can
maybe do it slightly differently but
typically that's how you affect the
world and the decision of doing a PhD is
like so abstract relative to what you're
actually doing in the world and I think
that's where credit assignment becomes
just completely beyond what any current
RL algorithm can do and we need
hierarchical reasoning at a level that
is just not available at all yet where
do you think we can pick up hierarchical
reasoning by which mechanisms yeah so
maybe let me highlight what I think the
limitations are of what already was done
20-30 years ago in fact you'll find
reasoning systems that reason over
relatively long horizons but the
problems that they were not grounded in
the real world so people would have to
hand design some kind of logical
dynamical descriptions of the world and
that didn't tie into perception and so
then time to real objects and so forth
and so that that was a big gap now with
deep learning we start having the
ability to really see with sensors
process that and understand what's in
the world and so it's a good time to try
to bring these things together one I see
a few ways of getting there one way to
get there would be to say deep learning
can get bolted on somehow to some of
these more traditional approaches now
bolted on would probably mean you need
to do some kind of end-to-end training
where you say my deep learning
processing somehow leads to a
representation that in Perm uses some
kind of traditional underlying dynamical
systems that can be used for planning
and that's for example the direction
Aviv Tamar and the North Korea touch
here have been pushing with causal info
gone and of course other people to that
that's that's one way can we somehow
force it into the form factor that is
amenable to reasoning
another direction we've been thinking
about for a long time and they didn't
make any progress on was more
information theoretic approaches so the
idea there was that what it means to
take high-level action is to take and
choose a latent variable now that tells
you a lot about what's gonna be the case
in the future because that's what it
means to to take a high-level action I
say what I decide I'm gonna navigate to
the gas station because need to get gas
for my car well that'll now take five
minutes to get there but the fact that I
get there I could already tell that from
the high-level action it took much
earlier that we had a very hard time
getting success with not saying it's a
dead-end necessarily but we had a lot of
trouble getting that to work and then we
start revisiting the notion of what are
we really trying to achieve what we're
trying to achieve is non ously hierarchy
per se but you could think about what
does hierarchy give us what it's we hope
it would give us is better credit
assignment kind of what is better credit
ominous is given is giving us it gives
us faster learning right and so faster
learning is ultimately maybe what we're
after and so that's what we ended up
with the RL squared paper on learning -
reinforcement learn which at a time
rocky duan LED and that's exactly the
meta learning approach or is say okay we
don't know how to design hierarchy we
know what we want to get from it let's
just enter an optimize for what want to
get from it and see if it might emerging
we saw things emerge the maze navigation
had consistent motion down hallways
which is what you want a hierarchical
control should say I want to go down
this hallway and then when there is an
option to take a turn I can this art
will take a turn or not and repeat even
had the notion of where have you been
before or not do not revisit places
you've been before it still didn't scale
yet to the real world kind of scenarios
I think you had in mind but it was some
sign of life that maybe you can meta
learn these hierarchal concepts I mean
it seems like through these meta
learning concepts get at the what I
think is one of the
hardest and most important problems of
AI which is transfer learning so it's
generalization how far along this
journey towards building general systems
are we being able to do transfer
learning well so there's some signs that
you can generalize a little bit but do
you think we're on the right path or
it's totally different breakthroughs are
needed to be able to transfer knowledge
between different learned models yeah
I'm I'm pretty tired on this and then I
think there are some very many there
there's just some very impressive
results already right I mean yes I would
say when even with the initial and a big
breakthrough in 2012 with Aleks net
right the initial the initial thing is
okay great this does better on imagenet
hands image recognition but then
immediately thereafter that was of
course the notion that Wow
what was learned on image net and you
now want to solve a new task you can
fine-tune Aleks net for new tasks and
that was often found to be the even
bigger deal that you learned something
that was reusable which was not often
the case before usually machine learning
you learned something for one scenario
and that was it and that's really
exciting I mean that's just a huge
application that's probably the biggest
success of transfer learning today in
terms of scope and impact that was huge
breakthrough and then recently I feel
like similar kind of but by scaling
things up it seems like this has been
expanded upon like people training even
bigger networks they might transfer even
better if you looked at for example some
of the opening eye results on language
models and some of the recent Google
results on language models they are
learned for just prediction and then
they get reused for other tasks and so I
think there is something there where
somehow if you train a big enough model
on enough things it seems to transfer
some deepmind results I thought were
very impressive unreal results where it
was learned to navigate mazes in ways
where it wasn't just
reinforcement learning going to have
other objectives was optimizing for so I
think there's a lot of interesting
results already I think maybe words hard
to wrap my head around this to which
extend or when do we call something
generalization right or the levels of
generalization involved in these
different tasks alright so you draw this
by the way just to frame things you've
heard you say somewhere it's the
difference between learning to master
versus learning to generalize that it's
a nice line to think about and it guess
you're saying that's a gray area of what
learning to master and learning to
generalize where once think I might have
heard this I might have heard it
somewhere else and I think it might have
been one of one of your interviews and
maybe the one with yo show Benjamin on
hundred percent sure but I like the
example I'm gonna act not sure who it
was but the example was essentially if
you use current deep learning techniques
what we're doing to predict let's say
the relative motion of our planets it
would do pretty well but then now if a
massive new mass enters our solar system
it would prompt predict what will happen
right and that's a different kind of
journal is a Shahnaz a generalization
that relies on the ultimate simplest
simplest explanation that we have
available today to explain the motion of
planets where I was just pattern
recognition could predict our current
solar system motion pretty well no
problem and so I think that's an example
of a kind of generalization that is a
little different from what we've
achieved so far and it's not clear if
just you know regularizing more I'm
forcing it to come up with a simpler
simpler simple experience but it's not
simple but that's what physics
researchers do right to say can I make
this even simpler how simple can I get
this what's a simplest equation I can
explain everything right yeah the master
equation for the entire dynamics of the
universe we haven't really pushed that
direction as hard in in deep learning I
would say not sure if it should be
pushed but it seems a kind of
generalization you get from that that
you don't get in our current methods
so far so I just talked to vladimir
vapnik for example who was a
statistician the statistical learning
and he kind of dreams of creating these
are the a equals e equals mc-squared for
learning right the general theory of
learning do you think that's a fruitless
pursuit in the near term in within the
next several decades I think that's a
really interesting pursuit and in the
following sense and that there is a lot
of evidence that the brain is pretty
modular and so I wouldn't maybe think of
it as the theory maybe the the
underlying theory but more kind of the
principle where there have been findings
where people who are blind will use the
part of the brain usually used for
vision for other functions and even
after some kind of if people will get
rewired in some way they might I'm able
to reuse parts of their brain for other
functions and so what that suggests is
some kind of modularity and I think it
is a pretty natural thing to strive
forward to see can we find that
modularity can we find this thing of
course it's not every part of the brain
is not exactly the same not everything
can be rewired arbitrarily but if you
think of things like the neocortex which
is pretty big part of the brain that
seems fairly modular from what the
findings so far can you design something
equally modular and if you can just grow
it it becomes more capable probably I
think that would be the kind of
interesting underlying principle to
shoot for that is not unrealistic do you
think you prefer math or empirical trial
and error for the discovery of the
essence of what it means to do something
intelligent
so reinforcement learning embodies both
groups right then prove that something
converges prove the bounds and then at
the same time a lot of those successes
are well let's try this and see if it
works so which do you gravitate towards
how do you think of those two parts of
your brain so
maybe I would prefer we could make the
progress with mathematics and the reason
maybe I would prefer that is because
because often if you have something you
can mathematically formalise you can
leapfrog a lot of experimentation and
experimentation takes a long time to get
through and a lot of trial and error
kind of reinforcement learning your
research process but you need to do a
lot of trial and error before you get to
a success so if we can leapfrog doubt in
my mind that's what the math is about
and hopefully once you do a bunch of
experiments you start seeing a pattern
you can do some derivations that
leapfrog some experiments but I agree
with you I mean in practice a lot of the
progress has been such that we have not
been able to find the math that allows
it to leapfrog ahead and we are kind of
making gradual progress one step at a
time a new experiment here a new
experiment there that gives us new
insights and gradually building up but
not getting to something yet where we're
just okay here's an equation that now
explains how you know that would be have
been two years of experimentation to get
there but this tells us what the results
going to be unfortunately not so much
yes not so much yeah but your hope is
there in trying to teach robots or
systems to do everyday tasks or even in
simulation what what do you think you're
more excited about imitation learning or
self play
so letting robots learn from humans or
letting robots plan their own to try to
figure out in their own way and
eventually play eventually interact with
humans or to solve whatever problem is
what's the more exciting to you what's
more promising you think as a research
direction so when we look at self play
what's so beautiful about it is goes
back to kind of the challenges in
reinforcement learning so the challenge
of reinforced learning is getting signal
and if you don't never succeed you don't
get any signal in self play you're on
both sides so one of you succeeds and
the beauty is also one of you fails and
so you see the contrast you see the one
version of me that it better
the other version and so every time you
play yourself you get signal and so
whenever you can turn something into
self play you're in a beautiful
situation where you can naturally learn
much more quickly than in most other
reinforced learning environments so I
think I think if somehow we can turn
more reinforcement learning problems
into self play formulations that would
go real really far so far south play has
been largely around games where there is
natural opponents but if we could do
self play if for other things and let's
say I don't know a robot learns to build
a house I mean that's a pretty advanced
thing to try to do for a robot but maybe
it tries to build a hut or something if
that can be done through self play it
would learn a lot more quickly if
somebody can figure that out and I think
that would be something where it goes
closer to kind of the mathematical leap
frogging where somebody figures out a
formalism to it's okay
any RL problem by playing this and this
idea you can turn it into a self play
problem where you get signal a lot more
easily
reality is many problems we don't know
how to turn the self lay and so either
we need to provide detailed reward that
doesn't just reward for achieving a goal
but rewards for making progress and that
becomes time-consuming and once you're
starting to do that let's say you want a
robot to do something you need to give
all this detailed reward well why not
just give a demonstration right because
why not just show the robot and now the
question is how do you show the robot
one way to show is to tally operate the
robot and then the robot really
experiences things and that's nice
because that's really high
signal-to-noise ratio data and we've
done a lot of that and you teach your
robot skills in just 10 minutes you can
teach your robot a new basic skill like
okay pick up the bottle place it
somewhere else that's a skill no matter
where the bottle starts maybe it always
goes on to a target or something
that's fairly is a teacher about with
tally up now what's even more
interesting if you can now teach robot
through third person learning where the
robot watches you do something and
doesn't experience it but just watches
it and says okay well if you're showing
me that that means I should be doing
this and I'm not gonna be using your
hand because I don't get to control your
hand but I'm gonna use my hand I'd do
that mapping and so that's where I think
one of the big breakthroughs has
happened this year this was led by
Chelsea Finn here it's almost like
machine translation for demonstrations
were you have a human demonstration and
the robot learns to translated into what
it means for the robot to do it and that
was a meta learning for a Malaysian
learn from one to get the other and that
I think opens up a lot of opportunities
to learn a lot more quickly so my focus
is on autonomous vehicles do you think
this approach of third-person watching
is about the autonomous driving is
amenable to this a kind of approach so
for autonomous driving I would say it's
third-person is slightly easier and the
reason I'm gonna say slightly easier to
do a third-person is because the hard
dynamics are very well understood so the
easier than of first-person you mean or
easier so I think the distinction
between third-person and first-person is
not a very important distinction for
autonomous driving they're very similar
because the distinction is really about
who turns the steering wheel and or
maybe I'll let me put it differently how
to get from a point where you are now to
a point let's say a couple meters in
front of you and that's a problem that's
very well understood and that's the only
distinction being third and first-person
there whereas with the robot
manipulation interaction forces are very
complex and it's still a very different
thing for autonomous driving I think
there is still the question imitation
versus RL so imitation gives you a lot
more signal I think where imitation is
lacking and needs some extra machinery
is it doesn't in its normal format
doesn't think about goals or objectives
and of course there are versions of
imitation learning inverse reinforce
learning type imitation which also
thinks about goals I think then we're
getting much closer but I think it's
very hard to think of a fully reactive
car generalizing well if it really
doesn't have a notion of objectives to
generalize well to the kind of general
that you would want you'd want more than
just that reactivity that you get from
just behavioral cloning / supervised
learning so a lot of the work whether
its self play
imitation learning would benefit
significantly from simulation from
effective simulation and you're doing a
lot of stuff in the physical world and
in simulation do you have hope for
greater and greater power of simulation
loop being boundless eventually to where
most of what we need to operate in the
physical world would could be simulated
to a degree that's directly transferable
to the physical world are we still very
far away from that so I think we could
even rephrase that question in some
sense please so the power of simulation
right simulators get better and better
of course become stronger and we can
learn more in simulation but there's
also another version which is where you
said the simulator doesn't even have to
be that precise as long as is somewhat
representative and instead of trying to
get one simulator that is sufficiently
precise to learn in and transfer really
well to the real world
I'm gonna build many simulators ensemble
of simulators ensemble of simulators not
any single one of them is sufficiently
representative of the real world such
that it would work if you train in there
but if you train in all of them then
there is something that's good in all of
them the real world will just be you
know another one that's you know cannot
identical to any one of them but just
another one of them another sample from
the distribution of simulators exact we
do live in a simulation so this is just
like oh one other one I'm not sure about
that video it's definitely a very
advanced simulator if it is yeah it's
pretty good one I've talked to this to
Russell is something you think about a
little bit too of course you're like
really trying to build these systems but
do you think about the future of AI a
lot of people have concerned about
safety how do you think about AI safety
as you build robots that are operating
in the physical world what what is uh
yeah how do you approach this problem in
an engineering kind of way in a
systematic way so what a robot is doing
things you
kind of have a few notions of safety to
worry about one is that Throwbot is
physically strong and of course could do
a lot of damage same for cars which we
can think of as robots do in some way
and this could be completely
unintentional so it could be not the
kind of long-term AI safety concerns
that okay a is smarter than us and now
what do we do but it could be just very
practical okay this robot if it makes a
mistake
whether the results going to be of
course simulation comes in a lot there
too to test in simulation it's a
difficult question and I'm always
wondering like I was wondering at let's
go back to drivings a lot of people know
driving well of course what do we do to
test somebody for driving right to get a
driver's license what do they really do
I mean you fill out some test and then
you drive and I mean perfume in suburban
California the driving test is just you
drive around the block pull over you do
a stop sign successfully and then you
know you pull over again and you pretty
much done and you're like okay if a
self-driving car did dad would you trust
it that it can drive and be like no
that's not enough for me to trust but
somehow for humans we've figured out
that somebody being able to do that it's
representative of them being able to do
a lot of other things and so I think
somehow for you must we figured out
representative tests of what it means if
you can do this what you can really do
of course testing you must you must all
want to be tested at all times
self-driving cars the robots can be
tested more often probably you can have
replicas that get testament are known to
be identical because they use the same
neural net and so forth but still I feel
like we don't have this kind of unit
tests or proper tests for for robots and
I think there's something very
interesting to be thought about there
especially as you update things your
software improves you have a better self
driving car suite you updated how do you
know it's indeed more capable on
everything than what you had before that
you didn't have any bad things creep
into it so I think that's a very
interesting direction of research that
there is no real solution yet
except that's somehow for you must we do
because we say okay you have a driving
test you passed you can go on the road
now and you must have accents every like
a million or ten million miles something
something pretty phenomenal compared to
that short test yeah that is being done
so let me ask you've mentioned
you mentioned that Andrew Aang by
example showed you the value of kindness
and to do you think the space of
policies good policies for humans and
for AI is populated by policies that
with kindness or ones that are the
opposite exploitation even evil so if
you just look at the sea of policies we
operate under as human beings or if AI
system had to operate in this real world
do you think it's really easy to find
policies that are full of kindness like
we naturally fall into them or is it
like a very hard optimization problem I
mean there is kind of two optimizations
happening for humans right so for you
most was kinda the very long-term
optimization which evolution has done
for us and we're kind of predisposed to
like certain things and that's in
sometimes what makes our learning easier
because I mean we know things like pain
and hunger and thirst and the fact that
we know about those is not something
that we were taught that's kind of
innate when we're hungry were unhappy
when we're thirsty were unhappy when we
have pain we're unhappy and ultimately
evolution built that into us to think
about this thing so so I think there is
a notion that it seems somehow humans
evolved in general to prefer to get
along in some ways but at the same time
also to be very territorial and kind of
centric to their own tribe is it like it
seems like that's the kind of space we
converge down to it I mean I'm not an
expert in anthropology but it seems like
we're very kind of good within our own
tribe but need to be taught but to be
nice to other tribes well if you look at
Steven Pinker he highlights is pretty
nicely in
better better angels of our nature where
he talks about violence decreasing over
time consistently so whatever attention
whatever teams we pick it seems that the
long arc of history goes towards us
getting along more and more so I hope so
so do you think that do you think it's
possible to cheat teach RL bass robots
the this kind of kindness this kind of
ability to interact with humans this
kind of policy even - let me ask let me
ask a fun one do you think it's possible
to teach RL based robot to love a human
being and to inspire that human to love
the robot back so - like RL based
algorithm that leads to a happy marriage
that's interesting question maybe I'll
oh I'll answer it with with another
question right I mean it's it but I'll
come back to it so another question you
can have is okay
I mean how close does some people's
happiness get from interacting with just
a really nice dog like I mean dogs you
come home that's what dogs did they
greet you they're excited it makes you
happy when you're coming home to your
dog just like okay this is exciting
they're always happy when I'm here and
if they don't greet you because maybe
whatever your partner took them on a
trip or something you might not be
nearly as happy when you get home right
and so the kind of it seems like the
level of reasoning a dog houses is
pretty sophisticated but then it's still
not yet at the level of human reasoning
and so it seems like we don't even need
to achieve human love reason to get like
very strong affection with humans and so
my thinking is why not right why
couldn't with an AI couldn't we achieve
the kind of level of affection that
humans feel among each other or with
friendly animals and so forth it's a
question is it a good thing for us or
not that misses another going right
because I mean
but I don't see why not why not yeah so
he almost says love was the answer maybe
he should say love is the objective
function and then RL is the answer maybe
I'll Peter thank you so much I don't
want to take up more of your time thank
you so much for talking today well
thanks for coming by great to have you
visit
you