Kind: captions
Language: en
the following is the conversation with
max tag mark his second time on the
podcast
in fact the previous conversation was
episode number one
of this very podcast he is a physicist
and artificial intelligence researcher
at mit co-founder of the future of life
institute
and author of life 3.0 being human
in the age of artificial intelligence
he's also the head of a bunch of
other huge fascinating projects and has
written
a lot of different things that you
should definitely check out he has been
one of the key humans
who has been outspoken about long-term
existential risks of ai
and also its exciting possibilities and
solutions to real-world problems
most recently at the intersection of ai
and physics
and also in re-engineering the
algorithms
that divide us by controlling the
information we see
and thereby creating bubbles and all
other kinds of
complex social phenomena that we see
today in general he's one of the most
passionate and brilliant people
i have the fortune of knowing i hope to
talk to him many more times
on this podcast in the future quick
mention of our sponsors
the jordan harbinger show for sigmatic
mushroom coffee
better help online therapy and
expressvpn
so the choice is wisdom caffeine
sanity or privacy choose wisely my
friends
and if you wish click the sponsor links
below to get a discount
to support this podcast as a side note
let me say that
much of the researchers in the machine
learning and
artificial intelligence communities do
not spend much time
thinking deeply about existential risks
of ai
because our current algorithms are seen
as useful but dumb
it's difficult to imagine how they may
become destructive
to the fabric of human civilization in
the foreseeable future
i understand this mindset but it's very
troublesome
to me this is both a dangerous and
uninspiring perspective
reminiscent of the lobster sitting in a
pot of lukewarm water
that a minute ago was cold i feel a
kinship with this lobster
i believe that already the algorithms
that drive our interaction on social
media
have an intelligence and power that far
outstrip the intelligence and power of
any one human being
now really is the time to think about
this to define the trajectory
of the interplay of technology and human
beings in our society
i think that the future of human
civilization very well may be at stake
over this very question
of the role of artificial intelligence
in our society
if you enjoy this thing subscribe on
youtube review it on apple podcast
follow on spotify
support on patreon or connect with me on
twitter
lex friedman and now here's my
conversation
with max tag mark so people might not
know this but you were actually episode
number one
of this podcast just a couple of years
ago and
now we're back and it so happens that a
lot of exciting things happened in
both physics and artificial intelligence
both fields that you're super passionate
about
can we try to catch up to some of the
exciting things happening in
artificial intelligence especially in
the context of
the way it's cracking open the different
problems of the sciences
yeah i'd love to especially now as we
start 2021 here
it's a really fun time to think about
what were the biggest
breakthroughs in ai not the ones
necessarily that media wrote about but
that
really matter and and what does that
mean for our ability to do better
science
what does it mean for our ability to
help people around the world and what
does it mean for new um
problems that they could cause if we're
not smart enough to avoid them so you
know
what do we learn basically from this yes
absolutely so
one of the amazing things you're part of
is the ai institute for artificial
intelligence and
fundamental interactions what's up with
this institute
what are you working on what are you
thinking about the idea
is something i'm very on fire with which
is basically ai meets physics
and you know it's been almost five years
now since i shifted my own
mit research from physics to machine
learning and in the beginning i noticed
a lot of my colleagues even though they
were polite about it well i kind of
what is max doing what is this weird
stuff he's lost his mind
then but then gradually i uh
together with some colleagues we're able
to persuade more and more
of the other professors in the our
physics department to get interested in
this
and and um now we got this amazing nsf
center so
20 million bucks for for the next five
years mit
and a bunch of neighboring universities
here also
and i noticed now those callings looking
at me funny have stopped asking
what the point is of this because it's
becoming more clear
and i really believe that of course ai
can help physics a lot
to do better physics but physics can
also help
ai a lot both by
building better hardware my colleague
marin soljacic for example
is working on an optical chip
for much faster machine learning where
the computation is done
not by moving electrons around and but
we're moving
photons around dramatically less energy
use
faster better um we can
we can also help ai a lot i think by
having a
different set of tools and a different
maybe more audacious attitude
you know ai has to significant extent
been
an engineering discipline where you're
just trying to make things that work
and being less more interested in maybe
selling them then figuring out exactly
how they work and proving theorems about
that they will always work
right contrast that with physics you
know when elon musk sends a rocket
to the international space station they
didn't just train with machine learning
oh let's fire it a little bit left more
to the left a bit more to the right
though that also missed let's try here
no you know we figured out newton's laws
of gravitation
and other and got other things and got a
really deep fundamental understanding
and that's what gives us such confidence
in
in rockets and my vision is that
in the future all machine learning
systems that actually have impact on on
people's lives will be understood at a
really really deep level
right so we trust them not because some
sales rep told us to
but because they've earned our trust we
can
and really safety critical things even
prove that they will always do
you know what we expect them to do
that's very much the physics mindset so
it's interesting if you look at big
breakthroughs that have happened in
machine learning
this year you know from dancing robots
you know it's pretty fantastic not just
because it's cool but
if you just think about not that many
years ago
this youtube video at this darpa
challenge where the mit robot
comes out as the car and face plants
yeah
how far we've come and just a few years
similarly alpha fold to
you know crushing the protein folding
problem
we can talk more about implications for
medical research and stuff but
hey you know that's huge progress
i you can look at
gpt-3 they can spout off
english text which sometimes really
really blows you away
you can look at the google deepmind's
mu0 which doesn't just kick your butt
and
go and chest and shogi but also in all
these atari games and you don't even
have to teach it the rules now
you know what all of those have in
common is besides being powerful
is we don't fully understand how they
work
and that's fine if it's just some
dancing robots and the worst thing that
can happen is the face plant right
or if they're playing go and the worst
thing that can happen is that they make
a bad move and lose the game right
it's less fine if that's what's
controlling your self-driving car
or your nuclear power plant and uh
we've seen already that even though
hollywood
had all these movies where they try to
make us worry about the wrong things
like machines turning evil the actual
bad things that have happened with
automation have not been machines
turning evil they've been caused by over
trust
in things we didn't understand as well
as we thought we did right even
very simple automated systems like
what the boeing put into the 737 max
right yes killed a lot of people was it
that that little simple system was evil
of course not
but we didn't understand it as well as
we should have right
and we trusted without understanding
exactly we over trust we didn't even
understand that we didn't understand
right
the humility is really at the core of
being a scientist
i think step one if you want to be a
scientist is don't ever fool yourself
into thinking you understand things when
you actually don't
yes right that's probably good advice
for humans in general
i think humility in general and it was
good but in science it's so spectacular
like why
did we have the wrong theory of gravity
ever from aristotle
onward and close until like galileo's
time like why would we believe something
so dumb
as that if i throw this water bottle
it's going to go up with constant speed
until it realizes that its natural
motion is down it changes its mind you
know
because we people just kind of assumed
aristotle was right he's an authority we
understand that
why did we believe things like that the
sun is going around the earth
why did we believe that time flows at
the same rate for everyone until
einstein
same exact mistake over and over again
we just
weren't humble enough to acknowledge
that we actually didn't know for sure
we assumed we knew so we didn't discover
the truth because we assumed
there was nothing there to be discovered
right there was something to be
discovered about the
737 max and if you had been a bit more
suspicious and tested it better we would
have found it
and it's the same thing with most harm
that's been done
by automation so far i would say so i
don't know if you did you
hear of a company called knight capital
no
so good that means you didn't invest in
them earlier
they deployed this automated trading
system yes
all nice and shiny they didn't
understand it as well as they thought
and it went about losing 10 million
bucks per minute
yeah for 44 minutes straight no until
someone presumably was like oh
shut this up you know was it evil no
it was again misplaced trust something
they didn't fully understand right and
um
there have been so many um even when
people have been killed by
robots it's quite rare still but in act
factory accidents it's in every single
case been not
malice just that the robot didn't
understand that a human is
different from an auto part or whatever
and and
we um so this is where i think there's
so so much
opportunity for physics approach where
you just
aim for a higher level of understanding
and if you look at the
all these systems that we talked about
from the from
reinforcement learning systems and
dancing robots
to all these neural networks that power
gpt 3
and and go playing software stuff
they're all basically black boxes
much like not so different from if you
teach a human something you have no idea
how their brain works right
except the human brain at least has been
error corrected
during many many centuries of evolution
in a way that these
some of these systems have not right and
my
my mit research is entirely focused on
demystifying this black box
intelligible intelligence is my slogan
that's a good line intelligible
intelligence yeah it's not that we
shouldn't settle for something that
seems intelligent but we should it
should be intelligible so that we
actually trust it because we understand
it right
like again elon trusts his rockets
because he understands newton's laws and
thrust and how everything works uh and
let me tell you what can i tell you why
i'm optimistic about this yes
i think i think there's we've made a bit
of a mistake
yeah where we some people still think
that somehow we're never going to
understand neural networks
and we're just going to have to learn to
live with this it's this very powerful
black box
basically for those you know haven't
spent time
building their own it's super simple
what happens inside you send in a long
list of numbers
and then you do a bunch of operations on
them
multiply by matrices et cetera et cetera
and some other numbers come out that's
the output of it
and then there are a bunch of knobs you
can tune
and when you change them you know it
affects the computation
the input output relation and then you
just give the computer some definition
of good
and it keeps optimizing these knobs
until it performs as
good as possible and often you go like
wow that's really good
this robot can dance or this machine is
beating me a chest now
and in the end you have something which
even though you can look inside it you
have
very little idea of how it works you
know you can print out
tables of all the millions of parameters
in there
is it crystal clear now how it's working
you know of course not right
so many of my colleagues seem willing to
settle for that and i'm like
no that's like the halfway point
uh some have even gone as far as sort of
guessing that the mr the instructability
of this is
where the some of the power comes from
and sort of some sort of mysticism
i think that's total nonsense i i think
the real power of
neural networks comes not from
inscrutability but from
differentiability and what i mean by
that
is simply that
the output depends changes only smoothly
if you
tweak your knobs and then you can use
all these powerful methods we have for
optimization in science we can just
tweak them a little bit and see did that
get better or worse
that's the fundamental idea of machine
learning that the machine itself
can keep optimizing until it gets better
suppose you wrote this
an algorithm instead in python or some
other programming language
and then what what the knobs did was
they just changed random letters in your
in your code that would just epically
fail right you change one thing and
instead of saying print it says
synt syntax error you don't even know
was that
for the better or for the worse right
this to me is
re this is what i believe is the
fundamental power of
neural networks and just to clarify the
changing of the different letters in a
program
would not be a differentiable process it
would make it an invalid program
typically and then you wouldn't even
know if you changed more letters if it
would make it work again right
so that's the magic of uh neural
networks
uh the inscriptibility the
differentiability that every
every setting of the parameters is a
program and you can tell is it better or
worse right and
so so you don't like the poetry of the
mystery of neural networks as the source
of its power
i generally like poetry but not in this
case
it's so misleading and it's above all it
it shorts changes us
it fails it makes us underestimate what
we can the good things we can accomplish
because
so what we've been doing in my group is
basically step one
train the mysterious neural network to
do something well
and then step two do some additional
ai techniques to see if we can now
transform this
black box into something equally
intelligent
that you can actually understand so for
example i'll give you one example this
ai feynman
project that we just published right so
we took the 100 most famous
or complicated equations from one of my
favorite
physics textbooks in fact the one that
got me into physics in the first place
the feynman lectures on physics and uh
so you have a formula you know maybe it
has
what goes into the formula is six
different
variables and then what comes out as one
so then you can make like a giant excel
spreadsheet with seven columns
you put in just random numbers for the
six columns for those six input
variables and then
you calculate with the formula the
seventh column the output so maybe it's
like the force
equals in the last column some function
of the other
and now the task is okay if i don't tell
you what the formula was
can you figure that out from looking at
my spreadsheet i gave you yes
this problem is called symbolic
regression
if i tell you that the formula is what
we call a linear formula
so it's just that the output is
some sum of all the things input the
times some constants
that's the famous easy problem we can
solve we do it
all the time in science and engineering
but the general one if it's more
complicated functions with logarithms or
cosines or other math
it's a very very hard one and probably
impossible to do
fast in general just because the number
of formulas with
n symbols you know just grows
exponentially just like the number of
passwords you can make
grow dramatically with length so
so we but we had this idea that if you
first have a neural network that can
actually approximate the formula you
just trained it
even if you don't understand how it
works that can be
a first step towards actually
understanding how it works
so that's what we do first
and then we study that neural network
now and
put in all sorts of other data that
wasn't in the original training data and
use that to discover simplifying
properties of the formula
and that lets us break it apart often
into many simpler pieces
and a kind of divide and conquer
approach that we so we were able to
solve all of those 100 formulas discover
them automatically
plus a whole bunch of other ones and
it's a
it's actually kind of humbling to see
that this code which
anyone who wants now is listening to
this can type pip
install ai fineman on the computer and
run it you know it can actually do
what johannes kepler spent four years
doing when he stared at mars data
until he's like funny eureka this is an
ellipse yeah
this will do it automatically for you in
one hour right or max planck
he was looking at at how much radiation
comes out from
at different wavelengths from a hot
object and discovered the famous
blackbody formula this discovers it
automatically
i'm actually excited about
seeing if we can discover not just old
formulas again
but new formulas that no one has seen
before
and do you like this process of using
kind of a neural network to
find some basic insights and then
dissecting the neural network to then
gain
the final so that's in that way
you've uh forcing the explainability
issue
uh you know really trying to analyze the
neural network
for the things it knows in order to come
up with the final
beautiful simple theory underlying the
whole
the initial the initial system that you
were looking at i love that
and and the reason i'm so optimistic
that it can be generalized
so much more is because that's exactly
what we
do as a human scientist think of galileo
whom we mentioned right
i bet when he was a little kid if his
dad threw him an apple
he would catch it why because
he had a neural network in his brain
that he had trained to predict
the parabolic orbit of apples that are
thrown under gravity
if you throw a tennis ball to a dog it
also has this same ability
of deep learning to figure out how the
ball is going to move and catch it
but galileo went one step further when
he got older
he went back and was like wait a minute
i can write down a formula yes y equals
x squared a parabola
you know and he helped revolutionize
physics as we know it right so there was
a basic neural network
in there from childhood that captured
like the base
the experiences of observing different
kinds of trajectories
and then he was able to go back in with
another extra little
neural network and analyze all those
experiences
and be like wait a minute there's a
deeper rule here
exactly he was able to distill out in
symbolic form
what that complicated black box nor like
was doing right not only did he the
formula he get
he got ultimately become more accurate
you know and similarly this is how he
how newton got newton's laws which is
why elon can send rockets to
the space station now right so it's not
only more accurate
but it's also simpler much simpler and
it's so simple that we can actually
describe it to our friends and each
other right
we've talked about it just in the
context of physics now but
hey you know isn't this what we're doing
when we're talking to each other also
we go around with our neural networks
just like dogs and cats and
chipmunks and bluejays and we experience
things in the world
but then we humans do this additional
step on top of that where we then
distill out certain high-level
knowledge that we've extracted from this
in a way that we can communicate it
to each other in a symbolic form in
english
in this case right so if we can do it
and we believe that we are information
processing entities
then we should be able to make machine
learning that does it also
well do you think the entire thing could
be learning
because they're uh this dissection
process like for ai feynman
the secondary stage feels like something
like
reasoning and the initial step feels
like more like the
the more basic kind of differentiable
learning
do you think the whole thing could be
differentiable learning
do you think the whole thing could be
basically neural networks on top of each
other it's like turtles all the way down
could be neural networks all the way
down i mean that's a really interesting
question we know that in
in your case it is neural networks all
the way down because
you have in your skull as a bunch of
neurons doing their thing right
yeah but uh if you ask the question
more generally what what algorithms are
your brain is your brain
are being used in your brain right i
think super interesting to compare
i think we've gone a little bit
backwards historically because
we humans first discovered good
old-fashioned ai the logic based ai that
we often called
gophi for good old-fashioned ai
and then more recently we did machine
learning
because it required bigger computers so
we had to discover it later
so we think of machine learning with
neural networks
as the modern thing and the logic based
ai as the old-fashioned thing
but if you look at evolution on earth
right it's actually been the other way
around
i would i would say that for example
an eagle has a better vision system
than i have using and
dogs are just as good at casting tennis
balls as i am
all this stuff which is done by training
a neural network
and not interpreting it in words you
know
it's something so many of our animal
friends can do at least as well as us
right what is it that we humans can do
that the chipmunks
and the eagles cannot it's more to do
with this logic based stuff right where
we can
extract out information in symbols
in language and now even with equations
if you're a scientist
right so basically what happened was
first we built these computers that
could
multiply numbers real fast and
manipulate symbols and we felt they were
pretty dumb
and that then we made neural networks
that can see as well as a cat can
and do a lot of this inscrutable black
box
neural networks what we humans can do
also is
put the two together in a useful way yes
artificially in our own brain
yes in our own brain so if we ever want
to get
artificial general intelligence that can
do all
jobs as well as humans can right then
that's what's going to be required
to be able to combine the neural
networks with with um
symbolic combine the old ai with a new
ai in a good way
we do it in our brains and there seems
to be basically two strategies i see in
industry now
one scares the heebie jeebies out of me
and the other one i find much more
encouraging
okay which one can we break them apart
which which are the two
the one that scares the heebie-jeebies
out of me is this attitude that we're
just gonna make ever bigger systems that
we still don't understand
until they can do be as smart as humans
i
what could possibly go wrong right yeah
i think it's just such a reckless thing
to do and unfortunately
and if we actually succeed as a species
to build artificial general intelligence
then we still have no clue how it works
i think at least 50 chance
we're going to be extinct before too
long it's just going to be an utter
epic uh own goal you know it's that 44
minutes
losing money problem or like the paper
clip problem like where
we don't understand how it works and
it's just in a matter of seconds runs
away in some kind of direction that's
going to be
very problematic even long before you
have to worry about the machines
themselves
somehow deciding to do things and to us
that we have to worry about people
using machines that are short of ai agin
power
to do bad things i mean just take your
moment
and if if anyone is not worried
particularly about
advanced ai just take 10 seconds
and just think about your least favorite
leader on the planet right now don't
tell me who it is
i want to keep this a political but just
see the face in front of you that person
for 10 seconds
yes now imagine that that person has
this incredibly powerful ai
under their control and can use it to
impose their will on the whole planet
how does that make you feel
yeah so the can can we break that apart
just
briefly for the 50
chance that we'll run into trouble with
this approach do you see
the bigger worry in that leader or
humans
using the system to do damage or
are you more worried and i think i'm
in this camp more worried about like
accidental
unintentional destruction of everything
so like humans trying to do good and
like
in a way where everyone agrees it's kind
of good it's just they're trying to do
good without understanding
because i think every evil leader in
history thought they're
to some degree thought they were trying
to do good oh yeah i'm sure hitler
thought he was doing
good yeah they're good too i've been
reading a lot about stalin i'm sure
stalin
is from he legitimately thought that
communism was good for the world and
that he was doing good
i think mao zedong thought what he was
doing with a great leap forward is good
too
yeah i'm actually concerned about both
of those
uh before i promised to answer this in
detail but before we do that
let me finish answering the first
question because i told you that there
were two
different boosts we could get to
artificial general intelligence and one
scares the species out of me
which is this one where we build
something we just say bigger neural
networks ever more hardware and just
train the heck out and more data and
poof now it's very powerful that i think
is the the most unsafe and reckless
approach
the alternative to that is the
intelligent
intelligible intelligence approach
instead where
we uh say neural networks is just a tool
to this like for the first step to get
the intuition
but then we're going to spend also
serious resources
sources on other ai techniques for
demystifying this black box and figuring
out what's it actually doing
so we can convert it into something
that's equally intelligent but that we
actually understand
what it's doing maybe we can even prove
theorems about it that
this car here will never be hacked
when it's driving because here's the
proof uh
there is a whole science of this it
doesn't work for neural networks
there are big black boxes but it works
well and we're certain other kinds of
codes right
that approach i think is much more
promising that's exactly why i'm working
on it frankly not just because i think
it's cool for science
but because i think the more we
understand
i mean these systems the better the
chance is that
we can make them do the things that are
good for us that are actually intended
not unintended so do you think it's
possible to prove
things about something as complicated as
a neural network
that's the hope well ideally there's no
reason there has to be a neural network
in the end
either right like we discovered newton's
laws of gravity
with neural network in newton's head
yes but that's not the way it's
programmed into the
navigation system of elon musk's rocket
anymore right it's written
in c plus plus or i don't know what
language he uses exactly
yeah and then there are software tools
called symbolic verification
the darpa and the us military has done a
lot of really great research on this
because they really want to understand
that when they build weapon systems they
don't just
go fire at random or malfunction right
and
there's even a whole operating system
called cell 3 that's been developed by a
darpa grant where
you can actually mathematically prove
that this thing can never be hacked
well one day i hope that will be
something you can say about the os
that's running on our laptops too
as you know we're not there but i think
we should be ambitious
frankly yeah and and
if we can use machine learning to help
do the proofs and so on as well
right then it's much easier to verify
that a proof is correct
than to come up with a proof in the
first place that's really the core idea
here
if someone comes on your on your podcast
and says they they proved the riemann
hypothesis
or some new sensational new theorem
it's not me oh it's much easier for some
one else to take some smart math grad
students to check oh there's an error
here on equation equation five
or this really checks out than it was to
discover the proof
yeah although some of those proofs are
pretty complicated but yes it's still
nevertheless much easier
to uh to verify the proof i love the
optimism
you know we kind of even with the
security of systems
there's a kind of cynicism that pervades
people who think about this which is
like oh it's hopeless
i mean in the same sense exactly like
you're saying when y'all now work so
let's go must understand what's
happening
uh with security people are just like
well it's always going there's always
going to be
um attack vectors yeah like uh
waste ways to attack the system but
you're right we're just very new with
these computational systems
we're new with these intelligent systems
and and it's
not out of the realm of possibility just
like people didn't understand the
movement of the stars and the planets
and so on
yeah it's it's entirely possible that
like
within hopefully soon but it could be
within a hundred years
we start to have an obvious like laws of
gravity about
intelligence and uh
god forbid about consciousness too that
one is
agreed you know i think of course if
you're
selling computers that get hacked a lot
that's in your interest as a company
that people think it's impossible to
make it safe so nobody's going to get
the idea of suing you but
i want to really inject optimism here it
the there it
it's it's absolutely possible to do
much better than we're doing now and you
know
your laptop does so much stuff you don't
need
the music player to be super safe in
your in your future
self-driving car right if someone hacks
it and starts playing music you don't
like
the world on end but what you can do is
you can break out and say the drive
computer
that controls your safety must be
completely physically
decoupled entirely from the
entertainment system and
it must physically be such that it can't
take on over-the-air updates
while you're driving and it can be it
can have
it's not that it can have ultimately a
some operating system on it which is
symbolically verified and proven
uh that that it's always going to do
what it's going to what it's supposed to
do right
we can basically have and companies
should take that into two they should
look at everything they do
and say what are the few systems in our
in our company
that threaten the whole life of the
company if they get hacked you know and
have
the highest standards for them and then
they can save money by going
for the el cheapo poorly understood
stuff for the rest you know
this this is very feasible i think and
coming back to
the bigger question about that you
worried about that that there will be
unintentional
failures i think there are two quite
separate risks here right
we talked a lot about one of them which
is that the goals are noble of the human
the human says i want this airplane to
not crash
because this is not muhammad atta now
flying the airplane right
and now there's this technical challenge
of making sure that the
the autopilot is actually going to
behave as as the pilot wants
if you set that aside there's also the
separate question how do you make sure
that the goals of the pilot are actually
aligned with the goals of the passenger
how do you
make sure very much more broadly that if
we can all agree as a species that we
would like things to kind of go well for
humanity as a whole
that the goals are aligned here the
alignment problem
and um yeah there's been a lot of
progress in in the sense that there's
suddenly
huge amounts of research going on on it
about it i'm very grateful to elon musk
for giving us that money five years ago
so we could launch the first research
program on
technical ai safety and alignment
there's a lot of stuff happening
but i think we need to do more than just
make sure little machines do always what
their owners do
you know that wouldn't have prevented
september 11. if muhammad after said
okay okay autopilot please fly into
world trade center you know and it's
like okay
that even happened in a different
situation there was this depressed
pilot named andreas lubitz right he told
his german wings passenger jet to fly
into the alps
he just told the computer to change the
altitude to 100 meters or something like
that and you know what the computer said
all right okay okay and it had the
freaking topographical map of the alps
in there it had gps
everything no one had bothered teaching
it even the basic kindergarten ethics of
like no we never want airplanes to fly
into
mountains under any circumstances
and so we have to think beyond just
the the technical
issues and think about how do we align
in general incentives on this planet
for the greater good so starting with
simple stuff like that
every airplane that has a computer in it
should be taught
whatever kindergarten ethics that's
smart enough to understand like
no don't fly into fixed objects if the
pilot tells you to do so
then go on autopilot mode
send an email to the cops and land at
the latest airport
nearest airport you know any car
with a forward-facing camera should just
be programmed by the
by the manufacturer so it will never
accelerate into a human ever
that would have avoid things like the
nice
attack and many horrible terrorist
vehicle attacks where they deliberately
did that right
this was not some sort of thing oh you
know us and china
different views on no there was not a
single car manufacturer in the world
in the world right who wanted the cars
to do this they just hadn't thought to
do the alignment and if you look at
more broadly problems that happen on
this planet
the vast majority have to do a poor
alignment i mean
think about let's go back really big
because i know this is you're so good
yeah in the very so long ago in
evolution we had these genes
and you they wanted to make copies of
themselves that's really all they cared
about
so they some genes said hey i'm going to
build a brain on this body i'm in so
that i can get better at making copies
myself yes
and then they decided for their benefit
to get
copied more to align your brain's
incentives with
their incentives so it didn't want
you to starve to death so it gave you an
incentive to eat
and it wanted you to make copies of of
the genes
so it gave you incentive to fall in love
and
do all sorts of naughty things to make
copies of it
of itself right yeah so that was
successful value alignment done
on the genes but they created something
more intelligent than themselves
but they made sure to try to align the
values but then something went a little
bit
raw against the idea of what the genes
wanted because
a lot of humans discovered hey you know
yeah we really like this
business about sex that the genes have
made us enjoy
but we don't want to have babies right
now yeah so we're going to hack the
genes
and use birth control and i really feel
like drinking a coca-cola right now but
i don't want to get a potbelly so i'm
going to drink diet coke you know
we have all these things we've figured
out because we're smarter than the jeans
how we can actually subvert their
intentions so
it's not surprising that this we humans
now when we're in the role of these
genes creating
other non-human entities with a lot of
power have to face the same exact
challenge how do we make other powerful
entities have
incentives that are aligned with ours
and so they won't hack them
corporations for example right we humans
decided to create corporations
because can benefit us greatly now all
of a sudden there's a supermarket i can
go buy food there i don't have to hunt
awesome and then to
make sure that this corporation would do
things that were good for us and not bad
for us we created institutions to keep
them in check
like if the local supermarket sells
poisonous food
then those
some the owners of a supermarket have to
spend some years reflecting
behind bars right so we created
incentives to get align them but of
course just like we were able to see
through this thing
and you developed birth control if
you're powerful corporation you also
have an incentive to try to hack
the institutions that are supposed to
govern you because you ultimately as a
corporation have an incentive to
maximize your profit
it's like you have an incentive to
maximize the enjoyment your brain has
not for your genes so if they can figure
out a way of
of bribing regulators then they're going
to do that
in the u.s we kind of caught on to that
and made laws against corruption and
bribery
then in the late 1800s
teddy roosevelt realized that no we were
still being kind of hacked because the
massachusetts railroad companies had
like a bigger budget than
the state of massachusetts and they were
doing a lot of very
corrupt stuff so he did the whole trust
busting thing to try to align
these other non-human entities the
companies again more with
the incentives of americans as a whole
um
it's not surprising though that you know
this is a battle you have to keep
fighting
now we have even larger companies than
we ever had before
and of course they're going to try to
again
support the institutions
not because you know i think people make
a mistake of getting all too um
black thinking about things in terms of
good and evil like arguing about whether
corporations are good or evil or whether
robots are good or evil
a robot isn't good or evil
it's tool and you can use it for great
things like robotic surgery or for bad
things
and a corporation also is a tool of
course
and if you have good incentives to
corporation it'll do great things like
start a hospital or a grocery store
if you have any bad incentives then it's
going to start maybe
marketing addictive drugs to people and
you'll have an opioid epidemic right
it's all about i don't want we should we
not make the mistake of getting into
some sort of fairy tale good evil thing
about
corporations or robots we should focus
on putting the right incentives in place
my optimistic vision is that if we can
do that you know
then we can really get good things we're
not doing so great with that right now
either on ai i think or on other
intelligent non-human entities like big
companies like we just have a new um
secretary of defense there's going to
start up now in
in the biden administration who is was
an active member of the board of
raytheon
for example yeah so you know
i have nothing against raytheon i'm all
i'm not a pacifist but
there's a obvious conflict of interest
if someone
is in the job where they decide who's
they're going to contract with
and i think somehow we have uh maybe we
need another teddy roosevelt to come
along again
and say hey you know we want what's good
for all americans
and we need to go do some serious
realigning again of the incentives that
we're giving
to these big companies and um
then we're going to be better off it
seems that naturally with human beings
just like you beautifully described the
history of this whole thing
uh of it all started with the genes and
they're probably pretty upset
by all the unintended consequences that
happened since
but the it seems that it kind of works
out like it's
in this collective intelligence that
emerges at the different levels
it seems to find sometimes last minute a
way to
realign the values or keep the values
aligned
it's almost um it finds a way like uh
different leaders different humans pop
up all over the place that
uh reset the system do you want i mean
do you have an explanation why that is
or is that just survivor bias
and also is that different somehow
fundamentally different than
with the ai systems where
you're no longer dealing with something
that was a direct
maybe companies are the same a direct
byproduct of the evolutionary process
i think there is one thing which has
changed that's why
i'm not all optimistic that's why i
think there's about a 50 percent chance
if we if we take the dumb route with um
artificial intelligence that
we will human humanity will be extinct
in this century
first just the big picture yeah
companies
need to have the right incentives even
governments right
we used to have governments usually
there were just some king
you know who was was the king because
his dad was the king you know
and and then there were some benefits of
having this powerful
kingdom because or empire of any sort
because then it could
prevent a lot of local squabbles so at
least everybody in that region would
stop warning against each other and
their incentives of different cities in
the kingdom became more aligned right
that's that was the whole selling point
harare yeah noah harari has a beautiful
piece on how empires were collaboration
enablers and then we also
hirari says invented money for that
reason so we could
have better alignment and we could do
trade even with people we didn't know
so this sort of stuff has been playing
out since time immemorial right
what's changed is that it happens on
ever larger scales right technology
keeps getting better because science
gets better so now we can communicate
over larger distances
transport things faster over larger
distances and so
the entities get ever bigger but our
planet is not getting bigger anymore
so in the past you could have one
experiment that just
totally screwed up like easter island
where they actually managed to have such
poor alignment that
when they went extinct people there
there was no one else to come back and
replace them right
if elon musk doesn't get us to mars and
then we
go extinct on a global scale and we're
not coming back
that's that's the fundamental difference
and that's an ex
mistake i would rather we don't make for
that reason
in the past of course history is full of
fiascos
right but it was never the whole planet
and and then okay now there's this nice
uninhabited land here some other people
could move in and
organize things better this is different
the second thing which is also different
is that technology
gives us so much more empowerment right
both to the good things and also to
screw up
in the stone age even if you had someone
whose goals were really poorly aligned
like maybe he was really pissed
off because his stone age girlfriend
dumped him and he just wanted to
if he wanted to like kill as many people
as he could yeah how many could he
really take out with a rock and a stick
before he was overpowered right right
just handful right yeah now
with today's technology if we have an
accidental nuclear war between
russia and the u.s which we almost have
about a dozen times and then we have a
nuclear winter it could take out seven
billion people
or six billion people we don't know uh
so this
the scale of the damage is bigger that
we can do and if
if um there's obviously no law of
physics that says that technology will
never get powerful enough that
we could wipe out our species entirely
that would just be
fantasy to think that science is somehow
doomed not get more powerful than that
right
and and it's not at all unfeasible in
our lifetime that someone could design a
designer pandemic
which spreads as easily as covet but
just basically kills everybody
we already had smallpox they killed
one-third of everybody who got it
and and um what do you think of the
here's an intuition maybe it's
completely naive and this optimistic
intuition i have which
it seems and maybe it's a biased
experience that i have
but it seems like the most brilliant
people i've met in my life
all are really like fundamentally good
human beings
and not like naive good like they really
want to do good for the world in a way
that
well maybe is aligned to my sense of
what good means
and so i have a sense that the
the people that will be defining the
very cutting edge of technology
there will be much more of the ones that
are doing good versus the ones that are
doing evil
so the race i'm optimistic on the
us always like last minute coming up
with a solution so
if there's an engineered pandemic that
has the
capability to destroy most of the human
civilization
it it feels like to me either leading up
to that
before or as it's going on there will be
we're able to rally the the collective
genius of the human species
i could tell by your smile that you're
uh at least some percentage uh
um doubtful but is that
could that be a fundamental law of human
nature
that evolution only creates it creates
uh
like karma is beneficial good is
beneficial and therefore will be all
right
i hope you're right i would really would
love
it if you're right if there's some sort
of law of nature that says that we
always get lucky in the last second
because of karma but you know
i pref i prefer uh
not playing it so close and gambling on
that
and i i think um in fact i think it can
be dangerous
to have too strong faith in that yes
because it it
makes us complacent like if someone
tells you you never have to worry about
your house burning down then you're not
going to put in a smoke detector because
why would you need to right
right even if it's sometimes very simple
precautions we don't take them
if you're like oh the government is
going to take care of everything for us
i can always trust my politicians
you know we advocate our own
responsibility i think it's a healthier
attitude to say yeah maybe things will
work out
maybe i'm actually gonna have to myself
step up and take responsibility
uh and the stakes are so huge i mean
if we do this right we can develop all
this ever more powerful technology and
cure all diseases
and create a future where humanity is
healthy and wealthy for not just the
next election cycle but like
billions of years throughout our
universe that's really worth working
hard for
and not just you know sitting and hoping
for some sort of fairy tale karma
well i i just mean so you're absolutely
right from the perspective of the
individual like for me
like the primary thing should be to take
responsibility and to
build the solutions that your skill set
allows
yeah which is a lot i think we
underestimate often very much how much
good we can do right if the if you or
anyone listening to this
is completely confident that our
government would do a perfect job on
handling any future crisis
with engineered pandemics or
future ai people out there
on on what actually happened in 2020
do you feel that the government by and
large around the world has handled this
flawlessly
uh that's a really sad and disappointing
reality that uh
hopefully is a wake-up call for
everybody uh for the scientists for the
for the re for the engineers for the
researchers in ai especially it was
disappointing to see
how inefficient we were as
spread collecting the right amount of
data in a privacy
preserving way and spreading that data
and utilizing that data to make
decisions
all that kind of stuff yeah i think when
something bad happens to me
i made myself uh a promise
many years ago that i would not
be a whiner so when something bad
happens to me
of course it's a process the
disappointment but then
i try to focus on what did i learn from
this that can make me a better person in
the future
and there's usually something to be
learned when i fail
and i think we should all ask ourselves
what
can we learn from the pandemic about how
we can do better in the future
and you mentioned there's a really good
lesson you know we were not as resilient
as we thought we were and we were not as
prepared
maybe as we wish we were you can even
see very stark contrast around the
planet
south korea right they have over 50
million people
do you know how many deaths they have
from covet last time i checked
no about 500
why is that well
the short answer is that they had
prepared
they were incredibly quick incredibly
quick to get on it
with very rapid testing and contact
tracing
and so on which is why they never had
more cases than they could contract
trace effectively right they never even
had to have the kind of big lockdowns we
had in the west
but the deeper answer to it's not just
the koreans are just somehow better
people
the reason i think they were better
prepared was because they
had already had a pretty bad hit
from the sars pandemic or which never
became a pandemic
something like 17 years ago i think so
it's kind of fresh memory that you know
we need to be prepared for pandemics
so they were right and
so maybe this is a lesson here for all
of us to draw
from covid that rather than just wait
for the next pandemic or the next
problem with ai getting out of control
or anything else
maybe we should just actually set aside
a tiny fraction of our gdp
to have people very systematically do
some horizon scanning and say okay
what are the things that could go wrong
and let's duke it out and see which are
the more likely ones and which are the
ones that are actually actionable
and then be prepared so
one of the observations as one little
ant
slash human that i am of disappointment
is the political division
over information
that has been observed that i observed
this year
that it seemed uh the discussion was
less about
um sort of uh
what happened and understanding what
happened deeply
and more about there's different
truths out there and it's like a
argument my truth is better than your
truth
and it's it's like red versus blue or
different like it was like this
ridiculous
discourse that doesn't seem to get at
any kind of notion of the truth
it's not like uh some kind of scientific
process even science got politicized in
ways that's very
heartbreaking to me uh you have an
exciting project
on the ai front uh of trying to
rethink one of the you mentioned
corporations
there's one of the other collective
intelligence systems that have emerged
all this is social networks and just to
spread
the internet is the spread of
information on the
uh the internet our ability to share
that information there's all different
kinds of news sources and so on
and so you said like that's from first
principles let's rethink
how we think about the news how we think
about information
can you talk about this uh amazing
effort that you're undertaking
well i'd love to this has been my big
covet project
it's been nights and weekends on ever
since the
lockdown to segue into this actually let
me come back to what you said earlier
that you
had this hope that in your experience
people who you felt were very talented
or often
idealistic and wanted to do good frankly
i feel the same about all people
by and large there are always exceptions
but i think the vast majority of
everybody regardless of education and
whatnot
really are fundamentally good right so
how can it be
that people still do so much nasty stuff
right
i think it has everything to do with
this with the information that we're
given
yes you know if you go into sweden
500 years ago and you start telling all
the farmers that those danes in denmark
they're so terrible people you know and
we have to
invade them yes because they've done all
these terrible things that you can't
fact check yourself
a lot of people swedes did that right
and if
and um we've seen we're seeing
so much of this today in the world
both geopolitically you know where we
are told that that
china is bad and russia is bad and
venezuela is bad and people in
those countries are often told that we
are bad and we also see it at a
micro level you know where people are
told that oh those who voted for the
other party are bad people
it's not just an intellectual
disagreement but they're
bad people and um we're getting ever
more divided
and so how do you reconcile this with
with this
intrinsic goodness i and people i
i think it's pretty obvious that it has
again to do with this with the
information they were fed
and given right we evolved to live in
small groups where you might know 30
people in total right so you
then had a system that was quite good
for assessing who you could trust and
who you could not and if someone told
you
that you know joe there is a jerk
but you had interacted with him yourself
and seen him in action and
and you would quickly realize maybe that
that's actually not
quite accurate right but now that we the
most people on the planet are people
we've never met
it's very important that we have a way
of trusting the information we're given
and so okay so where does the news
project come in well
throughout history you can go read
machiavelli you know from the 1400s and
you'll see how already then they were
busy manipulating people with propaganda
and stuff
propaganda is not new
at all and the incentives to manipulate
people is
just not new at all what is it that's
new
what's new is machine learning meets
propaganda
that's what's new that's why this has
gotten so much worse you know some
people like to blame
certain individuals like in my liberal
university bubble many people blame
donald trump and say it was his fault
i see it differently i think
what donald trump just had this extreme
skill at playing this game
in the machine learning algorithm age
a game he couldn't have played you know
10 years ago so what's changed which
changes
well facebook and google and other
companies
and i don't i'm not bad matching them i
have a lot of friends who work for these
companies
good people they deployed machine
learning algorithms
just to increase their profit a little
bit to just maximize the time
people spent watching ads and they had
totally underestimated how effective
they were going to be this was again the
black box
non-intelligible intelligence they just
noticed oh we're getting more ad revenue
great it took a long time until they
even realized why
and how and how damaging this was for
society
because of course what the machine
learning figured out was
that the by far most effective way of
gluing you to your little rectangle
was to show you things that triggered
strong emotions anger
etc resentment and uh
if it was true or not didn't really
matter it was also easier to find
stories that weren't true
if you weren't limited that's just the
limitations let's show people
that's a very limiting fact and before
and
long we got these amazing filter bubbles
on a scale we had never seen before
couple this to the fact that also
the online news media were so effective
that they
killed a lot of print journalism there's
only there's
less than half as many journalists now
in america i believe
as there was you know a generation ago
he just couldn't compete with the online
advertising
so all of a sudden most people
are not getting even reading newspapers
they get get their news from social
media
and most people only get news in their
little bubble
so along comes now some people like
donald trump
who've figured out what among the first
successful politicians to figure out how
to really play this new game
and become very very influential but i
think donald trump was a sim
well he he took advantage of it he
didn't
create the fundamental conditions were
created by
machine learning taking over the news
media
so this is what motivated my little
covid project here so
you know i said before machine learning
and tech in general is not evil but it's
also not good it's just a tool
that you can use for good things or bad
things and as it happens
machine learning and news was mainly
used by the big players
big tech to manipulate people watch as
many ads as possible
which have this unintended consequence
of really screwing up our democracy into
and fragmenting it into filter bubbles
so i thought well
machine learning algorithms are
basically free they can run on your
smartphone for free also if someone
gives them away to you right
there's no reason why they only have to
help the big guy
to manipulate the little guy they can
just as well help the little guy
to see through all the manipulation
attempts from the big guy so
this project is called you can go to
improve the news.org
the first thing we've built is that it's
a little news aggregator
looks a bit like google news except it
has these sliders on it to help you
break out or your filter bubble
so if you're reading you can click click
and go to your favorite topic
and then um if you just slide the left
right slider away all the way over to
the left
there's two sliders right yeah there's
the one the most obvious one is the one
that has left right labeled on it
you go to the left you get one set of
articles you go to the right you see a
very different truth yeah appearing oh
that's literally
left and right on the political spectrum
on the political yeah so if you're
reading about
immigration for example it it's very
very
noticeable and and i think step one
always if you want to
not get manipulated is just to be able
to recognize
the techniques people use so it's very
helpful to just see
how they spin things on the two sides
i think many people are under the
misconception that the main problem is
fake news it's not
we i had an amazing team of mit students
where we did an academic project
to use machine learning to detect the
main kinds of bias over the summer and
yes of course sometimes there's fake
news where someone just claims something
that's
false right like oh hillary clinton just
got divorced or something
yes but what we see much more of is
actually just omissions
if you go to there's some stories
which just won't be mentioned by the
left or the right because it doesn't
suit their agenda
and then they'll instead mention other
ones very very very much
so for example we've had
a lot a number of stories about
the trump family's financial dealings
and then there's been some a bunch of
stories about the biden families hunter
binds financial dealings right
surprise surprise they don't get equal
coverage on the left and the right
right one side loves to cover the biden
hunter biden stuff and one side loves to
cover the trump you
never guess which is which right yeah
but the great news is if you want to if
you're a
normal american citizen and you dislike
corruption in all its forms
then slide slide you can just get look
at both of the
sides and you'll see all the corrupts
those political corruption stories
it's really liberating to just take in
the both sides the spin on both sides
it somehow unlocks your mind to like
think on your own
to realize that that i don't know it's
the same thing that was useful right in
the soviet union times for
when when everybody was much more aware
that they're
surrounded by propaganda right it's so
interesting what you're saying actually
so noam chomsky
you know used to be our mit colleague
once said that
propaganda is to democracy what violence
is
to totalitarianism and and
what he means by that is if you have a
really totalitarian government you don't
need propaganda
right people will do what you do what
you want them to do
anyway out of fear right yes but
otherwise you need propaganda so i would
say actually that the propaganda is much
higher quality
in democracies much more believable and
it's
brilliant it's really striking when i
talk to colleagues
science colleagues like from russia and
china and so on
i noticed they are actually much more
aware of the propaganda in their own
media than many of my american
colleagues are about the
propaganda and western media that's
brilliant that means the propaganda in
the western media is just better yes
that's so big better in the west
oh man
that's good but when you when once you
realize that you realize there's also
something very optimistic there that you
can do about it right
because first of all omissions
as long as there's no outright
censorship you can just look at both
sides
and pretty quickly piece together a much
more accurate idea of what's actually
going on right and develop a natural
skepticism too
yeah just a analytical scientific
mind about how you're taking the
information yeah and i i think
i have to say sometimes i feel that some
of us in the academic bubble are
too arrogant about this and somehow
think oh it's just people who aren't as
educated when we are often just as
gullible also you know
we read only our media and and don't see
through things
anyone who looks at both sides like this
in comparison will immediately start
noticing
the shenanigans being pulled and you
know i think
what i tried to do with with this app is
that the
big tech has to some extent uh try to
blame the individual
for being manipulated much like big
tobacco
tried to blame the individuals entirely
for smoking and then
later on you know our government stepped
up and said actually you know
you can't just blame little kids for
starting to smoke we have to have more
responsible advertising and this and
that
i think it's a bit the same here it's
very convenient for a big tech to blame
so it's just people who are so dumb and
get fooled yeah
the blame usually comes in saying oh
it's just human psychology
people just want to hear what they
already believe but professor david rand
at mit
actually partly debunked that with a
really nice study showing that people
are tend to be interested
in hearing things that go against what
they believe
if it's presented in a respectful way
like suppose for example
that um you have a company
and you're just about to launch this
project and you're convinced it's going
to work and someone says you know
lex i hate to tell you this but this is
going to fail and here's why
would you be like shut up i don't want
to hear it
yeah would you you would be interested
right and also
if you're on an airplane back in the co
precovered times you know
and the guy next to you is clearly from
the opposite side of the political
spectrum
but is very respectful and polite to you
wouldn't you be kind of interested to
hear a bit about
how he or she thinks about things of
course
but it's not so easy to find out
respectful disagreement now because like
for example if you
are a democrat and you're like i want to
see something on the other side so you
just go
breitbart.com and then after the first
10 seconds
you feel deeply insulted by something
and then they
it's it's not going to work or if you
take someone who votes republican
and they go to something on the left and
they just get very offended very quickly
by them having put a deliberately ugly
picture of donald trump on the front
page or something
it doesn't really work so this news
aggregator also has this nuanced slider
which you can pull to the right and then
to make it easier to get exposed
to actually more sort of academic style
or more respectful um
portrayals of different views and and
finally the one
kind of bias i think people are mostly
aware of is
the left right right because it's so
obvious because
both left and right are very powerful
here right
both of them have well-funded tv
stations and newspapers and it's kind of
hard to miss
but there's another one the
establishment slider
which is it's also really fun i i i love
to play with it
that's more about corruption yes because
if you have a society
that where almost all the powerful
entities
want you to believe a certain thing
that's what you're going to read in
both the big media mainstream media on
the left and on the right of course
and the powerful companies can push back
very hard
like tobacco companies pushed back very
hard back in the day when peop
some newspaper started writing articles
about tobacco being dangerous
so it was hard to get a lot of coverage
about it initially
and also if you look geopolitically
right of course
in any country when you read their media
you're mainly going to be reading a lot
about articles about how
our country is the good guy and the
other countries are the bad guys
right so if you want to have a really
more nuanced understanding you know
like the germans used to be told that
the the
the british used to be told that the
french were the bad guys and the french
used to be told that
british were the bad guys now they
visit each other's countries a lot and
have a much more nuanced understanding i
don't think there's going to be any more
wars between france and germany
but on the geopolitical scale
it's just just as much as ever you know
a big cold war now you as china
and so on and if you want to if you want
to get a more nuanced
understanding of what's happening
geopolitically then it's really fun to
look at this
establishment slider because it turns
out there are tons of
little newspapers both on the left and
on the right
who sometimes challenge establishment
and say
you know maybe we shouldn't actually
invade iraq right now maybe this weapons
and mass destruction thing was bs
if you look at the journalism research
afterwards you can actually see that
quite clearly that both
cnn and fox were very pro
let's and get rid of saddam there are
weapons of mass destruction
then there were a lot of smaller
newspapers they were like wait a minute
this evidence seems a bit sketchy and
maybe we
but of course they were so hard to find
most people didn't even know they
existed right
yet it would have been better for
american national security if those
voices had also come up
i think it harmed america's national
security actually that we invaded iraq
and arguably there's a lot more interest
in
that kind of thinking too from those
small sources
so like the when you say big it's more
about
kind of uh the reach of the broadcast
but it's not big in terms of the
interest i think
i think there's a lot of interest in
that kind of anti-establishment or like
skepticism towards
you know out-of-the-box thinking there's
a lot of interest in that kind of thing
do you see this news project or
something like it being um
basically taking over the world as as
the main way we consume information like
what's how do we get
how do we get there like how do we you
know
so okay the idea is brilliant it's a
it's a you you're calling it your little
project
in 2020 but how does that become the new
way we consume
information i hope first of all just to
plan the little seed there because
normally the big barrier of doing
anything in media is you need a ton of
money but
this cost me no money at all i've just
been paying myself you
pay a tiny amount of money each month to
amazon to run the thing in their cloud
we're not there never will never be any
ads the point is not to make any money
off of it
and we just train machine learning
algorithms to classify the articles and
stuff so it just kind of runs by itself
so
if it actually gets good enough at some
point that it starts catching on it
could
scale and if other people carbon copy it
and make other versions that are better
that's the more the merrier
i think there's a real
opportunity for machine learning to
empower the individual
the list of the powerful players
uh it's as i said in the beginning here
it's been mostly the other way around so
far that the big players have the ai
and then they tell people this is the
truth
this is how it is but it can just as
well
go the other way around and when the
internet was born actually a lot of
people had this hope that maybe this
will
be a great thing for democracy make it
easier to find out about things
and maybe machine learning and things
like this can uh can
actually help again and i have to say i
think it's impo it's more important than
ever
now right because this is very linked
also to the whole
future of life as we discussed earlier
right we're getting this ever more
powerful attack you know
it's frank it's pretty clear if you look
on the one or two generation three
generation time scale that there are
only two ways this can end
geopolitically yeah either it ends great
for all humanity
or ends terribly for all of us
there's there's really no in between it
and
we're so stuck in and because you know
technology knows no borders
and you can't have people fighting
when the weapons just keep getting ever
more powerful
uh indefinitely eventually
lux the luck runs out and and you know
right like right now we have i love
america
but the the fact of the matter is
what's good for america is not opposites
in the long term to what's good for
other countries it it would be if this
was some sort of
zero-sum game like it was thousands of
years ago when the only way one country
could get more resources
was to take land from other countries
because that was basically the resource
right you look at the map of europe some
countries kept getting bigger and
smaller
endless wars but then since
1945 there hasn't been any war in
western europe and they all got way
richer because of tech so the optimistic
outcome is
that the big winner in this century is
going to be america and china
and russia and everybody else because
technology just makes us all
healthier and wealthier and we just find
some way of
keeping the peace on this planet but i
think
unfortunately there are some pretty
powerful forces right now that are
pushing in exactly the opposite
direction and trying to demonize other
countries
which just makes it more likely that
this ever more powerful tech
we're building is gonna use disastrous
ways
yeah for aggression versus cooperation
that kind of thing yeah even look at
look at this military ai now right
it's 120 it was so awesome to see these
dancing robots i loved it
right but one of the biggest growth
areas in robotics now
is of course autonomous weapons right
and and 2020 was like the
best marketing year ever for autonomous
weapons because
in both libya civil war and in
nagorno-karabakh
they made the decisive difference right
and everybody else is like watching this
oh yeah we want to build autonomous
weapons
too and in libya
you had on one hand our ally the united
arab emirates that were flying their
autonomous weapons that they bought from
china
bombing libyans and on the other side
you had our other ally turkey flying
their
drones and
they had no skin in the game any of
these other countries and of course
there was the libyans who really got
screwed
in nagorno-karabakh you had actually
again now turkey is sending drones built
by this company that was actually
founded by a guy who went to mit
aeroastro do you know that
background dr yeah so mit has a direct
responsibility for
ultimately this and a lot of civilians
were killed there you know and
so because it was militarily so
effective
now now suddenly there's a huge push oh
yeah yeah let's
go build ever more autonomy
into these these weapons and it's going
to be great
and uh i think actually
people who are obsessed about some sort
of future terminology scenario right now
or
should start focusing on the fact that
we have
two much more urgent threats happening
for machine learning one of them is the
whole
destruction of democracy that we've
talked about now where
where our flow of information is being
manipulated by machine learning and the
other one is
that right now you know this is the year
when the big arms race
in out of control arms race and at least
thomas weapons is going to start or
it's going to stop so you have a sense
that there is uh
like 2020 was a instrumental catalyst
for the race
of for the autonomous weapons race yeah
because it was the first year when
when they proved decisive in the
battlefield and
and these ones are still not fully
autonomous mostly they're remote
controlled right
but you know we could
very quickly make things about you know
the size and cost of a smartphone
which you just put in the gps
coordinates or the face of the one you
want to kill his skin color or whatever
and it flies away and you know does it
and
the the real good reason why the u.s and
all the other superpowers
should put the kibosh on this is the
same reason
we decided to put the kibosh on bio
weapons
so you know we gave the future of life
award that we can talk more about later
matthew messelston from harvard before
for convincing nixon to ban bioweapons
and
i asked him how did you do it and he was
like well
i just said look we don't want there to
be a 500
weapon of mass destruction that even
all our enemies can afford even
non-state actors
and nixon was like
good point you know it's in america's
interest that the power of weapons are
all really expensive
so only we can afford them or maybe some
more stable adversaries right
nuclear weapons are like that but
bioweapons were not like that that's why
we banned them
and that's why you never hear about them
now that's why we love biology
so you have a sense that it's possible
for the big
powerhouses in terms of the the big
nations in the world to agree that
autonomous weapons is not a race we want
to be on that it doesn't end well
yeah because we we know it's just going
to end in mass proliferation and every
terrorists terrorist
everywhere is going to have these super
cheap weapons that they will use against
us
and it and our and our politicians have
to constantly worry about being
assassinated every time they go outdoors
by some anonymous
little mini drone you know we don't want
that and
if even if the u.s and china and
everyone else could just agree that
you can only build these weapons if they
cost at least 10 million bucks
that would that would be a huge win for
the superpowers
and frankly for everybody the um
you don't and people often push back and
say well it's so hard to prevent
cheating but hey you could say the same
about bioweapons you know
take any of your rmit colleagues in
biology
of course they could build some nasty
bioweapon if they really wanted to
but first of all they don't want to
because they think it's disgusting
because of the stigma
and second even if there's some sort of
nut case in want to
it's very likely that some other grad
students or someone would rat them out
because everyone else thinks it's so
disgusting right
and in fact we now know there was even a
fair bit of cheating on the bioweapons
ban
but none no countries used them because
it was so stigmatized
that it just wasn't worth revealing that
they had cheated
you talk about drones but you kind of
think
that drones is the remote operation
which they are
mostly yes but you're not taking the
next
intellectual step of like where does
this go
you're kind of saying the problem with
drones is that you're removing yourself
from
direct violence therefore you're not
able to sort of maintain the common
humanity required to make the proper
decision strategically
but that's the criticism as opposed to
like if this is
automated and just exactly as you said
if you automate it and there's a race
then he's going to the technology and
you get better and better and better
which means getting cheaper and cheaper
and cheaper yeah
and unlike perhaps nuclear weapons which
is connected to
uh resources in a way like it's hard to
get the
it's hard to engineer yeah it feels like
it's you know um there's too much
overlap between the tech industry
and autonomous weapons to where you
could have smartphone type of
uh cheapness if you look at drones uh
you know it's a it's a you know for a
thousand dollars you can have an
incredible system that's able to
maintain flight autonomously for you and
take pictures and stuff
you could see that going into the
autonomous weapon space that's
um but like why is that not thought
about or discussed enough
in the public do you think you see those
dancing boston dynamics robots
and everybody has this kind of um
like as if this is like a far future
yeah they have this like
fear like oh this will be terminator in
like some
i don't know unspecified 20 30 40 years
and they don't think about well this is
like some
much less dramatic version of that is
actually happening
now it's not going to have it's not
going to be legit it's not going to be
dancing
but it's this already has the capability
to use artificial intelligence to
kill humans yeah the boston dynamics
leg robots i think the reason we imagine
them holding guns is just because you've
all seen arnold schwarzenegger right
right
that's our reference point that's pretty
useless
that's not going to be the main military
use of them they might be useful in law
enforcement
in the future and then there's a whole
debate about do you want robots showing
up at your house with guns telling you
to
uh who'll be perfectly obedient to
whatever dictator controls whatever
but let's leave that aside for a moment
and look at what's actually relevant now
so there's a spectrum of things you can
do with ai in the military and again
to put my card on the table i'm not the
pacifist i think we should
have good defense um
so for example a predator drone
is a room basically a fancy little
remote controlled airplane
right there's a human can piloting it
and the decision ultimately about
whether to kill somebody with it is made
by a human still
and this is a line i think we should
never cross
there's a current dod policy again you
have to have a human in the loop
i think algorithms should never make
life or death decisions they should be
left to humans
now why might we cross that line
well first of all these are expensive
right so
for example when uh when
when azerbaijan had all these drones and
armenia didn't have any they start
trying to jerry-rig little cheap things
fly around and but then of course their
minions would jam them
as areas would jam them and remote
control things can be jammed
that makes them inferior also there's a
bit of a time delay
between you know if we're piling
something far away
speed of light and the human has a
reaction time as well
it would be nice to eliminate that
jamming possibility in the time delay by
having it
fully autonomous but now you might be so
then
if you do but now you might be crossing
that exact line you might program it to
just
oh yeah the air drone go hover over
this country for a while and whenever
you find someone who
is a bad guy you know kill them
now the machine is making these sort of
decisions and you and some people who
defend this
still say well that's morally fine
because we are the good guys
and we will tell it the definition of
bad guy that we think
is moral but now
you'll be very naive to think that if
isis
buys that same drone that they're gonna
use our definition of bad guy
maybe for them bad guy is someone
wearing a us army uniform right
or maybe maybe there will be some
weird uh ethnic group who decides that
someone of another ethnic group they are
the bad guys right
the thing is human soldiers with all
our faults right we still have some
basic wiring
in us like no it's not okay to kill kids
and civilians and thomas weapon has none
of that
it's just going to do whatever is
programmed it's like the perfect adolf
eichmann
on steroids like they told him out of
icman you know he wanted to do this and
this than this to make the holocaust
more efficient and he was like
and off he went and did it right yeah do
we really want to make machines
that are like that like completely
amoral and we'll take the user's
definition of who's the bad guy
and do we then want to make them so
cheap that all our adversaries can have
them like
what could possibly go wrong that's the
that's i think the big argument for why
we want to
this year really put the kibosh on this
and i think you can tell there's a lot
of
very active debate even going on within
the u.s military and undoubtedly
in other militaries around the world
also about whether we should have some
sort of international agreement
to at least require that these weapons
have to be above a certain
size and cost you know so that
um things just don't uh totally spiral
out of control
and finally just for your question now
but is it possible to stop it
because some people tell me oh just give
up you know
but again so so matthew messelcen again
from harvard right who
the bio weapons hero he
had exactly this criticism also with bio
weapons people were like
how can you check for sure that the
russians aren't cheating
and um he told me this
i think really ingenious insight he said
you know max
some people think you have to have
inspections and things and you have to
make sure that people that you can catch
the cheaters with a hundred percent
chance you don't need a hundred percent
they said
one percent is usually enough
because i like if it's just an enemy if
it's another big state
like suppose china and the u.s have
signed the treaty
[Music]
drawing a certain line and saying yeah
these kind of drones are okay but these
fully autonomous ones are not
now suppose you are
china and you have cheated and secretly
developed some clandestine little thing
or you're thinking about doing it you
know what's your calculation that you do
well
you're like okay what's the probability
that we're going to get caught
if the probability is 100 of course
we're not going to do it
but if the probability is five percent
that we're going to get caught then it's
going to be like a huge embarrassment
for us
yeah and it doesn't have it's
we still have our nuclear weapons anyway
so it doesn't really
make an enormous difference in in terms
of deterring the u.s
you know and that fees the stigma that
you kind of
establish like this fabric this
universal stigma over the thing
exactly it's very reasonable for them to
say well you know we probably get away
with it but if we don't
then the u.s will know we cheated and
then they're going to go full tilt with
their program and say look the chinese
are cheaters and that's good
now we have all these weapons against us
and that's bad so
the stigma alone is very very powerful
and again
look what happened with bioweapons right
it's been 50 years now
yeah when was the last time you read
about a bioterrorism attack
the only deaths i really know about with
bioweapons that have happened
when we americans managed to kill some
of our own with anthrax you know the
idiot who sent them to tom daschle and
others and letters right
and similarly the in uh zverdlovsk in
the soviet union they had some anthrax
and some lab there maybe they were
cheating or who knows
and it leaked out and killed a bunch of
russians i'd say that's a pretty good
success right 50 years
just two own goals by the superpowers
and then nothing
and that's why whenever i ask anyone
what they think about biology they they
think it's great
they associate it with new cures new
diseases maybe a good vaccine
this is how i want to think about ai in
the future i want and i want others to
think about ai too
as a source of all these great solutions
to our problems
not as oh ai oh yeah
that's the reason i feel scared going
outside these days
yes it's kind of brilliant that the
bioweapons and nuclear weapons we've
figured out
i mean of course there's still a huge
source of danger but we figured out
some way of creating rules
and social stigma over these weapons
that then
creates a stability to our whatever that
game theoretic stability that is
exactly and we don't have that with ai
and you're kind of
screaming from the top of the uh the
mountain
about this that we need to find that
because um
just like you know it's very possible
you know with future of life as you
point out uh institute uh awards
pointed out that uh you know with
nuclear weapons
we could have destroyed ourselves quite
a few yeah times and it's um
you know it's a learning experience
that uh doesn't it's very costly we gave
you know this future life award
we gave it the first time to this guy
vasily arkhipov you know he was
on most people haven't even heard of
yeah can you say who he is
vasily arkhipov he um
has in my opinion made the
greatest positive contribution to
humanity of any human
in modern history and maybe it sounds
like hyperbole here like i'm just
over the top but let me tell you the
story and i think maybe you'll agree
so during the cuban missile crisis
we americans first didn't know that the
russians had sent four submarines
but we caught two of them and we didn't
know that
that so we dropped practice depth
charges on the one that he was on
to try to force it to the surface
but we didn't know that this nuclear
submarine actually was a nuclear
submarine with a nuclear torpedo
we also didn't know that they had
authorization to launch it without
clearance from moscow
and we also didn't know that they were
running out of electricity their
batteries were almost dead they were
running out of oxygen sailors were
fainting left and right
the temperature was about 110
120 fahrenheit on board it was really
hellish conditions really just
kind of doomsday and at that point these
giant explosions start happening from
the americans dropping these
the captain thought world war iii had
begun they
decided that they were going to launch
the nuclear torpedo and one of them
shouted you know we're all going to die
but we're not going to disgrace our navy
you know we don't know what would have
happened if there had to be a giant
mushroom cloud all of a sudden
you know and against the americans but
since everybody had their hands on the
triggers
it's pretty you don't have to be too
creative to think that it could have led
to an all-out nuclear war
and in which case we wouldn't be having
this conversation now right what
actually took place
was they needed three people to to
approve this
the captain had said yes there was the
communist party political officer he
also said yes let's do it
and the third man was this guy
vasiliyakipov who said yet
yeah uh he for some reason he was just
more chill than the others and he was
the right man at the right time
i don't want us as a species rely on the
right person being there at the right
time
you know we tracked down his family
living in relative poverty outside
moscow
when he flew his daughter he had passed
away
and flew them to london they had never
been to the west even it was incredibly
moving so you have to honor them for
this
uh the next year we gave this future
life award to
stanislav petrov have you heard of him
yes so he
he was in charge of the soviet early
warning
station which was built with soviet
technology
and honestly not that reliable it said
that there were five us missiles coming
in
again if they had launched at that point
we probably wouldn't be having this
conversation he decided
based on just mainly gut instinct to
just not tell
not to not escalate this and i'm very
glad he wasn't replaced by an ai that
was just
automatically following orders and then
we gave the third one to matthew
messelson
last year we gave this award to
these guys who actually use technology
for good
not avoiding something bad but for
something good the guys who
eliminated this disease which is way
worse than covent
that had killed a half a billion people
in in the past its final century
smallpox right so
we mentioned it earlier coved on average
kills less than one percent of people
who get it smallpox about 30 percent
and um they just ultimately victor donov
and bill feige most my colleagues have
never heard of either of them
um one american one russian they
did this amazing effort not only what
was
able to get the us and the soviet union
to team up against smallpox during the
cold war
but bill feige came up with this
ingenious strategy for making it
actually
go all the way to defeat the disease
with without funding for vaccinating
everyone and as a result we haven't had
any we went from 15 million deaths the
year i was born in smallpox
so what do we have in covet now a little
bit short of two million right yes
to zero deaths of course this year and
forever
there have been 200 million people they
estimate who would have died
since then by smallpox had it not been
for this so isn't science awesome can't
that
when you use it for good and the reason
we want to celebrate these sort of
people is
is to remind them of this science is so
awesome
when you use it for good and those
those awards actually uh the variety
there paint's a very
interesting picture so the the the first
two
are looking at it's kind of exciting to
think that these these
average humans in some sense that
they're products of you know billions of
other humans that came before them
evolution and some
little you said gut you know but there's
something
in there that that uh
stopped the annihilation of the human
race
and that's a magical thing but that's
like this deeply human thing
and then there's the other aspect where
that's also very human
which is to build solution to the to the
existential crises that we're facing
like to build it to take the
responsibility and to take
come up with different technologies and
so on yeah and both of those are
deeply human the gut and and the mind
whatever that is the best is when they
work together archipelago
i wish i could have met him of course
but he had passed away
he was really a fantastic military
officer
combining all all the best traits that
we in america admire in our military
because first of all he was very loyal
of course he never even told anyone
about this
during his whole life even though you
think he had some bragging rights right
but he just was like this is just
business just doing my job
it only came out later after his death
and and second
the reason he did the right thing was
not because he was some sort of liberal
or some sort not because he was
just oh you know uh peace and love
it was partly because he had been the
captain on another submarine that had a
nuclear reactor meltdown
and it was his heroism that
helped contain this that's why he died
of cancer later also but he's seen many
of his crew members die and
i think for him that gave him this gut
feeling that you know
if there's a nuclear war between the u.s
and the soviet union
the whole world is going to go through
what i saw
my dear crew members suffer through it
wasn't just an abstract thing for him i
think
it was real and second though not just
the gut the mind right he
he was for some reason very level-headed
personality a very smart guy
which is exactly what we want our best
fighter pilots to be also right i i
never forget neil armstrong when he's
landing on the moon and almost running
out of gas
and he doesn't even change when they say
30 seconds he doesn't even change the
tone of voice just keeps going arc
i think was just like that so when the
explosion starts going off and his
captain is screaming and we should nuke
them and
and all he's like
i don't think the americans are trying
to sink us
i think they're trying to send us a
message
that's a pretty badass yes coolness
because he said yeah if they wanted to
sink us
no and he said listen listen it's
alternating
one loud explosion on the left one on
the right
one on the left one on the right he was
the only one who noticed this pattern
and it is like i think this is them
trying to send us a signal
that they want us to surface and they're
not going to sink us
uh and somehow
this is how he then managed to
ultimately with his
combination of gut and also just
cool analytical thinking was able to
de-escalate the whole thing
and uh yeah so this is some of the best
in humanity i guess coming back to what
we talked about earlier is the
combination of the neural network
you know with uh i'm getting the tearing
up here getting emotional but
he was just he is one of my superheroes
com having both the
heart you know and the mind combined and
especially in that time
uh there's something about the i mean
this is a very
in america people are used to this kind
of idea of being the individual
um of like on your own thinking yeah i
think
under in the soviet union under
communism it's actually much harder
to do that oh yeah he didn't even he
even got
he didn't get any accolades either when
he came back for this right
uh they just wanted to hush the whole
thing up yeah there's echoes of that
with chernobyl there's all kinds of um
uh that's one that's that's a really
hopeful thing
that amidst big centralized powers
whether it's companies
or states there's still the power of the
individual
to think on their own to act but i think
we need to think of people like this
not as a panacea we can always count on
but rather as
a wake-up call you know so
because of them because of archipov we
are alive to learn from this lesson
to learn from the fact that we shouldn't
keep playing russian roulette and almost
have a nuclear war by mistake
now and then because relying on luck is
not a good long-term strategy
if you keep playing russian inlet over
and over again the probability of
surviving just drops exponentially with
time
yeah and if you have some probability of
having an accidental new core every year
the probability of not having one
also drops exponentially i think we can
do better than that so i i think
uh the message is very clear once in a
while happens
and um there's a lot of very concrete
things we can do
to reduce the risk of things like that
happening in the first place
on the ai front if we just link an
effort
yeah so you're friends with you often
talk with elon musk throughout history
did a lot of interesting things together
um he has a
a set of fears about the future of
artificial intelligence
agi do you have a sense
you've we've already talked about the
things we should be worried about with
ai do you have a sense of the shape of
his spheres
in particular about ai
of the which subset of what we've talked
about whether it's
creating you know it's that direction of
creating
sort of these giant computational
systems that are not explainable
they're not intelligible intelligence or
is it um is it the
and then like as a branch of that is it
the manipulation by
big corporations of that or individual
evil people
to use that for destruction or the
unintentional consequences
do you have a sense of where his
thinking is on this from
my many conversations with elon yeah i
i certainly have a model of how how he
thinks it's actually very
much like the way i think also i'll
elaborate on it a bit i just want to
push back on when you said evil people i
i don't think it's very helpful context
concept evil people yes sometimes people
do very very bad things
but they usually do it because they
think it's a good thing yes
because somehow other people had told
them that that was a good thing or given
them
incorrect information or or whatever
right um i i have
i believe in the fundamental goodness of
humanity that if we
educate people well and they find out
how things really are
people generally want to do good and be
good
hence the value alignment as yeah it's
it's
about information about knowledge and
then once we have that we'll
we'll uh we'll likely be able to
uh do good in the way that's aligned
with everybody else who thinks
yeah and it's not just the individual
people we have to align so
we we don't just want people to
be educated to know the way things
actually are and to treat each other
well
but we also need to align other
non-human entities we talked about
corporations there has to be
institutions so that what they do is
actually good for the country they're in
and
we should align do make sure that what
countries do is actually good for
the species as a whole etc uh coming
back to elon yeah
my my uh understanding of of
how elon sees this is really quite
similar to my own which is
one of the reasons i like him so much
and enjoy talking with him so much i
feel he's
quite different from most people in that
he
thinks much more than most people about
the really big picture
not just what's going to happen in the
next election cycle but
in millennia millions and billions of
years from now
right and if you when you look in this
more cosmic perspective it's so obvious
that we are gazing out into this
universe that as far as we can tell
is mostly dead with life being almost
imperceptibly tiny perturbation right
and and he sees this enormous
opportunity for our universe to come
alive
for us to become an interplanetary
species mars is obviously just
first stop on this cosmic
journey and precisely because he thinks
more long-term
it's much more clear to him than to most
people that what we do with this russian
roulette thing we keep playing with our
nukes
as a really poor strategy really
reckless strategy
and also that that we're just building
these ever more powerful ai systems that
we don't understand
it's also it's a really reckless
strategy i feel elon is a human
very much a humanist in the sense that
he wants an awesome
future for humanity he wants to be
us that control the machines
rather than the machines that control us
yes you know
and why shouldn't we insist on that
we're building them after all right
why should we build things that just
make us into some little cog in the
machinery that has no further say in the
matter right it's not my idea of
an inspiring future either yeah the if
if you think on the cosmic scale in
terms of both time and space
so much is put into perspective yeah
whenever i have a bad day that's what i
think about it immediately makes me feel
better
it makes me sad that for us individual
humans at least for now the ride ends
too quickly
we don't get to experience the cosmic
scale
yeah i mean i think of our universe
sometimes as an organism that has only
begun to wake up a tiny bit
uh just like you know when you're in the
the very first
little glimmers of consciousness you
have in the morning when you start
coming around for the coffee
before the coffee even before you get
out of bed before you even open your
eyes
you start start to wake up a little bit
there's something here you know
that's very much how i think of of where
we are you know those all those galaxies
out there you know i think they're
really beautiful but why are they
beautiful
they're beautiful because conscious
entities are actually
observing them and experiencing them
through our telescopes
if i you know i define consciousness
as subjective experience whether it be
colors or emotions or sounds
so beauty is an experience meaning is an
experience
purpose is an experience if there was no
conscious experience of
observing these galaxies they wouldn't
be beautiful if if we do something dumb
with advanced ai in the future here and
earth originating life goes extinct
and that was it for this if there is
nothing else with telescopes in our
universe then
it's kind of game over for me beauty and
meaning and purpose in our whole
universe and i think that would be just
such an opportunity lost frankly and i
think
when elon points this out he gets very
unfairly
maligned in the media for all the dumb
media bias reasons we talked about right
they print precisely the things about
elon out of context
that are really click-baity like he has
gotten so much flack
for this summoning the demon statement
yeah i happen to know exactly the
context because i was in the front row
when they gave that talk it was at mit
you'll be pleased to know it was the
aero astro
anniversary they had buzz aldrin there
from the moon landing
a whole house a kresge auditorium packed
with mit students
and he had this amazing q a it might
have gone for an hour and then we
talked about rockets and mars and
everything at the very end
this one student was actually in my
class asked him what about ai
elon makes this one comment and they
take this out of context
print it goes viral is it like with ai
we're summoning the demon something like
that
and try to cast him as some sort of doom
and gloom dude you know
yeah you know elon yes he's not the doom
and gloom
dude he he is such a positive visionary
and the whole reason he warns about this
is because he realizes more than most
what the opportunity cost is of screwing
up that there is so
much awesomeness in the future that we
can we can and our descendants can
enjoy if we don't screw up right i get
so pissed off when people try to cast
him in some sort of
technophobic luddite and then
at this point it's kind of ludicrous
when when i hear people say that people
who worry about
artificial general intelligence are
luddites because
of course if you look more closely you
have some of the most
outspoken people making warnings are
people like professor stuart russell
from berkeley who's written the
best-selling ai textbook you know
so claiming that he is a luddite who
doesn't understand ai
is the joke is really on the people who
said it but i think more broadly
this message is really not sunk in at
all
what it is that people worry about they
think that elon
and stuart russell and others are
worried about
the dancing robots picking up an
ar-15 and going on a rampage right
they think they're worried about robots
turning evil
they're not i'm not you know the risk is
not malice
it's it's competence the risk is just
that we build some systems that are
incredibly competent which means they're
always going to get
their goals accomplished even if they
clash with our goals
that's the risk why did
we humans you know drive the west
african black rhino extinct
is it because we're malicious evil
rhinoceros haters
no it's just because our goals didn't
align with the goals of those rhinos and
tough luck for the rhinos you know so
when i'm the point is just we don't want
to put ourselves in the position of
those rhinos
creating these something more powerful
than us
if we haven't first figured out how to
align the goals and i am optimistic i
think we could
do it if we worked really hard on it
because i spent a lot of time
around intelligent entities that were
more intelligent than me
my mom and my dad and i was little and
that was fine
because their goals were actually
aligned with mind quite well
but uh we've seen today many examples of
where the goals of our powerful systems
are not so aligned so those
click through optimization algorithms
that are polarized social media right
they were actually
pretty poorly aligned with what was good
for democracy it turned out
and um again almost all problems we've
had in machine learning again came
so far not from malice but from poor
alignment and it's
that's exactly why that's why we should
be concerned about it in the future do
you think it's possible that uh
with with systems like neurolink and
brain computer interfaces
you know again thinking of the cosmic
scale elon's talked about this
but others have as well throughout
history of
figuring out how the exact mechanism of
how to achieve that kind of alignment
so one of them is having a symbiosis
with ai
which is like coming up with clever ways
where we're like
stuck together in this weird uh
relationship
whether it's biological or in some kind
of other way
do you think there's that's a
possibility of
having that kind of symbiosis or do we
want to instead kind of focus on
this uh distinct entities
of us humans talking to these
intelligible
self-doubting ais maybe like stuart
russell thinks about he's like
these these we're we're self-doubting
and full of uncertainty and then have
rai systems that are full on certainty
we communicate back and forth
and in that way achieve symbiosis
i honestly don't know i would say that
because we don't know
for sure what if any of our which of any
of our ideas will work
but we do know that if we don't i'm
pretty convinced that if we don't get
any of these things to work and just
barge ahead then our species is you know
probably going to go extinct this
century
i think it this century you think like
you think we're facing
this crisis as a 21st century christ
like
this century would be remembered
on a hard drive and a hard drive
somewhere or
or maybe by future generations is like
uh like there will be
future future life institute awards for
people that have done something about ai
it could also end even worse where there
isn't we're not superseded by leaving
any ai behind either way we just
totally wipe out you know like on easter
island our century is long
no it's there are still 79 years left of
it right think about how
far we've come just in the last 30 years
so
we can talk more about what might go
wrong but you asked me this really good
question about what's the best strategy
is it neuralink
russell's approach or or whatever i
think um
like you know when when we did the
manhattan project
we didn't know if any of our four ideas
for enriching uranium
and getting out the uranium uranium-235
were going to work but we felt this was
really important
to get it before hitler did so you know
what we did we tried all four of them
here i think it's it's analogous where
there's the greatest threat that's ever
faced our species and of course u.s
national security by implication
we don't know if we don't have any
method that's guaranteed to work but we
have a lot of ideas
so we should invest pretty heavily in
pursuing all of them with an open mind
and hope that one of them at least works
these are
the good news is the century is long you
know
and it might take decades until we have
artificial general intelligence so we
have some time hopefully
but it takes a long time als to solve
these very very difficult problems
it's going to actually be it's the most
difficult problem we're ever trying to
solve as a species
so we have to start now so we don't have
rather than you know begin thinking
about it the night before
some people have had too much red bull
switching on and we have to
coming back to your question we have to
pursue all of these different avenues
and see
if you're my investment advisor and i
was trying to invest
in the future how do you think the human
species
is most likely to destroy itself in
this century
yeah in uh so if if the crisis many of
the crisis we're facing are really
before us within the next hundred years
how do we make explicit the
make known the the unknowns and solve
those problems
to to avoid the biggest starting with
the biggest
existential crisis so as your investment
advisor how are you planning to make
money
destroying yourselves i have to ask i
don't know it's uh it might be the
russian origins
somehow involved at the micro level of
detailed strategies of course this is
these are unsolved problems
for ai alignment we can break it into
three sub-problems
that are all unsolved i think with you
know you want first to make machines
understand our goals then adopt our
goals
and then retain our goals so
to hit on all three real quickly
the problem when andreas lubitz told his
autopilot to fly into the alps
was that the computer didn't even
understand
anything about his goals right it was
too dumb it could have understood
actually
but we would have had to put some effort
in as a
system designer that don't fly into
mountains
so that's the first challenge how do you
how do you program into computers
human values human goals we can start
rather than than saying oh it's so hard
we should start with the simple stuff
as i said self-driving cars airplanes
just put in all the goals that we all
agree on already
and then have a habit of whenever
machines get smarter so they can
understand one level higher goals you
know
put them into uh the second challenge is
uh
getting them to adopt the goals it's
easy for situations like that where you
just program it in but when you have
self-learning systems like children you
know
any parent knows that um
there's a difference between getting our
kids to understand what we want them to
do and to actually
adopt our goals right with with humans
with
children fortunately they
they go through this phrase first
they're too dumb to understand what we
want
our goals are and then they have this
period of
some years when they're both smart
enough to understand them and malleable
enough that we have a chance to
raise them well and then they become
teenagers it's
kind of too late but we have this window
with machines the challenges they might
the intelligence might grow so fast that
that window is pretty short
so that's a that's the research problem
the third one is how do you make sure
they keep the goals
if they keep learning more and getting
smarter
many sci-fi movies are about how you
have something which initially was a
line but then
things kind of go off keel and you know
my kids
were very very excited about their legos
when they were little
now they're just gathering dust in the
basement you know if we put
if we create machines that are really on
board with the goal of taking care of
humanity we don't want them to get as
bored with us and
and as my kids go with legos
so this is another research challenge
how can you make some sort of
recursively self-improving system retain
certain basic goals that said a lot of
adult people still play with legos so
maybe we succeeded with the legos
i like your optimism so not all ai
systems have to maintain the goals right
some
just some fraction yeah so there so
there is a there's a lot of
talented ai researchers now who have
heard of this and want to work on it
not so much funding for it yet
of the billions that go into building a
ai more powerful
it's only a minuscule fraction so far
going into the safety research
my attitude is generally we should not
try to slow down the technology but we
should greatly accelerate the investment
in this sort of safety research
um and also make sure it's been it's
this was very embarrassing last year but
you know the nsf decided to give out the
six of these big institutes we got one
of them for ai and science
you asked me about another one was
supposed to be for ai safety research
and they gave it people studying oceans
and climate and stuff
yeah so i'm all for starting oceans and
climates but we need to actually
have some money that actually goes into
ai safety research also and doesn't just
get grabbed by whatever
um that's a fantastic investment and
then at the higher level
you ask this question okay what can we
do you know what are the biggest risks
i think i think we cannot just consider
this to be only a technical problem
like again because if you solve only the
technical problem
can i play with your robot yes please if
we just
get our machines you know to just
blindly obey
the orders we give them so we can always
trust that it will do what we want
that might be great for the owner of the
robot that might not be so great for the
rest of humanity
if if that person is that least favorite
world leader
or whatever you imagine right so we have
to also take
look at the apply alignment not just to
machines
but to all the other powerful structures
that's why it's so important to
strengthen our democracy again as i said
to have institutions
make sure that the playing field is not
rigged so that
corporations are given the right
incentives to do the things
that both make profit and are good for
people to make sure that countries have
incentives to do things
that are both good for their people and
don't screw up the rest of the world
and this is not just something for ai
nerds you know to geek out on this is
the
interesting challenge for political
scientists economists and
so many other thinkers so one of the
magical things
that uh perhaps makes him
this earth quite unique is that
it's home to conscious beings so you
mentioned consciousness
perhaps as a small aside
because we didn't really get specific to
what how we might
do the alignment like you said it's just
a really important research problem but
yeah
do you think engineering consciousness
into ai systems
is is a a possibility
it's something that we might to one day
do or is there something
fundamental to consciousness that is uh
is there something about consciousness
that is fundamental to
humans and humans only i think it's
possible
i think both consciousness and
intelligence
are information processing certain types
of information processing
and that fundamentally it doesn't matter
whether the information is processed by
carbon atoms in neurons and brains or by
silicon atoms and
and so on in our technology
some people disagree this is what i
think is this physicist that i i
and that consciousness is the same kind
of
you said consciousness is information
processing so meaning
you know i i think you had a quote of
something like
it's information uh
knowing itself that kind of thing i
think consciousness is
the way information feels when it's
being processed what's being done
we don't know exactly what those complex
ways are it's clear that
most of the information processing in
our brains does not create an experience
we're not
even aware of it right like for example
um
you're not aware of your heartbeat
regulation right now even though it's
clearly being done by your body
right it's just kind of doing its own
thing when you go jogging
you there's a lot of complicated stuff
about how you put your foot down and we
know it's hard that's why robots used to
fall over so much
but you're mostly unaware about it your
brain just
your ceo consciousness module just sends
an email hey you know i'm going to keep
drawing along this path yeah the rest is
on autopilot right
so most of it is not conscious but
somehow
there are some of the information
processing which is
we don't know what what exactly
i i think this is a science problem that
i hope one day
we'll have some equation for or
something so we can be able to build a
consciousness detector and say yeah here
there is some consciousness here there's
not
oh don't boil that lobster because it's
feeling pain or
it's okay because it's not feeling pain
right now we treat this as sort of just
metaphysics
but it would be very useful in emergency
rooms to know if a patient
has locked in syndrome and is conscious
or if they are
actually just out and in the future if
you build a very very intelligent
helper robot to take care of you you
know i think you'd like to know
if you should feel guilty by shutting it
down or if
or if it's just like a zombie going
through the motions like a fancy tape
recorder right
and and once we can make progress on the
science of consciousness and
figure out what is conscious and what
isn't
then um we
assuming we want to create positive
experiences and not
suffering we'll probably choose to build
some machines that are deliberately
unconscious
that do cr you know incredibly
boring repetitive jobs in a iron mine
somewhere or whatever
and maybe we'll choose to create helper
robots for the elderly that are
conscious so
so that people don't just feel creeped
out that the you know the robot is just
faking it
when it acts like it's sad or happy like
i said elderly i think everybody gets
pretty deeply lonely in this world
and uh so there's a place i think for
everybody to have a connection with
conscious beings whether they're human
or otherwise but i know for sure that i
would
if i had a robot if i was going to
develop any kind of personal
emotional connection with it i would be
very creeped out if i knew it in an
intellectual level that the whole thing
was just the fraud
now today you can buy a little talking
doll
for it for a kid which will say things
and the little child will often think
that this is actually conscious
yes and even real secrets to it that
then go on the internet and
with all sorts of creepy repercussions
uh you know
i would not want to be just hacked and
tricked
like this if i was going to be
developing real emotional connections
with
the robot i would want to know that this
is actually real
it's acting conscious acting happy
because it actually feels it
and i i think this is not sci-fi i think
it's possible to measure to come up with
tools and make after we understand the
science of consciousness
you're saying is we'll be able to come
up with tools that can measure
consciousness
and definitively say like this thing is
experiencing
the things it says it's experiencing
kind of by definition if it is a
physical phenomena
information processing that and we know
that some information processing is
conscious and some isn't well then there
is something there to be discovered with
the methods of science
giulia tononi has stuck his neck out the
farthest and written down some equations
for a theory maybe that's right maybe
it's wrong
we certainly don't know but i i applaud
that kind of efforts to
sort of take this say this say this is
not just something
that philosophers can have beer and muse
about but something we can
measure we can study and coming being
that back to us
i think what we would probably choose to
do as i said is if we cannot figure this
out
choose to make to be quite mindful about
what sort of consciousness
if any we put in in different machines
that we have
we um and certainly
we wouldn't want to make them we should
not be making much machines that
suffer without us even knowing it right
and if
if at any point someone decides to
upload themselves
like ray kurzweil wants to do i don't
know if you've had him on your show do
we agree but then covet happens
so we're waiting it out a little bit you
know suppose he uploads himself into
this
robo-ray and it talks like him
and acts like him and laughs like him
and before he powers off his biological
body
he would probably be pretty disturbed if
he realized that there's no one home
this robot is not having any subjective
experience
right if we reply if humanity gets
replaced
by by uh wrote by machine descendants
which do all these cool things and build
spaceships and go to
intergalactic rock concerts and it
turned out
turns out that they are all unconscious
uh just going through the motions
wouldn't that be like the ultimate robo
zombie apocalypse
right just a play for empty benches yeah
i have a sense that there's some
kind of once we understand consciousness
better we'll understand that there's
some kind of continuum
and it would be a greater appreciation
and we'll probably understand
just like you said it'd be unfortunate
if it's the trick
we'll probably definitely understand
that love is indeed a trick
that will play on each other that we
humans are
we convince ourselves we're conscious
but we're really um
you know awesome trees and dolphins are
all the same kind of caution can i try
to cheer you up a little bit with the
philosophical thought here about the
last part
yes let's do it you know you might say
okay yeah love is just a
collaboration enabler okay and then
you'll
and then maybe you can go and get
depressed about that but i
i think that would be the wrong
conclusion actually you know like i know
that the only reason i enjoy food is
because
my genes hacked me and they don't want
me to starve to death
not because they care about me it's
consciously enjoying succulent
delights of pistachio ice cream but they
just they just want me to make copies of
them
the whole thing so in a sense the whole
the whole enjoyment of food is also a
scam
like this but does that mean i shouldn't
take pleasure in this pistachio ice
cream
i love pistachio ice cream and i can
tell you i
i have i know this is an experimental
fact i enjoy pistachio ice cream
every bit as much even though i
scientifically know exactly
why what kind of scam this was your
genes
really appreciate that you like the
pistachio ice cream well but i my mind
appreciates it too you know and
i have a conscious experience right now
ultimately all of my brain
is also just something the genes built
to copy themselves but so what
you know i'm grateful that yeah thanks
jeans for doing this but you know
now it's my brain that's in charge here
and i'm gonna enjoy my conscious
experience thank you very much and not
just pistachio ice cream
but also the love i feel for my amazing
wife
and all the other delights of being
conscious i don't
actually richard feynman i think said
this
so well he he is also the guy you know
really got me into physics
some art friend said that oh science
kind of just
is the party pooper it kind of ruins the
fun right
when like you have a beautiful flowers
as the artist
and then the scientist is going to
deconstruct that into just a blob of
quarks and electrons and
and feynman just pushed back on that in
such a beautiful way
which i think also can be used to push
back and make you pre
not feel guilty about falling in love so
so here's what feynman basically said
he said his friend you know yeah i can
also
as a scientist see that this is a
beautiful flower thank you very much
maybe i'm
i can't draw as good a painting as you
because i'm not as talented an artist
but yeah
i can really see the beauty in it and it
just it also looks beautiful to me
but in addition to that fyman said as a
scientist i see even more beauty
that the artist did not see right
suppose this is the
a flower on a blossoming apple tree you
could say this tree
has more beauty in it than just the
fruit the colors and the fragrance
this tree is made of air if i'm in roads
this is one of my favorite flying quotes
ever
and it took the carbon out of the air
and bound it in using the flaming heat
of the sun you know to turn the air into
a tree
and when you burn logs in your fireplace
it's really beautiful to think that this
is being reversed now the
tree is going the wood is going back in
the air and this flaming beautiful
dance of the fire that the artist can
see is the flaming light of the sun that
was
bound in to turn the air into a tree and
then the ashes
is the little residue that didn't come
from the air that the tree sucked out of
the ground you know
feynman said these are beautiful things
and science just
adds it doesn't subtract and i i
feel exactly that way about love and
about pistachio ice cream also
i can understand that it even there is
even more nuance to the whole thing yeah
right
at this very visceral level you can fall
in love just as much as someone who
knows nothing about neuroscience
but you can also approach appreciate
this even greater beauty
in it just like isn't it remarkable that
it came about from from this completely
lifeless universe just a bunch of
a hot blob of plasma expanding
and then over the eons you know
gradually
first the strong nuclear force decided
to combine
quarks together into nuclei and then the
electric force bound in
electrons and made atoms and then they
clustered from gravity and you got
planets and stars and
this and that then natural selection
came along and
and the genes had their little thing and
you started getting what
went from seeming like a completely
pointless universe so we're just trying
to increase entropy and
approach heat death into something that
looked more goal-oriented
isn't that kind of beautiful and then
this goal-orientedness
through evolution got ever more
sophisticated where you got ever more
and then you started getting this thing
which is kind of like deepmind's um
mu0 on steroids self the ultimate self
play
is not what what deepmind's ai does
against itself to
get better at go it's what all these
little cork blobs did against each other
in the in the game of of survival of the
fittest
you know when you had really dumb
bacteria living in a simple environment
there wasn't much incentive to get in
intelligent but then
the the life made environment more more
complex
and then there was more incentive to get
even smarter and
and that gave the other organisms more
incentive to also get smarter and then
here we are now just like like mew zero
learned to become world club master at
at go and chess from playing as itself
by just playing against itself all the
quarks here on our planet the electrons
have created giraffes and elephants
and humans pistachio ice cream i just
find that
really beautiful and i i me that just
adds to the enjoyment of
of love it doesn't subtract anything do
you feel a little more
i feel way better that was that was
incredible
so the this self play of quarks
taking back to the beginning of our
conversation a little bit you've
there's so many exciting possibilities
about artificial intelligence
understanding
the basic laws of physics do you think
ai will help us unlock there's been a
quite a bit of excitement throughout the
history of physics of
coming up with more and more general
simple laws that explain the nature of
our reality
and then the ultimate of that would be a
theory of everything that combines
everything together
do you think it's possible that well one
we humans
but perhaps ai systems will figure out
a theory of physics that unifies
all the laws of physics yeah i think
it's absolutely absolutely possible
i think it's it's very clear that we're
going to see a great boost to science
we're already seeing a boost actually
from from machine learning helping
science
alpha fold was an example you know the
decades-old protein folding problem
so and gradually yeah unless we go
extinct by doing something dumb like we
discussed i think
it's um
very likely that our understanding of
physics will become so good
that uh our technology will no longer
be limited by human intelligence but
instead be limited by the laws of
physics
so our tech today is limited by what
we've been able to invent right
i think as i ai progresses it'll just be
limited by the speed of light
and other physical limits which would
mean
it's going to be just dramatically
beyond you know where we are now
do you think it's a fundamentally
mathematical pursuit
of trying to understand like the laws of
this
that govern our universe from a
mathematical perspective so almost like
if it's ai it's exploring the space of
like theorems and those those kinds of
things or is there some other
is is there some other more
computational ideas
more sort of empirical ideas they're
both i would say
it's really interesting to look out at
the landscape of everything we call
science today
so here we come out with this big new
hammer it says machine learning on it
and ask you know
where are there some nails that you can
help with here that you can hammer
ultimately if machine learning gets the
point that it can do everything
better than us it will be able to help
across the whole space of
of science but maybe we can anchor it by
starting a little bit right now near
term
and see how we kind of move forward so
so like right now
first of all you have a lot of big data
science
where for example with telescopes we are
able to collect way more data
every hour than a grad student can just
pour over like in the old times right
and machine learning is already being
used very effectively even at mit i
define planets around other stars to
detect
exciting new signatures of new particle
physics in the sky and
to detect the ripples and the fabric of
space
time that we call gravitational waves
caused by enormous black holes crashing
into each other halfway across the
observable universe machine learning is
running and thinking
right now you know doing all these
things and it's really helping
all these experimental fields there is a
separate front of physics
computational physics which is getting
an enormous boost also
so um we had to do all our computations
by hand right
people would have these giant books with
tables of logarithms and oh my god
i just pains me to even think how
long time it would have taken to do
simple stuff then we started to get
little calculators and computers that
could do some basic math for us
now what we're starting to see is
kind of a shift from gophi
computational physics to
neural network computational physics
what i mean by that is
most computational physics would be done
by humans programming in the
intelligence of how to do the
computation into the computer
just as when gary kasparov got his
posterior kicked by ibm's deep blue in
chess
humans had programmed in exactly how to
play chess intelligence came from the
humans it wasn't learned right
mu zero can be not only
kasparov in chess but also fish which is
the best
sort of go-fi chess program by learning
and we're seeing more of that now that
shift beginning to happen in physics so
let me give you
an example so lattice qcd is an area of
physics whose goal is basically to take
the periodic table
and just compute the whole thing from
first principles
this is not the search for theory of
everything we already know the theory
that's supposed to produce as output the
periodic table
which atoms are stable how heavy they
are all that good stuff
their spectral lines it's called
it's a theory lattice qcd you can put it
on your t-shirt
or colleague frank wilcheck got the
nobel prize for working on it
but the math is just too hard for us to
solve we have not been able to start
with these equations and solve them to
the extent that we can predict oh yeah
and then there is carbon and this is
what the spectrum of the carbon atom
looks like
but uh awesome people are building these
super computer simulations
where you just put in these equations
and you
you make a big cubic lattice of space
or actually it's very small lattice
because you're going to the sub-atomic
scale down to the subatomic scale
and you're trying to solve this but it's
just so computationally expensive
that we still haven't been able to
calculate things as accurately as we
measure them in many cases
and now machine learning is really
revolutionizing this so my colleague
fiola shanahan and mit for example she's
been using this really cool machine
learning technique
called normalizing flows where she's
realized she can actually speed up the
calculation
dramatically by having the ai learn how
to do things faster
another area like this where
where we suck up an enormous amount of
supercomputer time to do physics
as black hole collisions so now that
we've done the sexy stuff of detecting
a bunch of this with ligo and other
experiments we want to be able to
know what we're seeing and so it's a
very simple
conceptual problem it's the two body
problem
newton solved it for classical gravity
hundreds of years ago but the two body
problem is still not
fully solved for black holes black
copper yes and nice nice gravity because
they won't disorbit each other forever
anymore two things they give off
gravitational waves
and make sure they crash into each other
and the game what you want to do is you
want to figure out okay
what kind of wave comes out as a
function of of the masses of the two
black holes
as a function of how they're spinning
relative to each other
etc and that is so hard it can take
months of super computer time on
massive numbers of cores to do it you
know
wouldn't it be great if you can use
machine learning
to greatly speed that up right um
now you can use the expensive old gofi
calculation as the truth and then see if
machine learning can figure out a
smarter faster way of getting the right
answer
yet another area like of computational
physics
these are probably the big three that
suck up the most computer
time lattice qcd black hole collisions
and cosmological simulations where you
take not a subatomic
thing and try to figure out the mass of
the proton but you
take something with it's enormous and
try to look at how all the galaxies get
formed in there oh wow
yeah there again there are a lot of very
cool ideas right now about how you can
use machine learning to
do this sort of better stuff better
the difference between this and the big
data is you kind of make the data
yourself
right so and then finally
we're looking over the physics landscape
and seeing what can we hammer with
machine learning right so we talked
about
experimental data big data discovering
cool stuff
that we humans then look more closely at
then we talked about taking the
expensive computations we're doing now
and figuring out how to do them much
faster and better
with ai and finally let's go really
theoretical
like so things like discovering
equations
having deep fundamental insights
this comes this is something
closest to what i've been doing in my
group we talked earlier about the whole
ai fineman project where if you just
have some data
how do you automatically discover
equations that
seem to describe this well that you can
then go back as a human and
then work with and test and explore and
uh you asked a really good question also
about if this is sort of a search
problem in some sense that's very deep
actually what you said because it is
suppose i ask you to prove some
mathematical theorem
what is a proof in math it's just a long
string of steps
logical steps that you can write out
with symbols
and once you find it it's very easy to
write a program to check whether it's a
valid
proof or not so why is it so hard to
prove it then well
because there are ridiculously many
possible candidate proofs you could
write down right
if this if the proof contains 10 000
symbols
even if there were only 10 options for
what each symbol could be
that's 10 to the power 1000 possible
proofs
which is way more than there are atoms
in our universe right so
you could say it's trivial to prove
these things you just write a computer
generate all strings and then check is
this the valid proof
no is this a valid proof no
and then you just keep doing this
forever
but there are a lot of but it is
fundamentally a search problem you just
want to search the space of all those
all strings
of symbols to find the one with find one
that is the proof
right and there's a whole area of
machine learning
called search how do you search with
some giant space to find the needle in
the haystack
it's easier in cases where there's a
clear measure of
of good like you're not just right or
wrong but this is better
and this is worse you can maybe get some
hints as to which direction to go in
that's why we talked about neural
networks work so well
i mean that's such a human thing of that
moment of genius of figuring out
the intuition of of good
essentially i mean we thought that that
is it
maybe it's not right we thought that
about chess right that exactly
that the ability to see like 10 15
sometimes 20 steps ahead was not a
calculation that humans were performing
it was some kind of weird intuition
about different patterns about board
positions about the relative positions
exactly
somehow stitching stuff together and a
lot of it is just like intuition
but then you you have like alpha i guess
zero be the first one that did uh
the self-play it just came up with this
it's it was able to learn to self-play
mechanism this kind of intuition
exactly but just like you said it's so
fascinating to think
well they're in the space of totally new
ideas can that be done
in developing theorems we know it can be
done by neural networks because we did
it with the neural networks in the
cranium those are the great
mathematicians of our
of humanity right and and i'm so glad
you brought up alpha zero because that's
the counter example
it turned out we were flattering
ourselves when we said
intuition is something different it's
only humans can do it it's not
information processing
if you if it used to be that way
again it's very it's really instructive
i think to compare
the chess computer deep blue that beat
kasparov
with alpha zero that beat lisa doll at
the go
because for deep blue there was no
intuition
there was some pro humans had programmed
in some intuition
after humans had played a lot of games
they told the computer you know
count the pawn as one point the bishop
is three points
rook is five points and so on you add it
all up and then you add some extra
points for past pawns and
subtract if the opponent has it and and
blah blah blah blah
and then what what deep blue did was
just search
just very brute force tried many many
moves ahead all these combinations and
the pruned research
and it could think much faster than
kasparov and it won
right and that i think
inflated our egos in a way it shouldn't
have because people started to say
yeah yeah it's just brute force search
but has no intuition yeah
alpha zero really
popped our bubble there yeah because
what alpha zero does yes it does also do
some of that
research but it also has this intuition
module
which in geek speak is called a value
function where it just looks at the
board
and comes up with a number for how good
is that position
the difference was no human told it how
good the position
is it just learned it
and mu zero is the is the coolest or
scariest
of all depending on your mood because if
the same
basic ai system will learn
what the good board position is
regardless of whether it's chess
or go or shogi or pac-man
or lady pac-man or breakout or space
invaders or any number a bunch of other
games you don't tell anything and it
gets this intuition
after a while for what's good so this is
very hopeful for science i think
because if if it can get intuition
what's a good position there
maybe it can also get intuition for what
are some good directions to go if you're
trying to prove something
if i i often one of the most fun things
in my science career is when i've been
able to prove some theorem about
something and
it's very heavily guided of course i
don't sit and try all random strings i
have a hunch that you know
this reminds me a little bit about this
other proof i've seen
for this thing so maybe i first what if
i try this
nah that didn't work out but but this
reminds me actually the way this
failed reminds me of that and so so
combining the intuition
that with all these brute force
capabilities i think i think it's going
to be able to help physics too
do you think that there will be a day
when an ai system
being the primary contributor let's say
plus wins the nobel prize in physics
obviously they'll give it to the humans
because we humans don't like to give
prizes to machines
it'll give it'll give it to the humans
behind the system you could argue that
ai has already been involved in some
nobel prizes probably maybe something
with black holes and stuff like that
yeah we don't like giving prizes to
other life forms
if someone wins a horse racing contest
they don't give the price the horse
either it's true
uh but do you think that's we might be
able to see something like that in our
lifetimes when
ai so like the first system i would say
that
makes us think about a nobel prize
seriously
is like alpha fold is making us think
about
in medicine physiology a nobel prize
perhaps discoveries that are direct
result of something that's discovered by
alpha fall do you think in physics
we might be able to see that in our
lifetimes i think
what's probably going to happen is more
of a blurring of the distinctions
so today
if somebody uses a computer to do a
computation that gives them the normal
price nobody's going to dream of giving
the price to the computer they're going
to be like that was just a tool
i i think for these things also people
are just going to
for a long time view the computer as a
tool but what's going to
change is that as the ubiqui the
ubiquity of
of machine learning i think
at some point in my lifetime finding a
human physicist who knows nothing about
machine learning is going to be about
almost as hard as
it is today finding a human physicist
who doesn't says oh i don't know
anything about computers
right or i don't use math it would just
be a ridiculous concept
you see but the thing is there is a
magic moment though
like with alpha zero when the system
surprises us in a way where the best
people in the world
truly learn something from the system in
a way where
you feel like it's another entity yeah
like the way people
the way magnus carlsen the way certain
people are looking at the work of alpha
zero
it's like uh it it truly is no longer a
tool
in the in the sense that it doesn't feel
like a tool
it feels like some other entity so
there's a magic difference like where
where you're like uh you know if an ai
system is able to come up with an
insight
that surprises everybody in a some
uh in some like major way that's a phase
shift
in our understanding of some particular
science or
some particular aspect to physics i feel
like that
is no longer a tool and then you can
start to say
uh that like it perhaps deserves the
prize
so for sure the more important the more
fundamental
transformation of the 21st century
science is exactly what you're saying
which is
probably everybody will be doing machine
learning
it's to some degree like if you want to
be successful
at uh unlocking the mysteries of science
you should be doing machine learning but
it's just exciting to think about like
whether there will be one that comes
along that's super surprising
and uh they'll make us question
like who the real inventors are in this
world yeah
yeah i think the question of isn't if
it's going to happen but when
and and but it's important in my mind
the time when that happens is also more
or less the same time
when we get artificial general
intelligence yes and then we have a lot
bigger things to worry about than
whether they should get the nobel prize
or not right
yeah because when you have
machines that can outperform our best
scientists
at science they can probably outperform
us
at a lot of other stuff as well
which can at a minimum you know make
them incredibly powerful agents in in
the world you know
and i think it's a mistake
to think we only have to start worrying
about
loss of control when machines get to agi
across the board where they can do
everything all our jobs long before that
they'll be hugely influential we talked
at length
about how the hacking of our minds
with um
algorithms trying to get us glued to our
screens right has already had a big
impact on on
society that was an incredibly dumb
algorithm in the grand scheme of things
right
just supervised machine learning yet
that had had huge impact
so so um i just don't want us to be
lulled into false sense of security and
think there won't be any societal impact
yeah until things reach human level
because it's happening already and it's
i was just thinking the other week
you know when i see some scaremonger
going oh the robots are coming the
implication
is always that they're coming to kill us
yeah and maybe you should have worried
about that if you were in
nagorno-karabakh
during the recent war there but more
seriously
the robots are coming right now but
they're mainly not coming to kill us
they're coming to hack us
[Music]
they're they're coming to hack our minds
into buying things that we maybe we
didn't need
to vote for people who may not have our
best interest in mind
and and it's kind of humbling i think
actually
as a human being to admit that it turns
out that our minds are actually much
hacked more hackable than we thought
and the ultimate insult is that we are
actually getting hacked by
the machine learning algorithms that are
in some objective sense much dumber than
us
you know but maybe we shouldn't be so
surprised because you know
how do you feel about the cute puppies
love them
so you know you would probably argue
that in some
across-the-board measure you're more
intelligent than they are but boy are
our cute puppies good at hacking us
right yeah they move into our house
persuade us to feed them and do all
these things what do they ever do but
for us
yeah other than being cute and making us
feel good right
so if puppies can hack us maybe we
shouldn't be so surprised if
a pretty dumb machine learning
algorithms can hack us too not to speak
of cats which is another level
and and i think we should to counter
your previous point about there
let us not think about evil creatures in
this world we can all agree that cats
are as close to objective evil as we can
get
but that's just me saying that okay so
uh you have you seen the
the cartoon i think it's maybe the onion
um with this incredibly cute kitten and
just says um
with underneath something that thinks
about murder all day
exactly that's that's accurate uh you've
mentioned offline that there might be a
link between
post biological agi and seti so last
time we
talked um you've you've talked about
this intuition that
we humans might be quite unique
in our galactic neighborhood perhaps our
galaxy
perhaps the entirety of the observable
universe we might be the only
intelligence civilization here which is
um
and you argue pretty well for that
thought
um so i have a few little questions
around this
one the scientific question
in which way would you be
if you were wrong in that intuition
in which way do you think you would be
surprised like
why were you wrong if we find out that
you ended up being wrong
like in which dimensions so like is it
because we can't
see them is it because the nature of
their
intelligence or the nature of their life
is
totally different than we can possibly
imagine
is it uh because the
i mean something about the great filters
and surviving them
or maybe because we're being protected
from signals so all those explanations
for
um for why we haven't
heard a big loud like red
light that says yeah we're here yeah
so there are actually two separate
things there that i could be wrong about
two separate claims that i made right
not them
one one of them is i made the claim i
think
most civilizations
when you're going from simple bacteria
like things to space
space colonizing civilizations they
spend
only a very very tiny fraction of their
other
of their life being where we are uh that
i could be wrong about
the other one i could be wrong about is
a quite different statement that i think
that actually
i'm guessing that we are the only
civilization in our observable universe
from which light has reached us so far
that that's actually
gotten far enough to invent telescopes
so let's talk about maybe both of them
in turn because they really are
different
the first one if if we look at
the n equals one the data point we have
on this planet
right so we spent um four and a half
billion years fussing around on this
planet with life right we got
and most of it it was pretty lame stuff
from an intelligence perspective
you know it's bacteria and then
the dinosaur is spent then but things
gradually accelerated
right then the dinosaurs spent over 100
million years stomping around here
without even inventing smartphones
and um and then very recently
you know it's only we've only spent 400
years going from newton
to us right yeah in terms of technology
and
we've look what we've done even you know
when i was a little kid there was no
internet even
so it's i think it's pretty likely for
in this
case of this planet right that we're
either gonna
really get our act together and
start spreading life into space the
century and doing all sorts of great
things or where you're gonna
gonna wipe out um it's a little hard
if i i couldn't be wrong in the sense
that maybe what happened on this
earth is very atypical and for some
reason what's more common on other
planets is that
they spend an enormously long time
futzing around with the ham radio and
things but they just never really
take it to the next level for reasons i
don't haven't understood
i'm humble and open to that but i would
bet
at least ten to one that our situation
is more typical
because the whole thing with moore's law
and accelerating technology it's pretty
obvious why it's happening
everything that grows exponentially we
call it an explosion whether it's a
population explosion or a nuclear
explosion it's always caused by the same
thing it's that
the next step triggers a step after that
so i
we tomorrow's technology today's
technology enables tomorrow's technology
and that enables
the next level and
as i think because the technology is
always better of course the steps can
come faster and faster
on the other question that i might be
wrong about that's the much more
controversial one i think
but before we close out on this thing
about if
the first one if it's true that most
civilizations spend only a very short
amount of their total time in the stage
say between um
inventing um telescopes or like
mastering electricity and leaving their
and doing space travel
if that's actually generally true but
then that should apply also elsewhere
out there so we we should be very very
we should
be very very surprised if we had find
some random civilization and we happen
to catch them exactly in that very very
short stage
it's much more likely that we find this
planet full of bacteria
yes or that we find some civilization
that's already post-biological and has
done some really cool
galactic construction projects in their
in their galaxy would we be able to
recognize them do you think is it
possible
that we just can't i mean this post
biological world
could it be just existing in some other
dimension it could it
could be just all a virtual reality game
for them or something i don't know that
that it changes completely we won't be
able to detect
we have to be honestly very humble about
this yeah i think that i said i think i
said earlier the number one
principle of being a scientist is you
have to be humble and willing to
acknowledge that everything we think
guess might be totally wrong
of course you can imagine some
civilization where they all decide to
become buddhists and very inward looking
and just move into their little virtual
reality and not disturb the
flora and fauna around them and we might
not notice them
but this is a numbers game right if you
have millions of civilizations out there
or billions of them
all it takes is one with a more
ambitious mentality
right that decides hey we are going to
go out and
settle a bunch of other solar systems
and maybe galaxies
and then it doesn't matter if they're a
bunch of quiet buddhists we're still
going to notice
yeah that expansionist one right and it
seems like
quite the stretch to assume that you
know we know even in our own galaxy that
there are
probably a billion or more planets that
are pretty earth-like
and many of them were formed over a
billion years before ours
so had a big head start so if you
actually assume also that life happens
kind of
automatically on an earth-like planet
i think it's it's pretty quite the
stretch to then go and say okay so
we are billions of another billion
civilizations out there that also have
our level of tech
and they all decided to become buddhists
and not a single one
decided to go like go hitler on the
galaxy and say we need
to go on and colonize or and or not and
not a single one decided for more
benevolent reasons to go out and get
more resources
that that seems seems like a bit of a
stretch frankly and this leads into the
the second thing you challenged me to be
that i might be wrong about
how rare or common is life you know
so francis drake when he wrote down the
drake equation multiplied together a
huge number of factors
and then we don't know any of them so we
know even less about
what you get when you multiply together
the whole product yeah
since then a lot of those factors have
become much better known
one of his big uncertainties was how
common is it that a solar system even
has a planet right
well now we know it's very common
earth-like planets we know
we have better diamonds many many of
them even in our galaxy
at the same time you know we have thanks
to i'm a big supporter of the seti
project
and its cousins and i think we should
keep doing this and we've learned a lot
we've we've learned that so far all we
have is still
unconvincing hints nothing more right
and and there are certainly many
scenarios where it would be
dead obvious if there were
100 million other human-like
civilizations in our galaxy
it would not be that hard to notice some
of them with today's technology and we
haven't right so
so what we can what we can say is well
okay
we can rule out that there is a human
level of civilization on the moon
and in fact in many nearby solar systems
where we
we cannot rule out of course that there
is something like earth
sitting in a galaxy 5 billion light
years away
but we've ruled out a lot and that's
already kind of shocking
given that they're all these planets
there you know so like where are they
where are they all that's the that's the
classic fermi paradox
yeah and and um so my argument which
might very well be wrong
it's very simple really it just goes
like this okay we have no clue about
this
it could be the the the probability of
getting life on a random planet it could
be
10 to the minus one a priori or ten to
the minus five
ten ten to the minus twenty ten to minus
thirty ten to the minus
forty basically every order of magnitude
is about equally likely
when you then do the math and ask how
close is our nearest neighbor
it's again equally likely that it's 10
to the 10 meters away 10 to 20 meters
away 10 to 30 meters away
we can we have some nerdy ways of
talking about this with bayesian
statistics and a uniform log prior but
that's irrelevant
this is the simple basic argument and
now comes the data so we can say okay
how many were there all these orders of
magnitude 10 to the 26 meters away
there's the edge of our observable
universe if it's farther than that light
hasn't even reached us yet
if it's uh less than 10 to the 16 meters
away well it's
within an earth survey it's no farther
away than the sun we can definitely rule
that out you know
um so i think about it like this a
priori before we look to telescopes you
know
it could be 10 10 meters 10 to 20 10 to
the 30 to the 40th and 50th and
equally likely anywhere here yeah uh
and now we've ruled out like this chunk
yeah and most of it is outside and here
is the edge of our observable universe
yes yep so
i'm certainly not saying i don't think
there's any life elsewhere in space
if space is infinite then you're
basically 100 guaranteed that there is
but the probability that there is life
that the nearest neighbor it happens to
be in this little region between
where we would have seen it already yeah
and where we will never see it there's
actually
significantly less than one i think and
and
i think there's a moral lesson from this
which is really important
which is to be good stewards
of this planet and this shot we've had
you know it can be very dangerous to say
oh you know it's fine if we nuke our
planet or ruin the climate
or mess it up with an unaligned ai
because
i know there is this nice star trek
fleet out there they're going to swoop
in
and take over where we failed just like
it wasn't the big deal that the easter
island
losers wiped themselves out it's a
dangerous way of lulling yourself into
false sense of security
if it's actually the case that it might
be up to us
and only us the whole future of
intelligent life in our observable
universe
then i think um it's both it really
puts a lot of responsibility on our
shoulders inspiring it's a little bit
terrifying but it's also inspiring
but it's empowering i think most people
because because the biggest problem
today is
i see this even when i teach right
so many people feel that it doesn't
matter what they do or we do
we feel disempowered oh it makes no
difference
this is about as far from that as you
can come but we realize that what we do
on our little spinning ball here in our
lifetime you know
could make the difference for the entire
future of life in our universe you know
how empowering is that yeah survival of
consciousness i mean and the
the other a very similar kind of uh
empowering aspect of the drake equation
is
say that there is a huge number of
intelligent civilizations that spring up
everywhere
but because of the drake equation which
is the lifetime of a civilization
yeah maybe many of them hit a wall and
yeah
and just like you said it's clear that
that for us the great
filter the one possible great filter
seems to be coming
you know in the next 100 years so it's
also empowering to
say okay well uh
we have a chance to not i mean the way
great filters work it does just get most
of them
exactly nick bostrom has articulated
this really beautifully too i i you know
every time
yet another search for life on mars
comes back negative or something i'm
like
yes are odds for
us surviving this yes you already made
the argument and brought rush there
right
but just unpack it right the point is we
already know
there's a crap ton of planets out there
that are earth-like and we also know
that
most of them do not seem to have
anything like our kind of life on them
so
what went wrong there's clearly one step
along the evolutionary at least one
filter roadblock in going from no life
to space faring life and um
where is it is it in front of us or is
it behind us
right if if there's no filter behind us
and we keep finding all sorts of
of uh little mice on
mars and or whatever right that's
actually very depressing because that
makes it much more likely that the
filter is in front of us
and that what actually is going on is
like the ultimate
dark joke that that whenever a
civilization
invents sufficiently powerful attack
it's just you just set your clock and
then after a while it goes poof
for one reason or other and wipes itself
out that will be wouldn't that be like
utterly depressing
if we're actually doomed whereas if it
turns out
that there is a really there is a great
filter early on
that for whatever reason seems to be
really hard to get to the stage of um
sexually reproducing organisms or even
the first
ribosome or whatever right or
or maybe you have lots of planets with
dinosaurs and cows but for some reason
they tend to get stuck there and never
invent smartphones
all of those are huge vic boosts for our
own odds
because been there done that you know
it doesn't matter how hard or unlikely
it was that we got past that
roadblock because we already did yeah
and
then then that makes it likely that the
future is in our own hands we're not
doomed so that's why that's why i think
the fact that like that the life is rare
in the universe it's not just something
that there's some evidence for
but also something we should actually
hope for
so that's the the end the mortality the
death of human civilization that we've
been discussing in life maybe prospering
beyond
any kind of great filter do you think
about your own
death does it make you sad that you may
not witness some of the
you know you lead a research group on
working some of the biggest questions in
the universe actually
um both on the physics and the ai side
does it make you sad that you may not be
able to see some of these
exciting things come to fruition that
we've been talking about
of course of course it sucks the fact
that i'm going to die
and i remember once when i was much
younger
my dad made this remark that life is
fundamentally tragic and i'm like what
are you talking about that you know
many years later i i felt now i feel
like i totally understand what he means
you know
we grow up very little kids and
everything is infinite and it's so cool
and then suddenly
we find out that actually you know
you got to only this this is that you
can get game over at some point
so of course it's it's it's um
it's something that's sad uh are you
afraid
no not in the sense that uh i think
anything terrible is going to happen
after i die or anything like that no i i
think it's really going to be game over
but it's more that um
it makes me very acutely aware of i just
have what a wonderful gift this is that
i get to be alive right now
and and is a steady reminder to just
live life to the fullest and really
enjoy it because it is finite you know
and i think actually and we know we all
get the
regular reminders when someone near and
dear to us dies that
that um
you know one day it's going to be our
turn it adds this kind of focus
i wonder what it would feel like
actually to be an immortal being
if they might even enjoy some of the
wonderful things of life a little bit
less
just because there isn't that um
finiteness yeah do you think that could
be a feature not a bug the fact that we
beings are finite maybe there's lessons
for
engineering in artificial intelligence
systems as well
that are conscious like do you think uh
it makes is it possible
that the reason the pistachio ice cream
is delicious
is the fact that you're going to die one
day and you will not have
all the pistachio ice cream that you
could eat
because of that fact well let me say two
things first of all i
it's actually quite profound what you're
saying i do think i appreciate the
pistachio ice cream a lot more
knowing that i will there's only finite
number of times i get to enjoy that
and i can only remember finding a number
of times in the past
and moreover my life is not so long that
it just starts to feel like things are
repeating themselves in general
it's so new and fresh i also think
though that
beth is a little bit overrated and in
the sense that the
it comes from a sort of outdated view of
physics and what life actually is
because
if you ask okay what is it that's going
to die exactly what what
am i really when i say it i feel sad
about
the idea of myself dying am i really sad
that this skin cell here is going to die
of course not because it's going to die
next week anyway and then i'll grow a
new one
right and it's not any of my cells that
i'm
associating really with with who i
really am
nor is it any of my atoms or quarks or
electrons
in fact basically all of my atoms get
replaced
on a regular basis right so what is it
that's really me
from a modern physics perspective it's
the information
in processing amy that's where my my
memory that's my memories
that's my my values
my dreams my passion my love
um it's that's what's
really fundamentally me and
frankly not all of that will die
when when my body dies like richard
feynman for example
his body died of cancer you know but
many of his
ideas that he felt were made him very
him actually live on i i
this is my own little personal tribute
to richard feynman right i try to keep a
little bit of him alive in myself
and i've even quoted him today right
yeah he almost came alive for a brief
moment in this conversation
yeah yeah and and this honestly gives me
some solace you know when i work as a
teacher i feel
if i can actually share a bit about
myself
that my students feel worthy enough to
copy and
adopt some part of things that they know
or
or they believe or aspire to
now i live on also a little bit in them
right
and i uh so
being a teacher is a little bit of what
i
that's something also that contributes
to making me
a little teeny bit less mortal right
because
i'm not at least not all gonna die all
at once right
and i find that a beautiful tribute to
people we generally respect
if we can if we can remember them and
carry
in us the things that we felt was the
most
awesome about them right then they live
on
and i'm getting a bit emotionally ever
but it's a very beautiful idea you bring
up there
i think we should stop this
old-fashioned materialism and just
equate
who we are with our quarks and electrons
there's no scientific basis for that
really and
it's also very uninspiring
now if you look a little bit towards the
future
right one thing which really sucks about
humans dying
is that even though some of their
teachings and memories and stories and
and ethics and so on will be copied by
those around them hopefully
a lot of it can't be copied and just
dies with them with a brain and that
really sucks
that that's the fundamental reason why
we find it so tragic
when someone goes from having all this
information there
more just gone ruined right
with with uh more post biological
intelligence
that's gonna shift a lot right
the only reason it's so hard to make a
backup of your brain
in its entirety is exactly because it
wasn't built for that right
if you have a future machine
intelligence
there's no reason for why it has to die
at all if it wants to up
if you want to copy it whatever is it
into some other quark blob
right you can copy not just sum it but
all of it right
and so so in that sense
you can get immortality because all the
information can be copied out of any
individual
entity uh it's and it's not just
mortality that will change if we get
more post biological life it's also
the with that i think very much the
whole
individual the whole individualism we
have now right the reason that we make
such a big difference between me
and you is exactly because we're a
little bit limited in how much we can
copy like i would just love to go like
this
and copy your russian skills russian
speaking skills yeah
wouldn't it be awesome but i can't i
have to actually work for years if you
want to get better on it and
but if you have if we were robots
just copy and paste freely then that
loses completely
uh it washes away the sense of what
immortality is
and also individuality a little bit
right we would start feeling much more
maybe we would feel much more
collaborative with each other if we can
just
hey you know i'll give you my rush you
can give me your russian and i'll give
you whatever some i can
and suddenly you can speak swedish maybe
that's less a bad trade for you but
whatever else you want from my brain
right yeah and and um
there have been a lot of sci-fi stories
about hive minds and so on
where where people where experiences can
be more broadly shared
uh and uh i think one
we don't i don't pretend to know what it
would feel like
to be a
super intelligent machine but i'm quite
confident that
however it feels about mortality and
individuality will be very very
different from how it is for
us well
for us mortality and finiteness seems to
be pretty important
at this particular moment and so all
good things must come to an end
just like this conversation i saw that
coming
sorry this is the world's worst
transition i could talk to you forever
it's such a huge honor that you spend
time with me
uh honor is mine thank you so much for
getting me essentially to start this
podcast by doing the first conversation
making me realize
uh falling in love with conversation in
itself
and thank you so much for inspiring so
many people in the world with your books
with your research with your talking and
with the
other the like this this ripple effect
of friends
including elon and everybody else that
you inspire so
thank you so much for talking today
thank you i feel like you're
so fortunate that this you're doing this
podcast and getting so many
interesting voices out there into the
ether
and not just the five second sound bites
but so many of the interviews of what
you do
you really let people go in into depth
in a way which we surely need in this
day and age and that i got to be number
one like i feel
super honored yeah you started thank you
so much max
thanks for listening to this
conversation with max tegmark and thank
you to our sponsors
the jordan harbinger show for sigmatic
mushroom coffee
better help online therapy and
expressvpn
so the choice is wisdom caffeine sanity
or privacy choose wisely my friends and
if you wish
click the sponsor links below to get a
discount and to support this podcast
and now let me leave you with some words
for max tag mark
if consciousness is the way that
information feels
when it's processed in certain ways then
it must be
substrate independent it's only the
structure of information processing that
matters
not the structure of the matter doing
the information processing
thank you for listening and hope to see
you next time