Kind: captions
Language: en
the following is a conversation with
Greg Brockman he's the co-founder and
CTO of open AI a world-class research
organization developing ideas and AI
with the goal of eventually creating a
safe and friendly artificial general
intelligence one that benefits and
empowers humanity open AI is not only a
source of publications algorithms tools
and datasets their mission is a catalyst
for an important public discourse about
our future with both narrow and general
intelligence systems this conversation
is part of the artificial intelligence
podcast at MIT and beyond if you enjoy
it subscribe on youtube itunes or simply
connect with me on twitter at Lex
Friedman spelled Fri D and now here's my
conversation with Greg Brockman so in
high school and right after you wrote a
draft of a chemistry textbook I saw that
that covers everything from basic
structure of the atom to quantum
mechanics so it's clear you have an
intuition and a passion for both the
physical world with chemistry and now
robotics to the digital world with AI
deep learning reinforcement learning so
on do you see the physical world in the
digital world is different and what do
you think is the gap a lot of it
actually boils down to iteration speed
right that I think that a lot of what
really motivates me is is building
things right is the I you know think
about mathematics for example where you
think you're really hard about a problem
you understand it you're right down in
this very obscure form that we call
proof but then this is in humanities
library right it's there forever this is
some truth that we've discovered you
know maybe only five people in your
field will ever read it now but somehow
you've kind of moved humanity forward
and so I actually used to really think
that I was going to be a mathematician
and then I actually started writing this
chemistry textbook one of my friends
told me you'll never publish it because
you don't have a PhD so instead I
decided to build a website and try to
promote my ideas that way and then I
discovered programming and I you know
that in programming you think hard about
a problem you understand it you write
down in a very obscure form that we call
a program but then once again it's in
humanities library right and anyone
could get the benefit from
and the scalability is massive and so I
think that the thing that really appeals
to me about the digital world is that
you can have this this this insane
leverage right a single individual with
an idea is able to affect the entire
planet and that's something I think is
really hard to do if you're moving
around physical atoms but you said
mathematics so if you look at the the
what thing you know over here our mind
do you ultimately see it as just math is
just information processing or is there
some other magic as you've seen if
you've seen through biology and
chemistry and so on I think it's really
interesting to think about humans is
just information processing systems and
that it seems like it's actually a
pretty good way of describing a lot of
kind of how the world works or a lot of
what we're capable of to think that that
you know again if you just look at
technological innovations over time that
in some ways the most transformative
innovation that we've had it has been
the computer right in some ways the
internet you know that what is the right
the Internet is not about these physical
cables it's about the fact that I am
suddenly able to instantly communicate
with any other human on the planet I'm
able to retrieve any piece of knowledge
that in some ways the human race has
ever had and that those are these insane
transformations do you see the our
society as a whole the collective as
another extension of the intelligence of
the human being so if you look at the
human being is an information processing
system
you mentioned the internet then
networking do you see us all together as
a civilization as a kind of intelligence
system yeah I think this is actually a
really interesting perspective to take
and to think about to you sort of have
this collective intelligence of all of
society the economy itself is this
superhuman machine that is optimizing
something right and it's all in some
ways a company has a will of its own
right that you have all these
individuals we're all pursuing their own
individual goals and thinking really
hard and thinking about the right things
to do but somehow the company does
something that is this emergent thing
and that is it so there's a really
useful abstraction and so I think that
in some ways you know we think of
ourselves as the most intelligent things
on the planet and the most powerful
things on the planet but there are
things that are bigger than us that
these systems that we all contribute to
and so I think actually you know it's a
it's interesting to think about if
you've read as a guys a models
foundation
right that that there's this concept of
psychohistory in there which is
effectively this that if you have
trillions or quadrillions of beings then
maybe you could actually predict what
that being that that huge macro being
will do I and almost independent of what
the individuals want I actually have a
second angle on this I think is
interesting which is thinking about a
technological determinism one thing that
I actually think a lot about with with
open a tie right is that we're kind of
coming on onto this insanely
transformational technology of general
intelligence right that will happen at
some point and there's a question of how
can you take actions that will actually
steer it to go better rather than worse
and that I think one question you need
to ask is as a scientist as an inventor
as a creator what impact can you have in
general right you look at things like
the telephone invented by two people in
the same day like what does that mean
what does that mean about the shape of
innovation and I think that what's going
on is everyone's building on the
shoulders of the same giants and so you
can kind of you can't really hope to
create something no one else ever would
you know if Einstein wasn't born someone
else would have come up with relativity
you know he changed the timeline a bit
right that maybe it would have taken
another 20 years but it wouldn't be that
fundamentally humanity would never
discover these these fundamental truths
so there's some kind of invisible
momentum that some people like Einstein
or open the eyes plugging into that's
anybody else can also plug into and
ultimately that wave takes us into a
certain direction that's me that's right
that's right and you know this kind of
seems to play out in a bunch of
different ways that there's some
exponential that is being ridden and
that the exponential itself which one it
is changes think about Moore's law an
entire industry set its clock to it for
50 years like how can that be right how
is that possible and yet somehow it
happened and so I think you can't hope
to ever invent something that no one
else will maybe you can change the
timeline a little bit but if you really
want to make a difference I think that
the thing that you really have to do the
only real degree of freedom you have is
to set the initial conditions under
which a technology is born and so you
think about the internet right that
there are lots of other competitors
trying to build similar things and the
internet one and that the initial
conditions where that was created by
this group that really valued people
being able to be you know anyone being
able to plug in this very academic
mindset of
being open and connected and I think
that the Internet for the next 40 years
really played out that way
you know maybe today things are starting
to shift in a different direction but I
think if those initial conditions were
really important to determine the next
40 years worth of progress that's really
beautifully put so another example of
that I think about you know I recently
looked at it I looked at Wikipedia the
formation of Wikipedia and I wonder what
the internet would be like if Wikipedia
had ads you know there's a interesting
argument that why they chose not to make
it put advertisement wikipedia i think
it's i think wikipedia is one of the
greatest resources we have on the
internet it's extremely surprising how
well it works and how well it was able
to aggregate all this kind of good
information and they essentially the
creator of wikipedia I don't know
there's probably some debates there but
set the initial conditions and now it
carried it itself forward that's really
interesting so you're the way you're
thinking about AGI or artificial
intelligences you're focused on setting
the initial conditions for the for the
progress that's right that's powerful
okay so look into the future if you
create an AGI system like one that can
ace the Turing test natural language
what do you think would be the
interactions you would have with it what
do you think are the questions you would
ask like what would be the first
question you would ask it her/him that's
right I think it at that point if you've
really built a powerful system that is
capable of shaping the future of
humanity the first question that you
really should ask is how do we make sure
that this plays out well and so that's
actually the first question that I would
ask a powerful AGI system is so you
wouldn't ask your colleague you wouldn't
ask like Ilya you would ask the AGI
system oh we've already had the
conversation with Ilya right and
everyone here and so you want as many
perspectives and a piece of wisdom as
you can for it for answering this
question so I don't think you
necessarily defer to whatever your
powerful system tells you but you use as
one input I like to try to figure out
what to do but and I guess fundamentally
what it really comes down to is if you
built something really powerful and you
think about think about for example the
creation of of shortly after the
creation of nuclear weapons right the
most important question the world was
what's the world order going to be like
how do we set ourselves up in
where we're going to be able to survive
this species with a GI I think the
question is slightly different right
that there is a question of how do we
make sure that we don't get the negative
effects but there's also the positive
side right you imagine that you know
like like what won't AGI be like like
what will be capable of and I think that
one of the core reasons that an AGI can
be powerful and transformative is
actually due to technological
development yeah right if you have
something that's capable as capable as a
human and that it's much more scalable
that you absolutely want that thing to
go read the whole scientific literature
and think about how to create cures for
all the diseases right you want it to
think about how to go and build
technologies to help us create material
abundance and to figure out societal
problems that we have trouble with like
how we're supposed to clean up the
environment and you know maybe you want
this to go and invent a bunch of little
robots that will go out and be
biodegradable and turn ocean debris into
harmless molecules and I think that that
that positive side is something that I
think people miss sometimes when
thinking about what an AGI will be like
and so I think that if you have a system
that's capable of all of that you
absolutely want its advice about how do
I make sure that we're using your your
capabilities in a positive way for
Humanity so what do you think about that
psychology that looks at all the
different possible trajectories of an
AGI system many of which perhaps the
majority of which are positive and
nevertheless focuses on the negative
trajectories I mean you get to interact
with folks you get to think about this
maybe within yourself as well you look
at sam harris and so on it seems to be
sorry to put it this way but almost more
fun to think about the negative
possibilities whatever that's deep in
our psychology what do you think about
that and how do we deal with it because
we want AI to help us so I think there's
kind of two problems so I entailed in
that question the first is more of the
question of how can you even picture
what a world with a new technology will
any like now imagine were in 1950 and
I'm trying to describe Buber to someone
apps and the internet yeah I mean your
yeah that's that's going to be extremely
complicated but it's imaginable
it's imaginable right but and now
imagine being a 1950 and predicting
goober right and you need to describe
the internet you need to describe GPS
you need to describe the fact that
everyone's going to have this phone in
their pocket and so I think that that
just the first truth is that it is hard
to picture how a transformative
technology will play out in the world
we've seen that before with technologies
that are far less transformative than AG
I will be and so I think that that one
piece is that it's just even hard to
imagine and to really put yourself in a
world where you can predict what that
that positive vision would be like and
you know I think the second thing is
that it is I think it is always easier
to support the negative side than the
positive side it's always easier to
destroy than create and you know less
than in a physical sense and more just
in an intellectual sense right because
you know I think that with creating
something you need to just get a bunch
of things right and to destroy you just
need to get one thing wrong yeah and so
I think that that what that means is
that I think a lot of people's thinking
dead ends as soon as they see the
negative story but that being said I
actually actually have some hope right I
think that the the positive vision is
something that I think can be something
that we can we can talk about I think
that just simply saying this fact of
yeah like there's positive there's
negatives everyone likes to draw them
the negative people have to respond well
to that message and say huh you're right
there's a part of this that we're not
talking about not thinking about and
that's actually something that's that's
that's I think really been a key part of
how we think about AGI at open AI right
you can kind of look at it as like okay
like opening eye talks about the fact
that there are risks and yet they're
trying to build this system like how do
you square this those two facts so do
you share the intuition that some people
have I mean from Sam Harris even Elon
Musk himself that it's tricky as you
develop AGI to keep it from slipping
into the existential threats into the
negative what's your intuition about how
hard is it to keep a
a development on the positive track and
you what's your intuition there to
answer the question you can really look
at how we structure open AI so we really
have three main arms
we have capabilities which is actually
doing the technical work and pushing
forward what these systems can do
there's safety which is working on
technical mechanisms to ensure that the
systems we build are lined with human
values and then there's policy which is
making sure that we have governance
mechanisms answering that question of
well whose values and so I think that
the technical safety one is the one that
people kind of talk about the most right
you talk about like think about you know
all of the dystopic AI movies a lot of
that is about not having good technical
safety in place and what we've been
finding is that you know I think that
actually a lot of people look at the
technical safety problem and think it's
just intractable right this question of
what do humans want how am I supposed to
write that down can I even write down
what I want no way and then they stop
there but the thing is we've already
built systems that are able to learn
things that humans can't specify you
know even the rules for how to recognize
if there's a cat or a dog in an image
turns out its intractable to write that
down and yet we're able to learn it and
that what we're seeing with systems we
build it open it yeah and there's still
an early proof of concept stage is that
you are able to learn human preferences
you're able to learn what humans want
from data and so that's kind of the core
focus for our technical safety team and
I think that they're actually we've had
some pretty encouraging updates in terms
of what we've been able to make work so
you have an intuition and a hope that
from data you know looking at the value
alignment problem from data we can build
systems that align with the collective
better angels of our nature so aligned
with the ethics and the morals of human
beings to even say this in a different
way I mean think about how do we align
in humans right think about like a human
baby can grow up to be an evil person or
a great person and a lot of that is from
learning from data right that you have
some feedback as a child is growing up
they get to see positive examples and so
I think that that just like them that
the the only example we have of a
general intelligence that is able to
learn from data I too
aligned with human values and to learn
values I think we shouldn't be surprised
that we can do the same sorts of
techniques or whether the same sort of
techniques end up being how we we saw
value alignment for AG eyes so let's go
even higher as I don't know if you've
read the book sapiens mm-hmm but there's
an idea that you know that as a
collective is us human beings who kind
of develop together and ideas that we
hold there's no in that context
objective truth we just kind of all
agree to certain ideas and hold them as
a collective if you have a sense that
there is in the world of good and evil
do you have a sense that to the first
approximation there are some things that
are good and that you could teach
systems to behave to be good so I think
that this actually blends into our third
team right which is the policy team and
this is the one the the aspect I think
people really talk about way less than
they should all right because imagine
that we built super-powerful systems
that we've managed to figure out all the
mechanisms for these things to do
whatever the operator wants the most
important question becomes who's the
operator what do they want and how is
that going to affect everyone else right
and and I think that this question of
what is good what are those values I
mean I think you don't even have to go
to those those very grand existential
places to start to realize how hard this
problem is you just look at different
countries and cultures across the world
and that there's there's a very
different conception of how the world
works and you know what what what kinds
of of ways that society wants to operate
and so I think that the really core
question is is is actually very concrete
um and I think it's not a question that
we have ready answers to right is how do
you have a world where all the different
countries that we have United States
China Russia and you know the hundreds
of other countries out there are able to
continue to not just operate in the way
that they see fit but in that the world
that emerges in these where you have
these very powerful systems I operating
alongside humans ends up being something
that empowers humans more that makes
like exhuming existence
be a more meaningful thing and the
people are happier in wealthier and able
to live more fulfilling lives it's not
nob vyas thing for how to design that
world once you have that very powerful
system so if we take a little step back
and we're having it like a fascinating
conversation and open eyes in many ways
a tech leader in the world and yet we're
thinking about these big existential
questions which is fascinating really
important I think you're a leader in
that space and it's a really important
space of just thinking how AI affect
society in a big-picture view so Oscar
Wilde said we're all in the gutter but
some of us are looking at the Stars and
I think open air has a charter that
looks to the Stars I would say to create
intelligence to create general
intelligence make it beneficial safe and
collaborative so can you tell me how
that came about how a mission like that
and the path to creating a mission like
that open yeah I was founded yeah so I
think that in some ways it really boils
down to taking a look at the landscape
alright so if you think about the
history of AI that basically for the
past 60 or 70 years people have thought
about this goal of what could happen if
you could automate human intellectual
labor right imagine you can build a
computer system that could do that
what becomes possible well out of sci-fi
that tells stories of various dystopian
and you know increasingly you have
movies like heard that tell you a little
bit about maybe more of a little bit
utopic vision I you think about the
impacts that we've seen from being able
to have bicycles for our minds and
computers and that I think that the the
impact of computers and the Internet has
just far outstripped what anyone really
could have predicted and so I think that
it's very clear that if you can build an
AI it will be the most transformative
technology that humans will ever create
and so what it boils down to then is a
question of well is there a path is
there hope is there a way to build such
a system and I think that for 60 or 70
years that people got excited and I they
you know ended up not being able to
deliver on the hopes that the people I
pinned on them and I think that then you
know that after you know two to winters
of AI development
that people I you know I think kind of
almost stopped daring to dream right the
really talking about a GI or thinking
about a GI became almost this taboo in
the community but I actually think that
people took the wrong lesson from AI
history and if you look back starting in
nineteen fifty nine is when the
perceptron was released and this is
basically you know one of the earliest
neural networks it was released to what
was perceived as this massive overhype
so in the New York Times in nineteen
fifty-nine you have this article saying
that you know the the perceptron will
one day recognize people call out their
names instantly translate speech between
languages and people at the time looked
at this and said this is Jack your
system can't do any of that and
basically spent ten years trying to
discredit the whole perceptron direction
and succeeded and all the funding dried
up and you know people kind of went in
other directions and you know the 80s
there was a resurgence and I'd always
heard that the resurgence in the 80s was
due to the invention of back propagation
and these these algorithms that got
people excited but actually the
causality was due to people building
larger computers that you can find these
these articles from the 80s saying that
the democratization of computing power
suddenly meant that you could run these
larger neural networks and then people
start to do all these amazing things the
backpropagation algorithm was invented
and you know that the the neural nets
people running were these tiny little
like 20 neuron neural nets right what
are you supposed to learn with 20
neurons and so of course they weren't
able to get great results and it really
wasn't until 2012 that this approach
that's almost the most simple natural
approach that people have come up with
in the 50s right in some ways even in
the 40s before there were computers with
a Pitts McCullen air and neuron suddenly
this became the best way of solving
problems right and I think there are
three core properties that deep learning
has that I think are very worth paying
attention to
the first is generality we have a very
small number of deep learning tools SGD
deep neural net maybe some some you know
RL and it solves this huge variety of
problems speech recognition machine
translation game playing all these
problems small set of tools so there's
the generality there's a second piece
which is the competence you want to
solve any of those problems throw it
forty years worth of
computer vision research replacing the
deep neural net it's kind of work better
and there's a third piece which is the
scalability right the one thing that has
been shown time and time again is that
you if you have a larger neural network
for a more compute more data at it
it will work better those three
properties together feel like essential
parts of building a general intelligence
now it doesn't just mean that if we
scale up what we have that we will have
an AGI right there are clearly missing
pieces they're missing ideas we need to
have answers for reasoning but I think
that the core here is that for the first
time it feels that we have a paradigm
that gives us hope the general
intelligence can be achievable and so as
soon as you believe that everything else
becomes comes into focus right if you
imagine that you may be able to and you
know that the timeline I think remains
uncertain on the but I think that that
you know certainly within our lifetimes
and possibly within a much shorter
period of time than then people would
expect if you can really build the most
transformative technology that will ever
exist you stop thinking about yourself
so much right and you start thinking
about just like how do you have a world
where this goes well and that you need
to think about the practicalities of how
do you build an organization and get
together a bunch of people and resources
and to make sure that people feel
motivated and ready to do it but I think
that then you start thinking about well
what if we succeed and how do we make
sure that when we succeed that the world
is actually the place that we want
ourselves to exist then and almost in
the Rawls the unveils sense of the word
and so that's kind of the broader
landscape and opening I was really
formed in 2015 with that high level
picture of AGI might be possible sooner
than people think and that we need to
try to do our best to make sure it's
going to go well and then we spent the
next couple years really trying to
figure out what does that mean how do we
do it
and you know I think that typically with
a company you start out very small so
you in a co-founder and you build a
product you got some users you get a
product market fit
you know then at some point you raise
some money you hire people you scale and
then you know down the road then the big
companies realize you exist and try to
kill you
and for opening I it was basically
everything in exactly the
order let me just pause for a second he
said a lot of things and let me just
admire the jarring aspect of what open
AI stands for which is daring to dream I
mean you said it's pretty powerful you
caught me off guard because I think
that's very true
the-the-the step of just daring to dream
about the possibilities of creating
intelligence in a positive in a safe way
but just even creating intelligence is a
much needed refreshing catalyst for the
AI community so that's that's the
starting point
okay so then formation of open AI was
just I just say that you know when we
were starting opening AI that kind of
the first question that we had is is it
too late to start a lab with a bunch of
the best people possible that was an
actual question so those were those that
was the core question of you know hey
there's dinner in July of 20 2015
and there's that was that was really
what we spent the whole time talking
about and you know cuz it's the you
think about kind of where AI was is that
it transitioned from being an academic
pursuit to an industrial pursuit and so
a lot of the best people were in these
big research labs and that we wanted to
start our own one that you know no
matter how much resources we could
accumulate it would be you know pale in
comparison to the big tech companies and
we knew that and there's a question of
are we going to be actually able to get
this thing off the ground you need
critical mass you can't just do you and
a co-founder build a product right you
really need to have a group of you know
five to ten people and we kind of
concluded it wasn't obviously impossible
so it seemed worth trying well you're
also dreamers so who knows right that's
right okay so speaking of that competing
with with the the big players let's talk
about some of the some of the tricky
things as you think through this process
of growing of seeing how you can develop
these systems a task at scale that
competes so you recently recently formed
open ILP a new cap profit company that
now carries the name open it so open has
now this official company the original
non profit company
still exists and carries the opening I
nonprofit name so can you explain what
this company is what the purpose of us
creation is and how did you arrive at
the decision yep to create it openly I
the whole entity and opening I LP as a
vehicle is trying to accomplish the
mission of ensuring that artificial
general intelligence benefits everyone
and the main way that we're trying to do
that is by actually trying to build
general intelligence ourselves and make
sure the benefits are distributed to the
world that's the primary way we're also
fine if someone else does this all right
it doesn't have to be us if someone else
is going to build an AGI and make sure
that the benefits don't get locked up in
one company or you know one one want
with one set of people like we're
actually fine with that and so those
ideas are baked into our Charter which
is kind of the the foundational document
that are describes kind of our values
and how we operate
but it's also really baked into the
structure of open at LP and so the way
that we've set up opening ILP is that in
the case where we succeed right if we
actually build what we're trying to
build then investors are able to get a
return and but that return is something
that is capped and so if you think of
AGI in terms of data the value that you
could really create you're talking about
the most transformative technology ever
created it's going to create orders of
magnitude more value than any existing
company and that all of that value will
be owned by the world like legally title
to the nonprofit to fulfill that mission
and so that's that's the structure so
the mission is a powerful one and it's a
it's one that I think most people would
agree with it's how we would hope a I
progresses and so how do you tie
yourself to that mission how do you make
sure you do not deviate from that
mission that you know other incentives
that are profit driven wouldn't don't
interfere with the mission so this was
actually a really core question for us
for the past couple years because you
know I'd say that like the way that our
history went was that for the first year
we were getting off the ground right we
had this high level picture but we
didn't know
exactly how we wanted to accomplish it
and really two years ago it's when we
first started realizing in order to
build a GI we're just going to need to
raise way more money than we can as a
nonprofit I mean you're talking many
billions of dollars and so the first
question is how are you supposed to do
that and stay true to this mission and
we looked at every legal structure out
there and concluded none of them were
quite right for what we wanted to do and
I guess it shouldn't be too surprising
if you're going to do something like
crazy unprecedented technology that
you're gonna have to come up with some
crazy unprecedent structure to do it in
and a lot of a lot of our conversation
was with people at opening I write the
people who really join because they
believe so much in this mission and
thinking about how do we actually raise
the resources to do it and also stay
true to to what we stand for and the
place you got to start is to really
align on what is it that we stand for
right what are those values what's
really important to us and so I'd say
that we spent about a year really
compiling the opening I'd charter and
that determines and if you even look at
the first the first line item in there
it says that look we expect we're gonna
have to marshal huge amounts of
resources but we're going to make sure
that we minimize conflicts of interest
with the mission and that kind of
aligning on all of those pieces was the
most important step towards figuring out
how do we structure a company that can
actually raise the resources to do what
we need to do I imagined open AI the
decision to create open ILP was a really
difficult one and there was a lot of
discussions as you mentioned for a year
and there was different ideas perhaps
detractors with an open AI sort of
different paths that you could have
taken what were those concerns what were
the different paths considered what was
that process of making that decision
like yep um but so if you look actually
at the opening I charter that there's
almost two paths embedded within it
there is we are primarily trying to
build AGI ourselves but we're also ok if
someone else does it and this is a weird
thing for a company it's really
interesting actually yeah there there is
an element of competition that you do
want to be the one that does it but at
the same time you're ok somebody else's
and you know we'll talk about that a
little bit that trade-off that's the day
that's really interesting and I think
this was the core tension as we were
designing open an ILP and really the
opening eye strategy is how do you make
sure that both you have a shot at being
a primary actor which really requires
building an organization raising massive
resources and really having the will to
go and execute on some really really
hard vision all right you need to really
sign up for a long period to go and take
on a lot of pain and a lot of risk and
to do that normally you just import the
startup mindset right and that you think
about okay like how do we how to execute
everyone you give this very competitive
angle but you also have the second angle
of saying that well the true mission
isn't for opening high to build a GI the
true mission is for AGI to go well for
Humanity and so how do you take all of
those first actions and make sure you
don't close the door on outcomes that
would actually be positive in fulfill
the mission and so I think it's a very
delicate balance right I think that
going 100% one direction or the other is
clearly not the correct answer and so I
think that even in terms of just how we
talk about opening I and think about it
there's just like like one thing that's
always in the back of my mind is to make
sure that we're not just saying opening
eyes goal is to build AGI right that
it's actually much broader than that
right that first of all I you know it's
not just AGI it's safe AGI that's very
important but secondly our goal isn't to
be the ones to build it our goal is to
make sure it goes well for the world and
so I think that figuring out how do you
balance all of those and to get people
to really come to the table and compile
the the like a single document that that
encompasses all of that wasn't trivial
so part of the challenge here is your
mission is I would say beautiful
empowering and a beacon of hope for
people in the research community and
just people thinking about AI so your
decisions are scrutinized more than I
think a regular profit driven company do
you feel the burden of this in the
creation of the Charter and just in the
way you operate yes so why do you lean
into the burden by creating such a
charter why not to keep it quiet I mean
it just boils down to the to the mission
right
I'm here and everyone else is here
because we think this is the most
important mission right dare to dream
all right so what do you think you can
be good for the world or create an a GI
system that's good when you're a
for-profit company from my perspective I
don't understand why profit interferes
with positive impact on society I don't
understand by Google that makes most of
its money from ads you can't also do
good for the world or other companies
Facebook anything I don't I don't
understand why those have to interfere
you know you can profit isn't the thing
in my view that affects the impact of a
company what affects the impact of the
company is the Charter is the culture is
the you know the people inside and
profit is the thing that just fuels
those people so what are your views
there yeah so I think that's a really
good question and there's there's
there's some some you know real like
long-standing debates in human society
that are wrapped up in it the way that I
think about it is just think about what
what are the most impactful nonprofits
in the world what are the most impactful
for profits in the world right is much
easier to lists the for profits that's
right and I think that there's there's
some real truth here that the system
that we set up the system for kind of
how you know today's world is organized
is one that that really allows for huge
impact and that that you know kind of
part of that is that you need to be you
know for profits are our self-sustaining
and able to to kind of you know build on
their own momentum and I think that's a
really powerful thing it's something
that when it turns out that we haven't
set the guardrails correctly causes
problems right think about logging
companies that go and DeForest you know
you know the rain forest that's really
bad we don't want that and it's actually
really interesting to me the kind of
this this question of how do you get
positive benefits out of a for-profit
company it's actually very similar to
how do you get positive benefits out of
an AGI right that you have this like
very powerful system it's more powerful
than any human and it's kind of
autonomous in some ways you know super
human and a lot of axes and somehow you
have to set the guardrails to get good
to happen but when you do the benefits
are massive and so I think that the when
when I think about nonprofit vs.
for-profit I think it's just not enough
happens in nonprofits they're very pure
but it's just kind of you know it's just
hard to do things they're in for profits
in some ways like too much happens but
if if kind of shaped in the right way it
can actually be very positive and so
with open NLP we're picking a road in
between now the thing I think is really
important to recognize is that the way
that we think about opening ILP is that
in the world where AGI actually happens
right in a world where we are successful
we build the most transformative
technology ever the amount of value
we're going to create will be
astronomical and so then in that case
that the if it the the cap that we have
will be a small fraction of the value we
create and the amount of value that goes
back to investors and employees looks
pretty similar to what would happen in a
pretty successful startup and that's
really the case that we're optimizing
for right that we're thinking about in
the success case making sure that the
value we create doesn't get locked up
and I expect that in another you know
for-profit companies that it's possible
to do something like that I think it's
not obvious how to do it right and I
think that as a for-profit company you
have a lot of fiduciary duty to your
shareholders and that there are certain
decisions you just cannot make in our
structure we've set it up so that we
have a fiduciary duty to the Charter
that we always get to make the decision
that is right for the Charter rather
than even if it comes at the expense of
our own stakeholders and and so I think
that when I think about what's really
important it's not really about
nonprofit vs. for-profit it's really a
question of if you build a GI and you
kind of you know humanities now in this
new age who benefits whose lives are
better and I think that what's really
important is to have an answer that is
everyone yeah which is one of the core
aspects of the Charter so one concern
people have not just with open the eye
but with Google Facebook Amazon anybody
really that's that's creating impact
that scale is how do we avoid as your
Charter says avoid enabling the use of
or AGI to unduly concentrate power why
would not a company like open a I keep
all the power of an AGI system to itself
the Charter the Charter so you know how
does the Charter actualize itself in day
to day so I think that first to zoom out
right there the way that we structure
the company is so that the the power
first sort of you know dictating the
actions that opening eye takes
ultimately rests with the board right
the board of the nonprofit I'm and the
board is set up in certain ways certain
certain restrictions that you can read
about in the opening hi LP blog post but
effectively the board is the is the
governing body for opening ILP and the
board has a duty to fulfill the mission
of the nonprofit and so that's kind of
how we tie how we thread all these
things together now there's a question
of so day to day how do people the
individuals who in some ways are the
most empowered ones ain't no the board
sort of gets to call the shots at the
high level but the people who are
actually executing are the employees the
way that people here on a day-to-day
basis who have the you know the the keys
to the technical Kingdom and their I
think that the answer looks a lot like
well how does any company's values get
actualized right I think that a lot of
that comes down to that you need people
who are here because they really believe
in that mission and they believe in the
Charter and that they are willing to
take actions that maybe are worse for
them but are better for the Charter and
that's something that's really baked
into the culture and honestly I think
it's I you know I think that that's one
of the things that we really have to
work to preserve as time goes on and
that's a really important part of how we
think about hiring people and bringing
people into opening I so there's people
here there's people here who could speak
up and say like hold on a second this is
totally against what we stand for
cultural eyes yeah yeah for sure I mean
I think that that we actually have I
think that's like a pretty important
part of how we operate and how we have
even again with designing the Charter
and designing open alp in the first
place that there has been a lot of
conversation with employees here and a
lot of times where employees said wait a
second this
seems like it's coming in the wrong
direction and let's talk about it and so
I think one thing that's that's I think
I really and you know here's here's
actually one thing I think is very
unique about us as a small company is
that if you're at a massive tech giant
that's a little bit hard for someone
who's aligned employee to go and talk to
the CEO and say I think that we're doing
this wrong and you know you look at
companies like Google that have had some
collective action from employees to you
know make ethical change around things
like maven and so maybe there are
mechanisms that other companies that
work but here super easy for anyone to
pull me aside to pull Sam aside to
Balilla aside and people do it all the
time one of the interesting things in
the Charter is this idea that it'd be
great if you could try to describe or
untangle switching from competition to
collaboration and late-stage AGI
development it was really interesting
this dance between competition and
collaboration how do you think about
that yeah assuming you can actually do
the technical side of AGI development I
think there's going to be two key
problems with figuring out how do you
actually deploy it make it go well the
first one of these is the run-up to
building the first AGI you look at how
self-driving cars are being developed
and it's a competitive race I'm the
thing that always happens in a
competitive race is that you have huge
amounts of pressure to get rid of safety
and so that's one thing we're very
concerned about right is that people
multiple teams figuring out we can
actually get there but you know if we
took the slower path that is more
guaranteed to be safe we will lose and
so we're going to take the fast path and
so the more that we can both ourselves
be in a position where we don't generate
that competitive race where we say if
the race is being run and that you know
someone else's is further ahead than we
are we're not gonna try to to leapfrog
we're gonna actually work with them
right we will help them succeed as long
as what they're trying to do is to
fulfill our mission then we're good we
don't have to build AGI ourselves and I
think that's a really important
commitment from us but it can't just be
unilateral right I think that's really
important that other players who are
serious about building AGI make similar
commitments right I think that that you
know again to the extent that everyone
believes that AGI should be something to
benefit everyone then it actually really
shouldn't matter which company builds it
and we should all be concerned about the
case where we just race so hard to get
there
that something goes wrong so what role
do you think government our favorite
entity has in setting policy and rules
about this domain from research to the
development to early stage to late stage
a a inhi development so I think that
first of all is really important the
government's in their right in some way
shape or form you know at the end of the
day we're talking about building
technology that will shape how the world
operates and that there needs to be
government as part of that answer and so
that's why we've we've we've done a
number of different congressional
testimonies we interact with a number of
different lawmakers and the you know
right now a lot of our message to them
is that it's not the time for regulation
it is the time for measurement right
that our main policy recommendation is
that people and you know the government
does this all the time with bodies like
NIST um spend time trying to figure out
just where the technology is how fast
it's moving and can really become
literate and up to speed with respect to
what to expect
so I think that today the answer really
is about about about measurement and I
think if there will be a time in place
where that will change and I think it's
a little bit hard to predict exactly I
what what exactly that trajectory should
look like so there will be a point
oh it's regulation federal in the United
States the government steps in and and
helps be the I don't want to say the
adult in the room to make sure that
there is strict rules may be
conservative rules that nobody can cross
well I think there's this kind of maybe
to two angles to it so today with narrow
AI applications that I think there are
already existing bodies that are
responsible and should be responsible
for regulation you think about for
example with self-driving cars that you
want the you know the National Highway
it's exactly to be very good mat that
makes sense right that basically what
we're saying is that we're going to have
these technological systems that are
going to be do performing applications
that humans already do great we already
have ways of thinking about standards
and safety for those so I think actually
empowering those regulators today is
also pretty important
and then I think for for a GI you know
that there's going to be a point where
we'll have better answers and I think
that maybe a similar approach of first
measurement and you know start thinking
about what the rules should be I think
it's really important that we don't
prematurely squash you know progress I
think it's very easy to kind of smother
the budding field and I think that's
something to really avoid but I don't
think it's the right way of doing it is
to say let's just try to blaze ahead and
not involve all these other stakeholders
so you've recently released a paper on
GPT two language modeling but did not
release the full model because you have
concerns about the possible negative
effects of the availability of such
model it's uh outside of just that
decision is super interesting because of
the discussion as at a societal level
the discourse it creates so it's
fascinating in that aspect but if you
think that's the specifics here at first
what are some negative effects that you
envisioned and of course what are some
of the positive effects yeah so again I
think to zoom out like the way that we
thought about GPT 2 is that with
language modeling we are clearly on a
trajectory right now where we scale up
our models and we get qualitatively
better performance right GPT 2 itself
was actually just a scale-up of a model
that we've released in the previous June
right and we just ran it at you know
much larger scale and we got these
results we're suddenly starting to write
coherent prose which was not something
we'd seen previously and what are we
doing now well we're gonna scale up GPT
2 by 10x by hundred X by thousand X and
we don't know what we're going to get
and so it's very clear that the model
that we were that we released last June
you know I think it's kind of like it's
it's it's it's a good academic toy it's
not something that we think is something
that can really have negative
applications or you know to the sense
that it can the positive of people being
able to play with it is you know far far
outweighs the possible harms you fast
forward to not GPT to buy GPU 20
and you think about what that's gonna be
like and I think that the capabilities
are going to be substantive and so if
there needs to be a point in between the
two where you say this is something
where we are drawing the line and that
we need to start thinking about the
safety aspects and I think for GPT too
we could have gone either way and in
fact when we had conversations
internally that we had a bunch of pros
and cons and it wasn't clear which one
which one outweighed the other and I
think that when we announced that hey we
decide not to release this model then
there was a bunch of conversation where
various people said it's so obvious that
you should have just released it there
other people said it's so obvious you
should not have released it and I think
that that almost definitionally means
that holding it back was the correct
decision right if it's contra if there's
if it's not obvious whether something is
beneficial or not you should probably
default to caution and so I think that
the overall landscape for how we think
about it is that this decision could
have gone either way there are great
arguments in both directions but for
future models down the road and possibly
sooner than you'd expect because you
know scaling these things up doesn't
have to take that long those ones but
you're definitely not going to want to
release into the wild and so I think
that we almost view this as a test case
and to see can we even design you know
how do you have a society or how do you
have a system that goes from having no
concept of responsible disclosure where
the mere idea of not releasing something
for safety reasons is unfamiliar to a
world where you say okay we have a
powerful model let's at least think
about it let's go through some process
and you think about the security
community it took them a long time to
design responsible disclosure right you
know you think about this question of
well I have a security exploit I send it
to the company the companies like tries
to prosecute me or just sit just ignores
it what do I do right and so you know
the alternatives of oh I just just
always publish your exploits that
doesn't seem good either right and so it
really took a long time and took this
this it was bigger than any individual
right is really about building the whole
community that believed that okay we'll
have this process where you send it to
the company you know if they don't act
in a certain time then you can go public
and you're not a bad person you've done
the right thing and I think that in AI
part of the response of gbt to just
proves that we don't have any concept of
this
so that's the high level picture um and
so I think that I think this was this
was a really important move to make and
we could have maybe delayed it for D BT
3 but I'm really glad we did it for GPT
too and so now you look at GPT 2 itself
and you think about the substance of
okay what are potential negative
applications so you have this model
that's been trained on the Internet
which you know it's also going to be a
bunch of very biased data a bunch of you
know very offensive content and there
and you can ask it to generate content
for you on basically any topic right you
just give it a prompt and we'll just
start start writing and all writes
content like you see on the internet you
know even down to like saying
advertisement in the middle of some of
its generations and you think about the
possibilities for generating fake news
or abusive content and you know it's
interesting seeing what people have done
with you know we released a smaller
version of GPT too and the people have
done things like try to generate now I
you know take my own Facebook message
history and generate more Facebook
messages like me and people generating
fake politician content or you know
there's a bunch of things there where
you at least have to think is this going
to be good for the world there's the
flip side which is I think that there's
a lot of awesome applications that we
really want to see like creative
applications in terms of if you have
sci-fi authors that can work with this
tool and come up with cool ideas like
that seems that seems awesome if we can
write better sci-fi through the use of
these tools and we've actually had a
bunch of people write in to us asking
hey can we use it for you know for a
variety of different creative
applications so the positive I actually
pretty easy to imagine there if you know
the usual
NLP applications are really interesting
but let's go there it's kind of
interesting to think about a world where
look at Twitter where that just fake
news but smarter and smarter BOTS being
able to spread in an interesting complex
in that working way in information that
just floods out us regular human beings
with our original thoughts so what are
your views of this world with deep
t20 right what are you how do we think
about again it's like one of those
things about in the 50s trying to
describe the the internet or the
smartphone what do you think about that
world the nature of information do we
and one possibility is that we'll always
try to design systems that identify it
robot versus human and we'll do so
successfully and so we will authenticate
that we're still human and the other
world is that we just accept the part
the fact that we're swimming in a sea of
fake news and just learn to swim there
well have you ever seen the there so you
know popular meme of of robot eye with a
physical physical arm and pen clicking
the I'm not a robot button yeah I think
I think the truth is that that really
trying to distinguish between robot and
human is a losing battle ultimately you
think it's a losing battle I think it's
a losing battle ultimately right I think
that that is that in terms of the
content in terms of the actions that you
can take I mean think about how captures
have gone
alright the captures used to be a very
nice simple you have this image all of
our OCR is terrible you put a couple of
of artifacts in it you know humans are
gonna be able to tell what what it is an
AI system wouldn't be able to today like
I can barely do CAPTCHAs yeah and I
think that that this is just kind of
where we're going I think CAPTCHAs where
we're a moment in time thing and as AI
you systems become more powerful that
they're being human capabilities that
can be measured in a very easy automated
way that the a eyes will not be capable
of I think that's just like it's just an
increasingly hard technical battle but
it's not that all hope is lost right and
you think about how do we already
authenticate ourselves right the you
know we have systems we have social
security numbers if you're in the u.s.
or you know you have you have uh you
know ways of identifying individual
people and having real world identity
tied to to digital identity seems like a
step towards you know authenticating the
source of content rather than the
content itself now there are problems
with that how can you have privacy and
unanimity in a world where the only
content you can really trust is or the
only way you can trust content is by
looking at where it comes from and so I
think that building out good reputation
networks maybe maybe one possible
solution but yeah I think that this this
question is it's not an obvious one and
I think that we you know maybe sooner
than we think we'll be in a world where
you know today I often will read a tweet
and be like I feel like a real human
wrote this or you know don't feel like
this is like genuine I feel like I kind
of judge the content a little bit and I
think in the future it just won't be the
case you will get for example the FCC
comments on net neutrality it came out
later that millions of those were
auto-generated and that the researchers
were able to do various statistical tik
techniques to do that what do you do in
a world where those statistical
techniques don't exist it's just
impossible to tell the difference
between humans at any highs and in fact
the the the the most persuasive
arguments are written by by AI all that
stuff it's not sci-fi anymore you okay
GPT to making a great argument for why
recycling is bad for the world you got
to read that be like huh you're right
yeah that's that's quite interesting I
mean ultimately it boils down to the
physical world being the last frontier
of proving so you said like basically
networks of people humans vouching for
humans in the physical world and somehow
the authentication and ends there I mean
if I had to ask you I mean you're way
too eloquent for a human so if I had to
ask you to authenticate like prove how
do I know you're not a robot and how do
you know I'm not a robot you know I
think that's so far were this in the
space this conversation we just had the
physical movements we did is the biggest
gap between us and AI systems is the
physical relation so maybe that's the
last frontier well here's another
question is is you know why why is why
is solving this problem important right
like what aspects are really important
to us I think that probably where we'll
end up is will hone in on what do we
really want out of knowing if we're
talking to a human and and I think that
again this comes down to identity and so
I think that the Internet of the future
I expect to be one that will have lots
of agents out there that will interact
with with you but I think that the
question of is this you know a real
flesh-and-blood human or is this an
automated system
be less important let's actually go
there it's GPT two is impressive and
let's look at GPT 20 why is it so bad
that all my friends are GPT 20 well why
is it so why is it so important on the
internet do you think to interact with
only human beings why can't we live in a
world where ideas can come from models
trained on human data yeah I think this
is I think is actually a really
interesting question this comes back to
the how do you even picture a world with
some new technology right and I think
that that one thing I think is important
is is you know Gosei honesty um and I
think that if you have you know almost
in the Turing test style sense sense of
technology you have a eyes that are
pretending to be humans and deceiving
you I think that is you know that that
feels like a bad thing right I think
that it's really important that we feel
like we're in control of our environment
right that we understand who we're
interacting with and if it's an AI or a
human um that that's not something we're
being deceived about but I think that
the flipside of can I have as a
meaningful of an interaction with an AI
as I can with a human well I actually
think here you can turn to sci-fi and
her I think is a great example of asking
this very question right and one thing I
really love about her is it really
starts out almost by asking how
meaningful are human virtual
relationships right and and then you
have a human who has a relationship with
an AI and that you really start to be
drawn into that right and that all of
your emotional buttons get triggered in
the same way as if there was a real
human that was on the other side of that
phone and so I think that that this is
one way of thinking about it is that I
think that we can have meaningful
interactions and that if there's a funny
joke some sense it doesn't really matter
if it was written by a human or an AI
but what you don't want anyway I think
we should really draw hard lines is
deception
and I think that as long as we're in a
world where you know why do why do we
build AI systems at all alright the
reason we want to build them is to
enhance human lives to make humans be
able to do more things to have human
humans feel more fulfilled and if we can
build AI systems that do that I you know
sign me up so the process of language
modeling
how far do you think it take us let's
look at movie her do you think a dialog
natural language conversation is
formulated by the Turing test for
example do you think that process could
be achieved through this kind of
unsupervised language modeling so I
think the Turing test in it seems real
form isn't just about language right
it's really about reasoning to write
that to really pass the Turing test I
should be able to teach calculus to
whoever's on the other side and have it
really understand calculus and be able
to you know go and solve new calculus
problems and so I think that to really
solve the Turing test we need more than
what we're seeing with language models
we need some way of plugging and
reasoning now how different will that be
from what we already do that's an open
question right might be that we need
some sequence of totally radical new
ideas or it might be that we just need
to kind of shape our existing systems in
a slightly different way but I think
that in terms of how far language
modeling will go it's already gone way
further than many people would have
expected right I think that things like
and I think there's a lot of really
interesting angles to poke in terms of
how much does GBT to understand physical
world like you know you you read a
little bit about fire under water in ng
bt - so it's like okay maybe it doesn't
quite understand what these things are
but at the same time I think that you
also see various things like smoke
coming from flame and you know a bunch
of these things that gbg - it has no
body it is no physical experience it's
just statically read data and I think
that I think that if the answer is like
we don't know yet
then these questions though we're
starting to be able to actually ask them
to physical systems the real systems
that exist and that's very exciting do
you think what's your intuition do you
think if you just scale language
modeling maintain like significantly
scale that reasoning can emerge from the
same exact mechanisms I think it's
unlikely that if we just scale
gbt - that will have reasoning in the
full-fledged way and I think that there
is like you know the type signature is a
little bit wrong right that like there's
something we do with that we call
thinking right where we spend a lot of
compute like a variable amount of
compute
get to better answers right I think a
little bit harder I get a better answer
and that that kind of type signature
isn't quite encoded in a gbt all right G
BT well kind of like it's been a long
time and it's like evolutionary history
baking and all this information getting
very very good at this predictive
process and then at runtime I just kind
of do one forward pass and and am able
to generate stuff and so you know there
might be small tweaks to what we do in
order to get the type signature right
for example well you know it's not
really one forward pass right you know
you generate symbol by symbol and so
maybe you generate like a whole sequence
of thoughts and you only keep like the
last bit or something right um but I
think that at the very least I would
expect you have to make changes like
that yeah yeah just exactly how we you
said think is the process of generating
thought by thought in the same kind of
way you like you said keep the last bit
the thing that we converge towards you
know and I think there's there's another
piece which is which is interesting
which is this out of distribution
generalization right that like thinking
somehow lets us do that right that we
have an experience a thing and yet
somehow we just kind of keep refine our
mental model of it this is again
something that feels tied to whatever
reasoning is and maybe it's a small
tweak to what we do maybe it's many
ideas and we'll take as many decades
yeah so the the assumption they're
generalization out of distribution is
that it's possible to create new new
ideas the pot you know it's possible
that nobody's ever creating new ideas
and then was scaling GPT 2 to GPT 20 you
would you would essentially generalize
to all possible thoughts the Aussie was
gonna have I think just to play devil's
ne how many new new story ideas have we
come up with since Shakespeare right
yeah exactly it's just all different
forms of love and drama and so on okay
not sure if you read bitter lesson a
recent blog post by Ray Sutton no I have
he basically says something that echoes
some of the ideas that you've been
talking about which is he says the
biggest lesson that can be read from so
many years of AI research is that
general methods the leverage
computation are ultimately going to
ultimately win out do you agree with
this so basically and openly I in
general about the ideas you are
exploring about coming up with methods
whether it's GPT to modeling or whether
its opening i-5 playing dota or a
general method is better than a more
fine-tuned expert to tuned a method yeah
so I think that well one thing that I
think was really interesting about the
reaction to that blog post was that a
lot of people have read this as saying
that compute is all that matters and
it's a very threatening idea right and I
don't think it's a true idea either
right it's very clear that we have
algorithmic ideas that have been very
important for making progress and to
really build a GI you want to push as
far as you can on the computational
scale and you want to push as far as you
can on human human ingenuity and so I
think you need both but I think the way
that you phrase the question is actually
very good right that it's really about
what kind of ideas should we be striving
for and absolutely if you can find a
scalable idea you'd pour more compute
into you pour more data into it it gets
better like that's that's the real Holy
Grail and so I think that the answer to
the question I think is yes that that's
really how we think about it that part
of why we're excited about the power of
deep learning the potential for building
an AGI is because we look at the system
that exists in the most successful AI
systems and we realize that you scale
those up they're gonna work better and I
think that that scalability is something
that really gives us hope for being able
to build transformative systems so I'll
tell you this is a partially an
emotional you know a thing that responds
that people often have is computers so
important for state-of-the-art
performance you know individual
developers maybe a 13 year old sitting
somewhere in Kansas or something like
that you know they're sitting they that
might not even have a GPU and or may
have a single GPU a 1080 or something
like that
and there's this feeling like well how
can I possibly compete or contribute to
this world of AI if scale is so
important so for if you can comment on
that and in general do you think we need
to also in the future focus on
democratizing compute resources more
more or as much as we democratize the
algorithms well so the way that I think
about it is that there's this space of a
possible progress right there's a space
of ideas and sort of systems that will
work that will move us forward and
there's a portion of that space and to
some extent increasingly significant
portion in that space that does just
require massive compute resources and
for that fit I think that the answer is
kind of clear and that part of why we
have this structure that we do is
because we think it's really important
to be pushing the scale and to be you
know building these large clusters and
systems but there's another part portion
of the space that isn't about the large
scale compute that are these ideas that
and again I think that for the a is to
really be impactful and really shine
that they should be ideas that if you
scaled them up would work way better
than they do at small scale um
but you can discover them without
massive computational resources and if
you look at the history of recent
developments you think about things like
began or the VA II that these are ones
that I think you could come up with them
without having and you know in practice
people did come up with with them
without having massive massive
computational resources alright I just
talked to being good fellow but the
thing is the initial gaen produce pretty
terrible results right so only because
it was in a very specific it was because
only because they're smart enough to
know that this is quite surprising can
generate anything that they know and do
you see a world there's that too
optimistic and dreamer like to imagine
that the compute resources are something
that's owned by governments and provided
as utility actually some extent this
this question reminds me of of blog post
from one of my former professors at
Harvard this guy map Matt Welsh
who was a systems professor I remember
sitting in his tenure talk right and you
know that he had literally just gotten
tenure he went to Google for the summer
and I then decided he wasn't going back
it's academia right and that kind of in
his bog post makes this point that look
as a systems researcher that I come with
these cool system ideas right and I kind
of a little proof of concept and the
best thing I can hope for is that the
people at Google or Yahoo
which was around at the time I will
implement it and like actually make it
work at scale
right that's like the dream for me right
I built the little thing and they the
big thing that's actually working and
for him he said I'm done with that I
want to be the person who's who's
actually doing this building and and
deploying and I think that there's a
similar dichotomy here right I think
that there are people who really
actually find value and I think it is a
valuable thing to do to be the person
who produces those ideas right who
builds the proof of a concept and yeah
you don't get to generate the coolest
possible Ganim ajiz but you invent it
again right and so that there's that
there's there's a real trade-off there
and I think that's a very personal
choice but I think there's value in both
sides do you think creating AGI
something or some new models would we
would see echoes of the brilliance even
at the prototype level so you would be
able to develop those ideas without
scale the initial so seeds you know I
always like to look at at examples that
exist right look at real precedent and
so take a look at the June 2018 model
that we released that we scaled up to
turn into GPT - and you can see that at
small scale it set some records right
this was you know the devotional GPT we
actually had some some cool generations
that weren't nearly as amazing and
really stunning as the GPT - ones every
but it was promising it was interesting
and so I think it is the case that with
a lot of these ideas
do you see prominence at small-scale but
there is an asterisk here a very big
asterisk which is sometimes we see
behaviors that emerge that are
qualitatively different from anything we
saw it's small scale and that the
original inventor of whatever algorithm
looks at and says I didn't think it
could do that this is what we saw in
DotA
all right so PPO was was created by John
Schulman who's a researcher here and and
with with dota we basically just ran PPO
at massive massive scale and I there's
some tweaks and in order to make it work
but fundamentally it's PPO with the core
and we were able to get this long-term
planning these behaviors to really play
out on a time scale that we just thought
was not possible and John looked at that
and it was like I didn't think it could
do that
that's what happens when you're at three
orders of magnitude more scale contest
to that yeah but it still has the same
flavors of you know at least echoes of
the expected billions although I suspect
with GPT is scaled more and more you
might get surprising things so yeah yeah
you're right it's it's interesting that
it's it's difficult to see how far an
idea will go when it's scaled it's an
open question
we've also at that point with with dota
and PPO like I mean here's a very
concrete one right it's like it's
actually one thing that's very
surprising about dota that I think
people don't really pay that much
attention to is the decree of
generalization out of distribution that
happens right that you have this AI
that's trained against other bots for
its entirety the entirety of its
existence sorry to take a step back and
you can't talk through in his you know a
story of dota a story of leading up to
opening high five and that passed and
what was the process of self play it's a
lot of training yeah yeah yeah yeah so
with donors dota
yeah it's a complex video game and we
started training we started trying to
solve dota because we felt like this was
a step towards the real world relative
to other games like chess or go right
those various free board games where you
just kind of have this board very
discrete moves dota starts to be much
more continuous time so you have this
huge variety of different actions that
you have a 45 minute game with all these
different units and it's got a lot of
messiness to it that really hasn't been
captured by previous games and famously
all of the hard-coded bots for dota were
terrible right just impossible to write
anything good for it because it's so
complex and so this seems like a really
good place to push what's the state of
the art in reinforcement learning and so
we started by focusing on the one versus
one version of the game and and and were
able to solve that we were able to beat
the world champions and that the
learning you know the skill curve was
this crazy exponential right it was like
constantly we were just scaling up that
we were fixing bugs and you know that
you look at the at the skill curve and
it was really very very smooth one it's
actually really interesting to see how
that like human iteration loop yielded
very steady exponential progress and to
want one side note first of all it's an
exceptionally popular video game this
effect is that there's a lot of
incredible human experts at that video
again so the benchmark the trying to
reach is very high and the other can you
talk about the approach that was used
initially and throughout training these
agents to play this game yep and so they
person that we used is self play and so
you have cue agents they don't know
anything they battle each other they
discover something a little bit good and
now they both know it and they just get
better and better and better without
bound and that's a really powerful idea
right that we then went from the one
versus one version of the game and
scaled up to four five versus five right
so you think about kind of like with
basketball where you have this like team
sport you know I need to do all this
coordination and we were able to push
the same idea the same self play to to
really get to the professional level at
the full thigh versus by version of the
game and and and the things I think are
really interesting here is that these
agents in some ways they're almost like
an insect like intelligence right where
the you know there's they've a lot in
common with how an insect is trained
right insect kind of lives in this
environment for a very long time or you
know the the ancestors of this insect
I've been around for a long time and had
a lot of experience it gets baked into
into into this agent and you know it's
not really smart in the sense of a human
right it's not able to go and learn
calculus but it's able to navigate its
environment extremely well and simple
they handle unexpected things in the
environment that's never seen before
pretty well and we see the same sort of
thing with our dota BOTS
right they're able to in within this
game they're able to play against humans
which are something that never existed
in its evolutionary environment totally
different playstyles from humans versus
the bots and yet it's able to handle it
extremely well and that's something I
think was very surprising to us was
something that doesn't really emerge
from what we've seen with PPO at smaller
scale writing the kind of scale we're
running the stuff out was you know I
could take a hundred thousand CPU cores
running with like hundreds of GPUs it's
probably about I you know like you know
it's something like hundreds of years of
experience going into this bot every
single real day and so that scale is
massive and we start to see very
different kinds of behaviors out of the
algorithms that we all know and love
Dora he mentioned beat the world expert
1v1
and then you didn't weren't able to win
505 this year yeah at the best in the
world so what's what's the comeback
story what's first of all talk through
that does exceptionally exciting event
and what's what's the following months
and this year look like yeah yeah so
well one thing that's interesting is
that you know we lose all the time
because we we so the dota team at
opening I we played the bot against
better players than our system all the
time or at least we used to it right
like you know the the first time we lost
publicly was we went up on stage at the
International and we played against some
of the best teams in the world and we
ended up losing both games but we gave
them a run for their money
right the both games were kind of 30
minutes 25 minutes and that they went
back and forth back and forth back and
forth and so I think that really shows
that we're at the professional level and
that kind of looking at those games we
think that the coin could have gone a
different direction and it could have
could have had some wins and so that was
actually very encouraging for us and you
know it's interesting because the
international was at a fixed time right
so we we knew exactly what day we were
going to be playing and we pushed as far
as we could as fast as we could
two weeks later we had a bot that had an
80% win rate versus the one that played
at ti so the march of progress you know
you should think of as a snapshot rather
than as an end state and so in fact well
we'll be announcing our our finals
pretty soon I actually think that we'll
announce our final match I prior to this
podcast being released Cassell's
there should be will be playing will be
playing against the the world champions
and you know for us it's really less
about like that the way that we think
about what's upcoming is the final
milestone the file competitive milestone
for the project right that our goal in
all of this isn't really about beating
humans at dota our goal is to push the
state of the art and reinforcement
learning and we've done that right and
we've actually learned a lot from our
system and that we have I you know I
think a lot of exciting next steps that
we want to take and so you know kind of
a final showcase of what we built we're
going to do this match but for us it's
not really the success or failure to see
you know do
do we have the coin flip go in our
direction or against where do you see
the field of deep learning heading in
the next few years what do you see the
work and reinforcement learning perhaps
heading and more specifically with open
AI all the exciting projects that you're
working on
what is 2019 hold for you massive scale
scale I will put a naturist on that and
just say you know I think that it's
about ideas plus scale you need both so
that's a really good point so the
question in terms of ideas you have a
lot of projects that are exploring
different areas of intelligence and the
question is when you when you think of
scale do you think about growing scale
those individual projects so do you
think about adding new projects and
society today in if you are thinking
about adding new projects or if you look
at the past what's the process of coming
up with new projects and new ideas so we
really have a life cycle of project here
so we start with a few people just
working on a small scale idea and
language is actually a very good example
of this that it was really you know one
person here who was pushing on language
for a long time I mean then you get
signs of life right and so this is like
let's say you know with with the
original gbt we had something that was
interesting and we said okay it's time
to scale this right it's time to put
more people on it put more computational
resources behind it and and then we just
kind of keep pushing and keep pushing
and the end state is something that
looks like dota or robotics where you
have a large team of you know 10 or 15
people that are running things at very
large scale and that you're able to
really have material engineering and and
and and you know sort of machine
learning science coming together to make
systems that work and get material
results that just would've been
impossible otherwise so we do that whole
lifecycle we've done it a number of
times you know typically end to end it's
probably to two years or so to do it I
you know the organization's been around
for three years so maybe we'll find it
we also have longer life cycle projects
but you know we we will work up to those
we have so so one one team that we were
actually just starting Illya and I are
kicking off a new team called the
reasoning team
and that this is to really try to tackle
how do you get neural networks to reason
and we think that this will be a
long-term project and we're very excited
about in terms of reasoning super
exciting topic woody what kind of
benchmarks what kind of tests of
reasoning oh do you envision what what
would if you set back with whatever
drink and you would be impressed that
this system is able to do something what
would that look like not fear improving
they are improving so some kind of logic
and especially mathematical logic I
think so right I think that there's
there's there's kind of other problems
that are dual to if you're improving in
particular you know you think about
programming I think about even like
security analysis of code that these all
kind of capture the same sorts of core
reasoning and being able to do some
amount of distribution generalization it
would be quite exciting if open ai
reasoning team was able to prove that P
equals NP that would be very nice I be
very very very exciting especially if it
turns out the P equals NP that'll be
interesting too
it just it would be ironic and humorous
you know so what problem stands out to
you is uh
the most exciting and challenging
impactful to the work for us as a
community in general and for open AI
this year he mentioned reasoning I think
that's that's a heck of a problem yeah
so I think reasoning is an important one
I think it's gonna be hard to get good
results in 2019 you know again just like
we think about the life cycle takes time
I think for 2019 language modeling seems
to be kind of on that ramp right it's at
the point that we have a technique that
works we want to scale 100 X thousand X
see what happens
awesome do you think we're living in a
simulation I think it's I think it's
hard to have a real opinion about it I
you know it's actually interesting I
separate out things that I think can
have like you know yield materially
different predictions about the world
from ones that are just kind of you know
fun fun to speculate about and I kind of
view simulation it's more like is there
a flying teapot between Mars and Jupiter
like maybe but it's a little bit hard to
know what that would mean for my life so
there is something actionable I'd so
some of the best work opening has done
is in the field of reinforcement
learning and some of the success of
reinforcement learning come from being
able to simulate the problem you trying
to solve so it do you have a hope for
reinforcement for the future of
reinforcement learning and for the
future of simulation like what we're
talking about autonomous vehicles or any
kind of system do you see that scaling
so we'll be able to simulate systems and
enhance be able to create a simulator
that echoes our real world and proving
once and for all even though you're
denying it that we're living in a
simulation question right so you know
kind for the core thereof like can we
use simulation for self-driving cars
take a look at our robotic system dactyl
right that was trained in simulation
using the DOTA system in fact and it
transfers to a physical robot and I
think everyone looks at our dota system
the wreck okay it's just a game how are
you ever going to escape to the real
world
and the answer is well we did it with
the physical robot the noble could
program and so I think the answer is
simulation goes a lot further than you
think if you apply the right techniques
to it now there's a question of you know
are the beings in that simulation gonna
wake up and have consciousness I think
that one seems a lot a lot harder to
again reason about I think that you know
you really should think about like where
where exactly just human consciousness
come from and our own self-awareness and
you know is it just that like once you
have like a complicated enough neural
net do you have to worry about the
agents feeling pain and I think there's
like interesting speculation to do there
but but you know again I think it's a
little bit hard to know for sure well
let me just keep with a speculation do
you think to create intelligence general
intelligence you need one consciousness
and to a body do you think any of those
elements are needed or as intelligence
something that's that's orthogonal to
those I'll stick to the kind of like the
the non grand answer first
right so the non grand answer is just to
look at you know what are we already
making work yoga GPG to a lot of people
would have said that even get these
kinds of results you need real-world
experience you need a body you need
grounding how are you supposed to reason
about any of these things how are you
supposed to like even kind of know about
smoke and fire and those things if
you've never experienced them and GPT
two shows
it you can actually go way further than
that kind of reasoning would predict so
I think that the the in terms of doing
any consciousness do we need a body it
seems the answer is probably not right
that we can probably just continue to
push kind of the systems we have they
already feel general they're not as
competent or as general or able to learn
as quickly as an aged guy would but you
know they're at least like kind of proto
AGI in some way and they don't need any
of those things now now let's move to
the grand answer which is you know if
our neural next Nets conscious already
would we ever know how can we tell right
yeah here's where the speculation starts
become become you know at least
interesting or fun and maybe a little
bit disturbing it depending on where you
take it but it certainly seems that when
we think about animals that there's some
continuum of consciousness you know my
cat I think is is conscious in some way
right I you know not as conscious as a
human and you could imagine that you
could build a little consciousness meter
right you pointed a cat gives you a
little reading we ran a human gives you
much bigger reading what would happen if
you pointed one of those at a dota
neural net and if your training of this
massive simulation do the neural nets
feel pain you know it becomes pretty
hard to know that the answer is no and
it becomes pretty hard to to really
think about what that would mean if the
answer were yes and it's very possible
you know for example you could imagine
that maybe the reason these humans are
have consciousness is because it's a
it's a convenient computational shortcut
all right if you think about it if you
have a being that wants to avoid pain
which seems pretty important to survive
in this environment I'm and once you
like you know eat food then that may be
the best way of doing it is to have a
being that's conscious right that you
know in order to succeed in the
environment you need to have those
properties and how are you supposed to
implement them and maybe this this
consciousness is way of doing that if
that's true then actually maybe we
should expect that really competent
reinforcement learning agents will also
have consciousness but you know it's a
big if and I think there a lot of other
arguments they can make in other
directions I think that's a really
interesting idea that even GPT to has
some degree of consciousness that's
something is actually not as crazy to
think about
it's useful to think about as we think
about what it means to create
intelligence of a dog intelligence of a
cat and the intelligence of human so
last question do you think we will ever
fall in love like in the movie her with
an artificial intelligence system or an
artificial intelligence system falling
in love with a human I hope so
if there's any better way to end it on
love so Greg thanks so much for talking
today thank you for having me
you