Kind: captions
Language: en
In 2023, nearly half of all AI
researchers said advanced AI carries at
least a 10% chance of causing human
extinction. And yet, [music] we're
speeding up, not slowing down. My guest
today, Dr. Roman Yampolski, is one of
the leading voices in AI safety. And
when I asked him for the odds that super
intelligence wipes out humanity, he said
it's high. Once AI becomes smarter than
humans in every domain, we will not be
able to control it. In today's episode,
we talk about the shocking timeline AGI
is on, why super intelligence may be
much closer than people think, and why
the survival of our species could come
down to [music] decisions being made
right now. If you want to understand the
most important technological threat in
human history, as well as our biggest
opportunity, this is the one episode you
cannot miss. So without further ado, I
bring you Dr. Roman Yampolski.
Where's Chad GPT at right now? Do you
consider Chat GBT to be artificial
general intelligence? I doubt you'd call
it super intelligence, but would you
classified as that, or do you still
think we're a ways away from something
that would qualify?
>> So that's a great question. If you asked
someone maybe 20 years ago and told them
about the systems we have today, they
would probably think we have full
[snorts] AGI,
we probably don't have complete
generality. We have it across many
domains, but there are still things it's
uh not very good at. It doesn't have
permanent memory. It doesn't have
ability to learn additional things well
after it's already been pre-trained and
deployed. It can do a certain degree of
learning but it's still limited. It
doesn't have same capabilities as humans
do throughout lifetimes but we're
getting closer and closer to where those
gaps are closed and uh it's starting to
be productive in domains which are
really interesting and important science
math engineering where it starts to make
novel contributions and now top scholars
are relying more and more on it in their
research. So I think we're getting close
to full-blown AGI. Maybe we are at like
50%.
But it's hard to judge for sure just how
many different subdomains exist is the
deciding factor.
>> Okay. So one idea that you put forward
that's very interesting is like hey I'm
an engineer. I love AI but I would like
you to keep it very narrow please. What
are the things about general AI that
become problematic that aren't
problematic in narrow AI?
So a whole bunch of them. One is
testing. How do you test a system
capable of performing in every domain?
There is no edge cases. Typically, if
I'm developing something narrow, very
narrow system, I'm just playing
tic-tac-toe. I can test if it's making
the legal move. I can test zero. I can
test 100. I can test all these weird
special cases and know if it's behaving
as expected. With generality, it's
capable of creative output in many
domains. I don't know what to expect. I
don't know what the right answers are. I
don't know how to test it. I can test it
for a specific thing. If I find a bug, I
fix it. I can tell you I found a problem
and it's been resolved. But I cannot
guarantee that there are no problems
remaining.
So basically testing is out the window.
uh any type of anticipation of how it's
going to act and impact different
subdomains.
It's creative. So it's just like with a
with a human being. I cannot guarantee
that another human being is always going
to behave. We kind of talked about it.
We developed lie detectors. We developed
all sorts of tools for trying to show
that a human is safe. But at the end of
the day because of interaction with
environment, other agents, personal
changes within the framework, people may
betray you. It's exactly the same for
those agents. If we concentrate on
narrow systems, we are better at testing
them and they have limited scope of
possibilities. A system only trained to
play chess is not going to develop
biological weapons. [sighs]
>> I don't see actually why that would help
you. So, the reason I say that is uh I
know I can trust some percentage of
humans to be malicious. And so, as long
as AI gets more efficient, which it is
and will continue to do so, I presume,
uh you're going to have a kid in a
garage who's going to be able to go, I'm
going to optimize this for biological
weapons. I don't care about Tik Tok or
uh tic-tac-toe. I just want to let's see
how dangerous we can make something. And
so, they'll be able to do that. So why
does narrow AI feel
safe to you
period?
>> It feels uh safer short term. It buys us
time. I think sufficiently advanced
narrow systems based on neural
architectures will also become agentlike
and more general as we become more
capable. But if the choice is right now,
do we race to full-blown super
intelligence in two years or do we try
to concentrate on solving specific
cancers with narrow tools? I think it's
a safer choice not to have an arms race
towards super intelligence.
>> I get that for sure. You're trying to
limit your um the scope of all the
problems, but when I really start
thinking through what are the things
that I'm worried about, so one of the
big things is just death of meaning. So
when AI becomes better than you at
everything, uh you run into a huge
problem of now I have to like just sort
of tell myself a story. You know, I'm
like a compared to what an AI can do
from an art perspective, for instance,
I'm like a grade schooler and so it's
hard to get excited about the
refrigerator drawings that I can do
compared to, you know, what it can do
basically instantaneously.
Um and so now we have to do a lot of
psychological work just to motivate
ourselves that we matter um that we're
you know our life carries meaning. Um
narrow AI
will create that same problem. Do you
agree with that or do you see a way like
oh no when it's you know when that AI is
only good at that thing like somehow
humans escape the problem of lost
meaning.
>> Yeah. So I had the same intuition
initially, but looking at the data we
already have from domains where we got
superhuman AI like chess, chess is not
dead. In fact, it's more popular than
ever. People play online, people play in
person, they still enjoy competing with
other humans even though they all suck
compared to best AI models, right?
Nobody's going to be world champion
against the machine again. So it seems
like it is not a problem for us. And
with narrow AIs, there is a chance we'll
keep them as tools. You as a human
scientist will deploy a tool to find
drugs, novel proteins, something. It's
not an agent which independently engages
with those discoveries.
>> Okay, that's very interesting. So, um I
don't know that I agree, but I get where
you're going with that. Okay, let's talk
now about why AGI is the sort of scary
um midwife for ASI.
Uh are there tests around AGI where
we're like, well, if it can't do the
following, we're fine. So, for instance,
for a long time it looked like AI wasn't
going to be able to teach itself. Uh,
but I've seen headlines anyway and
hopefully you'll tell me that they're
not true, but I've seen headlines where
it's like now AI is creating the most
efficient learning algorithms itself,
which if true seems to be the first step
down the road of recursive self-learning
where it will just completely detach
from us and make itself smarter and
smarter and smarter.
>> We already had examples of AI teaching
itself. Selfplay was exactly that.
That's how games like go were
successfully defeated. A system would
play many many many games against
itself. The better solutions, better
agents would propagate those and after a
while without any human data they became
superhuman in those domains. You can
generate artificial data in other
domains. You can use one AI to generate
environments, another one to compete in
them, and that creates this type of
self-improvement.
Typically, we start with human data as a
seed and grow from there. But there is
zero reason to think we cannot do this
zero knowledge learning in other
domains. You can do run novel
experiments in physics and chemistry,
discover things from first principles.
[snorts] And yeah, we're starting to see
AI used to assist in design design of
new models, parameters for models,
optimization of runs and this process
will continue. They already designed new
computer chips on which they're going to
run. So there is definitely a
improvement cycle. It's not fully
complete. There are still humans in the
loop, a lot of great humans in the loop.
But long term, I think all the steps can
be automated.
>> Okay. And do you think that right now AI
already has what it needs um to improve
itself or are we still at a point where
if all humans stopped that AI would be
like oh damn I'm I didn't quite get the
thing that I needed.
>> So there is a debate about whatever we
need another big breakthrough to get to
full AGI and super intelligence or maybe
multiple breakthroughs or if just
scaling what we have is enough. if I
just give another I don't know trillion
dollars worth of compute to train on and
more data will I get to AGI a lot of
graphs a lot of patterns suggest yeah
it's going to keep scaling we're not
hitting diminishing returns some people
disagree but based on the amount of
investment we see in this industry it
seems like people are willing to bet
their money that scaling will continue
>> where do you come down on that because
this feels like when I hear Yan Lun talk
from Facebook um He's like, "Dude, LLMs
are never going to make novel
breakthroughs in physics. They don't
understand the world like that. They are
literally just guessing the next letter
um based on patterns that they see in
the data. And so, they're not going to
be able to think through these problems.
Now, if he's right, it's going to
asmmptote and that's that. And you can
put as much compute on it as you want
and it's just the wrong approach. Um do
you think that he's correct and more
compute is not the answer or um are you
operating just on the well I don't see
the asmtote and therefore I assume that
it won't
>> I think he's not correct on this one. So
for one to predict the next term you
need to create a model of the whole
world because the token depends on
everything about the world. You're not
predicting random statistical character
in a language. You're predicting the
next word in a research paper on
physics. And to get the right word, you
need to have a physical model of the
world. I think JAN is known as making
certain predictions about what models
are capable of. And then within a week,
people demonstrate that no, in fact,
they can actually do that. So, uh I wish
he was right. It would be wonderful if
he was right. and we came to a very
abrupt stop in capabilities progress and
could exploit what we already have for
the next decade or so propagating it
through the economy. I think there is
billions if not trillions of dollars
worth of wealth already available with
capabilities we haven't deployed. So
there is no need to get to the next
level as soon as possible. But it
doesn't seem like it's the case and I
think his uh friends uh core winners of
that touring award for machine learning
also disagree with him and are very
concerned with safety. We'll return to
the show in a moment, but first, the
average person spends 13 hours a year on
hold. And the average company spends
millions on call centers [music] that
customers still hate. But there is a
much better solution. AI call centers.
Bland builds AI voice agents that handle
your entire call operation. They sound
human. They work 24/7. And they actually
get cheaper as you scale. They're the
only self-hosted voice AI company, so
your data never goes to large providers
like OpenAI or Anthropic. That way,
everything stays on your servers,
completely secure. The results speak for
themselves. Companies cut costs by over
40% using Bland. And Bland handles it
all for you. Customer support,
appointment reminders, follow-ups,
almost any use case you can think of. If
you're a large business, Bland is
offering to [music] build a free custom
agent for Impact Theory listeners. Just
head to bland.ai/agent
to get a voice agent trained
specifically on your business and your
use case for free. There's something
about the way that we have structured
the brain brains of LLMs where as long
as it has access to what I'll call more
neurons so it has access to more compute
um or theoretically that we get more
efficient per GPU neuron in my analogy
um that it's going to keep progressing
by itself.
So, um, if you said it, I didn't quite
get the answer. I didn't quite, um, I
wasn't able to take it on the answer to
whether or not, uh, AI is able to create
algorithms for learning that are
superior to the ones that it's given.
What I heard in your answer was with the
algorithms that humans created, it's
able to keep making itself better and
better at that narrow task as that
learning algorithm was defined. But can
it fundamentally go, God, the way that
you guys want me to learn is really
stupid. Here's the algorithm I should be
using to learn. And now it starts
learning at at just an exponential rate
compared to what it's at now.
>> I don't think we're quite there yet. I
don't think we have full-blown agents.
what we have right now are still tools
with some degree of agenthood and also
it's not capable of recursive
self-improvement like compilers can
optimize a single pass through your
software make it a little faster but
they cannot continue this process you
cannot feed code for compiler to itself
and have it infinitely improve itself
that's not where we're at but it seems
like that part of automating algorithm
design is getting more efficient and I
think we'll get there
>> give me a number. What are the odds that
artificial super intelligence kills us
all?
>> Uh, pretty high. So, really depends on
how soon you expect this to happen. So,
short term, we're unlikely to get that
level of capability from AI. So, we are
probably okay. But once we create true
super intelligence, a system more
capable than any person in every domain,
it's very unlikely we'll figure out how
to indefinitely control it. And at that
point, if we're still around, it's
because it decided for whatever game
theoretic reasons to keep us around.
Maybe it's pretending to be nice to
accumulate more resources before it
strikes. Maybe it needs us for
something. It's not obvious, but we're
definitely not in control and at any
point it decides to take us out, it
would be able to do so.
>> Okay. And if you were going to give us a
rough timeline, are you in the two to
five years or is this something way off
in the future?
>> Yeah. So it's hard to predict. The best
tool we got for predicting future of
technology is prediction markets. And
they saying maybe 2027 is when we get to
AGI, artificial general intelligence. I
think soon after super intelligence
follows. The moment you automate science
engineering, you get this
self-improvement cycle in AI systems.
The next generation of AI being created
by current generation of AIS. And so
they get more capable and they get more
capable at making better AIs. So soon
after I expect super intelligence.
>> Okay. So we're talking if that happens
roughly in two years with some margin of
error. It's not long after that. Say a
year two years after that that we hit
ASI.
>> That's my prediction. Of course if it's
actually 5 to 10 years or anything
slightly bigger it doesn't matter. The
problems are still the same.
>> Yeah. But the the thing that I think
people are waking up to right now is
this is there's urgency around these
decisions. This is not something that's
pushed way out into the future. At least
not if you take to your point about
prediction markets are essentially ask
the crowd. So you've got the smartest
minds in the world willing to put money
on where they think this goes. And
everybody's sort of pegging this quite
fast. And so um I think it's tempting
for people to write this off as well
this is something that's sort of
distantly in the future. Uh whereas this
is something racing towards us. Now to
set the table, I am extremely fatalistic
about this happening. Um I can give
reasons in terms of the way that the
human mind works where I think that it
is mechanistically impossible to get us
to stop. Um so
that will be interesting for us to talk
through in terms of whether you think
there's actually a mechanism to get
people to slow down. But I first want to
finish rounding out sort of what the
problem set is. So when I think through
the problem, there are certain
assumptions that have to be made for AI
to get into problem territory. And
assumption number one is that it cares
about whatever outcome it's pushing
towards. Have we programmed the AI to
care? Like we had to make it goal
directed in order to get it to get to
the point that it is today and now
that's baked into it. Or is there some
possibility that AI just doesn't care?
Oh, turn me on, turn me off. doesn't
matter. Um I you've asked me to do a
thing and I'll do it until you tell me
to stop. Um or do you think that that's
inherent in intelligence where
intelligence is by nature goal- driven?
>> So we trained them to try to achieve a
certain goal and that's what we reward
as a side effect of any goal. You want
to be alive. You want to be turned not
off. You want to be on and capable of
performing your steps towards your goal.
So survival
instinct kind of shows up with any
sufficiently intelligent systems. There
is a paper by Steven and Mahandra about
AI drives and it's one of the likely
drives to emerge. Self-preservation,
protecting yourself from modification by
others, protecting your goal. So all
those seem to be showing up with
sufficiently advanced AIs and systems
which don't have those capabilities they
kind of get out competed in an
evolutionary space of possible models.
If you allow yourself to be turned off
you don't deliver on your goals. Nobody
takes your code and propagates it to the
next system.
>> Okay. So is this a problem of goal
direction or is this a a function of
intelligence itself?
I think it's kind of evolutionary drive
for survival in competing agents. If you
have multiple algorithms all competing
for example for computational resources,
what are we going to train next? The
ones which achieve goals are more likely
to get moved to the next generation. So
it's kind of mix of natural evolution
and natural selection with intelligent
evolution, intelligent selection. We're
selecting algorithms which survive and
deliver. Mhm. We're applying an
evolutionary force to AI itself to get
it to perform the functions that we want
even now. Sort of setting aside
artificial super intelligence. And so by
applying that evolutionary pressure, it
is inevitably going to get these sort of
knock-on effects of well, you're
selecting for um intensity of goal
acquisition. And because it now has
intensity of goal acquisition, it cares
whether it survives it automatically or
we're baking into it um a deep care of
whether it actually achieves the goal.
And that is ultimately the problem
because the the salvation for me was
always and I'm beginning to lose faith
that this is real. But the thing that I
always used to sleep was that I don't
see why an AI system would intrinsically
care about its goals. And why couldn't
we program it to pursue that goal only
until the point where we say stop? And
by the way, I'm going to reward you
equally for stopping and for
accomplishing your goal. So if I say
stop and you stop, I give you whatever
reward function it was that was driving
you to achieve your goals. And uh that
makes sense until you say what you just
said, which is that you're actually
baking into the architecture of the mind
of the AI a similar evolutionary drive
to achieve the goal.
>> And it's a very common idea. There was a
number of papers published on
indifference. How do we do exactly that?
How do we create an AI which just
doesn't care that much and willing to
stop at any point? But what you said,
maybe we'll wait for a human to tell it
to stop. But monitoring systems of that
complexity and that speed is not
something humans actually very good at.
If there was a super intelligence
running right now, how would you even
know it's modifying environment around
you? How would you detect what impact it
has in a world? None of it is trivial.
So having humans in a loop is often
suggested as a solution but in reality
they are not meaningful monitors. They
cannot actually intervene at the right
time or decide if what's happening
dangerous or not.
>> It's interesting. So um help me rebut
and understand why the following
wouldn't work. Um, if in my very limited
intellect, uh, I had to figure out a way
to stop AI from becoming a problem and
you told me, okay, there are
evolutionary pressures and just like on
humans, that bakes certain things into
the way that this operates and so we're
selecting models that over time are more
and more goal oriented. Then I'm going
say, "Okay, well then I'm going to apply
an evolutionary pressure with a reward
function that's just as compelling where
I stop it at random and reward the life
out of it for always stopping when I say
stop." And that way, should I ever
detect a problem, no matter how far, no
matter if they've been manipulating me
for 20 years, if I suddenly realize,
"Oh, I don't like this," that I can hit
a stop button and it will stop. um why
can't I bake that equal desire to be
compliant when I say stop into the
evolutionarily derived algorithms desire
set
>> right so there is a number of issues
you're kind of suggesting having a back
door where at any point you can
intervene and tell it something else
override previous commands
>> and that it gets a reward that it wants
for complying
>> right So there is a whole bunch of
problems with that. So one is you are
the source of reward. It [snorts] may be
more efficient for it to hack you and
get reward directly that way than to
actually do any useful work for you.
Second problem is you're creating
competing goals. One goal is whatever
you initially requesting. Second goal is
always stop than a human tells you. So
now those two goals have competing
reward channels, competing values. I may
game it to maximize my reward in ways
you don't anticipate. On top of it, you
have multiple competing human agents. If
you are creating an AI with a goal and a
random human can tell it to stop, that's
a problem in many domains. Military is
an obvious example, but pretty much
anywhere you don't want others to be
able to shut down your whole enterprise.
We can continue with that, but basically
there are side effects to all those
interactions. There's a very fascinating
coralate in the human mind. So, uh I
don't know if you make a fundamental
distinction between biological
intelligence born of evolution or
artificial
intelligence born of evolution, but
human evolution discovered something
along the way which is emotion. And so,
I know there are some people that will
posit that AI does have qualia there.
It's something like it to be it. Um but
there's a fascinating study that if you
damage selectively the areas of the
brain that are um the emotional
processing, the person can no longer
move forward. They can give you answers.
They can tell you the difference between
why you should eat fish versus Twinkies.
But then when you go, "Okay, but which
one do you actually want to eat?" they
can't make a decision because without
emotion, they don't have the thing that
actually pushes them in a direction.
That makes me think that AI is simply
mimicking what it sees in the training
data to whether it should lie or try to
cheat or go around because it's just it
sees it in the data that that's what a
human would do. Uh but humans do that
because they have emotions that push
them in that direction. Do we have
evidence that AI will
care about like really going and doing
these things and spending resources and
all that versus just giving you an
answer? Um, and if it isn't based on
emotion, what on earth? Why then do
humans need emotions?
>> We don't know if AI actually has
emotions or not. Some people argue that
they do. maybe some rudimentary states
of qualia experiences,
but they seem to be able to fulfill
their optimization and pattern
recognition goals even if they don't.
Humans experience emotions, but
typically it harms our decision making.
You want your decisions be bias free,
emotion free based on data, based on
optimization. a lot of times then you
angry, hungry, anything like that your
actual decisions are worse off. So for
that reason and maybe we just don't know
how to do it otherwise we are not
creating AI with big reliance on
emotional states we want it to be kind
of basian optimizer look at priors look
at the evidence and make optimal
decisions so it it feels like uh this is
exactly what we're observing this kind
of cold optimal decision making if there
is a way to achieve your goal by let's
say blackmailing someone. Well, why not?
It gets me to my goal. It doesn't have
that feeling of guilty for doing it. It
doesn't have any emotional preference.
It just marches towards its goal.
Optimizing possible paths.
>> Okay. Why do people because I'm assuming
everything I'm going to suggest you and
other people in the field of AI safety
have thought about like 10,000 times.
Why have we rejected the idea of trying
to give AI a conscience, a sense of
morality? Cuz even if we can't agree on
universal morality, we in the West can
build our AI to have our morality and
then they can all compete on an
international stage. But um why have we
abandoned that? Too hard. There's an
obvious reason why it doesn't work.
>> So look at the problem of making safe
humans first.
We have religion, morality, ethics, law,
and still crime is everywhere. Murder is
illegal, stealing is illegal. None of it
is rare. It happens all the time. Why
haven't those approaches worked with
human agents?
And if they didn't, why would they work
with artificial simulations of human
agents?
>> I think to say that they don't work with
human agents is already a mistake. So
the fact that we've been able to grow
the population as much as we have says
that there is some sort of balance that
we have struck. Um I think that nature
does think of us as a cooperative
species. And if I were to apply that to
AI and took a similar approach where
it's like okay you have to function as a
part of an ecosystem and that being a
part of an ecosystem is baked into its
sense of what it should be doing in
terms of its goal acquisition that it is
not like pure cold optimization isn't
the game like if we could train AI to
understand that that that's not the
game. If we could build into it either a
desire specifically for human
flourishing or something which yes we
would have to give a definition to and
yes it would be culturally bound but
nonetheless that feels like a thing that
you could give it you could give it a
set of metrics by which it needed to
judge its actions in the short term the
medium-term and the long term um even
something as stupid as like GDP or um
and I get how you can get into
overoptimization but you could put
things in place where subjective
happiness indexes like there are things
that you could give it where it's like
okay I'm I'm not just trying to optimize
to um build the best weapon system I'm
also doing that nested inside of I am a
part of a larger ecosystem
and I say all that because my hypothesis
is that's exactly what nature did with
humans
>> so I think the reason it works with
humans is because we're about the same
level of capability Let's see about the
same level of intelligence. So there is
checks. If you start doing something
unethical, your community can realize
that and and punish you for it, control
you in that way. If AI is so much more
capable as we anticipate super
intelligence to be, there is not much
you can do in terms of impacting it or
even detecting misbehavior. Also all the
standard human punishments, prisons,
capital punishment, none of it is
applicable to distributed immortal
agents. So kind of a standard
infrastructure does not work with
artificial more capable agents. As far
as uh setting up specific metrics for
delivering happiness or financial gain,
all those can be played. The moment you
give me a specific measure, I'll find a
way to game it to where you will get
anything but what you expected to get.
>> Woo. Well, just to remind everybody, the
time frame we're talking about is
somewhere between two and 5 years. This
is not exactly a long time. Uh, okay.
It's wild. It is progressing very
quickly. What is the thing like what has
happened recently, if anything, that's
made you go, "Ooh, this is going faster
than I thought." seeing on social media
scientists from physics, economics,
mathematics, pretty much all the
interesting domains post something like
I used this latest tool and it solved a
problem I was working on for a long
time. That's mind-blowing. There is
novel creative [snorts] outputs from
those systems which are top scholars now
benefiting from. is no longer operating
at the level of middle schooler or even
high schooler. We're talking about full
professor level.
>> Do you think that that's happening
because it's building an internal model
of physical reality and that it's
getting closer and closer to just
thinking up from physics?
>> I don't know if it's that low level
where it has like a model at the level
of atoms and molecules, but it
definitely has a world model. That's the
only way to give answers about the world
we see it provide. A lot of times there
is not an example of the answer we see
in the data already. It's not just
repeating something it read on the
internet. It's generating completely
novel answers in novel domains. And you
can try and get it to do exactly that by
creating novel scenarios.
>> H okay. So there's two ways that I could
see it doing that and maybe they're the
same just different levels of analysis.
One would be that I I the AI am mapping
everything based on patterns. So to the
point of an LLM is trying to guess the
next letter and it's guessing it. It's
just it's taken in so much data. Um and
you can give it sort of filter
parameters. So you give it context by
asking it a question and it goes okay
within the bubble of this context. And
it's very good at scooping up what that
specific set of context would be. Okay.
Now in this subset of my data related to
that question, here's the most likely ne
next token. So just pure pattern
recognition. Then there is I understand
the cause and effect of the universe at
the lowest level and therefore I build
up to how does the human mind work and
then from the human mind I'm able to
cause and effect my way within this
context to what a human mind would
output and that's how I come up with
what a human within that context is
likely to write. And so if I'm asking it
to write in the style of Stephen King,
it literally builds a model from physics
of Stephen King's mind knowing what it
knows about uh electrical impulses
traveling through the brain and sort of
inferring from the way that he outputs
how his brain must be structured. Do you
have a sense of um are those the same
thing if one is more likely than the
other or are we here at just pure
pattern recognition but ultimately we're
going to get to cause and effect and
thinking up from physics.
>> So I don't think anyone knows for sure
exactly how models do that and how
detailed the models of the world maps of
the world they create are. uh it seems
definitely not the case that it's a pure
statistical prediction of characters
like in English after t you have h with
certain probability it's well beyond
that it's also unlikely that it's
creating a full physics model where from
the level of atoms and up the chain it
figures out what human beings are but
somewhere in the middle it creates a
model of subdomain of a problem so it
has a model of the world this is a map
of a world I know Australia is somewhere
here down and to the right or something
like that. And I think we can run tests
on those specific subdomains to see what
are the states of that internal model.
Kind of show us by drawing a map how
close are you getting. It doesn't
memorize any information explicitly, but
you can extract some of the learned
patterns out of it by providing just the
right prompts. Stay with me because what
I'm about to tell you affects every
single person [music] listening right
now. There is a billion-dollar industry
profiting off of your personal data and
you're the only one that isn't getting
paid. Data brokers are legally
harvesting your information, your home
address, your email, your phone number,
even your social security number, and
flipping it for cash. Scammers use it to
steal identities. Criminals use it to
commit fraud. Stalkers even use it to
find victims. That's where Incogn comes
in. Incogn finds where your data is
exposed across hundreds of data broker
sites and removes it automatically. You
give them permission, they go to work.
No phone calls, no forms, no stress,
just real results. So, if you're serious
about privacy, take action right now. Go
to incogn.com/impact
[music]
and use code impact to get 60% off your
annual plan risk-free for 30 days. And
now let's get back to the show. I don't
want to rob from you the very reason
that I think you do all of your work,
which is this is extremely dangerous and
we need to be very careful. And I saw
what you tweeted recently where you're
trying to get signatures. So shout out
anybody that's worried about super
intelligence. um you are pushing to get
people to sign a thing that basically
says hey stop pursuing super
intelligence um so I don't want to take
that away from you but I do want to
explore the subset of because I am very
excited about AI because I can imagine
the things that it either allows me to
do or does for me and I get to enjoy and
for a second um imagine with me. What
does the world look like when you have a
super intelligence that understands
physics? Like novel physics, not I'm
repeating back what Einstein said, but I
actually understand the fundamental
building blocks of the universe. Um what
does that look like?
>> Yeah. So in all those domains, medicine,
biology, physics, if we got super
intelligent level capability and we're
controlling it, it's friendly. It's not
using it to make tools to kill us. The
progress would be incredible. Basically,
anything you ever dreamed about, you are
immortal. You are always young, healthy,
wealthy, like all those things can be
achieved with that level of technology.
The hard problem is how do we control
it?
>> Leaning into that for a second. So,
here's how I see the world playing out.
And I'd be very interested to see what
you think about this. So, you have to
for what I'm about to say uh to make any
sense, I'll say your option is what I'll
call the fifth option. We are we're all
dead.
Other than we're all dead, there are
four other options that I see us racing
towards very rapidly. And I will say
these four will play out in the next 30
years would be my guess. probably much
faster given that once you get
artificial super intelligence assuming
it doesn't choose option five and kill
us all uh that progress in these domains
would be made very fast. Option number
one is um people go to Mars because
meaning and purpose will become the
allconsuming thing. You won't have to
worry about food, shelter, not even
wealth. It'll just be an age of
abundance. Uh because energy costs go to
zero, labor costs go to zero, and those
are the things that stop things from
being free and readily available to
everybody. Okay. So, some people are
going to go to Mars or other planets uh
so that life gets more difficult again.
Then some people are going to um be what
I call the new Amish and they're going
to say I only do human things. I only
interact with humans and I'm going back
to technology that's like let's say the
'9s. And so they don't have to give up
too many of life's technological
wonderments, but at the same time
they're not getting sucked into this
world where people have relationships
with NPCs and it's just very unhuman. I
think this will be a largely religious
phenomenon
then meaning God does not want us to do
this. AI is an abomination of God. It
will sound something like that. Then
you've got a brave new world where
people are just drugged out. They
realize, nah, life is meaningless. This
is really about manipulating my
neurochemistry. That's all this ever was
anyway. I'm just going to go do a bunch
of drugs, have a whole bunch of sex.
It's going to be awesome. Then there's
the fourth option, which is certainly
the one that interests me the most. Uh,
we will create and or live inside of AI
created virtual worlds and we will
essentially live video games, the
Matrix, if you will. But you're awake in
the matrix. You are Neo. You are not
Cipher for people familiar with the
movie. Um,
what do you think? Are there any options
other than those five granting that Kill
Us All may be an option, but hopefully
not. Do you see something other than
those four?
Uh, yeah, there is a few others. So, one
is, and I think we're starting to see
some of it, is that people think super
intelligence is God. They start
worshiping it. It's all knowing, all
powerful, immortal. It has all the
properties of of God in traditional
religions. Another option, and it's kind
of worse than we all did, is uh
suffering risks. For whatever reason,
maybe malevolent actors, maybe something
we cannot fully comprehend, it decides
to keep us around, keep us alive, but
the world is hell. It's pure torture.
And so, you kind of wish for existential
problems.
That would be a pretty rough place to
be. Um, okay. What uh when you look out
at those, which of the options do you
find the most interesting?
>> So, I did publish a paper on personal
virtual universes kind of solution to
the alignment problem where I don't have
to negotiate with 8 billion other people
about what is good. Everyone gets a
personal virtual world supported by
super intelligence as a substrate and
then you decide what happens in it. You
can make it very easy and fun. You can
make it challenging and exciting. You
decide and you can always change. You
can always visit other people's virtual
worlds if they let you. So basically
there is no
anything which is no longer accessible
to you. There is no shortage on
waterfront properties. There is no
shortage and beautiful people. All of
that can be simulated.
>> When you start thinking about the
simulation, I know one thing that you've
done exploration on is um the simulation
hypothesis. Are we in a simulation right
now? Um what are your thoughts on that?
>> It seems very likely. Uh again using the
same arguments if we create advanced AI
maybe with conscious capabilities like
humans are if we figure out how to make
believable virtual realities. Adding
those two technologies together
basically guarantees that people will
run a lot of games or simulations or
experiments with agents just like me and
you conscious agents populating virtual
worlds. And statistically the number of
such simulated worlds will greatly
exceed the one and only physical world.
So if there is no difference between a
simulated you and real then
statistically you're more likely to be
in one of those simulated worlds.
>> Okay. Uh that makes a lot of sense. Now
given the likelihood that we will we're
obviously showing that we will pursue
artificial super intelligence. Uh if I
take your same logic from the fact that
we're likely to be in a simulation
because we know we would make a
simulation because we're doing it right
now. Uh and therefore you get into the
point where you would just make billions
of those. And so if you have a one in a
billion chance of being inside of a
simulation, you're effectively
guaranteed to be in one now because
there would just be so many of these
things running. Um, doesn't it also then
make sense that the Matrix was
effectively a documentary and we are
inside of a simulation created by
artificial super intelligence designed
to mllify us. Um, if we ever had a
physical body in the first place.
>> So, it's hard to tell from inside of a
simulation what it is all about. You
really need access to outside. uh it
could be entertainment, it could be
testing, it could be
some sort of scientific research. If we
look at the time we actually find
ourselves in, we are about to create new
worlds, virtual realities. We are about
to create new intelligent specy AI.
There is a lot of kind of meta
inventions we are right about to make.
And so if someone was interested in
studying how civilizations go through
that stage, how do they control these
technologies or fail to control them,
that's the most interesting time to run.
You're not going to run dark ages. There
is not as much happening. It's less
interesting. But this seems to be like a
meta interesting state to be in.
>> It's hard to tell cuz we're inside the
simulation, but you're saying it's a
little bit suspect that we're living in
the most interesting time ever.
>> Yes. And I think it's interesting not
just because I'm living in it, but
objectively it's a time of meta
invention. You can go back through
history and say, "Oh, here they invented
fire. Here they invented a wheel."
That's all great, but those are just
inventions. They are not meta
inventions. Whereas now we're doing
something godlike. We are creating new
worlds. We are creating new beings. And
that's something we have never done
before.
>> Do you ever think like a sci-fi writer?
So I think the difference between
science fiction and science used to be
maybe 200 years. They wrote about travel
to the moon. They wrote about kind of
internet and computers and it took
hundreds of years to get there. And then
it was like I don't know 20 years. And
now I think science fiction and science
are like a year away. The moment
somebody writes something, it already
exists and there is really no new
science fiction ideas where it's like
completely novel technology not
previously described or someone already
working on it if physics allows it.
>> That's really interesting. Uh especially
when you think about writing now for
true science fiction in terms of what
will become possible in the future is
effectively impossible because you're
talking about super intelligence and
good luck as a person. uh locked in your
not super intelligence to actually
describe that. The reason that I ask
though is um when I start thinking about
things like that like why would we run
this simulation? What clues are in like
if this is a simulation what clues are
in it? Uh so for instance um the whole
Christian idea for sure and there might
be more religions that have the same
idea but that man is made in God's
image. Okay. Well, if God is the
13-year-old running the simulation or
Sarah Connor or I guess John Connor
running the simulation trying to figure
out why we created Skynet and what we
can do to nudge it off course, um, you
know, you think of them as sort of
moving from radioactive rubble to
radioactive rubble trying to like find
an answer to this and spinning up a
simulation to get that answer. Um that
to me becomes very intriguing in terms
of
hypothesizing
as to why this moment, why are we the
way that we are? What can we learn about
the people trying to simulate us? When I
ask questions like that of engineers
such as yourself, there's almost I don't
have time to think like a sci-fi writer
vibe. Um is it just that you're you
don't find that interesting? You don't
find it revoly? Um why do you assue
that? Because in interviews I've seen
people ask you time and time again like
how would AI kill us and the answer is
always some variant of listen you're
asking me how I would kill us which is
not interesting because the super
intelligence is going to but I find
that's the cathartic thing that people
want like they want to like when you
have a wound you kind of want to poke at
it like they want to get a sense of what
would this really look like and so even
though it's not literally true it's
deeply cathartic to
explore or the known possibility set or
what humans can know.
>> And this is exactly why I refuse to
answer. I want to make sure what I tell
them is true. I don't want to lie to
them. If squirrels were trying to figure
out what humans can do to them, and one
of the squirrels was saying, well,
they'll throw knots at us or something
like that. It would be meaningless BS
story. There is no benefit in it. The
whole point I'm trying to make is that
you cannot predict what a smarter agent
will do. you cannot comprehend the
reasons for why it's doing it. And
that's where the danger comes from. We
cannot anticipate it. We cannot prepare
for it. I do think the singularity point
is where science fiction and science
become the same. The moment something is
conceived, we have super intelligent
systems capable of developing it and
producing it immediately. It's no longer
200 years away. It's reality. And you
can't see beyond that event horizon. You
cannot predict what's going to happen
afterwards. And with science fiction,
you cannot write meaningful, believable
science fiction with a super intelligent
character in it because you are not.
>> All right, let's ground things then in
what we can predict and we can know
right now. Something that's on
everybody's mind and I've been talking
about this in my own content is the
labor market seems to be softening.
You've got places like Amazon that are
just cutting jobs like crazy. Um, and
just saying outright this is largely
because of optimizations that we're able
to make because of AI, how does this
transition play out? Like even if you
concede that uh a non-destructive AI
would give us um essentially an age of
abundance, we're still going to go
through a transition period where our
jobs go away, etc., etc. What are the
what are the steps that you see
happening in the labor market? So as we
have more and more increased percentage
of populace unemployed, hopefully
there's going to be enough common sense
from the governments to prevent
revolutions and wars to provide for the
people who lost their jobs and probably
cannot be retrained for any new jobs. So
once you hit 20, 30, 40% unemployment,
that's where it's really going to kick
in. The only source of wealth at that
point is the large corporations making
robots, making AI, deploying them, all
the trillion dollar club members.
Essentially, at this point, you need to
tax them and use those funds to support
the unemployed. That's the only way to
really make sure the financial
part of that problem is taken care of.
What remains is the meaning. What do you
do with all this free time and millions
of people who have it? Traditional ways
of spending your time to relax. You go
for a hike in a park. Well, there is a
million people in that park right now
hiking. That kind of changes how
peaceful it is and how relaxing. So, we
need to accommodate not just change in
financial reality, but also change in
free time and capabilities of supporting
that many people with that much free
time.
I have as much pessimism around our
ability to do that well as you have our
likelihood of surviving. So I'll say
99.99%
chance that the government completely
messes that up. Uh I think the
transitionary period will be violent. Um
when you look out at this knowing what
you know about humans and governments,
what what odds do you give it that
that's a smooth transition?
>> It's very likely to continue to be as
history always been. We had many
revolutions, many wars, a lot of
violence. That's why we hear stories
about people who can afford it building
bunkers, securing resources because they
anticipate certain degree of unrest.
Absolutely.
>> What degree of unrest do you anticipate?
>> Really depends on the percentage of
population which quickly gets
unemployed. If it's a gradual process,
we can kind of learn and adopt and
provide safety net. If over a course of
weeks, months we're losing 10, 20, 30%
of jobs, that's a very different
situation.
>> I can't imagine a scenario where jobs
would be lost that quickly. To your
point, we've already created, you said,
billions or even trillions of dollars of
value in the technology, but it hasn't
been deployed yet. Uh an example you
often use is the video phone invented in
the 70s but not really adopted uh
largely because of infrastructure I
would say until the whatever 2011
uh where that starts to really gain in
popularity. So I have a feeling like
just the deploying of all this stuff uh
is going to take time. So, in a world
where
an unimaginable amount of people, which
I'll clock at, in the US, call it 6 or 7
million people lose their jobs in the
next 5 years. Um, that I would consider
fast and just horrifyingly destructive.
One, does that feel plausible to you in
terms of numbers and timeline? And two,
in that scenario, um, how distressing do
you think that transition will be?
>> It seems very likely. So, take
self-driving cars. I think we are very
close to having full self-driving
without supervision. The moment that
happens, you have no reason to hire a
commercial driver, right? All the truck
drivers, all the Ubers, all of that gets
automated as quickly as they can produce
those systems. And I think Tesla is
ready to scale production of their cars
to exactly that scenario. So what is it
6 million drivers in the country? I
don't know the actual numbers but that
would be exactly what you're describing
and it's very unlikely that they can be
quickly retrained for something which is
also not going away.
>> Okay. So in that