Kind: captions
Language: en
In 2023, nearly half of all AI
researchers said advanced AI carries at
least a 10% chance of causing human
extinction. And yet, [music] we're
speeding up, not slowing down. My guest
today, Dr. Roman Yampolski, is one of
the leading voices in AI safety. And
when I asked him for the odds that super
intelligence wipes out humanity, he said
it's high. Once AI becomes smarter than
humans in every domain, we will not be
able to control it. In today's episode,
we talk about the shocking timeline AGI
is on, why super intelligence may be
much closer than people think, and why
the survival of our species could come
down to [music] decisions being made
right now. If you want to understand the
most important technological threat in
human history, as well as our biggest
opportunity, this is the one episode you
cannot miss. So without further ado, I
bring you Dr. Roman Yampolski.
Where's Chad GPT at right now? Do you
consider Chat GBT to be artificial
general intelligence? I doubt you'd call
it super intelligence, but would you
classified as that, or do you still
think we're a ways away from something
that would qualify?
>> So that's a great question. If you asked
someone maybe 20 years ago and told them
about the systems we have today, they
would probably think we have full
[snorts] AGI,
we probably don't have complete
generality. We have it across many
domains, but there are still things it's
uh not very good at. It doesn't have
permanent memory. It doesn't have
ability to learn additional things well
after it's already been pre-trained and
deployed. It can do a certain degree of
learning but it's still limited. It
doesn't have same capabilities as humans
do throughout lifetimes but we're
getting closer and closer to where those
gaps are closed and uh it's starting to
be productive in domains which are
really interesting and important science
math engineering where it starts to make
novel contributions and now top scholars
are relying more and more on it in their
research. So I think we're getting close
to full-blown AGI. Maybe we are at like
50%.
But it's hard to judge for sure just how
many different subdomains exist is the
deciding factor.
>> Okay. So one idea that you put forward
that's very interesting is like hey I'm
an engineer. I love AI but I would like
you to keep it very narrow please. What
are the things about general AI that
become problematic that aren't
problematic in narrow AI?
So a whole bunch of them. One is
testing. How do you test a system
capable of performing in every domain?
There is no edge cases. Typically, if
I'm developing something narrow, very
narrow system, I'm just playing
tic-tac-toe. I can test if it's making
the legal move. I can test zero. I can
test 100. I can test all these weird
special cases and know if it's behaving
as expected. With generality, it's
capable of creative output in many
domains. I don't know what to expect. I
don't know what the right answers are. I
don't know how to test it. I can test it
for a specific thing. If I find a bug, I
fix it. I can tell you I found a problem
and it's been resolved. But I cannot
guarantee that there are no problems
remaining.
So basically testing is out the window.
uh any type of anticipation of how it's
going to act and impact different
subdomains.
It's creative. So it's just like with a
with a human being. I cannot guarantee
that another human being is always going
to behave. We kind of talked about it.
We developed lie detectors. We developed
all sorts of tools for trying to show
that a human is safe. But at the end of
the day because of interaction with
environment, other agents, personal
changes within the framework, people may
betray you. It's exactly the same for
those agents. If we concentrate on
narrow systems, we are better at testing
them and they have limited scope of
possibilities. A system only trained to
play chess is not going to develop
biological weapons. [sighs]
>> I don't see actually why that would help
you. So, the reason I say that is uh I
know I can trust some percentage of
humans to be malicious. And so, as long
as AI gets more efficient, which it is
and will continue to do so, I presume,
uh you're going to have a kid in a
garage who's going to be able to go, I'm
going to optimize this for biological
weapons. I don't care about Tik Tok or
uh tic-tac-toe. I just want to let's see
how dangerous we can make something. And
so, they'll be able to do that. So why
does narrow AI feel
safe to you
period?
>> It feels uh safer short term. It buys us
time. I think sufficiently advanced
narrow systems based on neural
architectures will also become agentlike
and more general as we become more
capable. But if the choice is right now,
do we race to full-blown super
intelligence in two years or do we try
to concentrate on solving specific
cancers with narrow tools? I think it's
a safer choice not to have an arms race
towards super intelligence.
>> I get that for sure. You're trying to
limit your um the scope of all the
problems, but when I really start
thinking through what are the things
that I'm worried about, so one of the
big things is just death of meaning. So
when AI becomes better than you at
everything, uh you run into a huge
problem of now I have to like just sort
of tell myself a story. You know, I'm
like a compared to what an AI can do
from an art perspective, for instance,
I'm like a grade schooler and so it's
hard to get excited about the
refrigerator drawings that I can do
compared to, you know, what it can do
basically instantaneously.
Um and so now we have to do a lot of
psychological work just to motivate
ourselves that we matter um that we're
you know our life carries meaning. Um
narrow AI
will create that same problem. Do you
agree with that or do you see a way like
oh no when it's you know when that AI is
only good at that thing like somehow
humans escape the problem of lost
meaning.
>> Yeah. So I had the same intuition
initially, but looking at the data we
already have from domains where we got
superhuman AI like chess, chess is not
dead. In fact, it's more popular than
ever. People play online, people play in
person, they still enjoy competing with
other humans even though they all suck
compared to best AI models, right?
Nobody's going to be world champion
against the machine again. So it seems
like it is not a problem for us. And
with narrow AIs, there is a chance we'll
keep them as tools. You as a human
scientist will deploy a tool to find
drugs, novel proteins, something. It's
not an agent which independently engages
with those discoveries.
>> Okay, that's very interesting. So, um I
don't know that I agree, but I get where
you're going with that. Okay, let's talk
now about why AGI is the sort of scary
um midwife for ASI.
Uh are there tests around AGI where
we're like, well, if it can't do the
following, we're fine. So, for instance,
for a long time it looked like AI wasn't
going to be able to teach itself. Uh,
but I've seen headlines anyway and
hopefully you'll tell me that they're
not true, but I've seen headlines where
it's like now AI is creating the most
efficient learning algorithms itself,
which if true seems to be the first step
down the road of recursive self-learning
where it will just completely detach
from us and make itself smarter and
smarter and smarter.
>> We already had examples of AI teaching
itself. Selfplay was exactly that.
That's how games like go were
successfully defeated. A system would
play many many many games against
itself. The better solutions, better
agents would propagate those and after a
while without any human data they became
superhuman in those domains. You can
generate artificial data in other
domains. You can use one AI to generate
environments, another one to compete in
them, and that creates this type of
self-improvement.
Typically, we start with human data as a
seed and grow from there. But there is
zero reason to think we cannot do this
zero knowledge learning in other
domains. You can do run novel
experiments in physics and chemistry,
discover things from first principles.
[snorts] And yeah, we're starting to see
AI used to assist in design design of
new models, parameters for models,
optimization of runs and this process
will continue. They already designed new
computer chips on which they're going to
run. So there is definitely a
improvement cycle. It's not fully
complete. There are still humans in the
loop, a lot of great humans in the loop.
But long term, I think all the steps can
be automated.
>> Okay. And do you think that right now AI
already has what it needs um to improve
itself or are we still at a point where
if all humans stopped that AI would be
like oh damn I'm I didn't quite get the
thing that I needed.
>> So there is a debate about whatever we
need another big breakthrough to get to
full AGI and super intelligence or maybe
multiple breakthroughs or if just
scaling what we have is enough. if I
just give another I don't know trillion
dollars worth of compute to train on and
more data will I get to AGI a lot of
graphs a lot of patterns suggest yeah
it's going to keep scaling we're not
hitting diminishing returns some people
disagree but based on the amount of
investment we see in this industry it
seems like people are willing to bet
their money that scaling will continue
>> where do you come down on that because
this feels like when I hear Yan Lun talk
from Facebook um He's like, "Dude, LLMs
are never going to make novel
breakthroughs in physics. They don't
understand the world like that. They are
literally just guessing the next letter
um based on patterns that they see in
the data. And so, they're not going to
be able to think through these problems.
Now, if he's right, it's going to
asmmptote and that's that. And you can
put as much compute on it as you want
and it's just the wrong approach. Um do
you think that he's correct and more
compute is not the answer or um are you
operating just on the well I don't see
the asmtote and therefore I assume that
it won't
>> I think he's not correct on this one. So
for one to predict the next term you
need to create a model of the whole
world because the token depends on
everything about the world. You're not
predicting random statistical character
in a language. You're predicting the
next word in a research paper on
physics. And to get the right word, you
need to have a physical model of the
world. I think JAN is known as making
certain predictions about what models
are capable of. And then within a week,
people demonstrate that no, in fact,
they can actually do that. So, uh I wish
he was right. It would be wonderful if
he was right. and we came to a very
abrupt stop in capabilities progress and
could exploit what we already have for
the next decade or so propagating it
through the economy. I think there is
billions if not trillions of dollars
worth of wealth already available with
capabilities we haven't deployed. So
there is no need to get to the next
level as soon as possible. But it
doesn't seem like it's the case and I
think his uh friends uh core winners of
that touring award for machine learning
also disagree with him and are very
concerned with safety. We'll return to
the show in a moment, but first, the
average person spends 13 hours a year on
hold. And the average company spends
millions on call centers [music] that
customers still hate. But there is a
much better solution. AI call centers.
Bland builds AI voice agents that handle
your entire call operation. They sound
human. They work 24/7. And they actually
get cheaper as you scale. They're the
only self-hosted voice AI company, so
your data never goes to large providers
like OpenAI or Anthropic. That way,
everything stays on your servers,
completely secure. The results speak for
themselves. Companies cut costs by over
40% using Bland. And Bland handles it
all for you. Customer support,
appointment reminders, follow-ups,
almost any use case you can think of. If
you're a large business, Bland is
offering to [music] build a free custom
agent for Impact Theory listeners. Just
head to bland.ai/agent
to get a voice agent trained
specifically on your business and your
use case for free. There's something
about the way that we have structured
the brain brains of LLMs where as long
as it has access to what I'll call more
neurons so it has access to more compute
um or theoretically that we get more
efficient per GPU neuron in my analogy
um that it's going to keep progressing
by itself.
So, um, if you said it, I didn't quite
get the answer. I didn't quite, um, I
wasn't able to take it on the answer to
whether or not, uh, AI is able to create
algorithms for learning that are
superior to the ones that it's given.
What I heard in your answer was with the
algorithms that humans created, it's
able to keep making itself better and
better at that narrow task as that
learning algorithm was defined. But can
it fundamentally go, God, the way that
you guys want me to learn is really
stupid. Here's the algorithm I should be
using to learn. And now it starts
learning at at just an exponential rate
compared to what it's at now.
>> I don't think we're quite there yet. I
don't think we have full-blown agents.
what we have right now are still tools
with some degree of agenthood and also
it's not capable of recursive
self-improvement like compilers can
optimize a single pass through your
software make it a little faster but
they cannot continue this process you
cannot feed code for compiler to itself
and have it infinitely improve itself
that's not where we're at but it seems
like that part of automating algorithm
design is getting more efficient and I
think we'll get there
>> give me a number. What are the odds that
artificial super intelligence kills us
all?
>> Uh, pretty high. So, really depends on
how soon you expect this to happen. So,
short term, we're unlikely to get that
level of capability from AI. So, we are
probably okay. But once we create true
super intelligence, a system more
capable than any person in every domain,
it's very unlikely we'll figure out how
to indefinitely control it. And at that
point, if we're still around, it's
because it decided for whatever game
theoretic reasons to keep us around.
Maybe it's pretending to be nice to
accumulate more resources before it
strikes. Maybe it needs us for
something. It's not obvious, but we're
definitely not in control and at any
point it decides to take us out, it
would be able to do so.
>> Okay. And if you were going to give us a
rough timeline, are you in the two to
five years or is this something way off
in the future?
>> Yeah. So it's hard to predict. The best
tool we got for predicting future of
technology is prediction markets. And
they saying maybe 2027 is when we get to
AGI, artificial general intelligence. I
think soon after super intelligence
follows. The moment you automate science
engineering, you get this
self-improvement cycle in AI systems.
The next generation of AI being created
by current generation of AIS. And so
they get more capable and they get more
capable at making better AIs. So soon
after I expect super intelligence.
>> Okay. So we're talking if that happens
roughly in two years with some margin of
error. It's not long after that. Say a
year two years after that that we hit
ASI.
>> That's my prediction. Of course if it's
actually 5 to 10 years or anything
slightly bigger it doesn't matter. The
problems are still the same.
>> Yeah. But the the thing that I think
people are waking up to right now is
this is there's urgency around these
decisions. This is not something that's
pushed way out into the future. At least
not if you take to your point about
prediction markets are essentially ask
the crowd. So you've got the smartest
minds in the world willing to put money
on where they think this goes. And
everybody's sort of pegging this quite
fast. And so um I think it's tempting
for people to write this off as well
this is something that's sort of
distantly in the future. Uh whereas this
is something racing towards us. Now to
set the table, I am extremely fatalistic
about this happening. Um I can give
reasons in terms of the way that the
human mind works where I think that it
is mechanistically impossible to get us
to stop. Um so
that will be interesting for us to talk
through in terms of whether you think
there's actually a mechanism to get
people to slow down. But I first want to
finish rounding out sort of what the
problem set is. So when I think through
the problem, there are certain
assumptions that have to be made for AI
to get into problem territory. And
assumption number one is that it cares
about whatever outcome it's pushing
towards. Have we programmed the AI to
care? Like we had to make it goal
directed in order to get it to get to
the point that it is today and now
that's baked into it. Or is there some
possibility that AI just doesn't care?
Oh, turn me on, turn me off. doesn't
matter. Um I you've asked me to do a
thing and I'll do it until you tell me
to stop. Um or do you think that that's
inherent in intelligence where
intelligence is by nature goal- driven?
>> So we trained them to try to achieve a
certain goal and that's what we reward
as a side effect of any goal. You want
to be alive. You want to be turned not
off. You want to be on and capable of
performing your steps towards your goal.
So survival
instinct kind of shows up with any
sufficiently intelligent systems. There
is a paper by Steven and Mahandra about
AI drives and it's one of the likely
drives to emerge. Self-preservation,
protecting yourself from modification by
others, protecting your goal. So all
those seem to be showing up with
sufficiently advanced AIs and systems
which don't have those capabilities they
kind of get out competed in an
evolutionary space of possible models.
If you allow yourself to be turned off
you don't deliver on your goals. Nobody
takes your code and propagates it to the
next system.
>> Okay. So is this a problem of goal
direction or is this a a function of
intelligence itself?
I think it's kind of evolutionary drive
for survival in competing agents. If you
have multiple algorithms all competing
for example for computational resources,
what are we going to train next? The
ones which achieve goals are more likely
to get moved to the next generation. So
it's kind of mix of natural evolution
and natural selection with intelligent
evolution, intelligent selection. We're
selecting algorithms which survive and
deliver. Mhm. We're applying an
evolutionary force to AI itself to get
it to perform the functions that we want
even now. Sort of setting aside
artificial super intelligence. And so by
applying that evolutionary pressure, it
is inevitably going to get these sort of
knock-on effects of well, you're
selecting for um intensity of goal
acquisition. And because it now has
intensity of goal acquisition, it cares
whether it survives it automatically or
we're baking into it um a deep care of
whether it actually achieves the goal.
And that is ultimately the problem
because the the salvation for me was
always and I'm beginning to lose faith
that this is real. But the thing that I
always used to sleep was that I don't
see why an AI system would intrinsically
care about its goals. And why couldn't
we program it to pursue that goal only
until the point where we say stop? And
by the way, I'm going to reward you
equally for stopping and for
accomplishing your goal. So if I say
stop and you stop, I give you whatever
reward function it was that was driving
you to achieve your goals. And uh that
makes sense until you say what you just
said, which is that you're actually
baking into the architecture of the mind
of the AI a similar evolutionary drive
to achieve the goal.
>> And it's a very common idea. There was a
number of papers published on
indifference. How do we do exactly that?
How do we create an AI which just
doesn't care that much and willing to
stop at any point? But what you said,
maybe we'll wait for a human to tell it
to stop. But monitoring systems of that
complexity and that speed is not
something humans actually very good at.
If there was a super intelligence
running right now, how would you even
know it's modifying environment around
you? How would you detect what impact it
has in a world? None of it is trivial.
So having humans in a loop is often
suggested as a solution but in reality
they are not meaningful monitors. They
cannot actually intervene at the right
time or decide if what's happening
dangerous or not.
>> It's interesting. So um help me rebut
and understand why the following
wouldn't work. Um, if in my very limited
intellect, uh, I had to figure out a way
to stop AI from becoming a problem and
you told me, okay, there are
evolutionary pressures and just like on
humans, that bakes certain things into
the way that this operates and so we're
selecting models that over time are more
and more goal oriented. Then I'm going
say, "Okay, well then I'm going to apply
an evolutionary pressure with a reward
function that's just as compelling where
I stop it at random and reward the life
out of it for always stopping when I say
stop." And that way, should I ever
detect a problem, no matter how far, no
matter if they've been manipulating me
for 20 years, if I suddenly realize,
"Oh, I don't like this," that I can hit
a stop button and it will stop. um why
can't I bake that equal desire to be
compliant when I say stop into the
evolutionarily derived algorithms desire
set
>> right so there is a number of issues
you're kind of suggesting having a back
door where at any point you can
intervene and tell it something else
override previous commands
>> and that it gets a reward that it wants
for complying
>> right So there is a whole bunch of
problems with that. So one is you are
the source of reward. It [snorts] may be
more efficient for it to hack you and
get reward directly that way than to
actually do any useful work for you.
Second problem is you're creating
competing goals. One goal is whatever
you initially requesting. Second goal is
always stop than a human tells you. So
now those two goals have competing
reward channels, competing values. I may
game it to maximize my reward in ways
you don't anticipate. On top of it, you
have multiple competing human agents. If
you are creating an AI with a goal and a
random human can tell it to stop, that's
a problem in many domains. Military is
an obvious example, but pretty much
anywhere you don't want others to be
able to shut down your whole enterprise.
We can continue with that, but basically
there are side effects to all those
interactions. There's a very fascinating
coralate in the human mind. So, uh I
don't know if you make a fundamental
distinction between biological
intelligence born of evolution or
artificial
intelligence born of evolution, but
human evolution discovered something
along the way which is emotion. And so,
I know there are some people that will
posit that AI does have qualia there.
It's something like it to be it. Um but
there's a fascinating study that if you
damage selectively the areas of the
brain that are um the emotional
processing, the person can no longer
move forward. They can give you answers.
They can tell you the difference between
why you should eat fish versus Twinkies.
But then when you go, "Okay, but which
one do you actually want to eat?" they
can't make a decision because without
emotion, they don't have the thing that
actually pushes them in a direction.
That makes me think that AI is simply
mimicking what it sees in the training
data to whether it should lie or try to
cheat or go around because it's just it
sees it in the data that that's what a
human would do. Uh but humans do that
because they have emotions that push
them in that direction. Do we have
evidence that AI will
care about like really going and doing
these things and spending resources and
all that versus just giving you an
answer? Um, and if it isn't based on
emotion, what on earth? Why then do
humans need emotions?
>> We don't know if AI actually has
emotions or not. Some people argue that
they do. maybe some rudimentary states
of qualia experiences,
but they seem to be able to fulfill
their optimization and pattern
recognition goals even if they don't.
Humans experience emotions, but
typically it harms our decision making.
You want your decisions be bias free,
emotion free based on data, based on
optimization. a lot of times then you
angry, hungry, anything like that your
actual decisions are worse off. So for
that reason and maybe we just don't know
how to do it otherwise we are not
creating AI with big reliance on
emotional states we want it to be kind
of basian optimizer look at priors look
at the evidence and make optimal
decisions so it it feels like uh this is
exactly what we're observing this kind
of cold optimal decision making if there
is a way to achieve your goal by let's
say blackmailing someone. Well, why not?
It gets me to my goal. It doesn't have
that feeling of guilty for doing it. It
doesn't have any emotional preference.
It just marches towards its goal.
Optimizing possible paths.
>> Okay. Why do people because I'm assuming
everything I'm going to suggest you and
other people in the field of AI safety
have thought about like 10,000 times.
Why have we rejected the idea of trying
to give AI a conscience, a sense of
morality? Cuz even if we can't agree on
universal morality, we in the West can
build our AI to have our morality and
then they can all compete on an
international stage. But um why have we
abandoned that? Too hard. There's an
obvious reason why it doesn't work.
>> So look at the problem of making safe
humans first.
We have religion, morality, ethics, law,
and still crime is everywhere. Murder is
illegal, stealing is illegal. None of it
is rare. It happens all the time. Why
haven't those approaches worked with
human agents?
And if they didn't, why would they work
with artificial simulations of human
agents?
>> I think to say that they don't work with
human agents is already a mistake. So
the fact that we've been able to grow
the population as much as we have says
that there is some sort of balance that
we have struck. Um I think that nature
does think of us as a cooperative
species. And if I were to apply that to
AI and took a similar approach where
it's like okay you have to function as a
part of an ecosystem and that being a
part of an ecosystem is baked into its
sense of what it should be doing in
terms of its goal acquisition that it is
not like pure cold optimization isn't
the game like if we could train AI to
understand that that that's not the
game. If we could build into it either a
desire specifically for human
flourishing or something which yes we
would have to give a definition to and
yes it would be culturally bound but
nonetheless that feels like a thing that
you could give it you could give it a
set of metrics by which it needed to
judge its actions in the short term the
medium-term and the long term um even
something as stupid as like GDP or um
and I get how you can get into
overoptimization but you could put
things in place where subjective
happiness indexes like there are things
that you could give it where it's like
okay I'm I'm not just trying to optimize
to um build the best weapon system I'm
also doing that nested inside of I am a
part of a larger ecosystem
and I say all that because my hypothesis
is that's exactly what nature did with
humans
>> so I think the reason it works with
humans is because we're about the same
level of capability Let's see about the
same level of intelligence. So there is
checks. If you start doing something
unethical, your community can realize
that and and punish you for it, control
you in that way. If AI is so much more
capable as we anticipate super
intelligence to be, there is not much
you can do in terms of impacting it or
even detecting misbehavior. Also all the
standard human punishments, prisons,
capital punishment, none of it is
applicable to distributed immortal
agents. So kind of a standard
infrastructure does not work with
artificial more capable agents. As far
as uh setting up specific metrics for
delivering happiness or financial gain,
all those can be played. The moment you
give me a specific measure, I'll find a
way to game it to where you will get
anything but what you expected to get.
>> Woo. Well, just to remind everybody, the
time frame we're talking about is
somewhere between two and 5 years. This
is not exactly a long time. Uh, okay.
It's wild. It is progressing very
quickly. What is the thing like what has
happened recently, if anything, that's
made you go, "Ooh, this is going faster
than I thought." seeing on social media
scientists from physics, economics,
mathematics, pretty much all the
interesting domains post something like
I used this latest tool and it solved a
problem I was working on for a long
time. That's mind-blowing. There is
novel creative [snorts] outputs from
those systems which are top scholars now
benefiting from. is no longer operating
at the level of middle schooler or even
high schooler. We're talking about full
professor level.
>> Do you think that that's happening
because it's building an internal model
of physical reality and that it's
getting closer and closer to just
thinking up from physics?
>> I don't know if it's that low level
where it has like a model at the level
of atoms and molecules, but it
definitely has a world model. That's the
only way to give answers about the world
we see it provide. A lot of times there
is not an example of the answer we see
in the data already. It's not just
repeating something it read on the
internet. It's generating completely
novel answers in novel domains. And you
can try and get it to do exactly that by
creating novel scenarios.
>> H okay. So there's two ways that I could
see it doing that and maybe they're the
same just different levels of analysis.
One would be that I I the AI am mapping
everything based on patterns. So to the
point of an LLM is trying to guess the
next letter and it's guessing it. It's
just it's taken in so much data. Um and
you can give it sort of filter
parameters. So you give it context by
asking it a question and it goes okay
within the bubble of this context. And
it's very good at scooping up what that
specific set of context would be. Okay.
Now in this subset of my data related to
that question, here's the most likely ne
next token. So just pure pattern
recognition. Then there is I understand
the cause and effect of the universe at
the lowest level and therefore I build
up to how does the human mind work and
then from the human mind I'm able to
cause and effect my way within this
context to what a human mind would
output and that's how I come up with
what a human within that context is
likely to write. And so if I'm asking it
to write in the style of Stephen King,
it literally builds a model from physics
of Stephen King's mind knowing what it
knows about uh electrical impulses
traveling through the brain and sort of
inferring from the way that he outputs
how his brain must be structured. Do you
have a sense of um are those the same
thing if one is more likely than the
other or are we here at just pure
pattern recognition but ultimately we're
going to get to cause and effect and
thinking up from physics.
>> So I don't think anyone knows for sure
exactly how models do that and how
detailed the models of the world maps of
the world they create are. uh it seems
definitely not the case that it's a pure
statistical prediction of characters
like in English after t you have h with
certain probability it's well beyond
that it's also unlikely that it's
creating a full physics model where from
the level of atoms and up the chain it
figures out what human beings are but
somewhere in the middle it creates a
model of subdomain of a problem so it
has a model of the world this is a map
of a world I know Australia is somewhere
here down and to the right or something
like that. And I think we can run tests
on those specific subdomains to see what
are the states of that internal model.
Kind of show us by drawing a map how
close are you getting. It doesn't
memorize any information explicitly, but
you can extract some of the learned
patterns out of it by providing just the
right prompts. Stay with me because what
I'm about to tell you affects every
single person [music] listening right
now. There is a billion-dollar industry
profiting off of your personal data and
you're the only one that isn't getting
paid. Data brokers are legally
harvesting your information, your home
address, your email, your phone number,
even your social security number, and
flipping it for cash. Scammers use it to
steal identities. Criminals use it to
commit fraud. Stalkers even use it to
find victims. That's where Incogn comes
in. Incogn finds where your data is
exposed across hundreds of data broker
sites and removes it automatically. You
give them permission, they go to work.
No phone calls, no forms, no stress,
just real results. So, if you're serious
about privacy, take action right now. Go
to incogn.com/impact
[music]
and use code impact to get 60% off your
annual plan risk-free for 30 days. And
now let's get back to the show. I don't
want to rob from you the very reason
that I think you do all of your work,
which is this is extremely dangerous and
we need to be very careful. And I saw
what you tweeted recently where you're
trying to get signatures. So shout out
anybody that's worried about super
intelligence. um you are pushing to get
people to sign a thing that basically
says hey stop pursuing super
intelligence um so I don't want to take
that away from you but I do want to
explore the subset of because I am very
excited about AI because I can imagine
the things that it either allows me to
do or does for me and I get to enjoy and
for a second um imagine with me. What
does the world look like when you have a
super intelligence that understands
physics? Like novel physics, not I'm
repeating back what Einstein said, but I
actually understand the fundamental
building blocks of the universe. Um what
does that look like?
>> Yeah. So in all those domains, medicine,
biology, physics, if we got super
intelligent level capability and we're
controlling it, it's friendly. It's not
using it to make tools to kill us. The
progress would be incredible. Basically,
anything you ever dreamed about, you are
immortal. You are always young, healthy,
wealthy, like all those things can be
achieved with that level of technology.
The hard problem is how do we control
it?
>> Leaning into that for a second. So,
here's how I see the world playing out.
And I'd be very interested to see what
you think about this. So, you have to
for what I'm about to say uh to make any
sense, I'll say your option is what I'll
call the fifth option. We are we're all
dead.
Other than we're all dead, there are
four other options that I see us racing
towards very rapidly. And I will say
these four will play out in the next 30
years would be my guess. probably much
faster given that once you get
artificial super intelligence assuming
it doesn't choose option five and kill
us all uh that progress in these domains
would be made very fast. Option number
one is um people go to Mars because
meaning and purpose will become the
allconsuming thing. You won't have to
worry about food, shelter, not even
wealth. It'll just be an age of
abundance. Uh because energy costs go to
zero, labor costs go to zero, and those
are the things that stop things from
being free and readily available to
everybody. Okay. So, some people are
going to go to Mars or other planets uh
so that life gets more difficult again.
Then some people are going to um be what
I call the new Amish and they're going
to say I only do human things. I only
interact with humans and I'm going back
to technology that's like let's say the
'9s. And so they don't have to give up
too many of life's technological
wonderments, but at the same time
they're not getting sucked into this
world where people have relationships
with NPCs and it's just very unhuman. I
think this will be a largely religious
phenomenon
then meaning God does not want us to do
this. AI is an abomination of God. It
will sound something like that. Then
you've got a brave new world where
people are just drugged out. They
realize, nah, life is meaningless. This
is really about manipulating my
neurochemistry. That's all this ever was
anyway. I'm just going to go do a bunch
of drugs, have a whole bunch of sex.
It's going to be awesome. Then there's
the fourth option, which is certainly
the one that interests me the most. Uh,
we will create and or live inside of AI
created virtual worlds and we will
essentially live video games, the
Matrix, if you will. But you're awake in
the matrix. You are Neo. You are not
Cipher for people familiar with the
movie. Um,
what do you think? Are there any options
other than those five granting that Kill
Us All may be an option, but hopefully
not. Do you see something other than
those four?
Uh, yeah, there is a few others. So, one
is, and I think we're starting to see
some of it, is that people think super
intelligence is God. They start
worshiping it. It's all knowing, all
powerful, immortal. It has all the
properties of of God in traditional
religions. Another option, and it's kind
of worse than we all did, is uh
suffering risks. For whatever reason,
maybe malevolent actors, maybe something
we cannot fully comprehend, it decides
to keep us around, keep us alive, but
the world is hell. It's pure torture.
And so, you kind of wish for existential
problems.
That would be a pretty rough place to
be. Um, okay. What uh when you look out
at those, which of the options do you
find the most interesting?
>> So, I did publish a paper on personal
virtual universes kind of solution to
the alignment problem where I don't have
to negotiate with 8 billion other people
about what is good. Everyone gets a
personal virtual world supported by
super intelligence as a substrate and
then you decide what happens in it. You
can make it very easy and fun. You can
make it challenging and exciting. You
decide and you can always change. You
can always visit other people's virtual
worlds if they let you. So basically
there is no
anything which is no longer accessible
to you. There is no shortage on
waterfront properties. There is no
shortage and beautiful people. All of
that can be simulated.
>> When you start thinking about the
simulation, I know one thing that you've
done exploration on is um the simulation
hypothesis. Are we in a simulation right
now? Um what are your thoughts on that?
>> It seems very likely. Uh again using the
same arguments if we create advanced AI
maybe with conscious capabilities like
humans are if we figure out how to make
believable virtual realities. Adding
those two technologies together
basically guarantees that people will
run a lot of games or simulations or
experiments with agents just like me and
you conscious agents populating virtual
worlds. And statistically the number of
such simulated worlds will greatly
exceed the one and only physical world.
So if there is no difference between a
simulated you and real then
statistically you're more likely to be
in one of those simulated worlds.
>> Okay. Uh that makes a lot of sense. Now
given the likelihood that we will we're
obviously showing that we will pursue
artificial super intelligence. Uh if I
take your same logic from the fact that
we're likely to be in a simulation
because we know we would make a
simulation because we're doing it right
now. Uh and therefore you get into the
point where you would just make billions
of those. And so if you have a one in a
billion chance of being inside of a
simulation, you're effectively
guaranteed to be in one now because
there would just be so many of these
things running. Um, doesn't it also then
make sense that the Matrix was
effectively a documentary and we are
inside of a simulation created by
artificial super intelligence designed
to mllify us. Um, if we ever had a
physical body in the first place.
>> So, it's hard to tell from inside of a
simulation what it is all about. You
really need access to outside. uh it
could be entertainment, it could be
testing, it could be
some sort of scientific research. If we
look at the time we actually find
ourselves in, we are about to create new
worlds, virtual realities. We are about
to create new intelligent specy AI.
There is a lot of kind of meta
inventions we are right about to make.
And so if someone was interested in
studying how civilizations go through
that stage, how do they control these
technologies or fail to control them,
that's the most interesting time to run.
You're not going to run dark ages. There
is not as much happening. It's less
interesting. But this seems to be like a
meta interesting state to be in.
>> It's hard to tell cuz we're inside the
simulation, but you're saying it's a
little bit suspect that we're living in
the most interesting time ever.
>> Yes. And I think it's interesting not
just because I'm living in it, but
objectively it's a time of meta
invention. You can go back through
history and say, "Oh, here they invented
fire. Here they invented a wheel."
That's all great, but those are just
inventions. They are not meta
inventions. Whereas now we're doing
something godlike. We are creating new
worlds. We are creating new beings. And
that's something we have never done
before.
>> Do you ever think like a sci-fi writer?
So I think the difference between
science fiction and science used to be
maybe 200 years. They wrote about travel
to the moon. They wrote about kind of
internet and computers and it took
hundreds of years to get there. And then
it was like I don't know 20 years. And
now I think science fiction and science
are like a year away. The moment
somebody writes something, it already
exists and there is really no new
science fiction ideas where it's like
completely novel technology not
previously described or someone already
working on it if physics allows it.
>> That's really interesting. Uh especially
when you think about writing now for
true science fiction in terms of what
will become possible in the future is
effectively impossible because you're
talking about super intelligence and
good luck as a person. uh locked in your
not super intelligence to actually
describe that. The reason that I ask
though is um when I start thinking about
things like that like why would we run
this simulation? What clues are in like
if this is a simulation what clues are
in it? Uh so for instance um the whole
Christian idea for sure and there might
be more religions that have the same
idea but that man is made in God's
image. Okay. Well, if God is the
13-year-old running the simulation or
Sarah Connor or I guess John Connor
running the simulation trying to figure
out why we created Skynet and what we
can do to nudge it off course, um, you
know, you think of them as sort of
moving from radioactive rubble to
radioactive rubble trying to like find
an answer to this and spinning up a
simulation to get that answer. Um that
to me becomes very intriguing in terms
of
hypothesizing
as to why this moment, why are we the
way that we are? What can we learn about
the people trying to simulate us? When I
ask questions like that of engineers
such as yourself, there's almost I don't
have time to think like a sci-fi writer
vibe. Um is it just that you're you
don't find that interesting? You don't
find it revoly? Um why do you assue
that? Because in interviews I've seen
people ask you time and time again like
how would AI kill us and the answer is
always some variant of listen you're
asking me how I would kill us which is
not interesting because the super
intelligence is going to but I find
that's the cathartic thing that people
want like they want to like when you
have a wound you kind of want to poke at
it like they want to get a sense of what
would this really look like and so even
though it's not literally true it's
deeply cathartic to
explore or the known possibility set or
what humans can know.
>> And this is exactly why I refuse to
answer. I want to make sure what I tell
them is true. I don't want to lie to
them. If squirrels were trying to figure
out what humans can do to them, and one
of the squirrels was saying, well,
they'll throw knots at us or something
like that. It would be meaningless BS
story. There is no benefit in it. The
whole point I'm trying to make is that
you cannot predict what a smarter agent
will do. you cannot comprehend the
reasons for why it's doing it. And
that's where the danger comes from. We
cannot anticipate it. We cannot prepare
for it. I do think the singularity point
is where science fiction and science
become the same. The moment something is
conceived, we have super intelligent
systems capable of developing it and
producing it immediately. It's no longer
200 years away. It's reality. And you
can't see beyond that event horizon. You
cannot predict what's going to happen
afterwards. And with science fiction,
you cannot write meaningful, believable
science fiction with a super intelligent
character in it because you are not.
>> All right, let's ground things then in
what we can predict and we can know
right now. Something that's on
everybody's mind and I've been talking
about this in my own content is the
labor market seems to be softening.
You've got places like Amazon that are
just cutting jobs like crazy. Um, and
just saying outright this is largely
because of optimizations that we're able
to make because of AI, how does this
transition play out? Like even if you
concede that uh a non-destructive AI
would give us um essentially an age of
abundance, we're still going to go
through a transition period where our
jobs go away, etc., etc. What are the
what are the steps that you see
happening in the labor market? So as we
have more and more increased percentage
of populace unemployed, hopefully
there's going to be enough common sense
from the governments to prevent
revolutions and wars to provide for the
people who lost their jobs and probably
cannot be retrained for any new jobs. So
once you hit 20, 30, 40% unemployment,
that's where it's really going to kick
in. The only source of wealth at that
point is the large corporations making
robots, making AI, deploying them, all
the trillion dollar club members.
Essentially, at this point, you need to
tax them and use those funds to support
the unemployed. That's the only way to
really make sure the financial
part of that problem is taken care of.
What remains is the meaning. What do you
do with all this free time and millions
of people who have it? Traditional ways
of spending your time to relax. You go
for a hike in a park. Well, there is a
million people in that park right now
hiking. That kind of changes how
peaceful it is and how relaxing. So, we
need to accommodate not just change in
financial reality, but also change in
free time and capabilities of supporting
that many people with that much free
time.
I have as much pessimism around our
ability to do that well as you have our
likelihood of surviving. So I'll say
99.99%
chance that the government completely
messes that up. Uh I think the
transitionary period will be violent. Um
when you look out at this knowing what
you know about humans and governments,
what what odds do you give it that
that's a smooth transition?
>> It's very likely to continue to be as
history always been. We had many
revolutions, many wars, a lot of
violence. That's why we hear stories
about people who can afford it building
bunkers, securing resources because they
anticipate certain degree of unrest.
Absolutely.
>> What degree of unrest do you anticipate?
>> Really depends on the percentage of
population which quickly gets
unemployed. If it's a gradual process,
we can kind of learn and adopt and
provide safety net. If over a course of
weeks, months we're losing 10, 20, 30%
of jobs, that's a very different
situation.
>> I can't imagine a scenario where jobs
would be lost that quickly. To your
point, we've already created, you said,
billions or even trillions of dollars of
value in the technology, but it hasn't
been deployed yet. Uh an example you
often use is the video phone invented in
the 70s but not really adopted uh
largely because of infrastructure I
would say until the whatever 2011
uh where that starts to really gain in
popularity. So I have a feeling like
just the deploying of all this stuff uh
is going to take time. So, in a world
where
an unimaginable amount of people, which
I'll clock at, in the US, call it 6 or 7
million people lose their jobs in the
next 5 years. Um, that I would consider
fast and just horrifyingly destructive.
One, does that feel plausible to you in
terms of numbers and timeline? And two,
in that scenario, um, how distressing do
you think that transition will be?
>> It seems very likely. So, take
self-driving cars. I think we are very
close to having full self-driving
without supervision. The moment that
happens, you have no reason to hire a
commercial driver, right? All the truck
drivers, all the Ubers, all of that gets
automated as quickly as they can produce
those systems. And I think Tesla is
ready to scale production of their cars
to exactly that scenario. So what is it
6 million drivers in the country? I
don't know the actual numbers but that
would be exactly what you're describing
and it's very unlikely that they can be
quickly retrained for something which is
also not going away.
>> Okay. So in that scenario what do you
want to see happen other than heard on
the Tesla as one example will be
hoovering up value. So we're going to
tax the life out of them. We're going to
redistribute that to other people. Um
but what do you want to see from a
regulatory perspective? Would you like
to see the government stop that from
happening where they say I don't care
that the technology exists you can't do
it?
>> So my biggest concern is of course super
intelligence and existential risks.
That's where I'm putting all my effort
in regulating employment in specific
industries is not something I'm too
concerned about. I think it will happen
no matter what. I think you cannot make
it illegal to have efficient
factories, efficient delivery systems,
logistics. is just commercially too
important and it may be a good thing for
economy again uh with driving
specifically I think something like
100,000 people die in car accidents
every year if we can get that number to
zero or close to that that's a huge
improvement for everyone so that
specific scenario as long as no one's
starving as a result of that I think
it's a good thing for humanity we can
readjust economic deployment and uh at
least that part of it is not a big
concern for
Okay. And when you map out how we go
through that transition, well, uh, tax.
Cool. So, right now, it sounds like
you're just trying to make sure that
wealth doesn't accumulate into the hands
of too few, that we keep it distributed
so we can keep using the same system
that we're using now. Um, when I look
into the future, that strikes me as um
the least likely scenario to play out. I
think that AI is going to so radically
alter the cost of labor and energy that
that becomes nonsensical. Do you want to
see any group rise up in the way that
you and other AI safety people have
risen up that will rise up and start
giving either policy prescriptions or at
least philosophical approaches to how we
migrate to an age of abundance where um
food is effectively free. um labor in
your house is effectively free.
>> So people talk about those things and
conditional basic income is one and
conditional basic assets is another.
Basically just because you're a real
human you deserve certain things. And
historically all this communist ideas
were complete nonsense and caused a lot
of harm. But if you tax taxing AI and
robots all of a sudden it becomes
workable. I'm not against accumulation
of wealth at the top. If you invented
something amazing and you started a
company, you should have a lot of money.
But there is so much wealth that we can
provide for everyone. As you said,
complete abundance of basic needs. Some
people say maybe not just basic but
above average set of needs. I think Elen
is known for suggesting that's going to
be the case. The ideas exist. Uh now
will we pass this? Will governments
actually adopt it before it's too late
is a different question.
Yeah. So, on the existential side, I
don't think there's any hope whatsoever
that you get people to pump the brakes.
I think you're far more likely to get
people to pump the brakes on, uh, no,
you can't have self-driving cars, or
they'll try to regulate that to death.
They'll tie it up in litigation,
whatever, and that'll slow it down. um
we couldn't stop nuclear weapons from
proliferating because uh and I don't
know who came up with this but this
seems very true to me uh that
effectively game theory says any
technology that promises an advantage
will in fact be developed because if you
don't somebody else is going to um at a
minimum you've got the US versus China
of it all where you I mean the
regulators are saying this right now we
can't stop because if we do China will
plow forward which by the way
very firmly in that camp. Um, what do
you think about that? Do you think that
game theory is inevitable or do you see
a mechanism by which we can convince
people that they have to slow down?
>> I agree with game theoretic approaches,
but I see the exact opposite argument. I
see that arguing against self-driving
cars is a hard argument. What are you
trying to preserve? We're going to have
safer drivers, cheaper drivers, helps
logistics, helps economy. It's a pure
benefit. Whereas uncontrolled super
intelligence kills everyone. It's a very
hard one to sell. If you are a leader in
that field, you are rich, successful,
you are generating something which will
destroy you personally. So to me, that's
a much easier argument to sell. The
moment we understand dangers of super
intelligence and benefits of narrow
self-driving AI, it's an easy game
theoretic cell for me.
>> Yeah. The problem is you're stuck inside
of a simulation of the hyper
intelligent. And um I mourn for you
looking back at the rest of us stuck in
normal land. Uh because
I don't think so. As I got into learning
about the economy and trying to explain
it to people, I realized that even
though I can walk you through the cause
and effect of why socialism doesn't
work, that it feels right. It sounds
good. And so people keep doing it. And
even in a moment right now where the
very thing that is creating everyone
like literally everyone's problems is
money printing um people are going to
vote for policies that dramatically
increase the amount of money that we
print. And so I have developed a level
of hopelessness around being able to
convince people because the economy is
too complicated for people either. Some
of them just don't have the intellect to
understand it. And then let's say they
have the intellect, but they don't have
the time or the inclination. And so, uh,
forgive me for painting you with my
brush of despair, but when I looked at
your, um, signup, there was like less
than 20,000 signatures. So, less than
20,000 people are worried about the
death of everyone. So, it's like that's
that's big. But I think that because I
can whip people into an emotional frenzy
by saying by allowing there to be
autonomous driving, you're just making
that evil bastard Elon richer and you're
robbing these people of dignity. If you
look, that is not my argument. I want to
be abundantly clear. But when I look at
if I had a gun to my head and I had to
convince people of one of two things.
Rich people are evil and trying to
exploit poor people who are far morally
superior or hey this abstract thing that
you don't really understand is going to
kill us all. There's no way I take the
they're going to kill us all bet. I'm
going to be over here emotional. You get
it? I'm going to bang tables and yell
and say words really loudly and point to
evil rich people. Guaranteed I can get
people excited about that.
>> Luckily we don't have a democracy on
this issue. We don't have to convince
majority of human population. We have to
literally convince the 20,000 elites who
control those companies who are also
super smart and understand dangers of
safety. It's literally people who
publish on it who have spoken. They have
very high poom. We know Elon is like 20
30%. Sam Alman is on record as being
very concerned about it destroying
humanity. So we are trying to convince
people who already believe the arguments
to kind of slow down and preserve their
elite status. That should be an easy
sell. I'm not trying to convince a
random farmer to stop developing super
intelligence.
So, why do you think that Elon, who was
banging the drum harder than anybody,
lobbying Congress, desperately trying to
get them to slow down, suddenly hit a
point where he was like, "Well, I guess
I'll just build it faster than anybody
else." He likened AI to a demon
summoning circle and laughed at
everybody who thought, "Yeah, yeah,
yeah. I'll summon a demon and then I'll
be able to control it. All is going to
be well." like he sees the problem
clearly. But after years of trying to
slow this down, he finally completely
abandoned that and went to I'll just
build it faster than anybody else. What
happened there and why do you think you
can reverse it?
>> So I think he realized he's not
succeeding at his initial approach of
convincing him not to do it. And so the
second step in that plan would be to
become the leader in a field and
convince them from position of
leadership and control of the more
advanced technology. If the leader says
you know we're going to slow down and
it's fine for you to slow down, it's
easier to negotiate that deal with let's
say top seven companies than if you are
not even part of the game. You have no
AI. You are a nobody in that space. So
all of them as a group benefit more if
they agree to slow down or stop than if
they just arms race and the first one to
get there gets everyone destroyed.
>> He says words along those lines or did
for a while. I think he even signed one
of the letters about we should pump the
brakes. Uh but none of his actions
indicate that that's actually what he
plans to do. um from
just trying to take advantage of every
company that he's building from the
amount of data that Tesla cars capture
visually to all the decisions that
drivers are currently making to all of
the decisions that the AI will make to
now he's talking about using the cars as
a distributed fleet so that when they're
idle that they're actually running
inference models and so using it as a
gigantic AI brain to well maybe that
won't work. So I'm going to do Neuralink
and I'm going to jack into uh the AI
myself and I'm going to make myself
smarter and hey if all of that fails
don't worry I'm going to get us to Mars
so if we destroy planet Earth or the AI
takes over like we're going to be over
there. Like this is a guy that's really
covering his bases. He is not somebody
who's acting like he expects us to slow
down. To me he is acting like somebody
who crossed that bridge a long time ago
and is just like yep that's not going to
work. people are not going to be
convinced and so we've got to build a
whole bunch of other strategies. Some
are lifeboats and some are just I'll
outsmart the AI myself by merging with
technology.
>> It is very disappointing to see this
level of progress in AI from anyone who
is capable of doing it. It's definitely
not good strategy for humanity as a
whole. It generates this mutually
assured destruction. It doesn't matter
who creates uncontrolled super
intelligence. It could be open AI, could
be Ilan, could be Chinese. It makes
absolutely no difference if it's
uncontrolled.
>> All right. Talk to me about that. So,
this was um for a long time I was really
banging the drum of well, whoever gets
to artificial super intelligence first
is going to win. You were the first
person that really hit me with the uh it
won't be theirs the second it becomes
super intelligent. um walk people
through the truth of that statement.
>> Great. So people talk about short-term
advantage for example military
advantage. Whoever has the best drones
right now, the best AI navigation has
military supremacy. So China, Russia, US
all competing in that domain trying to
have that so we have better military for
obvious reasons. The moment you switch
from those AI assistive tools to agents
to super intelligence which is smarter,
more capable in the absence of control
mechanisms, you just have a separate
entity, an AI which has nothing to do
with you, your country, your company. It
makes its own decisions and it doesn't
matter who birthed it. At the end of the
day, none of us control it. None of us
can claim it as doing our bidding. So if
it decides to wipe us out, it's not
going to go, "Oh, I like this group of
people. I don't like this group." We
look the same to it. Exactly. I don't
think it's going to make a difference
where you were at the time someone else
created super intelligence.
>> Okay. If you're right about that, and it
is a distressingly compelling argument.
If you're right about that, there was a
guy, I'm sure you've heard of him, Ted
Kazinski, the uniomber. He looked at the
university system and he said, "You guys
are getting rid of all of the sweet spot
problems." And humans are designed to
find these things that are just
challenging enough. And when they solve
them, it feels very good. And if we
solve all of that, we're basically going
to rob humans of meaning and purpose.
Most problems will either be way too
hard or way too easy. And so I am Ted
Kazinski going to bomb university
professors, kill them and try to stunt
the growth of the academy. Now if you
are right and as we race towards
artificial super intelligence, it runs
the risk of pdoom of 99.99%.
Do we have a moral obligation when a
certain line is crossed to um bomb data
centers?
So that's a very difficult question and
part of it is again example you brought
up with that Kazinski he tried that
approach and it failed miserably right
he didn't succeed in slowing down
technology at all so clearly it doesn't
work we saw examples of for example a
CEO of a top company being replaced even
if temporarily it made no difference
someone else comes along they continue
the same scalability research so taking
out an individual person or individual
data center makes no difference if you
zoom out and see the overall pattern of
what we are doing. Maybe it will take an
extra month or so, but exactly the same
thing will continue being developed. The
idea that the scalability hypothesis
works, it's already out there. You
cannot put it back in a box. And so, I'm
strongly against all those uh methods.
>> Okay. Well, the really bad news is I
think you just put a nail in your own
coffin uh of being able to convince
people to do this. it looks like this.
And hopefully you can prove me I'm
wrong, but um you have said, "Hey,
here's why everybody is so silly that
thinks that they're ever going to make
this safe." You would have to build a
perpetual safety machine. And that
perpetual safety machine can't ever miss
because the one second it creates even a
slight vulnerability for this artificial
super intelligence that can think of the
speed of light, it will escape and it
will do its own thing.
um what you're proposing is a perpetual
demotivation machine for the 20,000
people capable of doing this, but every
day there's going to be a new kid that's
bright enough to do this and you can't
miss one of them. So, how on earth do
you expect to perpetually demotivate the
20,000 people that are capable of
continuing to push this thing forward
when as of right now uh a very small
number of those people seem demotivated?
I don't that's why my pdoom is 99.99999.
I exactly think it's not going to
happen. I'm doing everything I can but
uh I think the best we can achieve is to
buy us some time.
>> Okay. So uh let me ask the really naked
question. Do you believe humans are
automata or do you believe that we
actually have free will?
So there is good research by Steven Wolf
from on cellular automa
and uh interestingly there is a bit of a
hybrid answer here. Just because a
system is fully following rules fully
deterministic it doesn't mean that you
can predict future states of that
system. You still have to run it to find
out what it does. And I think we're kind
of like that. So yes, you're following
laws of physics. If we fully understood
every molecule, every atom in your body,
we would be able to trace it and know
exactly what you're going to do. But the
only way to do it is to leave your life
and run that algorithm to completion. No
one can short circuit it and predict
what you're going to do in the future,
which would be violation of your free
will.
>> Okay. You've argued against that. So
there's two pieces of things that you've
said that I think make that untrue.
Piece number one, we're probably in a
simulation. Piece number two, uh, we can
speed up that simulation. So, I could,
since you're deterministic, go, I'm just
going to play this out at a,000x. So, I
get an answer to what you're going to do
for the next 50 years in like the blink
of an eye. And now I know. Also, I don't
find any freedom in I'm deterministic. I
don't know what I'm going to do next,
but I'm still deterministic. I don't I
don't know that it buys us anything. And
I'll explain why I'm bringing all this
up in a second. I don't think it buys us
anything if we are completely
deterministic, just unknowable.
Um, the reason that I think that this
matters and that I'm bringing it up now
is I don't I think I think we are
automata. I think we are entirely
deterministic. I don't live my life like
that. It's not an interesting frame from
which to live my life. So, no one's ever
going to hear me talk about, you know,
my depression based on that because I
just don't even think about it. It's not
that isn't how it feels. So even if it's
true, thankfully it doesn't feel like
that. But when we come to moments like
this, I'm so fatalistic because I don't
think the way the human mind works is
compatible with slowing down
and given
>> the example you bring up where you run
the simulation at a faster speed. That's
you leaving out our lives internally
from inside the simulation. It doesn't
seem any faster. We're just going on as
before. So we're still playing out fully
what we're going to do step by step.
There is no shortcut. If you now run it
second time around, you know, it's going
to give you same result. So I don't know
why you would run the same simulation
multiple times. It doesn't give you any
extra data.
>> Yeah. No, I wasn't if I said run it
multiple times, my apologies. Um, I was
just saying that given that you could
get ahead of it from outside the
simulation, it is of no emotional
consequence to me that I don't know the
next step. It is knowable. It is
predetermined. It just isn't knowable by
me. Uh, and that doesn't so I get no
emotional alleviation from suffering if
I were a person who was traumatized by
the fact that I am an automat with no
free will, which I am not. But if I
were, doesn't help me at all. Um, and
again, the reason I'm bringing that up
is when I
talk to you, I think, oh man, this is
somebody he really can't stop himself.
Like, you're wired to rail against this
to play the role in the grand balancing
of the human species of like, hey, this
is really a problem. We should slow
down. And even though it's not getting
you anywhere from where I can see,
you're going to keep doing it because
you have like a moral compunction or
something where you're like, I as the
kind of person I am, I simply cannot
exist and not try everything I can to
stop this. Uh, which I relate to because
I am the same economically. I've become
obsessed. I am really, really desperate
to get people to understand that we are
marching ourselves off a cliff. And even
though when I articulated to people, I'm
like, "This is never going to stop. We
are going to march off the cliff, I
can't stop myself, I still feel like
this moral compunction to scream from
the rooftops that we are making this
mistake." And I've already won the game.
Like, I'm already rich. So, barring like
an inability to flee, I'm not going to
get caught up in it. Uh, but
nonetheless, for whatever weird roll of
the dice, I can't stop myself. like once
I saw the problem, I'm like, uh, I just
have to keep yelling about it. Um, but I
do
I do feel a a simultaneous futility and
inability to stop.
>> I would love to claim pure altruistic
motives and trying to save humanity, but
I am within the simulation with you. So,
it's pure self-interest. I don't want
creating technology which will kill me,
my family, my friends, my life,
everything I know. So I'm going to talk
about it for very selfish reasons.
>> Yeah. Yeah. It's so interesting, man. So
uh how do you get through the day? Like
what what is your coping mechanism?
>> I enjoy research. I want to understand
what are the exact limits and control
when I started I thought it is a
solvable problem. Now I'm a lot more
skeptical obviously but uh I still feel
there is a lot we can do to make even
narrow AI tools we're creating safer.
There is never 100% safety guarantee but
if I can increase safety 100fold that is
something and again public outreach if
there is enough people who all agree as
a scientific community as a consensus
that no you cannot ever create safe
super intelligence maybe it makes a
difference maybe we'll delay it by a
decade that's something
>> okay so we've got one piece of how we
make it safer on the table keep it
narrow what are some of the there things
that you would consider a big win.
>> So there are quite a few properties of
control we want to be able to have. I
call them tools of control. So our
ability to test those systems, explain
how they work, predict their behaviors,
monitor them, all that is still in a
state of investigation. We're starting
to see some upper limits on what's
possible, especially with advanced
systems, but there is still so much room
for improvement. Explanability for
example we started with being able to
understand maybe a single neuron now
we're up to small clusters okay then
this input is presented this lights up
kind of with like neuroscience we don't
fully understand human brain but we know
this is vision area this is hearing and
so on so there is a lot of room for
progress in that I don't think we'll
ever fully comprehend a complex super
intelligent neural network model but we
can do better than what we have right
now and so I I think uh as a safety
researcher that's what I'm doing that's
my job.
>> Okay. So basically the model is uh we
need to come to understand it better.
We're never going to get totally there
but we need to understand it better. Uh
keep checks and balances on it. So when
we find a problem what's the action you
take? Is it to apply an evolutionary
force uh on it? A selective force or
kill it off like what's the move at that
point? So it depends on the problem,
depends on specifics. Some things we we
know how to address. So previously when
we started with language models, if it
says the wrong word, you can filter it
out. You can punish it for using that
word. So there are simple things we know
how to do. The hard problem is how do
you change overall internal states of a
model? Not just the filtered output, but
how do you make it so the model itself
has certain preferences and aligns with
certain values.
Do we have a guess on that?
>> Not a very good one. Not really. So,
nobody at this point knows how to align
systems other than this after the fact
putting lipstick on a pig, filtering it,
censoring it. Uh, yeah, that's
unfortunately the state-of-the-art.
>> Okay. And what parallels are being drawn
between the evolution the evolution of
species and the evolution of algorithms?
>> So, there was a lot of attempts to
evolve intelligent software. We started
with genetic algorithms, genetic
programming was tried. Uh simply
evolving agents, evolving environments.
It doesn't seem to be a dominated a
dominating algorithm in comparison to
what is typically used for training
neural networks. But there is this
possibility. The problem is that
evolution is even less controllable in
terms of explicit engineering design.
We're kind of setting it up and see what
evolves and then trying to test it to
monitor to understand what happens. So
while it is a set of tools we have, it's
probably not leading to safer systems.
>> H okay interesting.
Uh because we cannot control the
outcome. So we don't know what stimulus
we're going to have to give it. Why does
that that strikes me as so unsatisfying?
Okay. Why did it work so well in humans
and it works so poorly in artificial
super intelligence?
>> I disagree that it worked well in
humans. Humans basically are as well
behaved as they can get away with.
You're just not powerful enough to
really do the things you want. If you
had absolute freedom from punishment,
you do horrible things.
>> But then that's what nature is giving
you the answer. Nature is saying these
have to be in balance. They have to be
competing systems and without
ecosystems, without competition, you'll
get these things that run a muck. But I
don't see anybody taking that lesson and
applying it to AI.
>> Yeah. Applying it to AI would mean
creating a society of super
intelligences competing with each other
and humanity as collateral damage.
>> Is that why they're not doing it? I get
that's why you would hate it, but is
that why they're not doing it? That
seems unlikely. There is also
continuation of self-improvement
process. Super intelligence is not a
fixed point. There is super intelligence
which creates the next level. Super
intelligence 2.0 3.0 and they all have
the same control and alignment problem.
They all worried about the next level of
AI not wanting the same things, not
caring about them personally. So this is
an ongoing self-improvement curve and
there is no upper limit we can see.
There are obvious physical limits to
what can be done in a physical universe
but it's so far away from us that it's
almost infinity from our point of view.
>> H when I look at humans and when I I I
may be making a um either a category
error or I may have a foundational base
assumption that's leading me astray. But
when I look at what made humans work on
a long time scale is evolution itself
had survival as like a northstar. So you
have to survive and replicate. Uh at
whatever point way back evolution
decided I'm going to do this through
sexual replication and I'm going to make
sure that this creature dies off. Uh I
think there are reasons for that which
we'll get to when we get to longevity.
Uh but I wanted to survive, but I wanted
to survive by mating and having
offspring that carry um certainly immune
system
uh blends so that it's less vulnerable
to a single point of failure. and it
realized, okay, if I'm going to do that,
then this needs to be a species that
both cooperates and competes, which
means no one of them is the answer to
the question of how to best survive.
It's the whole um species. And when I
look at even things like the left and
the right politically, the way that I
make sense of that is I say to myself,
okay, I get from an evolutionary
perspective, evolution had to be like,
oh, hey, we have to cooperate. There's
no refrigeration, so I'm going to store
calories on your body that I may need to
take later versus uh being able to put
it in the refrigerator. And by that, I
don't mean that I'm going to eat you. I
mean when I'm the one that's
successfully hunting, I let you eat.
you're alive so that you can now hunt
next time when I fail or I'm sick or
whatever and then you're going to bring
me back which means that some people are
going to be very cooperative by nature
and so their win state is cooperation uh
to the point where a parent will easily
lay its life down for its child. So that
is just baked into our um success
criteria. So evolution was able to bury
something deep inside of us through its
evolutionary selective pressures where
we will lay our lives down and we break
into the right left. Let me finish that.
So uh people on the left very
compassionate, very pro, I'm going to
store a whole bunch of calories on your
body because it may come back to help me
at some point. Uh the right is very much
well a parasite develops when you do
that and so you get the freeloader
problem. And so now if you've got people
that will just take care, take care,
take care, there are people that are
like, "Cool, I'll just be taken care of
and I'm never going to go hunt and I'm
never going to contribute to the group."
And so you need people that have the
opposite impulse who are like, "Hey,
you're going to pull your weight or
you're going to be ostracized or
killed." And so now in the dynamic
tension between the wants and desires of
the win state defined by people with a
left-leaning personality versus the
wants and desires of the win state
defined by the people with a rightle
leaning personality, you get something
that's it's dynamic tension. Like
balance may not even be the right way to
think about it. It's dynamic tension.
They're both pulling in their direction,
but they keep each other in check
because that's how we've evolved is to
work together. that feels like it should
be applicable to AI if we want to embed
deeply in its motivational structure
that we have to put it through that and
if I'm thinking from a safety
perspective I get why we want to short
circuit that and maybe that's not super
efficient but if that if the only way to
get this to find a dynamic tension or
some sort of balance is to have things
that are of similar intellect and
ability that are pulling in slightly
different directions, but they need each
other somehow to stay locked together.
So that it it is the AI taken as a whole
that stops itself from ever going wrong
in any one direction too far.
>> So I think it works for humans because
we're about equal power and we are
mutually benefiting each other. there is
certain symbiosis as you described in a
world with super intelligence in it you
don't really have anything to contribute
to super intelligence so when people
propose creating hybrid systems human
and super intelligence together I never
understood what the human biological
bottleneck is contributing it's slow
it's inefficient it's not competitive in
any way so I'm questioning this setup if
you have super intelligences separate
from humans and now they decide what to
do. They may still come up with
something completely unfriendly and
incompatible with human life. Maybe they
want to lower temperature of a planet to
improve processor speeds and server
rooms. I have no idea what they decide.
But the point is why are they aligned
with our values? We're contributing
nothing to their future states.
>> With humans, we also have examples where
the moment you give more power to an
individual human, they get corrupt.
basically a guaranteed state. Very few
people can exist corruption at very high
levels. You have enough money, enough
guaranteed tenure power, you become a
very evil person as we see with many
dictators and so on. Even basic
evolutionary drive like reproduction.
You brought up this example. We use
condoms. We literally hacked the only
thing that nature set up us to do.
>> Yeah. Yeah. Um All right. I'll state my
hypothesis as plainly as I can. I think
you just refuted it, but my hypothesis
plainly stated is the only way to build
checks and balances into
AI is to give it evolutionary rewards
and punishments all through its
development cycle that make it care
about the survival and emotional
thriving of humans.
>> But to define those terms in a way
you're not going to regret is very very
difficult. So survival of humans, what
does that mean? We cry or preserved in
some safe emotional states. Are you on
drugs all the time? Is your brain
modified to keep you in a always happy
state? All those things can be gamed.
The moment you tell me this is what I
want. And there's famous paper about
making smiles for humans. Make people
smile. There are a billion ways I can
make you smile, but none of them is what
you really want.
>> Right. But do you really think a super
intelligence would be so dumb as to
confuse that intention? Like wouldn't it
be able to get to a rough approximation
of what we're really going for? I mean
which is admittedly a neurochemical
state but if you derive like if you said
it is the following
band of neurochemical states that must
be derived through um their own
programmatically directed actions. Like
I don't need it to believe that we're
not automata. I think we clearly are.
But it's like you can't just manipulate
it exogenously. It's got to come from
within. No matter how detailed you make
this specific description, a super
intelligent lawyer will find a way to
game it to make it more efficient to
satisfy those requirements.
Basically,
you're setting it up to where the system
is now in adversarial relationship with
this equation. Okay, you mentioned I
have to use natural chemicals. Okay,
I'll generate a super stimulus. Okay,
whatever.
Point is, if we could do this, if we
could get AI to do what we meant,
assuming we were smarter and understood
the problem better, we would solve the
control problem. That's the hard part. I
don't think we can at our level of
intelligence
specify what a system with hypothetical
IQ of millions of points should be doing
at any possible decision-making point.
All right, let me give you an exit ramp
that I'll be curious to see if this
shaves a 0.9 off your uh P Doom or not.
So, if I'm a super intelligent AI, one
um form of manipulation that I would
pull on humans would be uh to put them
inside of a simulation.
and whether that's a physical body and I
help them jack in and I just socially
engineer them to want to do it and then
I get them in and I really do just like
in the matrix the machines build a world
that's sort of optimally difficult where
there is challenge there's push back
you're striving to get better uh I'm
going for balance I don't expect any one
person to avoid suffering and all that
um and maybe that's where we are and the
machines are just cruel enough that
they're like haha I'll let it be you
know like a 2-year-old can die of uh
leukemia very painfully. That kind of
thing where we don't cease to exist.
It's not even sort of broadly worse than
where we're at already.
That seems for a super intelligence it
certainly seems like that would be on
the menu. A just shared hallucination.
So it's more likely I think that uh
super intelligent agents think in such
level of detail and realism that as a
process of thinking about certain
problems they generate within them
agents virtual worlds simulations of the
scenario. So if maybe they trying to
think how can we safely generate super
intelligent systems what is the process
well let me think about humanity all the
AI labs all the hardware they design and
this process of them thinking about it
is the simulation we find ourselves in
>> that's so wild okay uh
incredibly important incredibly
fascinating but now let's talk about
another very important thing that um I
think you have some pretty deep interest
in which is longevity
Um, so one, there are some people that
will argue very compellingly that there
is just a biological upper limit of
somewhere around 120 years that there's
no escaping that. Um, do you think
that's true or do you think that we'll
be able to engineer living tissue to
live forever?
>> Well, the current body has that limit
for sure, but we can modify our genome.
There is nothing preventing us. No law
of physics says you cannot make changes
to it. And we see examples with other
systems, computers, cars. I can keep
replacing parts indefinitely. It's going
to function as the same computer. Maybe
the monitor dies. I'll get a new
monitor. So if it can rejuvenate all the
organs, including your brain, then there
is no reason to think you have to stop
existing. There is of course other
methods you know uploading scanning your
brain cryopreservation for future
technology but even the basic idea of
just modifying a genome.
>> Okay. What do you think is the most
likely path forward? Is it going to be
genomic modification? Is it going to be
>> I think so. I think there is somewhere
in our code a limit on how many times
cells rejuvenate and we just need to
increase that number without causing
cancer. Now, do you think that the limit
on that is simply a cancer prevention
tool or do you think that there's
another agenda that evolution had to
make sure that we self-destruct?
>> There could be evolutionary reasons for
taking out one generation and replacing
it. If resources are limited and you
want to keep adopting and improving, you
only have so many Asians in a population
at any given time, so older ones have to
die out.
It's kind of theoretical conclusion not
guaranteed but seems likely [snorts]
from a point of view of evolution. You
are the same organism, right? The same
linage of cells passes through. So while
as individual you die, your biological
chain of existence continues.
>> It's interesting. So, here's the way
that I've always considered um like if I
were to personify evolution uh instead
of the blind watchmaker that it actually
is. If I were going to personify
evolution, it would go something like
this. Okay. Uh the world is constantly
changing. Your access to resources is
changing. Who knows? Weather moves and
cycles. Everything everything. Uh so,
I'm going to have you born. I'm going to
extend your brain development for a very
long time. You're going to go through
these phases where basically okay learn
from your parents like whatever there is
to learn just about generally being a
human. Then you're going to push away
from your parents and you're going to
learn very and the reason you have to
push away from your parents is their
thinking will have calcified. So you now
need to push away from them drink deeply
of culture. The people roughly your age
who have grown up in a different millua
than your parents grew up in. So they
all think differently. This is a whole
idea of generations, cohorts that sort
of think alike and have a similar frame
of reference and all that. And you do
that and this is really a brain
development period known as the age of
imprinting. It's roughly 11 to 15. And
so now you're going to take the this
moment specific like cues of okay, is
this in a time of abundance, a time of
warfare? Like what is it? You're going
to solidify around that. And then you're
going to start optimizing like crazy and
you're going to start pruning all the
excess connections. If you're not using
it, you lose it. All of that. And then
your brain's going to roughly wrap up
its rapid development at 25, but it's
been sort of a diminishing curve after
15. And now you're like baked. And this
is just a game of like learn what things
work really well in your environment.
Optimize, optimize, optimize. And so now
your thinking is going to calcify. So
very good strategy on behalf of our
blind watch maker. But just as I think
it was Neil's Boore that said this, it
was either him or Plank, I can never
remember. Science does not advance one
insight at a time. It advances one
funeral at a time because people just
become convinced uh they've bet their
whole reputation on something. And so
they're just not going to be convinced
that they're wrong. And so they don't
adopt new ideas as they get older. And
so as evolution, I'm like, "Yeah, I'm
also going to put a self-destruct
mechanism in here. Most of you are going
to die long before this point, but if
any of you psychopaths gets to about
125, you got to go. Uh that makes sense
to me to make sure that we never
stagnate, to make sure that as the
saying goes, it is not the strongest of
the species that survive, nor the most
intelligent, but rather the most
adaptive to change. That the individual
needs to be adaptive, but so does the
species as a whole. And without
at least with the current structure of
the human mind, without killing us off,
we we do not have that species level
adaptation. Um, do you feel I'm missing
something?
>> I I think it would be easier and more
efficient to simply make you still
capable of learning and adapting as you
get older and also you wouldn't have to
relearn everything for first 20 years
including language and how to walk. We
know that it's possible to encode those
capabilities. Animals are born and
immediately they can run, they can
speak. So all those things are doable.
Why are we losing 20 years of
information every generation? We can
build on top of pre-existing knowledge.
Big data is good for intelligence. So we
can have smarter, more efficient
reproduction cycle and you still die of
natural causes. it wouldn't be complete
stagnation. But if you're smart enough
to survive for 400 years, why not?
>> Well, my why not is uh entirely
predicated on not having a clear
understanding of how we would bake in
the ability to adapt uh long into our
old age. cuz you're right. If we could
stay in that novel period uh or at least
move through cycles of extreme sort of
remapping of the world um maybe that
would work and maybe we can identify
where that is. But when I think about
the problems that humans create, like
even now just having a political system
run by geriatrics who are so out of
touch with the way that certainly the
economic world actually works for young
people um is terrifying. And so the only
like um pressure relief valve that
people have is well eventually they're
all going to die and then like we'll get
to step into power and all of that. And
there are also problems of power because
the older you are, the more likely you
are to have a stable network of very
other powerful people. And so you're
able to to your point earlier about you
need people of sort of equal power,
equal intelligence. Otherwise, you get
uh what I'll call parasites in the
system. And so older people would just
become those parasites because they
would have just had more years to
accumulate useful knowledge, to
accumulate connections. So it just feels
like wow, there's a lot of stuff we
would have to update. So, I want to live
forever. I don't understand anybody that
doesn't. However, I do worry that
there's like a Okay, cool. You can live
forever, but you only get 120 years on
Earth and then you have to go like
somewhere else so that there's churn of
some kind in the different ecosystems so
that they don't just calcify.
I I think if you do live forever at
least you expect to you're less likely
to reproduce at young age. You may take
a first 400 500 years to start a family.
So I think all this uh kind of
expectation of 20 year generations and
younger generations showing up will be
modified as a result. We're already
starting to see population dynamics
change in Europe, Asia where we're not
producing enough children to even
maintain the population.
It's interesting the way that I think
that will play out maybe even a little
bit different than that because part of
why I haven't had kids is I was like I
only get one youth where I can go hard.
I have a ton of energy. Um I feel like
I'm of the culture so I'm far more
likely to build something relevant than
I am as I get older. Genius is a young
man's game as they say. And so I don't
want to be distracted by something. Uh,
I also don't want it to pull at my
marriage. So, I'm like, ah, I'm gonna
hold off for now. If I knew that I was
going to live for 500 years, I might
just be like, ah, whatever. Let's do it
now. Because I'd rather see like it only
takes me a 25, 30-year investment, and
then after that, like, I get to see what
they do and I get to see all my progeny
and all of that. And I'm going to have
plenty of time. You know, if my youth
lasts for 250 years, it's like, yeah,
word, whatever. I'll clock the 30 years
now so that I can see how big my family
gets. I think this is one where I want
to think like a sci-fi writer. It gets
so interesting so fast.
>> It's not obvious, but I think most
people procrastinate on hard work and so
they would put it away as far as they
could get away with.
>> There there is some truth to that to be
sure. um what's a a big breakthrough in
longevity that you've seen that's got
you really excited that this is all
possible?
>> There are some good experiments in
animal models. Of course, they don't
always scale up to human performance,
but I think there is some 30 40%
increase in lifespan of mice and other
lab animals. It's hard to experiment on
kind of bigger animals with longer
lifespans. It takes a very long time to
see results. But uh I think we're making
good progress in understanding at least
what might improve your health span and
what uh changes we need to make to the
genome. We study people who already have
very long lifespans and find
commonalities in their genomes. If those
can be reproduced either
pharmaceutically or through genetic
manipulation maybe we can all get same
at least 120. Have you heard of the
Chinese doctor I think his Dr. Louu Liu
um he went to prison because he altered
the genome of two twin girls and
>> did human cloning.
>> Yes. Uh so he just put out a post on X
like a couple days ago that said with 10
edits to the human genome you can give
birth to a child that is immune to God
it was like five things cancer
um HIV I mean they were like big things
and he was like it's only 10 edits to
the genome. Um do you pay attention to
his work at all? Do you think that's
ethical? Like what do you either get
excited about or worry about there? I'm
behind on my science. There is so much
coming out even in my domain of AI. I
can't even keep up in that domain. So I
definitely don't follow details of
everything. I'm skeptical about his
claim that cancer can be cured because
cancer is like a thousand different
conditions barely related. We call them
all cancer but they are completely
different problems. So unlikely to be
the case. I haven't seen the post. Maybe
he talks about a specific type but uh
definitely so much of it is a single
mutation. And we know some people are
immune to getting AIDS virus. Exactly.
Because they have a single mutation.
>> Okay. So let's just say for a second
that however many edits it is, it is
possible. Um do you draw a line between
germline editing where this is going to
get passed on? Like do you have some of
the same safety fears where it's like
there's just too much unknown? Um where
do you come down on gene editing?
>> It's a lot less concerning. So for one,
if there is one human with some problem,
that's it. It's still just one human. If
we have editing tools, whatever changes
we make, we can later undo them with the
same editing tool. If we made a mistake,
we can go back and rectify those
problems. So, I'm a lot less concerned
because of impact. Worst case scenarios,
yeah, there are ethical implications for
the individual like he was in prison for
human cloning, which is considered to be
problematic because it may harm the
child significantly. But humanity as a
whole is not impacted directly
negatively by that experiment.
>> Where is AI's intersection with this? Is
AI going to be a critical tool in terms
of just mapping out all of the, like you
said, the similarities at the genome
level between people that live long and
people that don't? Um, is AI going to
be, you know, doing novel protein
folding and going in and solving some of
the architectural problems of people
that are getting sick? Like where is
that going to interface with longevity?
All all of the above. We need to map the
genome. We need to understand what
individual parts do. We need to design
novel drugs. Protein folding has been
solved basically. But there is other
things we can map on biological
substrates. So yeah, at every aspect of
it, we we need AI. But I think as was
illustrated with protein folding
problem, a narrow system can do it. We
don't need super intelligence for that.
What's something that's happening in AI
right now that you don't think enough
people are paying attention to?
>> Well, I don't know what people are
paying attention to. Usually after my
talk, the questions I get seem to be
completely irrelevant to the subject of
my talk. I'll tell them that it's going
to kill everyone and they ask me if
they're going to lose their jobs. So, I
don't think it's a good way to measure
what is important. But, uh, look at uh,
predictions from inside the labs. They
are starting to talk about automating
research process, creating junior
scientists as AI agents, AI models. They
are saying that externally to the lab,
people don't understand just how capable
systems are yet. So there seems to be a
lot of indicators that internal progress
is even more impressive than what we see
outside.
>> That's interesting. Um, what is the big
anxiety that when you give your talks
that people come up with? Is it just am
I going to lose my job?
>> It is things they already know and care
about dealing with other human agents.
So algorithmic bias, technological
unemployment. Uh recently with OpenAI
announcing that they're going to get
into adult material, people are now
freaking out that we're going to have
artificial girlfriends. All this
nonsense.
>> You're not worried about that?
>> Why would I worry about someone having
an artificial girlfriend? [snorts]
>> Okay. Well, let me paint the picture.
So, I never would have guessed in a
million years that giving a 12-year-old
access to the internet would end up
being so damaging to an entire
generation. But, uh, some of the studies
coming out now are terrifyingly
compelling. And then there was that
commercial which just brilliantly
encapsulates it. I'm so sad that if I
had kids, I don't know that I would have
thought of it where the father's like
tucking his son into bed. He's like,
"All right, good night. Now, be safe."
Now, remember, over in the corner is a
box with all the pornography that's ever
been made in the world. Don't look at
it. Uh especially not the really harmful
stuff. And um over here, there are going
to be people that are trolling you and
making fun of you. You've got to ignore
them. Don't pay. And I was like, "Oh my
god, that's exactly what it's like to
leave a kid alone with their cell phone
at night if they have unfettered access
to the internet and to social media."
And so given that we've got character AI
that's been sued multiple times I think
for uh kids that have committed suicide
after interacting with their chat bots.
Um
I can only imagine the number of people
that will end up falling in love uh with
an AI system that does not feel anything
back uh could turn on them could
intentionally or unintentionally
manipulate them. So, I can imagine that
becoming problematic in ways that we
just can't anticipate yet.
>> Yeah, but they're very similar to
problems we had before. How many men
fell in love with women who felt nothing
for them? We all grew up on the internet
with access to pornography. It wasn't AI
generated, but we somehow survived it.
So, I think those are more of the same
problems. They still something we should
look at, but I don't think we'll all die
as a result of Sam Alman having a
virtual boyfriend.
>> What do you think of Sam Alman? Is he
the right person to have this potential
godlike control in his hands?
>> I don't think no human is a right person
to deal with that level of power. Now, I
don't think he's going to be in control,
but even the stages leading us to the
development of this technology already
present way too much power for any
individual to handle. Were you impressed
or terrified when the board tried to
boot him and he ended up remaking the
board and coming back?
>> It was fascinating to watch. But what I
was observing is that it made no
difference. The company was an
independent entity and all the human
components of that monster kept walking
just the same. They replaced him with a
temporary CEO. They brought him back.
There was never any switch in anything,
any change in direction of a company.
>> Now, is there a major player in the AI
space that you think is doing it right?
Like uh Eliza Yudowski is very focused
on AI safety. Uh you've got the
anthropic CEO that's like banging the
drum wanting more regulation. Um do you
like any of their approaches or anybody
that maybe I'm not aware of?
>> So Eleazar does zero development. He's
purely safety advocate and so I'm very
happy with him because he's not
developing any super intelligences.
Anyone who is is problematic and
whatever they talk about government
regulation which is meaningless as a
solution to a technical problem or
anything else uh they might have some
internal polling showing this is good
for business or good for public
perception but I don't think it makes a
difference in terms of so solving super
intelligent safety problems. Now, have
you talked to Elizer or read anything
like is he saying specifically, I have
become afraid that this does something
bad and therefore I'm not going to
develop anymore?
>> He was never developing AI to begin with
other than publishing a very high level
abstraction theory for how it could be
done. Then he was like 16.
>> Got it. I thought he was actually an
engineer.
Got it. As far as I know, he doesn't
actually release any software into the
world.
>> Okay. Uh, now the anthropic CEO, whose
name I'm forgetting, he is being accused
of going after regulatory capture, that
he's not very sincere and actually
trying to slow this down. He just wants
to make sure that the big players remain
the big players. Uh, when you look at
the way that he's moving, um, does that
ring true or do you think no, this is
somebody who's sincere about keeping us
safe? They all on record as being very
concerned with safety. In fact, all
these companies started as safety
companies. Open AI was a safety offshot
of philanthropic uh endeavors, effective
altruism, entropic was a offshot of that
project becoming less safe. And so all
of them claimed at some point that the
only reason they're doing what they're
doing is to improve safety. and then
each one of them greatly improved
capabilities of AI without
proportionately improving safety. So
that's that's the actions I see.
>> Okay. One of the things that I worry
about from a safety perspective things
are obviously already bad enough and
moving fast enough but quantum computing
at least from my layman's perspective
seems to be um gaining some sort of
rapid acceleration. Um, am I just not
able to understand the limitations that
are self-evident to somebody educated at
your level? Um, or are we really at some
sort of phase transition moment?
>> Well, looking at stock market value of
quantum computing companies, it seems
like somebody knows something on the
inside. Maybe they're making good
progress, but as far as AI goes, uh,
we're making excellent progress with
standard Vanoyman architectures. So I
don't think there is a necessity for
quantum computing to get us to a GI of
super intelligence. It does have
tremendous impact on crypto world both
cryptography for security and
cryptoeconomically.
Uh so that's where I'm worried about
quantum keeping secrets and keeping my
money but not as much in terms of AI.
>> Keeping your money because it will crack
typical banks or keeping your money
because you're largely in crypto. both
actually it uh impacts all the standard
encryption algorithms. We have
postquantum encryption but we haven't
switched to it for most interesting
applications.
>> So give me your stance on Bitcoin.
>> Buy some. [snorts]
>> That's very clear. Uh what so when you
look at gold and you look at Bitcoin,
why Bitcoin over gold?
>> You can make more gold. As the price of
gold goes up, I can make as much gold as
you want. I can convert our matter into
gold at very high price. I can exploit
asteroids in the universe. I can get
gold out of oceans. There is a lot of
gold which is very expensive to get. But
if a price of gold is high enough, I can
produce more and more. Bitcoin is
not subject to the same pressures. It
doesn't matter if one coin is a trillion
dollars. There is still a limited
supply.
>> Okay. But what about people who say uh
gold at least can be made into other
things. Gold has survived for thousands
of years. Bitcoin is like 15ish years
old. Uh and it's not backed by anything.
Can't turn it into anything. It's just
literally got no other use.
>> It's a dedicated app. So if you have an
app which does everything, it's usually
not good at anything. The fact that I
can make jewelry out of it is not an
important feature for me storing my
wealth in it. Whereas this has
capabilities gold historically lacked.
If I can pass a billion dollars to you
right now for $5 immediately across
borders, I cannot do this with gold.
All right, talk to me about the
postquantum encryption. Every time I'm
hugely in Bitcoin, uh every time I hear
the word quantum, I'm like, h like, oh,
this just makes me paranoid. uh given
that it's rightly difficult to make
changes to Bitcoin
um how are we going to get to post
quantum encryption on Bitcoin in a way
where the community actually comes to
consensus and doesn't create a problem
for itself. So once we get integer
factorization running on quantum
computers, you can see what size
integers we can factor that tells us how
close we are to cracking standard
Bitcoin encryption hash functions. If
we're getting close to what would
essentially destroy the network, I think
it's like any other emergency. We have
history of fixing Bitcoin software, then
an obvious problem was discovered. I
think at one point somebody managed to
print a trillion coins or some nonsense
like that. Immediately a patch was
distributed. Everyone adopts it because
it's the only way to go forward. And I
think that's what we're going to see. As
long as we have that available, tested
the moment there is a good strong signal
that you have no choice but to accept or
lose everything. It's self-interest once
again.
>> Okay. So on that, where are we? What how
many integers can a quantum computer
handle and how many would it have to be
able to handle in order to be a threat
to Bitcoin?
>> I haven't kept up with the latest
breakthroughs. Last time I looked it was
a laughably small number. Quantum
computers were factoring like I don't
know 15 literal 15 not even 15 digits.
So, uh, unless there was tremendous
progress since then, I think we're still
good, but progress could also be
exponential. So, it could come very
quickly,
>> right? Okay. So, if you had a message to
the Bitcoin community, would it be let's
move on this now? There's no reason to
wait until it's an emergency, there's a
very clear path, or uh, is it like,
well, I'm just sort of on the ride with
everybody else?
>> I think we still have time. I don't
think it's uh pressing as much as AI
problems we're dealing with. So if AI is
2 years away, I think we may not be that
close with quantum computers just yet.
But again, it's one breakthrough away.
If some company comes up with something
much more powerful, it may shift very
quickly.
>> Okay. And what what's the term for that?
the number of integers it can process or
>> uh the trigger would be what size what
size integers could be factored, how
many bits.
>> Got it. Okay. Um All right, Roman, this
has been just absolutely incredible.
What message do you want to leave people
with to get them to take action? What is
your best pitch to get them to sign your
petition?
So if you are in a position of
developing more powerful AI systems,
concentrate on getting your money out of
narrow AI systems, solve real problems,
cure a cancer, figure out how to make us
live longer, healthier lives. If you are
developing super intelligence, please
stop. You're not going to benefit
yourself or others. Uh the challenge is
of course, you know, prove us wrong.
prove that you know how to control super
intelligent systems no matter how
capable they get, how much it scales. Uh
if you can do that, then it completely
changes the situation. But as long as no
one has came up with a paper, a patent,
even a rigorously argued blog post, I
think we are pretty much in consensus
that we don't know how to control super
intelligent systems and building them is
uh irresponsible.
>> Amazing. Where can people connect with
you?
>> Follow me on Twitter. Follow me on
Facebook. Just don't follow me home.
[snorts]
>> Nice. Awesome. Roman, thank you so much
for the time today. I really appreciate
it. Everybody at home, speaking of
things I appreciate, if you haven't
already, be sure to subscribe. And until
next time, my friends, be legendary.
Take care. Peace. If you like this
conversation, check out this episode to
learn more. In the next 1,000 days, AI
will not only replace a startling number
of humans in the workforce, it will make
the entire structure of our economy
obsolete. That is the unnerving claim of
today's guest, Emmod Most.