Kind: captions
Language: en
the following is a conversation with
George Hotz he's the founder of comma AI
a machine learning based vehicle
automation company he is most certainly
an outspoken personality in the field of
AI and technology in general he first
gained recognition for being the first
person to carry or unlock an iPhone and
since then he's done quite a few
interesting things at the intersection
of hardware and software this is the
artificial intelligence podcast if you
enjoy it subscribe on YouTube give it
five stars on iTunes supported on
patreon or simply connect with me on
Twitter at lex friedman spelled fri d-m
a.m. and i'd like to give a special
thank you to Jennifer from Canada for
her support of the podcast on patreon
merci beaucoup Jennifer she's been a
friend and an engineering colleague for
many years since I was in grad school
your support means a lot and inspires me
to keep this series going and now here's
my conversation with George Hotz do you
think we're living in a simulation
yes but it may be unfalsifiable what do
you mean by unfalsifiable so if the
simulation is designed in such a way
that they did like a formal proof to
show that no information can get in and
out and if their hardware is designed to
for the anything in the simulation to
always keep the hardware in spec it may
be impossible to prove whether we're in
a simulation or not
so they've designed it such there's the
closed system you can't get outside of
the system
well maybe it's one of three worlds
we're either in a simulation which can
be exploited we're in a simulation which
not only can't be exploited but like the
same things too about VMs I'm a really
well-designed VM you can't even detect
if you're in a VM or not that's
brilliant
so where it's yeah so the simulation is
running in a virtual machine but now in
reality all VMs have wasted the fact
that's the point I mean is it
yeah you've done quite a bit of hacking
yourself and so you should know that
really any complicated system will have
ways in and out so this isn't
necessarily true going forward I spent
my time away from comma I learned
and said dependently typed like it's a
language for writing math proofs and if
you write code that compiles in a
language like that
it is correct by definition the types
check it's correct and so it's possible
that the simulation is written in a
language like this in which case yeah
yeah but that can't be sufficiently
expressive a language like that all
weekend it can be yeah okay well so all
right so the simulation doesn't have to
be tiring complete if it has a scheduled
end date looks like it does actually
with entropy and you know I don't think
that a simulation that results in
something as complicated in universe
would have a formal proof of correctness
right as as possible of course we have
no idea how good their tooling is and we
have no idea how complicated the
universe computer really is it may be
quite simple it's just very large right
it's very it's definitely very large but
the fundamental rules might be super
simple yeah Conway's gonna live kind of
stuff right so if you could hack it so
imagine the simulation that is hackable
if you could hack it what would you
change about the you know like how would
you approach hacking a simulation the
reason I gave that talk I by the way I'm
not familiar with the talk he gave I
just read that you talked about escaping
the simulation yeah like that so maybe
you can tell me a little bit about the
theme and the message there - it wasn't
a very practical talk about how to
actually escape a simulation it was more
about a way of restructuring an
us-versus-them narrative if we continue
on the path we're going with technology
I think we're in big trouble
like as a species and not just as a
species but even as me as an individual
member of the species so if we could
change rhetoric to be more like to think
upwards like to think about that we're
in a simulation and how we could get out
already we'd be on the right path what
you actually do once you do that while I
assume I would have acquired way more
intelligence in the process of doing
that so I'll just ask that so the the
thinking upwards what kind of ideas what
kind of breakthrough ideas do you think
thinking in that way could inspire and
what did you say upwards upwards into
space are you thinking sort of
exploration in all forms the space
narrative that held for the modernist
generation doesn't hold as well for the
postmodern generation
what's the space narrator we're talking
about the same space the dimensional
space like going a little ace is like
building like yuan mosque like we're
gonna build rockets we're gonna go to
Mars we're gonna colonize the universe
and the narrative your friend was born
in the Soviet Union you're referring to
the race to space the race to space
explore okay that was a great modernist
narrative
it doesn't seem to hold the same weight
in today's culture I'm hoping for good
postmodern narratives that replace it so
think let's think so you work a lot with
AI so the eyes one formulation of that
narrative there could be also I don't
know how much you do in VR and they are
yeah that's another eye I know less
about it but every time I play with it
and our research is fascinating that
virtual world are you are you interested
in the virtual world I would like to
move to a virtual reality in terms of
your work
no I would like to physically move there
the apartment I can rent in the cloud is
way better in the apartment I can rent
in the real world well it's all relative
isn't it because others will have very
nice departments too so you'll be
inferior in the virtual world that's not
how I view the world right I don't view
the world I mean it's very like like
almost zero-sum issue a to view the
world say like my great apartment isn't
great because my neighbor has one - no
my great apartment is great because like
look at this dishwasher man yeah you
just touch the dish and it's washed
right and that is great in and of itself
if I have the only apartment or if
everybody had the apartment I don't care
so you have fundamental gratitude the
the world first learned of Geo ha George
Hotz in August 2007 maybe before then
but certainly in August 2007 when you
were the first person to unlock carry
unlock an iPhone how did you get into
hacking what was the first system you
discovered vulnerabilities for and broke
into so that was really kind of the
first thing I had I had a book in in
2006 called grey hat hacking and I guess
I realized that if you acquired these
sort of powers you could control the
world but I didn't really know that much
about computers back then I started with
electronics the first iPhone hack was
physical card work um you had to open it
up and pull an address line high and it
was because I didn't really know about
software exploitation I learned that all
in the next few years and I got very
good at it but back then I knew about
like how men
chips are connected to processors and he
knew about software and programming
he didn't didn't know I'll really see
you the view of the world and computers
was physical was the most hard work
actually if you read the code that I
released with that
in August 2007 it's atrocious
the language was it a C say yes and in a
broken sort of state machine SC I didn't
know how to program man so how did you
learn to program
what was your journey cuz I mean we'll
talk about it you've live streams from
your programming man this is a chaotic
beautiful mess how did you arrive at
that years and years of practice I
interned at Google after the summer
after the iPhone unlock and I did a
contract for them where I built hardware
for for Street View and I wrote a
software library to interact with it and
it was terrible code and for the first
time I got feedback from people who I
respected saying you know like don't
write code like this now of course just
getting that feedback is not enough the
way that I really got good was I wanted
to write this thing like that could
emulate and then visualize like armed
binaries because I wanted to hack the
iPhone better and I didn't like that I
couldn't like see what that I couldn't
single step through the processor
because I had no debugger on there
especially for the low level things like
the boot ROM in the bootloader so I
tried to build this tool to do it
and I built the tool once and it was
terrible I built the tool second times
it was terrible
I built the tool third time this by the
time I was at Facebook it was kind of
okay
and then I built the tool fourth time
when I was a Google intern again in 2014
and that was the first time I was like
this is finally usable how do you
pronounce this kira-kira yeah
so it's essentially the most efficient
way to visualize the change of state of
the computer as the program is running
that's what I mean by debugger yeah it's
a timeless debugger so you can rewind
just as easily as going forward think
about if you're using gdb you have to
put a watch on a variable if you want to
see if that variable changes and Kure
you can just click on that variable and
then it shows every single time when
that variable was changed or accessed
think about it like get for your
computers uh the run lock so there's
like a deep log of of the state of the
computer as the program runs and you can
rewind why isn't that maybe it is maybe
you can educate me what isn't that kind
of debugging used more often ah because
the tooling is bad
well two things one if you're trying to
debug chrome chrome is a 200 megabyte
binary that runs slowly on desktops so
that's going to be really hard to use
for that but it's really good to use for
like CTFs and for boot roms and for
small parts of code so it's it's hard if
you're trying to debug like massive
systems what's the CTF and what's the
boot ROM the boot ROM is the first code
that executes it's the minute you give
power to your iPhone okay and CTF were
these competitions that I played capture
the flag to capture the flag I was going
to ask you about that what are those
LaVette I watched a couple videos on
YouTube those look fascinating what have
you learned about maybe at the high
level of vulnerability of systems from
these competitions the like I feel like
like in the heyday of CTFs you had all
of the best security people in the world
challenging each other and coming up
with new toy exploitable things over
here and then everybody okay who can
break it and when you break it you get
like there's like a file on the server
called flag and then there's a program
running listening on a socket that's
vulnerable so you write an exploit you
she'll and then you cat flag and then
you type the flag into like a web-based
scoreboard and you get points so the
goal is essentially to find an exploit
in the system that allows you to run
shell to run arbitrary code on that
system that's one of the categories
that's like the PO noble category
vulnerable
yeah horrible it's like you know you
pwned the program you are it's a program
yeah yeah you know for personally I
apologize I'm gonna I'm gonna say it's
because I'm Russian but maybe you can
help educate me some video game like
misspell to own way back in the Mia and
there's just I wonder if there's a
definition I'll have to go to urban
dictionary for it okay so what was the
heyday seat yeah by the way but was it
what decade are we talking about I think
like I mean maybe I'm biased because
it's the era that that that I played but
like 2011 to 2015 because the modern CTF
scene is similar to the modern
competitive programming scene you have
people who like do drills you have
people who practice and then once you've
done that you've turned it lesson to a
game of generic computer skill and more
into a game of okay you memorize you you
drill on these five categories and then
before that it wasn't it didn't have
like as much attention as it had I don't
know they were like I won $30,000 ones
in Korea for one of these competitions
oh crap they were they were that so that
means I mean money's money but that
means there was probably good people
there exactly yeah are the challenges
human constructive or are they grounded
in some real flaws and real systems
usually they're human constructed but
they're usually inspired by real flaws
what kind of systems are imagined is
really focused on mobile like what has
vulnerabilities these days is it does
primarily mobile systems like Android
everything does No yeah of course the
price has kind of gone up because less
and less people can find them and what's
happened in security is now if you want
to like jailbreak an iPhone you don't
need one exploit anymore you need nine
nine chained together what women yeah
Wow okay so it's really so what's the
but what's the benefit speaking higher
level philosophically about hacking I
mean it sounds from everything I've seen
about you you just love the challenge
and you don't want to do anything you
don't want to bring that exploit out
into the world and doing the actual let
it run wild you just want to solve it
and then you go on to the next thing oh
yeah I mean doing criminal stuffs not
really worth it and I'll actually use
the same argument for why I don't do
defense for why I don't do crime
if you want to defend a system say the
system has ten holes right if you find
nine of those holes as a defender you
still lose because the attacker gets in
through the last one if you're an
attacker you only have to find one out
of the ten but if you're a criminal if
you log on with a VPN nine out of the
ten times but one time you forget you're
done because you're caught okay because
you only have to mess up once to be
caught as a criminal yeah that's why I'm
not a criminal
but okay let me uh that's having a
discussion with somebody just at a high
level about nuclear weapons actually why
we're having blowing ourselves up yet
and my feeling is all the smart people
in the world look at the distribution of
smart people smart people are generally
good and then this other person I was
talking to Sean Carroll the physicist
and you were saying no good and bad
people are evenly distributed amongst
everybody my sense was good hackers are
in general good people and they don't
want to mess with the world what's your
sense I'm not even sure about that like
I have a nice life crime wouldn't get me
anything
but if you're good and you have these
skills you probably have a nice life too
right like you can use the father things
but is there an ethical is there some is
there a little voice in your head that
says well yeah if you could hack
something to where you could hurt people
and you could earn a lot of money doing
it though not hurt physically perhaps
but disrupt her life in some kind of way
it is there a little voice that says um
what two things one I don't really care
about money
so like the money wouldn't be an
incentive the thrill might be an
incentive but when I was 19 I read crime
and punishment right that was another
that was another great one that talked
me out of ever really doing crime Oh cuz
it's like that's gonna be me I'd get
away with it whatever just went in my
head even if I got away with it you know
and then you do crime for long enough
you'll never get away with it that's
right in the end that's a good reason to
be good I wouldn't say good I just say
I'm not bad you're a talented programmer
and a hacker in a good positive sense of
the word award you've played around
found vulnerabilities in various systems
what have you learned broadly about the
design of systems and so on from that
from that whole process you learn to not
take things for what people say they are
but you look at things for what they
actually are
yeah I understand that's what you tell
me it is but what does it do man and you
have nice visualization tools to really
know what it's really doing oh I wish
I'm a better programmer now than I was
in 2014 I said Kira that was the first
tool that I wrote that was usable I
wouldn't say the code was great I still
wouldn't say my code is great so how was
your evolution as a programmer except
practice he went he started with C at
which point did you pick up Python
because you're pretty big and Python
though now yeah in uh in college I went
to Carnegie Mellon when I was 22 um I
went back I'm like I'm gonna take all
your hardest CS courses we'll see how I
do right like did I miss anything by not
having a real undergraduate education
took operating systems compilers AI and
they're like a freshman reader math
course and operating says some of these
some of those classes you mentioned
actually they're great at least one the
2012 circuit 2012 operating systems and
compilers we're two of the best classes
I've ever taken my life because you
write an operating system and you write
a compiler I wrote my operating system
in C and I wrote my compiler in Haskell
but classical well somehow I picked up
Python that semester as well I started
using it for the CTS actually that's
when I really started to get into CTF
and CTF you're all to race against the
clock so I can't write things and say oh
there's a clock component so you really
want to use the programming language you
can be fastest than 48 hours pone as
many of these challenges you can pone
yeah you got like a hundred points a
challenge whatever team gets the most
you were both the Facebook and Google
for a brief stint yeah well the project
zero actually at Google for five months
where you develop kara what was project
zero about in general speak what what
just curious about the security efforts
in these companies
well product zero started the same time
I I went there what what years are there
2015 2015 so that was right at the
beginning of project it's small it's
Google's offensive security team I'll
try to give I'll try to give the best
public facing explanation that I can so
the idea is basically these
vulnerabilities exist in the world
nation states have them some high
powered bad actors have them
sometime people will find these
vulnerabilities and submit them in bug
bounties to the companies but a lot of
the companies don't really care it only
fix the bug there's no it doesn't hurt
for there to be a vulnerability so
project zero is like we're gonna do it
different we're going to announce a
vulnerability and we're going to give
them 90 days to fix it and then whether
they fix it or not we're gonna drop the
drop the zero day oh wow we're gonna
drop the weapon that's so cool that is
so cool I love that deadlines though
that's so cool give him real deadlines
yeah and I think it's done a lot for
moving the industry forward I watched
your coding sessions on the stream
downline you code things up basic
projects usually from scratch I would
say sort of as a programmer myself just
watching you that you type really fast
and your brain works in both brilliant
and chaotic ways I don't know if that's
always true but certainly for the live
streams so it's it's interesting to me
because I'm more I'm much slower and
systematic and careful and you just move
I mean probably an order of magnitude
faster some curious is there a method to
your madness
is this just who you are there's pros
and cons there's pros and cons to my
programming style and I'm aware of them
like if you ask me to like like get
something up and working quickly with
like an API that's kind of undocumented
I will do this super fast because I will
throw things at it until it works if you
ask me to take a vector and rotate it 90
degrees and then flip it over the XY
plane I'll spam program for two hours
and won't get it all because it's
something that you could do with a sheet
of paper think through design and then
just you really just throw stuff at the
wall and you get so good at it that it
usually works I should become better at
the other kind as well sometimes I'll do
things pathetically it's nowhere near as
entertaining on the twitch streams I do
exaggerate it a bit on the edge games as
well the twitch streams I mean what do
you want to see a game or you want to
see actions permit me right I'll show
you a PM for programming yes I recommend
people go to I think I watched
I was probably several hours you put
like I've actually left you programming
in the background while I was
programming because you made me you it
was it was like watching a really good
gamer
it's like energizes you because you're
like moving so fast it so it's it's
awesome it's inspiring and so it made me
jealous that like because my own program
is inadequate in terms of speed Oh as I
was like so I'm twice as frantic on the
live streams as I am when I code without
oh it's super entertaining so I I wasn't
even paying attention to where you were
coding which is great it's just watching
you switch windows and VAM I guess is
driven screen I've developed a workflow
Facebook and talk about how do you learn
new programming tools ideas techniques
these days what's your like methodology
for learning new things so I wrote for
comma the distributed file systems out
in the world are extremely complex like
if you want to install something like
like like Saif Saif is I think the like
open infrastructure to should be a file
system or there's like newer ones like
seaweed FS but these are all like 10,000
plus line projects I think some of them
are even 100,000 line and just
configuring them as a nightmare so I
wrote I wrote one um
it's 200 lines and it's it uses like
nginx to the live servers and has low
master server that I wrote and go and
the way I go this if I would say that
I'm proud per line of any code I wrote
maybe there's some exploits that I think
are beautiful and then this this is 200
lines and just the way that I thought
about it I think was very good and the
reason it's very good is because that
was the fourth version of it that I
wrote and I had three versions that I
threw away you mentioned you see go I
ready go yeah and go so is that a
functional language I forget what goes
they go is Google's language right I'm a
functional it's some it's like in a way
it's C++ but easier it's it's strongly
typed it has a nice ecosystem erotic
when I first looked at it I was like
this is like Python but it takes twice
as long to do anything yeah
now that I've open pilot is migrating to
sea but it still has large Python
components I now understand why Python
doesn't work for large code bases and
why you want something like Oh
interesting so why why doesn't Python
work for so even most speaking for
myself at least like we do a lot of
stuff basically demo level work with
autonomous vehicles and most of the work
is Python yeah why doesn't Python work
for large code bases because well lack
of type checking is a big errors
creeping yeah and like you don't know
the compiler can tell you like nothing
right so everything is either you know
like like syntax errors fine but if you
misspell a variable and Python the
compiler won't catch that there's like
linters that can catch it some other
time
there's no types this is really the
biggest downside and then will Python
slow but that's not related to it well
maybe the kind of related to its that's
lacking so what's what's in your toolbox
these days is a Python what else go I
need to move on something else but my
adventure interdependently type
languages I love these languages they
just have like syntax from the 80s what
do you think about JavaScript
yes thanks Nick tomorrow typescript
javascript is the whole ecosystem is
unbelievably confusing
NPM updates a package from zero to two
to zero to five and that breaks your
babble linter which translates your es5
into es6 which doesn't run on so why do
I have to compile my JavaScript again
huh it may be the future though if you
think about I mean I've embraced
JavaScript recently because just like
I've continually embraced PHP it seems
that these worst possible languages live
on for long is that cockroaches never
die yeah well it's in the browser and
it's fast it's fast yeah it's in the
browser and compute mites they become
you know the browser it's unclear what
the role the browser's in terms of
distributed computation in the future so
javascript is definitely here to stay
yeah interesting if
Tom's vehicles will run on JavaScript
one day I mean you have to consider
these possibilities well all our debug
tools are JavaScript
we actually just open-source them we
have a tool Explorer which you can
annotate your dis engagements and we
have tool cabana which lets you analyze
the canned traffic from the car so
basically any time you're visualizing
something about the log you using
javascript yeah well the web is the best
UI toolkit by far yeah um so and then
you know what you're voting in
JavaScript we have a react guy he's good
he acts nice let's get into it so let's
talk to Thomas vehicles you found it
comma a let's at a high level how did
you get into the world the vehicle
automation can you also just for people
who don't know tell the story of comma
yeah sure so I was working at this AI
startup and a friend approached me and
he's like dude I don't know where this
is going but the coolest applied AI
problem today is self-driving cars I'm
like well absolutely do you want to meet
with UI mosque and he's looking for
somebody to build a vision system for
auto pilot this is when they were still
on ap one they were still using mobile I
kneel on back then was looking for a
replacement and he brought me in and we
talked about a contract where I would
deliver something that meets mobile eye
level performance I would get paid
twelve million dollars if I could
deliver it tomorrow and I would lose 1
million dollars for every month I didn't
deliver yeah so I was like ok this is a
great deal this is a super exciting
challenge you know what even if it takes
me 10 months I get two million dollars
it's good maybe I can finish up in five
maybe I don't finish it at all and I get
paid nothing and I'll work for twelve
months for free so maybe I just take a
pause on that I'm also curious about
this because I've been working on
robotics for a long time and I'm curious
to see a person like you just step in
and sort of somewhat naive but brilliant
right so that's though that's the best
place to be because you basically
full-steam take on a problem how
confident how from that time because you
know a lot more now at that time how
hard do you think it is to solve all of
autonomous driving I remember I
suggested to Elon in the meeting I'm
putting
GPU behind each camera to keep the
compute local this is an incredibly
stupid idea I leave the meeting 10
minutes later and I'm like I could have
spent a little bit of time thinking
about this problem was I would just send
all your cameras to one big GPU you're
much better off doing that oh sorry you
said behind every camera you have a
small GPU I was like oh I'll put the
first few layers of my comm there Oh
like why did I say that that's possible
it's possible but it's a bad idea it's
not obviously a bad idea pretty obvious
but whether it's actually a bad idea or
not I left that meeting with Elon like
beating myself up I'm like why did I say
something stupid yeah you haven't given
I'm at least like thought through every
aspect yes he's very sharp too like
usually in life I get away with saying
stupid things and then kind of course
alright right away he called me out
about it and like usually in life I get
away with saying stupid things and then
like people will you know people a lot
of times people don't even notice and
I'll like correct it and bring the
conversation back but with Elon it was
like nope like okay well that's not at
all why the contract fell through I was
much more prepared the second time I met
him yeah but in general huh how hard did
you think it is like 12 months is uh-oh
is it tough timeline oh I just thought
I'd clone mob like you three I didn't
think I'd solve level five self-driving
or anything so the goal there was to do
lane-keeping good good link keeping I
saw my friend showed me the outputs from
a mobile I in the office from a mobile I
was just basically two lanes at a
position of a lead car mm-hm
like I can I can gather a dataset and
train this net in in weeks and I did
well first time I tried the
implementation of mobile I and the test
I was really surprised how good it is
it's quite incredibly good because I
thought it's just because I've done a
lot of computation I thought it'd be a
lot harder to create a system that
that's stable so I was personally
surprised you know have to admit it
because I was kind of skeptical before
trying it because I thought it would go
in and out a lot more it would get
disengaged a lot more and it's pretty
robust so what how how hard is the
problem we need to when you tackled it
I think a p1 was great like Elon talked
about dis engagements on the 405 down in
LA we'd like the lane marks were kind of
faded and the mobile eye system would
drop out uh like I had something up and
working that I would say was like the
same quality in three months same
quality but how do you know you you say
stuff like that yeah confidently but you
can't and I love it but well the
question is you can't you're kind of
going by feel because he not solely
absolutely like like I would take I
hadn't I borrowed my friends Tesla yeah
I would take ap one out for a drive yeah
and then I would take my system out for
a dry and seems reasonably like the same
so the four or five how hard is it to
create something that could actually be
a product that's deployed I mean I've
read an article or you on this
respondent said something by you saying
that to build autopilot is is more
complicated than a single George Hotz a
level job how hard is that job to create
something that would work across the
globe Lee what are the global ease the
challenge but Elon followed that up by
saying it's gonna take two years in a
company of ten people yeah and Here I am
four years later with a company of
twelve people and I think we still have
another two to go two years so yeah so
what do you think what do you think
about the hottest is progressing with
autopilot v2 v3
I think we've kept pace with them pretty
well
I think navigator autopilot is terrible
we had some demo features internally of
the same stuff and we would test it and
I'm like I'm not shipping this even as
like open-source software to people what
do you think is do
Consumer Reports does a great job of
describing it like when it makes a lane
change it does it worse than a human
you shouldn't ship things like autopilot
open pilot they Lane keep better than a
human if you turn it on for a stretch of
highway like an hour long it's never
gonna touch a lane line human will touch
probably a lane line twice you just
inspired me I don't know if you're
grounded and data on that I read labor
okay but no but that's interesting uh I
wonder actually how often we touch Lane
lines in general like a little bit cuz
it is okay I could answer that question
pretty easily with the common data side
yeah I'm curious I've never answered it
I don't know yeah I just - is like my
person it feels right that's interesting
because every time you touch the lane
that's the source of a little bit of
stress and kind of lane-keeping is
removing that stress that's all to me
the big the biggest value-add honestly
is just removing the stress of having to
stay in lane and I think honestly I
don't think people fully realize first
of all that that's a big value add but
also that that's all it is
and that not only I find it a huge value
add I drove down when we moved to San
Diego I drove down our Enterprise
rent-a-car and I missed it so I missed
having the system so much it's so much
more tiring to drive without it it's it
is that Lane centering that's the key
feature yeah
and in a way it's the only feature that
actually adds value to people's lives
and autonomous vehicles today way mode
does not add value to people's lives
it's a more expensive lower slower uber
maybe someday it'll be this big cliff
where it adds value but I don't usually
do this vessei I haven't talked to is
that this is good because I haven't I
have intuitively but I think we're
making it explicit now I I actually
believe that really good lane-keeping is
a reason to buy a car will be a reason
to buy a car is a huge value add I've
never until we just started talking
about it haven't really quite realized
that that I've felt with elan chase of
level four is not the correct chase it
was on because you should just say Tesla
has the best as if from a testing
perspective say Tesla has the best
lane-keeping coming I should say coming
I is the best link keeping and that is
it yeah yeah does do you think well you
have to do the longitudinal as well
you can't just Lane keep you have to do
a cc but a cc is much more forgiving
than lanky especially on the highway oh
by the way are you uh calming eyes
camera only correct oh no we use the
radar we from the car you were able to
get to open it um we can't do a camera
only now it's gotten to the point but we
leave the radar there is like a it's
it's fusion now okay so let's maybe talk
through some of the system specs on the
hardware or what it what's what's the
hardware side of what you're providing
what's the capabilities in the software
side would open pilot and so on so open
pilot as the the box that we sell that
it runs on it's a phone in a plastic
case it's nothing special we sell it
without the software so you're like you
know you buy the phone it's just easy
it'll be easy setup but it's sold with
no software
open pilot right now is about to be 0.6
when it gets to 1.0 I think we'll be
ready for a consumer product we're not
gonna add any new features we're just
gonna make the lane-keeping really
really good
so what do we have right now it's a
snapdragon 820
say so many IMX 298 forward-facing
camera driver monitoring camera and
she's a selfie cam on the phone and a
can transceiver biffle's little thing
calls pandas and they talk over USB to
the phone and then they have three
canvases that they talk to the car one
of those campuses is the radar CANbus
one of them is the main car CANbus and
the other one is the proxy camera CANbus
we leave the existing camera in place so
we don't turn a DB off right now we
still turn a TV off if you're using our
longitudinal but we're gonna fix that
before 1.0 you got it wow that's cool so
in its can both way so how are you able
to control vehicles so we proxy the
vehicles that we work with already have
Lane Keeping Assist system so Lane
Keeping Assist can mean a huge variety
of things it can mean it will apply a
small torque to the wheel after you've
already crossed a lane line by a foot
which is the system in the older Toyotas
versus like I think Tesla still calls it
Lane Keeping Assist where it'll keep you
perfectly in the center of the lane on
the highway you can control like you
would in joystick the cars these so
these cars already have the capability
of drive-by-wire so is it is it trivial
to convert a car that it operates with
it open pile is able to control the
steering Oh a new car or a car that we
so we have support now for 45 different
makes of cars what are one of the cars
general mostly Hondas and Toyotas we
support almost every Honda and Toyota
made this year and then a bunch of GM's
bunch of Subarus which it doesn't have
to be like a Prius it could be Coral as
well okay the 2020 Corolla is the best
car with open pilot it just came out
there the actuator has less lag than the
older Corolla
I think I started watching video with
your eye the way you make videos is
awesome literally the dealerships
streaming stream for an hour
yeah and basically like if stuff goes a
little wrong you're just like you just
go with it yeah I love it what's real
yeah that's real that's that's it's
that's so beautiful and it's so in
contrast to the way other companies
would put together a video like that how
do I like to do it like good I mean if
you become super rich one day is
successful I hope you keep it that way
because I think that's actually what
people love that kind of genuine oh it's
all that has value to me yeah my money
has no if I sell out to like make money
and I sold out it doesn't matter what do
I get yacht I don't I got and I think
Tesla's actually has a small inkling of
that as well with autonomy day they did
reveal more than I mean of course
there's marketing communications you can
tell but it's more than most companies
will reveal which is I hope they go
towards a direction more other companies
GM Ford oh Jessa Tesla's gonna win level
5 they really are so let's talk about it
you think you're focused on level 2
currently currently we're gonna be one
to two years behind Tesla getting to
level five okay we're interested right
we're into it you're in I'm just saying
once Tesla gets it we're one to two
years behind
I'm not making any timeline on when
Tesla's that's right you did that's
brilliant
I'm sorry Tesla investors if you think
you're gonna have an autonomous robot
taxi fleet by the end of the year yes
that's all bet against that so that what
do you think about this the most level
four companies are kind of just doing
their usual safety driver during full
autonomy kind of testing and then Tesla
does basically trying to go from
lane-keeping to full autonomy what do
you think about that approach how
successful would it be a ton better
approach because Tesla is gathering data
on a scale that none of them are they're
putting real users behind the behind the
wheel of the car
it's I think the only strategy that
works the incremental well so there's a
few components to test approach that's
that's more than just incrementally you
spoke with is the one is the software so
over-the-air software updates necessity
I mean way more ease have those - those
aren't but there was differentiating
from the automaker's right no link
keeping assist systems have no cars with
lane keeping system have that except
Tesla yeah and the other one is the data
the other direction which is the ability
to query the data I don't think they're
actually collecting as much days people
think but the ability to turn on
collection and turn it off so I'm both
in the robotics world in the the
psychology human factors world many
people believe that level to autonomy is
problematic because of the human factor
like the more the task is automated the
more there's a vigilance decrement you
start to fall asleep you start to become
complacent start texting more and so on
do you worry about that
because if we're talking about
transition from lane-keeping to full
autonomy if you're spending eighty
percent of the time not supervising
machine do you worry about what that
means to the safety of the drivers one
we don't consider open pilot to be 1.0
until we have 100% driver monitoring you
you can cheat right now our driver
monitoring system there's a few ways to
cheat it there pretty obvious we're
working on making that better before we
ship a consumer product that can drive
cars I want to make sure that I have
driver monitoring that you can't cheat
what's like a successful driver
monitoring system look like it's keep
its is it all buzz just keeping your
eyes on the road um well a few things so
that's what we went with it first for
driver monitoring I'm checking I'm
actually looking at where your head is
looking but cameras know about my
resolution eyes are a little bit hard to
get well head is this big I mean that is
good and actually a lot of it just as
psychology wise to have that monitor
constantly there it reminds you that you
have to be paying attention but we want
to go further we just hired someone
full-time to come onto the driver
monitoring I want to detect phone in
frame and I want to make sure you're not
sleeping
how much does the camera see of the body
this one not enough not enough the next
one everything
what's interesting fish Atkins we have
we're doing just data collection that
real-time but fish eye is a beautiful
mouth being able to capture the body and
the smartphone is really like the
biggest problem I'll show you I can show
you one of the pictures from from our
finder system
awesome so you're basically saying the
driver monitoring will be the answer to
that um I think the other point that the
original paper is is good as well you're
not asking a human to supervise a
machine without giving them meat they
can take over at a time right our safety
model you can take over we disengage on
both the gas or the brake we don't
disengage on steering I don't feel you
have to but we disengage on gas or brake
so it's very easy for you to take over
and it's very easy for you to re-engage
that switching should be super cheap
yeah the cars that require even
autopilot requires a double press that's
almost I said I like that yeah and then
then the cancel um to cancel in
autopilot you either have to press
cancel which no one knows where that is
so they press the brake but a lot of
things you don't you want to press the
brake you want present ass
so you should cancel on gas or wiggle
the steering wheel which is bad as well
wow that's brilliant I haven't heard
anyone articulate at that point I like
what this is all I think about
it's because I think I think actually
Tesla has done a better job than most
automakers at making that frictionless
but you just described that it could be
even better I love super cruise as an
experience once it's engaged yeah I
don't know if you've used it but getting
the thing to try to engage him yeah I've
used this of Germany's super cruise a
lot so what's their thoughts on the
super Cruiser system in June disengage
super cruise and it falls back to ACC so
my car's like still accelerating it
feels weird otherwise when you actually
have super cruise engaged on the highway
it is phenomenal we bought that Cadillac
we just sold it but we bought it just to
like experience this and I wanted
everyone in the office to be like this
is what we're striving to build GM
pioneering with the driver monitoring
you know you like their driver
monitoring system it has some bugs
if there's a sun shining back year it'll
be blind to you by overall mostly yeah
that's so cool you know the stuff that's
uh I don't often talk to people that
because it's such a rare car
unfortunately they bought one yes
possibly for us we lost like by 25k the
deprecation but a Philips worth it
I was very pleasantly surprised that GM
system was so innovative and really that
wasn't advertised much wasn't talked
about much yeah and I was nervous that
it would die that they would disappear
my eyes did they put it on the wrong car
they should've put it on the bolt and
not some weird Cadillac that nobody
bought I think that's gonna be into
they're saying at least is going to be
into their entire fleet so what do you
think about it if as long as we're on
the driver monitoring what do you think
about you know I must claim that driver
monitoring is not needed normally I love
his claims that one is stupid
that one is stupid and you know he's not
gonna have his level five fleet by the
end of the year hopefully he's like okay
I was wrong I'm gonna add driver
monitoring because when these systems
get to the point that they're only
messing up once every thousand miles
you absolutely need driver monitor so
let me play Delta because I agree with
you but let me play devil's advocate so
one possibility is that without driver
monitoring people are able to monitor
the self-regulate monitor themselves you
know that so your idea is seeing all the
people sleeping in decimals uh yeah well
I'm a little skeptical of all the people
sleeping in Tesla's because I have I've
stopped paying attention to that kind of
stuff because I want to see real data
there's too much glorified it doesn't
feel scientific to me so I want to know
you know what how many people are really
sleeping in Tesla's vs. sleeping I've I
was driving here sleep-deprived in a car
with no automation I was falling asleep
I agree that it's high P it's just like
you know what if you under I've am
wondering I think I rented a my last
autopilot experience was I rented a
model
three in march and drove it around the
wheel thing is annoying and the reason
the wheel thing is annoying we use the
wheel thing as well but we don't
disengage on wheel for Tesla you have to
touch the wheel just enough you should
trigger the torque sensor to tell it
that you're there but not enough as to
disengage it which don't use it for two
things
you disengage one wheel you don't have
to that whole experience Wow beautiful
put that all those elements even if you
don't have driver monitoring that whole
experience needs to be better driver
monitoring I think would make I mean I
think super cruise is a better
experience once it's engaged over
autopilot
I think super cruise is our transition
to engagement and disengagement are
significantly worse yeah so there's a
tricky thing because if I were to
criticize super cruise is uh it's a
little too crude and uh I think it's
like six seconds or something if you
look off-road you'll start warning you
it's some ridiculously long period of
time and just the way it I think it's
basically it's a binary chili adapter it
yeah it's it just needs to learn more
about you and used to communicate what
it sees about you more like I'm not you
know Tesla shows what it sees about the
external world it would be nice the
supercruise would tell us what it sees
about the internal world it's even worse
than that you press the button to engage
and it just says super cruise
unavailable yeah why why yeah that
transparency is good we've renamed the
driver monitoring packet to driver state
service state we have car state packet
which has the state of the car driver
state packet which I stay the driver so
what does itah make their BAC
must be do you think that's possible
with computer vision absolutely so to me
it's an open question I don't haven't
looked into too much they actually had
quite seriously looked at the literature
it's not obvious to me that from the
eyes and so on you can tell you might
need to stuff from the car as well yeah
you might need how they're controlling
the car right and that's fundamentally
at the end of the day what you care
about you but I think especially when
people are really drunk they're not
controlling the car nearly
smoothly as they would look at them
walking right there the car is like an
extension of the body so I think you
could totally detect and if you could
fix people who drunk distracted asleep
if you fix those three yeah this is
that's huge so what are the current
limitations of open pilot what are the
main problems that still need to be
solved um we're hopefully fixing a few
of them in 0-6 we're not as good as auto
pilot at stop cars so if you're coming
up to a red light at like 55 so it's the
radar stopped car problem which is
responsible to auto pilot accidents it's
hard to differentiate a stopped car from
a like signpost yes that ecology um so
you have to fuse you have to do this
visually there's no way from the radar
data to tell the difference maybe you
could make a map but I really believe in
mapping at all anymore um really what
you don't believe in mapping no so you
basically the open pilot solution is
saying react to the environment is just
like human doing beings and then
eventually when you want to do navigate
on open pilot I'll train the net to look
at ways all runways in the background
I'll train a car using GPS at all we use
it to crown trees we use it to very
carefully ground treat the paths we have
a stack which can recover a relative to
10 centimeters over one minute and then
we use that to ground truth exactly
where the car went in that local part of
the environment but it's all local how
are you testing in general just for
yourself like experiments stuff all
right were you were you located San
Diego San Diego yeah okay Oh what you
basically drive around there then
collect some data and watch on Florence
we have a simulator now and we have our
simulators really cool our simulator is
not it's not like a unity based
simulator our simulator lets us load in
real estate what I mean we can load in a
drive and simulate what the system would
have done on the historical data ooh
nice interesting so what yeah right now
we're only using it for testing but as
soon as we start using it for training
what's your feeling about the real world
versus simulation do you like simulation
for training if this moves to training
Chuck
we have to distinguish two types of
simulators right there's a simulator
that light is completely fake I could
get my car to drive around in GTA mm-hmm
um I feel that this kind of simulator is
useless you're never there's so many my
analogy here is like okay fine you're
not solving the computer vision problem
but you're solving the computer graphics
problem right and you don't think you
can get very far about creating ultra
realistic graphics no because you can
create ultra realistic graphics of the
road now create alter a realistic
behavioral models of the other cars oh
well I'll just use my self-driving no
you won't you need real you need actual
human behavior because that's what
you're trying to learn the dead driving
does not have a spec the definition of
driving is what humans do when they
drive whatever way mode does I don't
think it's driving right well I think if
you win more than others its if there's
any useful reinforcement learning I've
seen it used quite well I study
pedestrians a lot too is try to train
models from real data of how pedestrians
move and try to use reinforcement
learning models to make pedestrians move
in human-like ways by that point you've
already gone so many layers you detected
a pedestrian did you did you hand code
the feature vector of their state did
you guys learn anything from computer
vision before deep learning well okay
you know I feel like this is a
perception to you is the sticking point
does that mean what what's what's the
hardest part of the stack here there is
no human understandable feature vector
separating perception and planning
that's the best way I can I can put that
there is no so it's all together and
it's it's a that's a joint problem so
you can take localization localization
and planning there is a human
understandable feature vector between
these two things I mean okay so I have
like three degrees position three
degrees orientation and those
derivatives maybe those second
derivatives right that's human
understandable that's physical the
between perception and planning
um so like way Moe has a perception
stack and then a planner um and one of
the things way matters right is they
have a simulator that can separate those
two they can like replay their
perception data and test their system
which is what I'm talking about about
like the two different kinds of
simulators there's the kind that can
work on real data and is the kind of
can't work on real data now the problem
is that I don't think you can hand code
a feature vector right like like you
have some lists of like well here's my
list of cars on the scenes here's my
list of pedestrians in the scene this
isn't what humans are doing what are
humans doing global some something
you're saying that's too difficult to
handle I'm saying that there is no state
vector given a perfect I could give you
the best team of engineers in the world
to build a perception system and the
best team to build a planner all you
have to do is define the state vector
that separates those two I'm missing the
state vector that separates those two
what do you mean so what is the output
of your perception system
I'll put it the perception system it's
theirs okay well there's several ways to
do it one is this lamp components
localization the other is drivable area
drivable space drivable space and then
there's the different objects in the
scene and different objects in the scene
over time maybe to give you input to
then try to start modeling the
trajectories of those objects sure
that's it I can give you a concrete
example of something you missed what's
that
so say there's a bush in the scene
humans understand that when they see
this bush that there may or may not be a
car behind that bush drivable area and a
list of objects does not include that
humans are doing this constantly at the
simplest intersections so now you have
to talk about occluded area right right
but even that what do you mean by
occluded okay so I can't see it well if
it's the other side of a house I don't
care what's the likelihood that there's
a car in that occluded area right and if
you say okay we'll add that I can come
up with 10 more examples that you can't
add
certainly occluded area would be
something that simulator would have
because it's simulating the entire you
know occlusion is part of it a part of a
vision stack pleasures that what I'm
saying is if you have a hand engineered
if your perception system output can be
written in a spec document it is
incomplete yeah idem you know certainly
it's it's hard to argue with that
because in the end that's going to be
true yes I'll tell you what the output
of our perception system is was that
it's a thousand it's a thousand twenty
four dimensional vector training
underling
oh no not it's a thousand twenty four
dimensions of who knows what because its
operating on real data yeah yeah
and that's the perception that's the
perception stake right think about a
think about an autoencoder four phases
alright if you have an autoencoder four
phases and you say it has 256 dimensions
in middle and I'm taking a face over
here and projecting it to a face over
here yeah can you hand label all 256 of
those dimensions
well no but those are generated
automatically but they but even if you
tried to do it by hand could you come up
with a spec for your and between your
encoder and your decoder
no no because that's not it is it wasn't
designed but there no no but if you
could design it if you could design a
face Reconstructor system could you come
up with a spec no but I think we're
missing here a little bit I think the
the you're just being very poetic about
expressing a fundamental problem of
simulators that they're going to be
missing so much that the feature vector
would just look fundamentally different
from in the simulated world in the real
world I'm not making a claim about
simulators I'm making a claim about the
spec division between perception and
planning and planning even in your
system just in general right just in
general if you're trying to build a car
that drives if you're trying to hand
code the output of your perception
system like saying like here's a list of
all the cars in the scene here's a list
of all the people here's a list of the
included areas here's a vector of
drivable areas
insufficient and if you start to believe
that you realize that what Wayman crews
are doing is impossible currently what
we're doing is the perception problem
it's converting the scene into a
chessboard you yeah and then you reason
some basic reasoning around that
chessboard yeah and you're saying that
really there's a lot missing there first
of all why are we talking about this cuz
isn't this a full autonomy is this
something you think about oh I want to
win self-driving cars so you're really
your definition of win includes level of
fool five level five I don't think level
four is a real thing I want to build I
want to build the alphago of driving so
so alphago is really end to end yeah is
uh yeah it's end to end and do you think
this whole problem is those that also
kind of what you're getting at with the
perception and the planning is that this
whole problem the right way to do it is
really to learn the entire thing I'll
argue that not only is it the right way
it's the only way that's going to exceed
human performance well certainly true
for go everyone who tried to hand code
go things built human inferior things
and then someone came along and wrote
some 10,000 line thing that doesn't know
anything about go that beat everybody
it's 10,000 lines true in that sense the
the open question then that maybe I can
ask you is uh driving is much harder
than go the open question is how much
harder so how because I think the AH
mosque approach here with planning and
perception it's similar to what you're
describing which is really turning into
not some kind of modular thing but
really do formulate is a learning
problem and it solves a learning problem
of scale so how many years put one is
how many years would it take to solve
this problem or just how hard is this
freaking problem well the cool thing is
I think there's a lot of value that we
can deliver along the way I think that
you can build lane-keeping assist
actually plus adaptive cruise control
plus okay looking at ways extends to
like all of driving yeah most of driving
varies
oh your adaptive cruise control treats
red lights like cars okay
so let's jump around with you you
mentioned that you didn't like navigate
an autopilot yeah
what advice how would you make it better
do you think as a feature that if it's
done really well it's a good feature I
think that it's too reliant on like hand
coded hacks for like how does navigate
an autopilot do a lane change it
actually does the same lane change every
time and it feels mechanical humans do
different lane changes human sometime
will do a slow one sometimes do a fast
one navigate an autopilot at least every
time I used it it did the identical
language how do you learn I mean this is
a fundamental thing actually yeah is uh
the braking and an accelerating
something that's still test the probably
does it better than most cars but it
still doesn't do a great job of creating
a comfortable natural experience and
navigate on autopilot just lane changes
an extension of that so how do you learn
to do natural lane change so we have it
and I can talk about how it works so I
feel that we have the solution for
lateral but we don't yet have the
solution for longitudinal there's a few
reasons longitudinal is harder than
lateral the lane change component the
way that we train on it very simply is
like our model has an input for whether
it's doing a lane change or not and then
when we train the end-to-end model we
hand label all the lane changes because
you have to I struggled a long time
about not wanting to do that but I think
you have to because you order the
training data for the train data right
well we actually we have an automatic
ground truth or which automatically
labels all the lane changes was that
possible to automatically label interest
yeah and detect the lane I see when it
crosses it right I don't have to get
that that high percent accuracy but it's
like 95 good enough
now I set the bit when it's doing the
lane change in the end-to-end learning
and then I set it to zero when it's not
doing a lane change so now if I wanted
to do a lane change a test time I just
put the bit to a 1 and I'll do later
yeah but so if you look at the space of
lane change you know some percentage not
a hundred percent that we make as humans
is not a pleasant experience because we
messed some part of it up yeah it's
nerve-racking to change even look at the
seizure des accelerate how do we label
the ones that are natural and feel good
you know that's the because that's your
ultimate criticism the current Oh
navigate not apologies doesn't feel good
well the current navigator on autopilot
is a hand coded policy written by an
engineer in a room who probably went out
and tested it a few times on the 280
probably a more a better version of that
but yes that's how we would have written
it a comment yeah Tesla they tested it
and it might have been two engineers
yeah no but so if you learn the lane
change if you learn how to do a lane
change from data just like just like you
have a label that says lane change and
then you put it in when you want to do
the lane change it'll automatically do
the lane change that's appropriate for
the situation now to get it the problem
of some humans do bad lane changes we
haven't worked too much on this problem
yet it's not that much of a problem in
practice my theory is that all good
drivers are good in the same way and all
bad drivers are bad in different ways
and we've we've seen some data to back
this up well beautifully put so you just
basically if that's true yeah hypothesis
then you know task is to discover the
good drivers um the good drivers stand
out because they're in one cluster and
the bad drivers are scattered all over
the place and your net learns the
cluster yeah that's uh
so you just learned from the good
drivers and they're easy to cluster we
learned from all of them and that
automatically learns the policy that's
like the majority but we'll eventually
probably afterthought so if that theory
is true I hope it's true because the the
counter theory is there is many clusters
maybe
but rarely many clusters of good drivers
because if there's one cluster of good
drivers you can at least discover a set
of policies you can learn a set of
policies which would be good universally
yeah that would be a nice that would be
nice if it's true and you're saying that
there are some evidence that let's say
lane changes can be clustered into four
clusters right right there's this finite
level of I would argue that all four of
those are good clusters all the things
that are random are noise and probably
bad and which one of the four you pick
or maybe it's Tanner maybe it's twenty
you can learn them it's context
dependent it depends on the scene and
the hope is it's not too dependent on
the driver
yeah the hope is that it all washes out
the hope is that there's that the
distribution is not bimodal the hope is
that it's a nice gas man so what advice
would you give to Tessa how to fix how
to improve navigate an autopilot the
lessons you've learned from Kamiya
the only real advice I would give to
Tesla is please put driver monitoring in
your cars with respect to improvement
you can't do that anymore I said to
interrupt but you know there's a
practical nature of many of hundreds of
thousands of cars being produced that
don't have a good driver facing camera
the model 3 has a selfie cam is it not
good enough did they not have put IR
LEDs for night that's a good question
but I do know that the is fisheye in its
relatively low resolution so it's really
not this I he wasn't
it wasn't designed for Arman you can
hope that you can kind of scrape up and
and and have something from it yeah but
put it in today put it in today today
every time I've heard Carpathia talk
about the problem and talking about life
software 2.0 and how the machine
learning is gobbling up everything I
think this is absolutely the right
strategy I think that he didn't write
navigate on autopilot I think somebody
else did and kind of hacked it on top of
that stuff I think what Carpathia says
wait a second why did we hand code this
lane change policy with all these magic
numbers we're gonna learn it from data
they'll fix it they already know what to
do there well that that's that's Andres
job is to turn everything into a
learning problem and collect a huge
amount of data the the reality is though
not every problem could be turned into a
learning problem in the short term in
the end
everything would be a learning problem
the reality is like if you want to build
alpha vehicles today it will likely
involve no learning and that's that's
the the reality is so at which point
does learning start it's the crutch
statement that lidar is a crutch
on which point will learning get up to
part of human performance it's all over
human performance and imagenet
classification under ivan is the
question still it is a question I'll say
this I'm I'm here to play for 10 years
I'm not here to try to I'm here to play
for 10 years and make money along the
way I'm not here to try to promise
people that I'm gonna have my l5 taxi
Network up and working in two years do
you think those mistake yes what do you
think there was the motivation behind
saying that other companies are also
promising alpha vehicles with their
different approaches in 2020 2021 2022
if anybody would like to bet me that
those things do not pan out I will I
will bet you even money even money I'll
bet you as much as you want so are you
worried about what's going to happen
because you're not in full agreement on
that I was going to happen when 2022 21
come around and nobody has fleets of
autonomous vehicles no you can look at
the history if you go back five years
ago they were all promised by 2018 and
2017 but they weren't that strong of
promises I mean Ford really declared
pretty that I think not many have
declared as as like definitively as they
have now these dates well okay so let's
separate l4 and l5 do I think that it's
possible for way mo to continue to kind
of like like hack on their system until
it gets to level 4 in Chandler Arizona
yes knows no safety driver Chandler
Arizona yeah but by OSI which year are
we talking about
oh I even think that's possible by like
2020 2021 but level 4 Chandler Arizona
not level 5 New York City level 4
meaning some very defined streets it
works out really well very defined
streets and then
these streets are pretty empty if most
of the streets are covered in way MOS we
mo can kind of change the definition of
what driving is hmm right if your
self-driving network is the majority of
cars in an area they only need to be
safe with respect to each other and all
the humans will need to learn to adapt
to them now go drive in downtown New
York
oh yeah that's already you can talk
about autonomy in like like fun farms it
already works great because you can
really just follow the GPS line so what
does success look like for comm AI what
what are the milestones like where you
can sit back with some champagne and say
we did it boys and girls
well it's never over yeah but don't be
let's drink champagne everything
straight so what is a good what are some
wins um a big milestone that we're
hoping for by mid next year is
profitability of the company and we're
gonna have to revisit the idea of
selling a consumer product but it's not
gonna be like the comma one when we do
it it's gonna be perfect
open pilot has gotten so much better in
the last two years we're gonna have a
few a few features we're gonna have a
hundred percent driver monitoring we're
gonna disable no safety features in the
car um actually I think it'd be really
cool we're doing right now our project
this week is we're analyzing the data
set and looking for all the AEP triggers
from the manufacturer systems we have a
better data set on that than the
manufacturers how much does how many
does Toyota have ten million miles of
real-world driving to know how many
times they're AUB triggered so let me
give you cuz yes right
financial advice yeah cuz I work with a
lot of automakers and one possible
source of money for you which I'll be
excited to see you take on is basically
selling the data so which is something
that most people are not selling in a
way we're here here at automaker but
creating we've done this actually at MIT
not for money purposes but you could do
it for significant money purposes and
make the world a better place by
creating a consortium where automakers
would pay in and then they get to have
free access to the data and I I think a
lot of people are really hungry for that
and would pay significant amount of
money for it here's the problem with
that I like this idea all in theory he'd
be very easy for me to give them access
to my servers and we already have all
open source tools to access this data
it's in a great format we have a great
pipeline but they're gonna put me in the
room with some business development guy
mm-hmm and I'm gonna have to talk to
this guy and he's not gonna know most of
the words I'm saying I'm not willing to
tolerate that okay but I think I agree
with you I'm the same way but you just
tell them the terms and there's no
discussion needed if if I could just
tell them the terms yeah and then like
all right who wants access to my data I
will sell it to you for let's say you
want to go on a subscription I'll sell
you 400 a month any 100k mo 100k month
I'll give you access to the data
subscription yeah yeah I think that's
kind of fair came up with that number
off the top of my head if somebody sends
me like a three line email where it's
like we would like to pay a hundred K
month to get access to your data we
would agree to like reasonable privacy
terms of the people who are in the data
set I would be happy to do it but that's
not gonna be the email the email is
gonna be hey do you have some time in
the next month where we can sit down and
we can I don't have time for that we're
moving too fast yeah you could politely
respond to that email but not saying I
don't have any time for your
yeah you say oh well unfortunately these
are the terms and so this is we try to
we brought the cost down for you in
order to minimize the friction of
education after here's the whatever it
is 1 2 million years dollars a year and
you have access and it's not like I get
that email from like but okay am I gonna
reach out am I gonna hire a business
development person who's gonna reach out
to the automaker's no way yeah okay
if they reached into me I'm not gonna
ignore the email I'll come back with
something straight yeah if you're
willing just pay honeycomb all the facts
they don't man I'm happy to to set that
up that's what
my engineering time but actually quite
insightful view you're right yeah
probably because many of the automakers
are quite a bit of old-school yeah there
will be need to reach out and they want
it but they they'll need to be some some
communication you right mobile eye
circuit 2015 had the lowest R&D spend of
any chip maker like purpur and you look
at all the people who work for them and
it's all business development people
because the car companies are impossible
to work with yeah so you're you have no
patience for that and you're you're
legit Android huh I have something to do
right like like it's not like it's not
like I don't like I don't mean to like
be a dick and say like I don't have
patience for that but it's like that
stuff doesn't help us with our goal of
winning self-driving cars if I want
money in the short term if I showed off
like the actual like the learning tech
that we have it's it's somewhat sad like
it's years and years ahead of everybody
else's not so maybe not Tesla's I think
Tesla has similar stuff to us actually
yeah I think Tesla's similar stuff but
when you compare it to like what the
Toyota Research Institute has you're not
even close to what we have no comment
but I also can't I have to take your
comments I ain't into ative Lee believe
you but I have to take it with a grain
of salt because I mean you you are an
inspiration because you basically don't
care about a lot of things that other
companies care about you don't try to
in a sense like make up stuff
so to drive a valuation you're really
very real and you're trying to solve the
problem and admire that a lot what I
don't necessarily fully can't trust you
on I do respect it's like how good it is
right
I can only but I also know how bad
others are and so I'll say I'll say two
things about don't trust but verify
right I'll say two things about that one
is try get in a twenty twenty Corolla
and try open pal 0.6 when it comes out
next month I think already you'll look
at this and you'll be like them this is
already really good and then I could be
doing that all with hand labelers and
all with with like like the same
approach that like Mobileye uses when we
release a model that no law
has the lanes in it that only outputs a
path mm-hmm
then think about how we did that machine
learning and then right away when you
see and that's gonna be an open pilot
that's gonna be an open pilot before 1.0
when you see that model you'll know that
everything I'm saying is true because
how else did I get that model good one
of the things too about the simulator oh
yeah yeah this is super exciting that's
super exciting and uh but like you know
I listened to your talk with Kyle and
Kyle was originally building the the
after market system and he gave up on it
because of technical challenges yeah
because of the fact that he's gonna have
to support twenty to fifty cars we
support forty five because what is he
gonna do when the manufacturer ABS
system triggers we have alerts and
warnings to deal with all of that in all
the cars and how is he going to formally
verify it well I got ten million miles
of data it's probably better it's
probably better verified than the spec
yeah I'm glad you're here talking to me
this is I'll remember this day is this
interesting if you look at Kyle's from
from Cruz I'm sure they have a large
number of business development folks and
you work with he's working with GM you
could work with agro a I working with
Ford it's interesting because chances
that you fail business-wise like
bankrupt are pretty high yeah and and
yet it's the Android model is you're
actually taking on the problem so that's
really inspiring I mean well I have a
long-term way for kamma to make money
too and one of the nice things when you
really take on the problem which is my
hope for autopilot for example is things
you don't expect ways to make money or
create value that you don't expect will
pop up oh I've known how to do it
since kind of 2017 is the first time I
said it well which part to know it to
know how to do which part our long-term
plan is to be a car insurance company
insurance yeah I love it yeah yeah what
I make driving twice is safe not only
that I have the best date is that you
know who statistically is the safest
drivers and oh oh we see you we see you
driving unsafely we're not going to
insure you and that that causes a like
bifurcation in the market because the
only people who can't get common
insurance or the bad drivers Geico can
insure them their premiums
crazy higher premiums are crazy low
would win contracts take over that whole
market okay so if we win if we went but
that's I'm saying like how do you turn
comma into a ten billion dollar company
is that that's right
so you you know a musk who else who else
is thinking like this and working like
this in your view who are the
competitors are there people seriously I
don't think anyone that I'm aware of as
seriously taking on lane-keeping you
know like to worse a huge business that
turns eventually into full autonomy that
then creates yeah like that creates
other businesses on top of it and so on
thinks insurance thinks all kinds of
ideas like that do you know who anyone
else thinking like this not really
that's interesting I mean it my sense is
everybody turns to that in like four or
five years like Ford once the autonomy
doesn't feel fall through but at this
time Elon to the iOS by the way he paved
the way for all I was not i OS true I
would not be doing comma AI today if it
was not for those conversations with
Elon and if it were not for him saying
like yeah I think he said like well
obviously we're not gonna use Leiter we
use cameras humans use cameras so what
do you think about that how important is
lidar everybody else is on l5 is using
lidar what are your thoughts on his
provocative statement that lidar is a
crutch see sometimes we'll say dumb
things like the driver monitoring thing
but sometimes we'll say absolutely
completely 100% obviously true things
yeah of course lidar is a crutch it's
not even a good crutch you're not even
using it they're using it for
localization yeah which isn't good in
the first place if you have to localize
your car to centimetres in order to
drive like yeah they're not drive it
currently not doing much machine
learning I thought polite our data
meaning like to help you in the tasks of
general tasks of perception the main
goal of those light hours on those cars
I think is actually localization more
than perception or at least that's what
they use them for yeah that's true if
you want to localize two centimeters you
can't use GPS the fanciest GPS in the
world can't do it especially if you're
under tree cover and stuff
flatter I can do it pretty easily see
really they're not taking on I mean in
some research they're doing they're
using it for perception but and they're
certainly not which sad they're not
fusing it well lay vision they do use it
for perception I'm not saying they don't
use it for perception but the thing that
they have vision based and radar based
perception systems as well you could
remove the lidar and and and keep around
a lot of the dynamic object perception
you want to get centimeter accurate
localization good luck doing that with
anything else so what should Cruz lame-o
do like what would you be your advice to
them now anyway Mo's actually there's I
mean they're doing they're serious way
mo out of all of them equate so serious
about the long game if everybody fell
five is a lot is requires fifty years I
think when will be the only one left
standing at the end with the forgiving
the financial backing if they have
Google box um I'll say nice things about
both lame-o and Cruz let's do it nice is
good way mo is by far the furthest along
with technology way mo has a three to
five year lead on all the competitors um
if that if the way mo looking stack
works mm-hmm maybe three year lead if
the way mo looking stack works they have
a three year lead now I argue that way
mo has spent too much money
to recapitalize to gain back their
losses in those three years also
self-driving cars have no network effect
like that yeah goober has a network
effect you have a market you have
drivers and you have riders self-driving
cars you have capital and you have
riders there's no network effect if I
want to blanket a new city in
self-driving cars i buy the
off-the-shelf Chinese knockoff
self-driving cars and I buy enough up
from the city I can't do that with
drivers and that's why Ober has a first
mover advantage that no self-driving car
company will can you uh disentangle that
a little bit uber you're not talking
about uber the autonomous vehicle number
you talked about the uber cars okay yeah
I'm over I open for business in Austin
Texas listen I need to attract both
sides of the market I need to both get
drivers
my platform and riders on my platinum
and I need to keep them both
sufficiently happy right riders aren't
going to use it if it takes more than
five minutes for an uber to show up
drivers aren't gonna use it if they have
to sit around all day and there's no
riders so you have to carefully balance
a market and whenever you have to
carefully balance a market there's a
great first mover advantage because
there's a switching cost for everybody
right the drivers and the riders would
have to switch at the same time let's
even say that you know um let's say
Luber shows up in Luber somehow you know
agrees to do things that add a bigger
you know you know we're just gonna we've
done it more efficiently right Luber is
only takes five percent of a cot instead
of the ten percent that Hooper takes no
one is gonna switch because the
switching cost is higher than that five
percent so you actually can in markets
like that you have a first mover
advantage yeah autonomous vehicles of
the level five variety have no first
mover advantage if the technology
becomes commoditized say I want to go to
a new city look at the scooters it's
gonna look a lot more like scooters
every person with a checkbook can
blanket a city in scooters and that's
why you have 10 different scooter
companies yeah which one's gonna win
it's a race to the bottom it's terrible
market to begin because there's no
market for scooters and scooters don't
get a say and whether they want to be
bought and deployed to a city or not
right so the yeah we're gonna entice the
scooters with subsidies and deals so
whenever you have to invest that capital
that's it doesn't it doesn't come back
yeah that they can't be your main
criticism over the way mo approach oh
I'm saying even if it does technically
work even if it does technically work
that's a problem yeah I don't know I if
I were to say I I would I would say
you're already there I haven't even
thought about that but I would say the
bigger challenge is the technical
approach so way most cruises and not
just the technical approach but of
creating value I still don't understand
how you beat uber the the human driven
cars in terms of financially it doesn't
it doesn't make sense to me that people
want to want to get an autonomous
vehicle I don't understand how you make
money in the long term
like real long-term but it just feels
like there's too much capital investment
needed oh and they're gonna be worse
than ubers because they're gonna they're
gonna stop for every little you know
thing everywhere um actually a nice
thing about Cruz that was my nice thing
about wait another three years that it
wasn't nice oh that's three years
technically ahead of everybody their
tech stack is is great my nice thing
about Cruz is GM buying them was a great
move for GM for 1 billion dollars GM
bought an insurance policy against way
mo they put Cruz is three years behind
way mo hmm that means Google will get a
monopoly on the technology for at most
three years and technology works
you might not even be right about the
three years it might be less might be
less crews actually might not be that
far behind I don't know how much way mo
has waffled around or how much of it
actually is just that long tail yeah
okay if that's the best you could say
there's some nice things it that's more
of a nice thing for GM that that's a
smart insurance policy it's just more
insurance policy I mean I think that's
how I I can't see crews working out any
other for crews to leapfrog way mo would
really surprise me yeah so let's talk
about like the underlying assumptions of
everything is we're not going to
leapfrog Tesla Tesla would have to
seriously mess up for us because you're
okay so the way you leapfrog right is
you come up with an idea or you take a
direction perhaps secretly that the
other people aren't taking and so cruise
way mo even Aurora no Aurora tzuke's is
the same stack as well they're all the
same codebase even and they're all the
same DARPA urban challenge codebase
so the question is do you think there's
a room for brilliance and innovation
there that will change everything like
say okay so I'll give you examples it
could be if revolution and mapping for
example that allow you to map things do
HD maps of the whole world all weather
conditions
really well or revolutionist simulation
to where the the what you said before
becomes incorrect that kind of thing I
knew room for breakthrough innovation um
what I said before about oh they
actually get the whole thing well I'll
say this about we divide driving into
three problems and I actually haven't
solved the third yet but I have an idea
how to do it so there's the static the
static driving problem is assuming you
are the only car on the road right right
and this problem can be solved 100% with
mapping and localization this is why
farms work the way they do if all you
have to deal with is the static problem
and you can statically schedule your
machines right it's the same as like
statically scheduling processes you can
statically schedule your tractors to
never hit each other on their paths all
right because then you know the speed
they go at so so that's the static
driving problem Maps only helps you with
the static driving problem yeah the
question about static driving yeah you
just made it sound like it's really easy
that was really easy how easy how well
because the whole drifting out of lane
when when Tesla drifts out of lane is
failing on the fundamental static
driving problem Tesla is drifting out of
lane the static driving problem is not
easy for the world the static driving
problem is easy for one route and one
route in one weather condition with one
state of lane markings and like no
deterioration no cracks in the road I'm
assuming you have a perfect localizer so
that's all for the weather condition and
me the lane marking condition that's the
problem is how could you how do you have
a perfect you can build perfect
localizers are not that hard to build
okay come on now with with wood lighter
why don't ya wood lighter okay yeah but
you use lighter right like use lidar
build a perfect localizer building a
perfect localizer without lidar it's
gonna be it's gonna be hard you can get
ten centimeters without liner you can
get one centimeter with lidar maybe
concern about the one or ten centimeter
I'm concerned if every once in a while
you're just way off yeah so this is why
you have to carefully make sure you're
always tracking your position you want
to use light
camera fusion but you can get the
reliability of that system up to a
hundred thousand miles and then you
write some fallback condition where it's
not that bad if you're way off right I
think that you can get it to the point
it's like özil D that you're you're
never in a case where you're way off and
you don't know it yeah okay so this is
brilliant so that's the static static we
can especially with lidar and good HD
maps you can solve that problem easy
no you just the static static very
typical for you to say something's easy
I got it it's not as challenging as the
other ones okay well it's okay maybe
it's obvious how to solve it the third
one's the hardest well where do we get
and a lot of people don't even think
about the third one and even I see it as
different from the second one so the
second one is dynamic the second one is
like say there's an obvious examples
like a car stopped at a red light
right you can't have that car in your
map yeah because you don't know whether
that car is gonna be there or not so you
have to detect that car in real time and
then you have to you know do the
appropriate action right also that car
is not a fixed object that car may move
and you have to predict with that car
will dim alright so this is the dynamic
problem yeah do you have to deal with
this um this involves again like you're
gonna need models of other people's
behavior do you are you including in
that and I don't want to step on on the
third one oh but if I are you including
in that you're influenced and people I
guess the third okay that's the moon we
call it the counterfactual yeah I
believe that I just talked to Judea
pearl who's obsessed with
counterfactuals oh yeah yeah so the
static and the dynamic yeah
our approach right now for lateral will
scale completely to the static a dynamic
the counterfactual the only way I have
to do it yet they don't give you thing
that I want to do once we have all these
cars is I want to do reinforcement
learning on the world I'm always gonna
turn the exploiter up to max I'm not
gonna have them explore but the only
real way to get at the counterfactual is
to do reinforcement learning because the
other agents are humans so that's
fascinating that you break you down like
that I agree completely I've set my life
thinking about this beautiful
they're so and part of it because you're
slightly insane because not my life just
the last four years no no you have like
some some nonzero percent of your brain
has a madman in it which that's a really
good feature but there's a safety
component to it that I think when this
sort of counterfactuals and so on that
would just freak people out how do you
even start to think about just in
general I mean you've you've had some
friction with Nitza and so on I am
frankly exhausted by safety engineers
the the prioritization on safety over
innovation to a degree where it kills in
my view kills safety in the long term so
the counterfactual thing they just just
actually exploring this world of how do
you interact with dynamic objects and so
on how do you how do you think about
safety you can do reinforcement learning
without ever exploring and I said that
like so you can think about you're in
like a reinforcement learning it's
usually called like a temperature
parameter and your temperature parameter
is how often you deviate from the Arg
max I could always set that to zero and
still learn and I feel that you'd always
want that set to zero on your actual
system got you but the problem is you
first don't know very much and so you're
going to make mistakes so the learning
the exploration happens to ready yeah
but okay so the consequences of a
mistake yeah open pilot and autopilot
are making mistakes left and right yeah
we have we have we have 700 daily active
users a thousand weekly active users
open pilot makes tens of thousands of
mistakes a week these mistakes have zero
consequences these mistakes are oh it I
wanted to take this exit and it went
straight so I'm just gonna carefully
touch the wheel humans the humans catch
them
and the human disengagement is labeling
that reinforcement learning in a
completely consequence-free way so
driver monitoring is the way you ensure
they keep yes they keep paying attention
how is your messaging say I gave you a
billion dollars you would be scaling and
now
oh my fact it's guy couldn't scale with
any amount of money I'd raise money if I
could if I had way to scale yeah you're
not focusing I don't know I don't know
how to do Oh like I guess I could sell
it to more people but I want to make the
system better better I don't know I mean
but what's the messaging here I got a
chance to talk to you on and and he he
basically said that the human factor
doesn't matter you know the human
doesn't matter because the system will
perform there would be sort of a sorry
to use the term but like a singular like
a point where it gets just much better
and so the human it won't won't really
matter but it seems like that human
caching the system when it gets into
trouble is like the thing which will
make something like reinforcement
learning work so how do you how do you
think messaging for Tesla for you should
chant for the industry in general should
change I think my messaging is pretty
clear at least like our messaging wasn't
that clear in the beginning and I do
kind of fault myself for that we are
proud right now to be a level 2 system
we are proud to be level 2 if we talk
about level 4 it's not what the current
hardware it's not gonna be just a
magical OTA upgrade it's gonna be new
hardware it's gonna be very carefully
thought-out right now we are proud to be
level 2 and we have a rigorous safety
model I mean not like like okay rigorous
who knows what that means but we at
least have a safety model and we make it
explicit is in safety MD and open pilot
and it says seriously though safety dot
MD Android so well this is this is the
safety model and I like to have
conversations like if like you know
sometimes people will come to you and
they're like your systems not safe okay
have you read my safety Doc's would you
like to have an intelligent conversation
about this and the answer is always no
they just like scream about it runs
Python okay what so you're saying that
that because pythons not real-time
Python not being real-time never causes
disengagement disengagement SAR caused
by you know the model is QM but safety
dad MD says the following first and
foremost the driver must be paying
attention at all times
I don't can I do I still consider the
software to be alpha software until we
can actually enforce that statement but
I feel it's very well
communicated to our users two more
things one is the user must be able to
easily take control of the vehicle at
all times
mm-hmm so if you step on the gas or
brake with open pilot
it gives full manual control back to the
user or press the cancel button step 2
the car will never react so quickly we
define so quickly to be about one second
that you can't react in time and we do
this by enforcing torque limits braking
limits and acceleration limits so we
have um like our torque limits way lower
than Tesla's this is another potential
if I could tweak autopilot
I would lower their torque limit or
would a driver monitoring um because
autopilot can jerk the wheel hard yeah
open pilot can it's we we limit um and
all this code is open source readable
and I believe now it's all misery C
compliant misra is like the automotive
coding standard um at first I you know
I've come to respect I've been reading
like the standards lately and I've come
to respect them they're actually written
by very smart people yeah they're
brilliant people actually they have a
lot of experience there's sometimes a
little too cautious but in this case it
pays off miss was written by like
computer scientists and you tell them as
a language they use you can tell by the
language they use they talk about like
whether certain conditions in misra are
decidable or undecidable you mean like
the halting problem and yes well all
right you've earned my respect I will
tell you carefully what you have to say
and we want to make our code compliant
with that all right so you're proud
level two and reform so you were the
founder and I think CEO of comm AI then
you were the head of research what the
heck are you know what's your connection
to come AI the president but I'm one of
those like unelect unelected presidents
of like like a small dictatorship
country not one of those like elected
presidents oh so you're like Putin when
he was like yeah I got sure so there's
uh what's the governance structure
what's the what's the future of commie I
finance I mean as a business do you want
you just focused on getting things right
now making some small amount of money
and mean to
and then one that works it works in each
scale our burn rate is about 200k a
month and our revenue is about 100k a
month so we need to 4x our revenue but
uh we haven't like tried very hard at
that yet
and the revenue is basically selling
stuff online yeah we sell stuff shopped
a comment at AI is there other well okay
so you you'll have to figure out that's
our that's our only see but to me that's
like respectable revenues yeah we make
it by selling products to consumers
we're honest and transparent about what
they are most actually level for
companies right because you could easily
start blowing up like smoke like over
selling the hype and feeding into
getting some fundraisers oh you're the
guy you're genius because you hacked the
iPhone oh I hate that I hate that yeah I
can trade my social capital for more
money
yeah I did it once I almost regret it
doing the first of it well on a small
tangent what's your you seem to not like
Fame and yet you're also drawn to fame
what were you on we're on you where are
you on that currently have you had some
introspection some soul-searching
yeah I actually I've come to a pretty
stable position on that like after the
first time I realized that I don't want
attention from the masses
I want attention from people who I
respect who you respect I can give a
list of people so are these like Elon
must have characters yeah
well actually you know what I'll make it
more broad than that I won't make it
about a person I respect skill I respect
people who have skills right and I would
like to like be I'm not gonna say famous
but be like known among more people who
have like real skills who in cars doers
do you think have skill not do you
respect Oh Kyle vote has skill a lot of
people away mo have skill and I respect
them I I respect them as engineers like
I can think I mean I think about all the
times in my life where I've been like
dead set on approaches and they turn out
to be wrong
so I mean this might I might be wrong I
accept that I accept that there's a
decent chance that I'm I'm wrong and
actually I mean having talked to Chris
Urmson sterling anderson i those those
guys I mean I deeply respect Chris I
just admire the guy he's legit can you
drive a car through the desert when
everybody thinks it's impossible that is
that's legit and then I also really
respect the people who are like writing
the infrastructure of the world like the
linus torvalds and the chris lab they're
doing the real work I know they're doing
the real work this every dog that Chris
Ladin you realize especially when
they're humble it's like you realize oh
you guys were just using your oh yeah
all the hard work they did him that's
incredible what do you think mr. Anthony
lowendahl ski
what do you he's a he's another mad
genius sharp guy oh yeah what do you
think he might long-term become a
competitor Oh
tu cama well so I think that he has the
other right approach I think that right
now there's two right approaches one is
what we're doing and one is what he's
doing can you describe I think it's
called pronto a certain you thing did do
you know what what the approaches
actually don't know embark is also doing
the same sort of thing the idea is
almost that you want to so if you're I
can't partner with Honda and Toyota
Honda and Toyota are uh like four
hundred thousand person companies it's
not even a company at that point like I
don't think of it like I don't personify
it I think of it like an object but a
trucker
drives for a fleet maybe that has like
some truckers are independent some
truckers Drive for fleets with a hundred
trucks there are tons of independent
trucking companies out there start a
trucking company and drive your costs
down or figure out how to drive down the
cost of trucking another company that I
really respect is uh not oh I should I
respect their business model no auto
sells a driver monitoring camera and
they sell it to fleet owners if I that's
right if I owned a fleet of cars and I
could pay you know 40 bucks a month to
monitor my employees
this is gonna like reduces accidents 18%
yeah it's it's so like that in the space
that is like the business model that I
like most respect is there creating
value today yeah which is uh that's a
huge one is how do we create value today
with some of this then the link keeping
things huge and it sounds like you're
creeping in or full steam ahead on the
driver monitoring - yeah which I think
actually were the short-term value if
you can get right I still I'm not a huge
fan of the statement that everything is
to have driver monitoring but will I
agree with that completely but I'm that
statement usually misses the point that
to get the experience of it right is not
trivial oh no not at all in fact like so
right now we have I think the time out
depends on speed of the car but we want
to depend on like the scenes day if
you're on like an empty Highway it's
very different if you don't pay
attention then if light you're like
coming up to a traffic light and
long-term it should probably learn from
from the driver because that's to do I
watched a lot of video we've built a
smartphone detector just to analyze how
people are using smartphones and people
are using it very differently and
there's a it's a texting styles there's
videos yeah like I got billions of miles
of people driving cars in this moment I
spent a large fraction of my time just
watching videos because it's never fails
to to learn like it never I've never
failed from a video watching session to
learn something I didn't know before
fact I usually like when I eat lunch
I'll sit especially when the weather is
good and just watch pedestrians with an
eye to understand like from a computer
vision I just to see can this model can
you predict what are the decisions made
and there's so many things that we don't
understand this is what I mean about the
state vector
yeah it's I'm trying to always think
like Gamma understanding in my human
brain how do we convert that into how
hard is the learning problem here I
guess is the fundamental question so
something that from a hacking
perspective this is always comes up
especially with folks well first the
most popular question is
the trolley problem right so that's not
a sort of a serious problem there are
some ethical questions I think that
arise maybe will you want to met you or
do you think there's any ethical serious
ethical questions that we have a
solution to the trolley problem Akane
aye well so there is actually an alert
in our code ethical dilemma detected
it's not triggered yeah we don't we
don't how you have to detect the ethical
dilemmas but we're a level two system so
we're going to disengage and leave that
decision to the human you're such a
troll hey no but the trolley problem
deserves to be trolled yeah that's a
beautiful answer actually I know I gave
it to someone who was like sometimes
people ask like you asked about the
trolley problems like you can have a
kind of discussion about it like boo you
get someone who's like really like
earnest about it because it's the kind
of thing where if you ask a bunch of
people in an office whether we should
use a sequal stack or no sequel stack if
they're not that technical they have no
opinion but if you ask them what color
they want to paint the office everyone
has an opinion on that and that's why
the trolley problem is that's it I mean
it's a beautiful answer
yeah we're able to detect the problem
and were able to pass it on to the human
yeah I've never never heard anyone say
it nice escape route okay but proud
level - I'm proud level - I love it so
the other thing that people cope you
know have some concern about with AI in
general is hacking so how hard is it do
you think to hack a nataas vehicle
either through physical access or
through the more sort of popular now
these adversarial examples on the
sensors be adversarial examples one you
want to see some adversarial examples
that affect humans hmm right oh well
there used to be a stop sign here but I
put a black bag over the stop sign and
then people ran it all right adversarial
yeah right like like like there's tons
of human adversarial examples - um the
question in general about like security
if you saw something something just came
out today I'm like there are always such
high P headlines about like how navigate
on autopilot was fooled by a GPS spoof
to take an exit right at least that's
all they could do was take an exit if
your car is relying on GPS in order to
have a safe driving policy they're doing
something
if you're relying and this is why v2v is
such a terrible idea
v2v now relies on both parties getting
communication right this is not even so
I think of safety security is like a
special case of safety right safety is
like we put a little you know piece of
caution tape around the hole so that
people won't walk into it by accident
security is I put a 10 foot fence around
the hole so you actually physically
cannot climb into it with barbed wire on
the top and stuff right so like if
you're designing systems that are like
unreliable they're definitely not secure
your car should always do something safe
using its local sensors and then the
local sensor should be hardwired and
then could somebody hack into your can
boss and turn your steering wheel on
your brakes
yes but they could do it before common
AI too so let's think out of the box and
some things so do you think
teleoperation has a role in any of this
so remotely stepping in and controlling
the cars no I think that if safety if
the safety operation by design requires
a constant link to the cars I think it
doesn't work so that's the same argument
using for v2i VTV well there's a lot of
non safety critical stuff you can do
with v2 I like v2 I liked v2 I weigh
more than V B because Vita I is is
already like I already have internet in
the car right there's a lot of great
stuff you can do with v2 I um like for
example you can well where I already
have v2 Waze is V die right ways can
route me around traffic jams that's a
great example of v2 I mm-hmm and then
okay the car automatically talks to that
same service like improving the
experience but it's not a fundamental
fallback for safety know if any of your
if any of your if any of your things
that require wireless communication are
more than qm like have a nozzle rating
you should you previously said that life
is work and then you don't do anything
to relax so how do you think about hard
work well what is it what do you think
it takes to
as great things you know there's a lot
of people saying that there needs to be
some balance you know you need to in
order to accomplish great things you
need to take some time off each of
reflects and so on now and then some
people are just insanely working burning
the candle at both ends how do you think
about that I think I was trolling in the
Siraj interview when I said that
off camera right before I spoke a little
bit we'd like get out spot this is a
joke right like I do nothing it relaxed
look where I am I'm at a party right
yeah
that's true so no of course I I don't um
what I say that life is work though I
mean that like I think that what gives
my life meaning is work I don't mean
that every minute of the day you should
be working I actually think this is not
the best way to maximize results I think
that if you're working 12 hours a day
you should be working smarter and not
harder
well so it gives work gives you meaning
for some people other source of meaning
is personal relationships yeah like
family and so on
you've also in that interview of Sirach
or does the the trolling mentioned that
one of the things you look forward to in
the future is AI girlfriends yes so at
the topic that I'm all very much
fascinated by not necessarily
girlfriends but just forming a deep
connection with AI what kind of system
do you imagine when you say AI
girlfriend whether you were trolling or
not know that one I'm very serious about
and I'm serious about that on both a
shallow level and a deep level I think
that VR brothels are coming soon and are
gonna be really cool it's not cheating
if it's a robot I see the slogan already
but there's I don't know if you've
watched it just watched the black mirror
episode i watch the one year yeah yeah
oh the the Ashley - one way da no where
there's two friends were having sex with
each other and mo in the VR game your
game it's just two guys but yeah one of
them was was a female and yeah there's
another mind-blowing concept that in VR
you don't have to be the form you can be
to animals having sex
weird I mean I'll see you I said the
software Maps the nerve endings right
yeah yeah they they sweep a lot of the
fascinating really difficult technical
challenges under the rock like assuming
it's possible to do the mapping of the
nerve endings then I wish yeah I saw
that the way they did it with a little
like stim unit on the head that'd be
amazing so wanna know on a shallow level
like you could set up like almost a
brothel with like real dolls and oculus
quests right some good software I think
it vehicle novelty experience you know
on a deeper like emotional level I mean
yeah I would really like to fall in love
with with with the machine do you see
yourself having a long-term relationship
of the kind monogamous relationship that
we have now with a robot with a a AI
system even not even just the robot so I
think about maybe my ideal future when I
was fifteen I read eliezer yudkowsky
early writings mmm-hmm
on the singularity and like that AI is
going to surpass human intelligence
massively he made some Moore's law based
predictions that I mostly agree with and
then I really struggled for the next
couple years of my life like why should
I even bother to learn anything it's all
gonna be meaningless when the machines
show up right maybe maybe when I was
that young I was still a little bit more
pure and really like clung to that and
I'm like wow the machines ain't here yet
you know and I seem to be pretty good at
this stuff
let's uh let's try my best you know like
what's the worst that happens but the
best possible future I see is me sort of
merging with the Machine and the way
that I personify this is in a long-term
monogamous relationship with a machine
oh you don't think there's room for
another human in your life if you really
truly merge with another machine I mean
I see merging I see like the best
interface to my brain is like the same
relationship and to merge with an AI
right
does that merging feel like I see yeah
I've seen couples who've been together
for a long time and like I almost think
of them as one person like couples who
spend all their time together and that's
that's how you're actually putting what
does that merging actually looks like
it's not just a nice channel like a lot
of people imagine it's just an efficient
link search link to Wikipedia or
something I don't believe in that but
it's more you're saying that there's the
same kind of the same kind of
relationship you have one other human
that's a deep relationship is that's
what merging looks like that's that's
pretty uh I don't believe that link is
possible um I think that that link so
you're like oh me to download Wikipedia
right to my brain yeah my reading speed
is not limited by my eyes my reading
speed is limited by my inner processing
locally and to like bootstrap that
sounds kind of unclear how to do it and
horrify but if I am with somebody and
I'll use a somebody who is making a
super sophisticated model of me and then
running simulations on that model I'm
not gonna get into the question whether
the simulations are conscious or not I
don't really want to know what it's
doing
um but using those simulations to play
out hypothetical futures for me deciding
what things to say to me to guide me
along a path and that's how I envision
it
so on that path to AI of superhuman
level intelligence you've mentioned that
you believe in the singularity that
singularity is coming yeah again could
be trolling could be not could be part
I'm all trolling his truth in it I don't
know what that means anymore what is the
singularity yeah so that's that's really
the question how many years do you think
before the singularity what form do you
think it will take does that mean
fundamental shifts and capabilities of
AI does it mean some other kind of ideas
um maybe this is just my roots but so I
can buy a human beings worth of compute
for like a million bucks that I it's
about one TPU pod v3 I want like I think
they claim a hundred peda flops that's
being generous I think humans are
actually more like twenty so that's like
five humans that's pretty good
Google needs to sell their teepees um
but I could buy I could buy I could buy
GPUs I could buy a stack of like by 1080
tea eyes build data center full of them
four million box I can get a human worth
of compute but when you look at the
total number of flops in the world when
you look at human flops which goes up
very very slowly with the population and
machine flops which goes up
exponentially but it's still nowhere
near I think that's the key thing to
talk about when the singularity happened
when most flops in the world are silicon
and not biological that's kind of the
crossing point like they are now the
dominant species on the planet and just
looking at how technology is progressing
when do you think that could possibly
happen
you think go to happen in your lifetime
oh yeah definitely my lifetime I've done
the math I like 2038 because it's the
UNIX timestamp rollover yeah beautifully
put
so you've you said that the meaning of
life has to win if you look five years
into the future what does winning look
like so
[Music]
hi there's a lot of I can go into like
technical depth to what I mean by that
to win um it may not mean I was
criticized for that in the comments like
doesn't this guy want to like save the
penguins in Antarctica or like you know
listen to what I'm saying I'm not
talking about like I have a yacht or
something I am an agent I am put into
this world and I don't really know what
my purpose is but if you're a
reinforcement if you're if you're an
intelligent agent and you're put into a
world what is the ideal thing to do well
the ideal thing mathematically you go
back to like Schmitt Hoover theories
about this is to build a compressive
model of the world to build a maximally
compressive to explore the world such
that your exploration function maximizes
the derivative of compression of the
past mid Hoover has a paper about this
and like I took that kind of as like a
personal goal function so what I mean to
win I mean like maybe maybe this is
religious but like I think that in the
future I might be given a real purpose
or I may decide this purpose myself and
then at that point now I know what the
game is and I know how to win I think
right now I'm still just trying to
figure out what the game is but once I
know so you have you have imperfect
information you have a lot of
uncertainty about the reward function
and you're discovering it exactly the
purpose is that's that's the better way
to put it the purpose is to maximize it
while you have it a lot of uncertainty
around it and you're both reducing the
uncertainty and maximizing at the same
time yeah and so that's at the technical
level what is the if you believe in the
universal prior yeah
what is the universal reward function
that's the better way to put it so that
when it's interesting I think I speak
for everyone in saying that I wonder
what that reward function is for you and
I look forward to seeing that in five
years in ten years I think a lot of
people who do myself right and cheering
you on man so I'm I'm a happy you exist
and I wish you the best of luck thanks
for talking today man
thank you this is a lot of fun
you