Kind: captions
Language: en
the following is a conversation with
Chris lner his second time in the
podcast he's one of the most brilliant
engineers in modern Computing having
created llvm compiler infrastructure
project the clang compiler the Swift
programming language a lot of key
contributions to tensor flown tpus as
part of Google he served as vice
president of autopilot software Tesla
was a software innovator and leader at
Apple and now is at SciFi as senior vice
president of platform engineering
looking to revolutionize chip design to
make it faster better and cheaper quick
mention of each sponsor followed by some
thoughts related to the episode first
sponsor is blinkist an app that
summarizes key ideas from thousands of
books I use it almost every day to learn
new things or to pick which books I want
to read or listen to next second is
neuro the maker of functional sugar-free
gum and mints that I use to supercharge
my mind with caffeine alanine and B
vitamins third is master class online
courses from the best people in the
world on each of the topics covered from
Rockets to game design to Poker to
writing and to guitar and finally cash
app the app I use to send money to
friends for food drinks and
unfortunately lost bets please check out
the sponsors in the description to get a
discount and to support this podcast as
a side note let me say that Chris has
been in inspiration to me on a human
level because he is so damn good as an
engineer and leader of Engineers and yet
he's able to stay humble especially
humble enough to hear the voices of
disagreement and to learn from them he
was supportive of me in this podcast
from the early days and for that I'm
forever grateful to be honest most of my
life no one really believed that I would
amount to much so when another human
being looks at me and makes me feel like
I might be someone special it can be
truly inspired ing that's a lesson for
educators the weird kid in the corner
with a dream is someone who might need
your love and support in order for that
dream to
flourish if you enjoy this thing
subscribe on YouTube review it with five
stars and apple podcast follow on
Spotify support on patreon or connect
with me on Twitter at Lex Freedman and
now here's my conversation with Chris
lner what are the strongest qualities of
Steve Jobs Elon Musk and the Great and
Powerful Jeff Dean since you've gotten
the chance to work with each you're
starting with an easy question there um
these are three very different people I
guess you could do maybe a pair wise
comparison between them instead of a
group comparison so if you look at Steve
Jobs and Elon um I worked a lot more
with Elon than I did with Steve um they
have a lot of commonality they're both
um Visionary in their own way they're
both very demanding in their own way um
my sense is Steve Steve is much more
human factor focused where Elon is more
technology focused what does human
factor mean Steve is trying to build
things that feel good that people love
that affect people's lives how they live
he's looking into into the future a
little bit in terms of um what people
want um where I think the Elon focuses
more on uh learning how exponentials
work and predicting the development of
those Steve worked with a lot of
Engineers that was one of the things
that reading the biography and how how
can a designer essentially talk to
engineers and like get their respect I
think so I did not work very closely
with Steve I'm not an expert at all my
sense is that he uh pushed people really
hard but then when he got an explanation
that made sense to him then he would let
go and um he did actually have a lot of
respect for engineering and but he also
knew when to push and you know when you
can read people well you can know when
they're holding back and when you can
get a little bit more out of them and I
think he was very good at that I mean if
you if you compare the other the other
folks so Jeff Dean right Jeff Dean's an
amazing guy he's super smart um as as
are the other guys um Jeff is a really
really really nice guy well-meaning he's
a classic googler he uh uh wants people
to be happy he combines it with
Brilliance so he can pull people
together in a in a really great way he's
definitely not a CEO type I don't think
he would even want to be that um do you
know if he still programs oh yeah
definitely programs Jeff is an amazing
engineer today right and that has never
changed so um it's really hard to
compare Jeff to to either of those two
um he uh I think that Jeff leads through
technology and building it himself and
then pulling people in and inspiring
them and so I think that that's um a one
of the amazing things about Jeff but
each of these people you know with their
pros and cons all are really
inspirational and have achieved amazing
things so it's been a it's been I've
been very fortunate to work with these
guys for yourself you've LED large teams
you've done so many incredible difficult
technical challenges is there something
you've picked up from them about how to
lead yeah well so I mean I think
leadership is really hard it really
depends on what you're looking for there
um I think you really need to know what
you're talking about so being grounded
on the product on the technology on the
business on the mission is really
important being uh understanding what
people are looking for for why they're
there one of the most amazing things
about Tesla is the unifying Vision right
people are there because they believe in
clean energy and El electrification all
these kinds of things um uh the others
is to understand what really motivates
people how to get the best people how to
how to build a plan that actually can be
executed right there's so many different
aspects of leadership and it really
depends on the time the place the
problems you know you know there's a lot
of issues that don't need to be solved
and so if you focus on the right things
and prioritize well that can really help
move things two interesting things you
mentioned one is you really have to know
what you're talking about how you've
uh you've worked on a lot of very
challenging technical things sure
so I kind of assume you were born uh
technically Savvy but assuming that's
not the case uh how did how did you
develop technical expertise like even at
Google you worked on I don't know how
many projects but really challenging
very varied compilers tpus Hardware
Cloud stuff bunch of different things um
the thing that I've become comfortable
as I've more comfortable with as I've uh
gained experience is uh being okay with
not knowing and so a major part of
leadership is actually it's not about
having the right answer it's about
getting the right answer and so if
you're working in a team of amazing
people right and many of these places
many of these companies all have amazing
people it's the question of how do you
get people together how do you get how
do you build trust how do you get people
to open up how do you people get people
to you know be vulnerable sometimes with
an idea that maybe isn't good enough but
it's the start of something beautiful
how do you um how do you provide an
environment where you're not just like
top down Thou shalt do the thing that I
tell you to do right but you're
encouraging people to be part of the
solution and uh and providing a safe
space where if you're not doing the
right thing they're willing to tell you
about it right so you're okay asking
dumb questions yeah dumb questions are
my specialty yeah well I so I've been in
the hardware room recently and I don't
know much at all about how chips are
designed I know a lot about using them I
know some of the principles and the ARs
technical level of this but level it
turns out that if you ask a lot of dumb
questions you get smarter really really
quick and when you're surrounded by
people that want to teach and learn
themselves uh it can be a beautiful
thing uh so let's talk about programming
languages if it's okay at the High this
absurd philosophical level cuz I I don't
get romantic on me Lex I I will forever
get
romantic and uh Tor here I apologize uh
why do programming languages even
matter okay well thank you very much so
you're saying why should why why should
you care about anyone programming
language or why do why do we care about
programming computers or no why why do
we why do we care about programming
language design creating effective
programming
languages uh
choosing a you know one programming
languages versus another programming
language why we keep struggling and
improving through the evolution of these
programming languages sure sure sure
okay so so I mean I think you have to
come back to what what are we trying to
do here right so we have these these uh
beasts called computers that are very
good at specific kinds of things and we
think it's useful to have them do it for
us right uh now you have this question
of how best to express that because you
have a human brain still that has an
idea in its head and you want to achieve
something right so well there's lots of
ways of doing this you can go directly
to the machine and speak Assembly
Language and then you can express
directly what the computer understands
that's fine um you can then have higher
and higher and higher levels of
exraction up until machine learning and
you're designing an neural net to do the
work for you um the question is where
where along this way do you want to stop
and what benefits do you get out of
doing so and so programming languages in
general you have C you have Fortran and
Java and a Pascal Swift you have lots of
different things um they all have
different trade-offs and they're T
tackling different parts of the problems
now one of the things that most
programming languages do is they're
trying to make it so that you have
pretty basic things like portability
across different Hardware so you've got
I'm going to run on an Intel PC I'm
going to run on RIS 5 PC I'm going to
run on a arm phone or something like
that fine um I want to write one program
and have it portable and this is
something the assembly doesn't do now
when you start looking at the space of
programming languages this is where I
think it's fun
because programming languages all have
trade-offs and most people will walk up
to them and they look at the surface
level of syntax and say oh I like Curly
braces or I like tabs or I like you know
semicolons or not or whatever right
subjective fairly subjective very
shallow things but programming languages
when done right can actually be very
powerful and the the benefit they bring
is expression
okay and if you look at programming
languages there's really kind of two
different levels to them one is the down
in the dirt nuts and bolts of how do you
get the computer to be efficient stuff
like that how they work type systems
compiler stuff things like that the
other is the UI and the UI for
programming language is really a design
problem and a lot of people don't think
about it that way and the UI you mean
all that stuff with the braces and yeah
all that stuff to the UI and what it is
and UI means user interface um and so
what what's really going on is it's the
interface between the guts and the human
and humans are hard right humans have
feelings they have things they like they
have things they don't like and a lot of
people treat programming languages as
though humans are just kind of abstract
creatures that cannot be predicted but
it turns out that actually there are
there is better and
worse like people can tell when a
program language is good or when it was
an accident right and uh one of the
things with Swift in particular is that
a tremendous amount of time by a t
tremendous number of people have been
put into really polishing and Mak it
feel good but it also has really good
nuts and bolts underneath it you said
that uh Swift makes a lot of people feel
good how do you get to that
point
so how do you predict that um you know
tens of thousands hundreds of thousands
of people are going to enjoy using this
the user experience of this programming
language well you can you can look at it
in terms of better and worse right so if
you have to write lots of boiler plate
or something like that you will feel
unproductive and so that's a bad thing
you can look at it in terms of safety if
like C for example is what's called a
memory unsafe language and so you get
dangling pointers and you get all these
kind of bugs that then you have to spend
tons of time debugging and it's a real
pain in the butt and you feel
unproductive and so by subtracting these
things from the experience you get um
you know happier people but uh uh again
keep interrupting I'm sorry uh but so
hard to deal
with if you look at the people people
that are most productive on stack
Overflow they are uh they have a set of
priorities yeah that may not always
correlate perfectly with the experience
of the majority of users you know like
if you look at the most uploaded uh
quote unquote correct answer on stack
Overflow it
usually really um sort of uh prioritizes
like safe code proper code stable
code uh you know that kind of stuff as
opposed to like if I want to use go-to
statements in my basic right uh I'm I
want to use go-to State like what if 99%
of people want to use go-to statements
or use completely improper you know
unsafe syntax I I don't think that
people actually like if you boil it down
and you get below the surface level
people don't actually care about go-tos
or if statements for things like this
they care about achieving a goal yeah
right so the real question is I want to
set up a web server and I want to do a
thing and I whatever like how how
quickly can I achieve that right and so
the from a programming language
perspective there's really two things
that that matter there one is what
libraries exist and then how quickly can
you put it together and what are the
tools around that look like right and uh
and when you want to build a library
that's missing what do you do okay now
this is where you see huge Divergence in
the force between worlds okay and so you
look at python for example python is
really good at assembling things but
it's not so great at building all the
libraries and so what you get because of
performance reasons other things like
this is you get python layered on top of
C MH for example and that means that
doing certain kinds of things well it
doesn't really make sense to do in
Python instead you do it in C and then
you rapid and then you have you're
living in Two Worlds and Two Worlds
never is really great because tooling
and the the debugger doesn't work right
and like all these kinds of things can
you clarify a little bit what uh what
you mean by python is not good at
building libraries meaning doesn't make
certain kinds of libraries no but just
the actual meaning of the sentence yeah
uh meaning like it's not conducive to
developers to come in and add libraries
or it's it's or the langu or is it the
The Duality of the it's a dance between
Python and c and Python's amazing
Python's a great language I did not mean
to say that python is is bad for
libraries what what I meant to say is um
there python there are libraries that
Python's really good at they you can
write in Python but there are other
things like if you want to build a
machine learning framework you're not
going to build a machine learning
framework in Python because of
performance for example or you want GPU
acceleration or things like this instead
what you do is you write a bunch of C or
C++ code or something like that and then
you talk to it from python right and so
this is because of decisions that were
made in the python design and um and
those decisions have other
counterbalancing forces but but the
trick when you start looking at this
from a programming language perspective
is you start say okay cool how do I
build this catalog of libraries that are
really powerful and how do I make it so
that then they can be assembled into
ways that feel good and they generally
work the first time because when you're
talking about building a thing you have
to include the debugging the fixing the
turnaround cycle the development cycle
all that kind of stuff in in into the
process of building the thing it's not
just about pounding out the code and so
this is where things like um you know
catching bugs at compile time is
valuable for example um but if you dive
into the details on this Swift for
example has certain things like value
semantics which is this fancy way of
saying that when you uh treat a treat a
variable like a value um uh it acts like
a mathematical object would okay so in
you have used pytorch a little bit in
pytorch you have tensors tensors are uh
ND nend dimensional grid of numbers very
simple you can do plus and other
operators on them it's all totally fine
but why do you need to clone a tensor
sometimes have you ever run into that uh
yeah okay and so why is that why do you
need to clone a tensor it's the usual
object thing that's in Python so in
Python and just like with Java and many
other languages this isn't unique to
python in Python it has a thing called
reference semantics which is the nerdy
way of explaining this and what that
means is you actually have a pointer to
a thing instead of the thing okay now
this is due to a bunch of implementation
details that you don't want to go into
but in Swift you have this thing called
value sematics and so when you have a
tensor in Swift it is a value if you
copy it it looks like you have a unique
copy and if you go change one of those
copies then uh it doesn't update the
other one because you just made a copy
of this thing right so that that's like
highly error prone in uh at least
computer science math Centric
disciplines about python that like the
the thing you would expect to behave
like math like math it doesn't behave
like math and in fact uh quietly doesn't
behave like math and then can ruin the
entirety of your math exactly well and
then it puts you in debugging land again
yeah right now now you just want to get
something done and you're like wait wait
a second where you need where do I need
to put clone in what level of this stack
which is very complicated which I
thought I was reusing somebody's library
and now I need to understand it to know
where to clone a thing thing right and
hard to debug by the way exactly right
and so this is where programming
languages really matter right so in
Swift having value sematics so that um
both you get the benefit of math working
like math right but also the efficiency
that comes with certain advantages there
certain implementation details there
really benefit you as a programmer right
you clarify the value sematics like how
how do you know that a thing should be
treated like a value yeah so so Swift uh
has a pretty strong culture and good
language support for defining values and
so if you have an array so tensors are
one example of that the machine learning
folks are very used to um just think
about arrays same thing where you have
an array you put uh you create an array
you put two or three or four things into
it and then you pass it off to another
function mhm what happens if that that
uh function adds some more things to it
well you'll see it on the side that you
pass it in right this is called
reference semantics now what if you
pass an array off to a function it
Scrolls it away in some dictionary or
some other data structure somewhere
right well it thought that you just
handed it that array then you return
back and that that that reference to
that array still exists in the callar
and they go and put more stuff in it
right the the person you handed it off
to may have thought they had the only
reference to that and so they didn't
know what they that this was going to
change underneath the covers and so this
is where you end up having to do clone
so like I was past a thing I'm not sure
if I have the only
version of it so now I have to clone it
so what value sematics does is it allows
you to say hey I have a so in Swift it
defaults to value sematics for oh so
defaults to value sematics and then
because most things should Valu then it
makes sense for that to be the default
and one of the important things about
that is that arrays and dictionaries and
all these other collections or
aggregations of other things also have
value semantics and so when you pass
this around uh to different parts of
your program you don't have to do these
defensive copies
and so this is this is great for two
sides right it's great because you
define away the bug which is a big deal
for productivity the the number one
thing most people care about but it's
also good for performance because when
you're doing a clone so you pass the
array down to the thing it's like I
don't know if anybody else has it I have
to clone it well you just did a copy of
a bunch of data it could be big and then
it could be that the thing that called
you is not keeping track of the old
thing so you just made a copy of it and
you may not have had to yeah and so the
way the value sematics work is in Swift
is it uses this thing called copy on
right which means that you get you get
the benefit of safety cool and
performance and it has another special
trick because um if you think of certain
languages like Java for example they
have immutable strings and so what
they're trying to do is they provide
value semantics by having pure
immutability functional languages have
pure immutability in lots of different
places and this provides a much safer
model than it provides
valics um the problem with this is if
you have immutability everything
everything is expensive everything
requires a copy um for example in Java
if you have a string X and A String y
you pen them together we have to
allocate a new string to hold XY oh if
they're immutable well and strings
strings in Java are immutable and if
there's there's optimizations for short
ones and it's it's complicated but but
generally uh think about them as a
separate allocation and so when you
append them together you have to go
allocate a third thing mhm because
somebody might have a pointer to either
of the other ones right and you can't go
change them so you have to go allocate a
third thing um because of the beauty of
how the Swift value SM system works out
if you have a string and Swift and you
say hey put in X right and they say
append on y z w what it knows that
there's only one reference to that and
so it can do an inpl
update and so you're not allocating tons
of stuff on the side you're not you
don't have all those problems when you
pass it off you can know you have the
only reference if you pass it off to
multiple different people but nobody
changes it they can all share the same
thing so you get a lot of the benefit of
of purely mutable design and so you get
a really nice sweet spot that I haven't
seen in other languages yeah that's like
I thought I thought there was going to
be a a philosophical like narrative here
that you're going to have to pay a cost
for it CU it sounds like uh I think
value semantics is beneficial for easing
of debugging or minim izing the risk of
Errors like bringing the errors closer
to the
source um bringing the symptom of the
air closer to the source of the air
however you say that and but you're
saying there's not a performance cost
either if you implement correctly well
so there there's trade-offs with
everything and so if you are doing very
low-level stuff then sometimes you can
noce cost but then what you're doing is
you're saying what is the right default
so um coming back to you interface when
you when you talk about programming
languages one of the ma ma major things
that Swift does that makes people love
it that is not obvious when it comes to
designing language is this UI principle
of progressive disclosure of complexity
okay so Swift like many languages is
very powerful the question is when do
you have to learn the power as a
user so Swift like python allows you to
start with like print hello world MH
right certain other languages uh start
with like public static void Main
like all the ceremony right and so you
go to teach you teach a new person hey W
welcome to this new thing let's talk
about Public Access Control classes wait
what's that string system.out do
printland like packages like God right
and so instead if you take this and you
say hey we need you need we need
packages you know modules we need we
need powerful things like classes we
need data structures we need like all
these things the question is how do you
factor after the complexity and how do
you make it so that the normal case
scenario is you're dealing with things
that work the right way the right way
give you good performance the right by
default but then as a power user if you
want to dive down to it you have full c
c performance full control over low-l
pointers you can call Malik if you want
to call Malik this is not recommended on
the first page of every tutorial but
it's actually really important when you
want to get work done right and so being
able to have that is really the design
in programm language design design and
design is really really hard it's
something that I think a lot of people
kind of um outside of UI again a lot of
people just think is uh subjective like
there's nothing you know it's just like
Curly braces or whatever it's just like
somebody's preference but actually good
design is something you can feel and uh
how many people are involved with good
design so if we looked at Swift but look
at historically I mean this might touch
like uh it's almost like a Steve jobs
question too like how much dictatorial
decision- making is required
versus um
collaborative and we'll talk about how
all that can go wrong or right but yeah
well Swift so I can't speak to in
general all design everywhere uh so the
way it works with swift is that um
there's a core team and so a core team
is uh six or seven people is something
like that that is people that have been
working with swift since very early days
and so I and by early days is not that
long ago okay yeah so it's it it became
public in 2014 so it's been six years
public now but um but still that's
enough time that there's a story arc
there okay and there's mistakes have
been made that then get fixed and you
learn something and then you you know
and so uh what the core team does is it
provides continuity and so you want to
have a okay well there's a big hole that
we want to fill we know we want to fill
it so don't do other things that invade
that space until we fill the hole right
there there's a boulder that's missing
here we want to do we will do that
Boulder even though it's not today keep
keep out of that space and the whole
team remembers of the remembers the myth
of the boulder that's there yeah yeah
there's a general sense of what the
future looks like in Broad strokes and a
shared understanding of that combined
with a shared understanding of what has
happened in the past that worked out
well and didn't work out well um the
next level out is you have the uh what's
called the Swift Evolution community and
you've got in that case hundreds of
people that really care passionately
about the way Swift evolves and that's
like an amazing thing to again uh the
court team doesn't necessarily need to
come up with all the good ideas you got
hundreds of people out there that care
about something and they come up with
really good ideas too and that provides
this like tumbling rock tumbler for
ideas and so the the evolution process
is you know a lot of people in a
discourse form they're like hashing it
out and trying to like talk about okay
well would should we go left or right or
if we did this what would be good and
you know here you're talking about
hundreds of people so you're not going
to get consensus necessarily not obvious
consensus and so there's a proposal
process that uh then allows the core
team and the community to work this out
and what the core team does is it aims
to get consensus out of the community
and provide gu guard rails but also
provide
long-term make sure we're going the
right direction kind of things so does
that group represent like the how much
people will love the user interface like
you think to capture that well I mean
it's something we talk about a lot
something we care about how well we how
well we do that Up For Debate but I
think that we've done pretty well so far
is the beginner in mind like cuz you
said the progressive disclosure complex
yeah so we care a lot about uh a lot
about that a lot about power a lot about
efficiency a lot about there are many
factors to good design and you have to
figure out a way to kind of work your
way through that and so if you like
think about like a language I love is
lisp probably still because I use Zac
but I haven't done anything any serious
working list but it has a ridiculous
amount of parentheses yeah um I've also
you know with Java and
C++ uh the
braces um you know I I like I I I
enjoyed the comfort of being between
braces you know python is really sorry
to interrupt just like and last thing to
me as a design if I was a language
designer uh God for bit is I would be
very surprised that python with no
braces would nevertheless
somehow be comforting also so like I can
see Arguments for all of these but look
at this this is evidence that it's not
about braces versus tevs right exactly
you're good that's a good point right so
like you know there there's there's
evidence that but see like it's one of
the most argued about things oh yeah of
course just like tabs and spaces which
it does I mean there's one obvious right
answer but it doesn't it doesn't
actually matter what's that come on
we're friends like come on what are you
trying to do to me here people are going
to yeah half the people are going to
tune out yeah um so so do you're able to
identify things that don't really matter
for the experience well no no no it's
it's it's always a really hard so the
easy decisions are easy right I mean you
you C fine those are not the interesting
ones the hard ones are the ones that are
most interesting right the hard ones are
the places where hey we want to do a
thing everybody agrees we should do it
there's one proposal on the table but it
has all these bad things associated with
it well okay what are we going to do
about that do we just take it do we
delay it do we say hey well maybe
there's this other feature that if we do
that first this will work out better um
how does this if if we do this are we
painting ourselves into a corner right
and so this is where again you're having
that core team of people that uh has
some continuity and has perspective has
some of the historical understanding is
really valuable because you get um it's
not just like one brain you get the
power of multiple people coming together
to make good decisions and then you get
the best out of all these people and you
also can harness the the community
around it what about like the decision
of whether like in Python having one
type or having you know uh strict typing
yeah yeah let's talk about this so so um
I I like how you put that by the way
like so so many people would say that
python doesn't have types doesn't have
types yeah I've listened to you enough
to where okay I'm I'm a fan of yours and
listened to way too many podcast and
videos talking about this oh yeah so I
would argue that python has one type and
so um so like when you import python
into Swift which by the way works really
well you have everything comes in as a
python object now here there trade-offs
because um uh you know it depends on
what you're optimizing for and python is
a super successful language for a really
good reason um because it has one type
uh you get duck typing for free and
things like this but also you're pushing
you're making it very easy to to pound
out code on the one hand but you're also
making it very easy to introduce uh
complicated bugs the have debug and you
pass the string into something that
expects an integer and it doesn't
immediately die it goes all the way down
the stack trace and you find yourself in
the middle of some code that you really
didn't want to know anything about and
it blows up and you're just saying well
what did I do wrong right and so types
are good and bad and they have
trade-offs they're good for performance
and certain other things depending on
where you're coming from but it's it's
all about trade-offs and so this is this
is what design is Right design is about
weighing tradeoffs and trying to
understand the ramifications of the the
things that you're weighing like types
or not or one type or many types um but
also within many types how powerful do
you make that type system is another
very complicated question uh with lots
of trade-offs it's very interesting by
the way uh but uh but that's like one
one
dimension and there's a bunch of other
dimensions jit compiled versus static
compiled garbage collected versus
reference counted versus memory man
manual memory management versus you know
like in like all these different
trade-offs and how you balance them are
what make a programm language good
currency y so and all those things I
guess uh when you're designing the
language you also have to think of how
that's going to get all compiled down to
if you care about performance yeah well
and go back to list right so list also I
would say JavaScript is another example
of a very simple language right and so
one of the so I also love lisp I don't
use it as much as maybe you do or you
did no I think we're both everyone who
loves lisp it's like you love it's like
I don't know I love Frank Sinatra but
like how often do I seriously listen to
Franks sure but but but you look at that
or you look at JavaScript which is
another very different but relatively
simple language and there's certain
things that don't exist in the language
but there's there is inherent complexity
to the problems that we're trying to
model and so what happens to the
complexity in the case of uh both of
them for example you say well what about
about large scale software development
okay well you need something like
packages neither language has a like
language affordance for packages and so
what you get is patterns you get things
like npn you get things like you know
like these ecosystems that get built
around and I'm a believer that if you
don't uh model at least the most
important inherent complexity in the
language then what ends up happening is
that complexity gets pushed
elsewhere and when it gets pushed
elsewhere sometimes that's great because
often building things as libraries is
very flexible and very powerful and
allows you to evolve and things like
that but often it leads to a lot of uh
unnecessary Divergence in the force and
fragmentation and and when that happens
you just get kind of a mess yeah and so
the question is how do you how do you
balance that uh don't put too much stuff
in the language because that's really
expensive and makes things complicated
but how do you model enough of the
inherent complexity of the problem that
um you provide the framework and the
structure for people to think about Al
so so the the the the key thing to think
about with uh with programming languages
and you think about what a programming
language is there for is it's about
making a human more productive right and
so like there's an old I think it's
Steve Jobs quote about um it's a bicycle
for the mind right you can you can you
can definitely
walk but you'll get there a lot faster
if you can bicycle on your way and a
programming language is a bicycle for
the mind yeah is basically a wow that's
a really interesting way to think about
it by by raising the level of
abstraction now you can fit more things
in your head by being being able to just
directly leverage somebody's Library you
can now get something done quickly um in
the case of Swift swift UI is this new
framework that Apple has released
recently for doing UI programming and it
has this declarative programming model
which defines away entire classes of
bugs it's make it builds on value
sematics and many other nice Swift
things and what this does allows you
just get way more done with way less
code and now your productivity as a
developer is much higher right and so
that that's really the what programming
languages should be about is it's not
about tabs versus spaces or curly braces
or whatever it's about how productive do
you make the person and you can only see
that when you have libraries that were
built with the right intention that the
language was designed for and with Swift
I think we're still a little bit early
um but Swift UI and many other things
that are coming out now are really
showing that and I think that they're
opening people's eyes it's kind of
interesting to think about like
how that you know then knowledge of
something of how good the bicycle is how
people learn about that you know so I've
used C++ now this is not going to be a
trash talking session about C++ but use
C++ for a really long go there if you
want I have the
scars I I feel like I spent many years
without realizing like there's languages
that could for my particular LIF style
brain style thinking style there's
languages that that could make me a lot
more productive uh in the debugging
stage in the just the development stage
and thinking like the bicycle for the
mind I can fit more stuff into my
Python's a great example of that right I
mean a machine learning framework in
Python is a great example of that it's
just very high abstraction level and so
you can be thinking about things on a
like very high level Al algorithmic
level instead of thinking about okay
well am I copying this tensor to a GPU
or not right it's not it's not what you
want to be thinking about and as I was
telling you I mean I guess I guess the
question I head is uh you know how does
a person like me or in general people
discover more
productive uh you know languages like
how I was as I've been telling you
offline I've been looking for like a
project to work on in Swift so I can
really uh try it out as I mean my
intuition was like doing a hello world
is not going to get me there uh to to to
get me to experience the power of the
language you need a few weeks to change
your metabolism exactly I put uh that
that's one of the problems with people
with diets like I I'm I'm actually
currently to go in parallel but in a
small tangent is I've been recently
eating only meat okay okay and okay so
most people are like uh think that's
horribly unhealthy or whatever you have
like a million It Whatever the science
is it just doesn't sound right well so
so back when I was in college we did the
Atkins diet that was that was a thing
similar and but if you you have to
always give these things a chance I mean
with dieting always not dieting but just
the things that you like if I eat
personally if I eat meat just everything
I could be super F or more focused than
usual I just feel great I I've been I've
been running a lot you know doing
push-ups and pull-ups and so on I mean
python is similar in that sense for me
where you going with
this I mean literally I just I felt I
had like a stupid smile on my face when
I first started using python I could uh
code up really quick things like I like
I I would see the world I'll be
empowered to write a script to to um you
know to do some basic data processing to
rename files on my computer yeah right
like Pearl didn't do that for me uh uh I
mean kind a little bit well and again
like none of these are about which which
is best or something like that but there
there's definitely better and worse here
but it clicks right well yeah it if you
look at Pearl for example you get bogged
down in uh scalers versus arrays versus
hashes versus type Globs and like all
that kind of stuff and and Python's like
yeah let's not do this right and some of
is debuging like everyone has different
priorities but for me it's could I
create systems for myself that Empower
me to debug quickly like I've always
been a big fan even just crude like
asserts like
always uh stating things that should be
true uh which in Python I found myself
doing more because of type all these
kinds of stuff well you could think of
types in a programming language as being
kind of assert yeah they get check at
compile time right um so how do you
learn a new thing well so this or how do
how do people learn new things right
this this is hard uh people don't like
to change people generally don't like
change around them either and so uh
we're all very slow to adapt and change
and usually there's a catalyst that's
required to to force yourself over the
over over the
so for learning a programming language
it really comes down to finding an
excuse like build a thing that that's
that the language is actually good for
that the ecosystem is ready for um and
so um and so if you were to write an IOS
app for example that would be the easy
case obviously you would use Swift for
that right there are other
Android Swift runs on Android oh does it
oh yeah yeah Swift runs in lots of
places so uh okay so Swift Swift swift
is built on top of lvm lvm runs
everywhere lvm for example builds the
Android kernel oh wow okay so um okay I
didn't realize this yeah so Swift swift
is very portable runs on Windows there's
it runs on lots of different things and
Swift side Swift UI and then there's a
thing called UI kit so can I build an
app with Swift uh well so that that's
the thing is the ecosystem is what
matters there so Swift UI and uiit are
Apple Technologies okay got it and so
they happen to like Swift eii happens to
be written in Swift but it's an apple
proprietary framework that um Apple
loves and wants to keep on its platform
which makes total sense you go go to
Android and you don't have that Library
yeah right and so Android has a
different ecosystem of things that
hasn't been built out and doesn't work
as well with Swift and so you can
totally use Swift to do uh like
arithmetic and things like this but
building a UI with Swift on Android is
not not not a not a great experience
right now so so if I wanted to uh so
learn Swift what's the pro I mean the
one practical different version of that
is um Swift for tensorflow for example
and one of the inspiring things for me
with both tensorflow and pytorch is how
quickly the community can like switch
from different libraries y like you
could see some of the communi switching
to pytorch now but it could it's very
easy to see and then tensor flow is
really stepping up its game and then
there's no reason why I think it the way
it works basically there has to be one
GitHub repo like one paper steps up to
get gets people excited gets people
excited and they're like ah I have to
learn this Swift for what what Swift
again like and then they learn and they
fall in love with I mean that's what
happen PTO has there has be a reason a
catalyst yeah and so and and there I
mean people don't like change but it
turns out that once you've worked with
one or two programming languages they're
the basics are pretty similar and so one
of the fun things about learning
programming languages even even maybe
list I don't know if you agree with this
is that when you start doing that you
start learning new
things because you have a new way to do
things and you're forced to do them and
that forces you to explore and it puts
you in learning mode and when you get in
learning mode your mind kind of opens a
little bit and you can you can see
things in a new way even when you go
back to the old place right yeah it's
totally well Lis is functional yeah uh
stuff but I wish there was a kind of
window maybe you can tell me if there is
uh there you go this this a question
uh to ask what is the most beautiful
feature in a programming language before
I ask it let me say like with python I
remember when I saw list
comprehensions okay was like when I like
really took it in yeah it I don't know I
just loved it it was like fun to do like
it was fun to do that kind of um uh yeah
there was something about it to be able
to filter through a list and to create a
new list all in a single line was
elegant I could all get into my head and
it just made me um fall in love with the
language so is there let me ask you a
question uh is there what do use the
most beautiful feature in uh in a
programming languages that you've ever
encountered in Swift maybe and then
outside of Swift I think the thing that
I like the most from a programming
language so so I think the thing you
have to think about with the programming
language again what is the goal you're
trying to get people to get things done
quickly and so you need libraries you
need high quality libraries and then you
need a user base around them that can
assemble them and do cool things with
them right and so to me the question is
what enables high quality
libraries okay yeah and there's a huge
divide in the world between libraries
who enable highquality
libraries versus um the ones that put
special stuff in the language so
programming languages that enable high
quality libr high quality libraries got
it so so and what I mean by that is
expressive libraries that then feel like
a natural integrated part of the
language itself MH so um an example of
this in Swift is that int and float and
also array and string things like this
these are all part of the library like
int is not hard-coded into
Swift and so what that means is that
because int is just a Library Thing
defined in the standard Library along
with strings and arrays and all the
other things that come with the standard
Library um well hopefully you do like
int but anything that any language
features that you needed to Define int
you can also use in your own types so if
you want to define a uh querian or
something like this right um well it
doesn't come in the standard Library um
there's a very special set of people
that care a lot about this but those
people are also important it's not it's
not about classism right it's not about
the people who care about instant floats
are more important than the people who
care about querian and so to me the
beautiful things about about programming
languages is when you allow those
communities to to build high quality
libraries that feel native that feel
like they're built into the built into
the compiler without having to
be what does it mean for the int to be
part of not hardcoded in so is it like
how so what is an what is an INT okay
int is just a integer in this case it's
like a you know like a 64-bit integer or
something like this but so like the
64-bit is hardcoded or no no none of
that's hardcoded so int int if you go
look at how it's implemented is it's
just a struct and Swift and so it's a
struct and then how do you add two
structs well you define plus and so you
can Define Plus on int well you can
Define Plus on your thing too you can
Define uh int has like an is OD method
or something like that on it and so yeah
you can add methods onto things yeah uh
so you can you can def find operators
like how it behaves yeah that to is
beautiful when there there's something
about the language which enables others
to create libraries which are um not
hacky yeah that feel that feel native
and so one of the best examples of this
is lisp MH right because in lisp all
like all the libraries are basically
part of the language right you write
term rewrite systems and things like
this and so can you as a counter example
provide what makes it difficult to write
a library that's native is it the python
C well so well so one example I'll give
you two examples um Java and C++ or Java
and C um they both allow you to Define
your own types um but int is hard code
in the language okay well why well in in
Java for example coming back to this
whole reference semantic value semantic
thing um int gets passed around by
value yeah but if you if you make if you
make like a pair or something like that
a complex number right it's a it's a
class in Java and now it gets passed
around by reference by pointer and so
now you lose value sematics right you
lost math okay well that's not great
right if if you can do something with in
why can't I do it with my type yeah
right so that's
that's the the negative side of the
thing I find beautiful is when you can
solve that when you can have full
expressivity where where you as a user
of the language have as much or almost
as much power as the people who
implemented all the standard built-in
stuff because what that enables is that
enables truly beautiful libraries you
know it's kind of weird cuz I've gotten
used to that uh that's one I guess other
aspect of program language design you
have to think you know the old uh first
principles thinking like why are we
doing it this way by the way I mean I
remember cuz I was thinking about the
wallers operator and I'll ask you about
it later but it it hit me that like the
equal sign for assignment yeah like why
are we using the equal sign for
assignment and that's not the only
solution right so if you look at Pascal
they use colon equals for assignment and
equals for um for equality and they use
like less than greater than instead of
the not equal thing like there are other
answers here so but like and yeah like
ask you all but how do you then
decide uh to break convention to say you
know what
this everybody's doing it wrong we're
gonna do it right yeah so so it's like
an Roi like return on investment
tradeoff right so if you do something
weird let's just say like not like colon
equal instead of equal for assignment
that would be weird with today's
aesthetic right and so you'd say cool
this is theoretically better but is it
better in which ways like what do I get
out of that do I Define away class of
bugs well one of the class of bugs that
c has is that you can use like you know
if x equals without equals equals f x
equals y yeah right well turns out you
can solve that problem in lots of ways
clang for example GCC all these
compilers will detect that as a as a
likely bug produce a warning do they
yeah I feel like they didn't or clang do
GCC didn't and it's like one of the
important things about programming
language design is like you're literally
creating suffering in the
world okay like like I feel I mean one
way to see it is the bicycle for the
mine but the other way is to like
minimizing suffering well you have to
decide if it's worth it right and so
let's come back to that okay but um but
if you if you look at this and again
this is where there's a lot of detail
that goes into each of these things um
uh equal and C returns a
value y that's messed up that allows you
to say xal yals Z like that works in C
yeah um is it messed up you know most
people think it's messed up by think uh
it it is very by messed up what I mean
is it is very rarely used for good and
it's often used for bugs yeah right and
so that's a good definition of up yeah
you could use you know it's it's a in
hindsight this was not such a great idea
right now one of the things with swift
that is really powerful and one of the
reasons it's actually good um versus it
being full of good ideas is that um when
when we launched Swift one we announced
that it was public people could use it
people could build apps but it was going
to change and break okay when Swift 2
came out we said hey it's open source
and there's this open process which
people can uh help evolve and direct the
language so the community at large like
Swift users can now help shape the
language as it is and what happened is
that part as part of that process is a
lot of really bad mistakes got taken out
so for example Swift used to have the C
style Plus+ and minus minus operators
like what does it mean when you put it
before versus
after right well that got cargo culted
from C into Swift early on what's cargo
culted cargo culted means uh brought
forward without really considering
considering it okay um this is maybe not
the most py term but um have to look it
up an urban dictionary yeah yeah so it
got pulled it got pulled into C without
or it got pulled into Swift without very
good consideration and we went through
this process and one of the first things
got ripped out was plus plus and minus
minus because they lead to confusion
they have very low value over saying you
know X plus equals 1 and X Plus equal 1
is way more clear and so when you're
optimizing for teachability and Clarity
and bugs and this multi-dimensional
space that you're looking at um things
like that really matter and so being uh
first principles on where you're coming
from and what you're trying to achieve
and being anchored on the objective is
really important well let me ask you
about uh the most uh sort of this this
uh this this podcast isn't about
information it's about drama so let me
talk to you about some drama so you
mentioned Pascal and colon equals uh
there's something that's called the
wallrus operator okay and uh python uh
in Python 3.8 added the walus operator
and the reason I think it's
interesting uh it's not just because of
the feature it does it's it has the same
kind of expression feature you can
mention to see that it Returns the value
of the assignment
and maybe you can comment on that in
general but on the other side of it it's
also the thing that that uh toppled the
dictator uh so okay it finally drove
Guido to uh step down from edfl the
toxicity of the community so maybe um
what do you think about the wallus
operator in in Python is there an
equivalent thing in Swift that
really uh stress tested the community
and uh and then on the flip side what do
you think about AGA stepping down over
it yeah if well if like if I look past
the details of the W walrus operator one
of the things that makes it most
polarizing is that it's syntactic
sugar okay what do you mean by syntactic
Sugar it means you can take something
that already exists in the language and
you can express it in a more concise way
so okay I'm going to play Do's advocate
so uh this is great uh is that objective
or subjective statement like can you can
you argue that basically anything is
syntactic sugar or no uh
no you not everything is is syntactic
sugar so for example um the type system
like can you have classes versus uh
versus uh like do you have types or not
right so so one type versus many types
is not something that affects syntactic
sugar and so if you say I want to have
the ability to Define types I have to
have all this like language mechanics to
Define classes and oh now I have to have
inheritance and I have like have all
this stuff that's just making the
language more complicated mhm that's not
that's not about sugaring it um Swift
has sugar so like Swift has this thing
called IFL and it has uh various
operators that used to
conisy uh specific use cases so the
problem with syntactic sugar when you're
talking about hey I have a thing that
takes a lot to write and I have a new
way to write it you have this like
horrible trade-off which becomes almost
completely subjective which is how often
does this happen and does it matter and
one of the things that is true about
human psychology particularly when
you're talking about introducing a new
thing is that uh people over
overestimate the burden of learning
something and so it looks foreign when
you haven't gotten used to it but if it
was there from the beginning of course
it's just part of python like
unquestionably like this is this is just
a thing I know and it's not a new thing
that you're worried about learning it's
just part of part of the deal now with
Guido uh I I don't know Guido
well um yeah have you a pass cross much
yeah I've met him a couple of times but
I don't know Guido well but the the
sense that I got out of that whole
dynamic was that he had put the not just
the decision maker weight on his
shoulders but it was so tied to his
personal identity that um he took it
personally and he felt the need and he
kind of put himself in the situation of
being the person instead of building a
base of support around him I mean he
this is probably not quite literally
true but by too much so there's too much
too much concentrated on him right and
so and that can wear you down well yeah
particularly because people then say
Guido you're a horrible person I hate
this thing blah blah blah blah blah blah
blah and sure it's like you know maybe
1% of the community that's doing that
but Python's got a big community and 1%
of of millions of people is a lot of
hate mail and that just from human
factor will just wear on you what to to
clarify it looked from just what I saw
in the messaging for the let's not look
at the million python users but at the
python core developers it feels like the
majority the big majority on a vote were
opposed to it okay I'm not that close to
it so I don't knowen so so this okay so
the situation is like
literally uh yeah I mean the majority of
the core developers are against so I and
they weren't they weren't even like
against
it it was uh there was a feel well they
were against it but the they against it
wasn't like this is a bad idea they were
more like we don't see why this is a
good idea and what that results in is
there's a stalling feeling like you you
just slow things down now from my
perspective now you could argue this and
I think it's very it's very interesting
if we look at politics today and the way
Congress works it slowed down everything
it's a dampener yeah it's a dampener but
like that's a dangerous thing too
because if it dampens things like you
know dampening results what are you
talking about like it's a low pass
filter but if you need billions of
dollars injected into the economy or
trillions of dollars then suddenly stuff
happens right and so for sure so you're
talking I'm not defending our political
situation just to be clear but you're
talking about like a a global pandemic I
I was hoping we could fix like the
Health Care system and the education
like you know I I'm not I'm not a
politics person I don't I don't I don't
know um when it comes to languages the
community is kind of right in terms of
it's a very high burden to add something
to a language so as soon as you add
something you have a community of people
building on it and you can't remove it
okay and if there's a community of
people that feel really uncomfortable
with it then taking it slow I think is
is is an important thing to do and
there's no rush particularly if with
something that's 25 years old and is
very established and you know it's not
like coming coming into its own um what
about features
so I I think that the issue with with
Guido is that maybe this is a case where
he realized it had outgrown him and it
went from being or the language the
language so python I mean Guido is
amazing but but python isn't about Guido
anymore it's about the users and to a
certain extent the users own it and you
know py Guido spent years of his life a
significant fraction of his career on
Python and from his perspective I
imagine he's like well but this is my
thing I should be be able to do the
thing I think is right but you can also
understand the users where they feel
like you know this is my thing I use
this like and um and I don't know it's
it's a hard it's a hard thing but what
if we could talk about leadership in
this cuz it's so interesting to me I'm
going to I'm going to make I'm going to
wear hopefully somebody makes it if not
I'll make it a w operator shirt because
I think it represents to me maybe it's
my Russian roots or something uh you
know it's the burden of leadership like
I feel like to push
back I feel like progress can only like
most difficult decisions just like you
said there'll be a lot of divis
divisiveness over especially in the
passionate
Community it just feels like leaders
need to take those risky decisions that
that if you like listen that with some
nonzero probability maybe even a high
probability would be the wrong decision
but they have to use their gut and make
that decision well this this this is
like one of the things where you see uh
amazing Founders the founders understand
exactly what's happened and what how the
company got there and are willing to say
to we have been doing thing X the the
last 20 years but today we're going to
do thing why and they make a major pivot
for the whole Company the company lines
up behind them they move and it's the
right thing but then when the founder
dies the successor doesn't always feel
that that um agency to be able to make
those kinds of decisions yeah even
though they're a CEO they could
theoretically do whatever there's two
reasons for that in my opinion or in
many cases it's always different but um
one of which is they weren't there for
all the decisions that were made and so
they don't know the principles in which
those decisions were made and once the
principles change you're you should be
obligated to change what you're doing
and change direction right and so if you
don't know how you got to where you are
it seems like gospel and you know you're
not going to question it you may not
understand that it really is the right
thing to do so you just may not see it
that's so brilliant I never thought of
it that way like it's it's so much
higher burden when as a leader you step
into a thing that's already worked for a
long time yeah yeah well and if you
change it and it doesn't work out now
you're the the person who screwed it up
people always second guess that yeah and
the second thing is that even if you
decide to make a change even if you're
theoretically in charge you're just
you're just a person that thinks they're
in charge meanwhile you have to motivate
the troops you have to explain it to
them in terms of understand you have to
get them to buy into and believe in it
because if they don't then they're not
going to be able to make the turn even
if you tell them you know their bonuses
are going to be curtailed they're just
not going to like buy into it you know
and so there's only so much power you
have as a leader and you have to
understand what that what those
limitations are are you still bdfl
you've been bdfl of some stuff uh you're
very heavy on the
be the benevolent uh benevolent dictated
for Life uh I guess lvm you're still so
I still lead the lvm world uh I mean
what's the role of uh so then on Swift
you said that there's a group of people
yeah so if you contrast python with
Swift right one of the reasons so
everybody on the core team takes the
role really seriously and I think we all
really care about where Swift goes but
you're almost delegating the final
decision- making to the wisdom of the
group and so it doesn't become
personal and also when you're talking
with the community so yeah some people
are very annoyed at certain decisions
get made um there's a certain faith in
the process because it's a very
transparent process and when a decision
gets made a full rationale is provided
things like this these are almost
defense mechanisms to help both guide
future discussions and provide case law
kind like Supreme Court does about this
decision was made for this reason and
here's the rationale and what we want to
see more of or less of um but it's a way
to provide a defense mechanism so that
when somebody's griping about it they're
not saying that person did the wrong
thing they're saying well this this
thing sucks
and and later they move on and they they
get over it yeah the analogy of the
Supreme Court I think is really is
really good but then okay not to get
person on the Swift team but like is
there is there div like it just seems
like it's impossible for their for
division not to emerge well each each of
the humans on the the Swift core team
for example are different and the
membership of the Swift core team
changes slowly over time which is I
think a healthy thing and so each of
these different humans have different
opinions trust me it's not it's not a
sing singular consciousness of by any
stretch of the imagination you've got
three major organizations including
Apple Google and sci-fi all kind of
working together and um it's a small
group of people but you need High trust
you need again it comes back to the
principles of what you're trying to
achieve and understanding you know what
what you're optimizing for and I think
that starting with strong principles and
working towards decisions is always a
good way to both make wise decisions in
general but then be able to communicate
them to people so that they can buy into
them and that that is hard and so you
mentioned lvm lvm is uh going to be 20
years old uh this December so it's it's
showing its own age you have like like a
like a like a dragon cake plant or you
have a no we should definitely do that
yeah if we can have a
uh pandemic cake pandemic cake everybody
gets a slice of cake and it gets you
know sent through email
um but the uh uh but lvm has had tons of
its own challenges over time too right
and one of the challenges that um the
lvm community has in my opinion is that
it has a whole bunch of people that um
have been working at lvm for 10 years
right because this happen some somehow
and lvm has always been one way but it
needs to be a different way right and
they've worked on it for like 10 years
is a long time to work on something and
you know you you suddenly can't see the
faults in the thing that you're working
on and lvm has lots of problems and we
need to address them and we need to make
it better and if we don't make it better
then somebody else will come up with a
better idea right and so it's just kind
of of that age where the community is
like in danger of getting too calcified
and um and so I'm happy to see new
projects joining and new things mixing
it up you know Fortran is now a new a
new thing in the Elum Community which is
hilarious and good
I've been trying to find uh on this
little tangent find people who program
in Cobalt or Fortran Fortran especially
to talk to they're hard to find yeah
yeah look to the uh scientific Community
they still use forun quite a bit
interesting thing you kind of mentioned
with lvm or just in general that if
something evolve you're not able to see
the faults so do you uh fall in love
with the thing over time or do you start
hating everything about the thing over
time
well so so my my my personal Folly is
that um I see maybe not all but many of
the faults and they gr on me and I don't
have time to go fix them yeah and they
get magnified over time well and they
may not get magnified but they never get
fixed it's like sand underneath you you
know it's just like raiding against you
and it's like s underneath your
fingernails or something it's just like
you know it's there you can't get rid of
it um and so the the problem is that if
other people don't see it right nobody
ever get like I can't go I don't have
time to go write the code and fix it
anymore but then uh people are resistant
to change and so you say hey we should
go fix this thing they're like oh yeah
that sounds risky well is it the right
thing or not are the challenges uh the
group dynamics or is it also just
technical I mean some of these features
like yeah I think uh as an observer is
almost like a fan in in the uh you know
as a spectator of the whole thing it I
don't often think about you know some
things might actually be technically
difficult to implement an example of
this is we we built this new compiler
framework called ml yes ml is this a
whole new framework it's not many people
think it's about machine learning the ml
stands for multi-level because compiler
people can't name things very well I
guess can we can we dig into what ml IR
is yeah so when you look at compilers
compilers have historically been
solutions for a given space so lvm is a
it's really good for dealing CPUs let's
just say at a high level you look at um
Java Java has a jvm the jvm is very good
for garbage collected languages that
need Dynamic compilation and it's very
optimized for specific space and so
hotspot is one of the compilers that
gets used in that space and that
compiler is really good at that kind of
stuff um usually when you build these
domain specific compilers you end up
building whole thing from scratch for
each
domain uh what's a domain so what what
we what's this what's the scope of a
domain Al so here I would say like if
you look at Swift there's several
different parts to the Swift compiler um
one of which is covered by um the LM
part of it there's also a highle piece
that's specific to Swift and there's a
huge amount of redundancy between those
two different infrastructures and a lot
of re reimplemented stuff that is
similar but different what is llvm
Define lvm is effectively an
infrastructure so you can mix and match
it in different ways it's built
libraries you can use it for different
things but it's really good at CPUs and
gpus CPUs and like the tip of the
iceberg on gpus it's not really great at
gpus okay um but it turns out languages
that that then use it to talk to CPUs it
um and so it turns out there's a lot of
Hardware out there that is custom
accelerators so machine learning for
example there are a lot of uh Matrix
multiply accelerators and things like
this there there's a whole world of
Hardware synthesis so we're using ml to
build circuits okay and so you're
compiling for a domain of
transistors and so what ml does is it
provides a tremendous amount of compiler
infrastructure that allows you to build
these domain specific compilers in a
much faster way and have the result be
good if we're if we're thinking about
the future now we're talking about like
as6 like so anything yeah yeah so if we
project into the future it's very
possible that the number of these kinds
of as6 very
specific um infrastructure thing
architecture things uh like multiplies
exponentially I hope so yeah so that's
ml so what ml what ml does is it allows
you to build these compilers very
efficiently right now one of the things
that coming back to the lvm thing and
then we'll go to Hardware is um lvm is
is a specific compiler for specific
domain mlr is now this very general very
flexible thing that can solve lots of
different kinds of problems so lvm is a
subset of what ml does so m is I mean
it's an ambitious project then yeah it's
a very ambitious project yeah and so to
make it even more confusing ml has
joined the lvm umbrella project so it's
part of the lvm family right um but
where this comes full circle is now
folks that work on the lvm part the
classic part that's 20 years old um
aren't aware of all the cool new things
that have been done in the new the new
thing that you know mlr was built by me
and many other people that knew a lot
about lvm and so we fixed a lot of the
mistakes that lived in
LV so now you have this community
Dynamic where it's like well there's
this new thing but it's not familiar
nobody knows it it feels like it's new
and so let's not trust it and so it's
just really interesting to see the
cultural social Dynamic that comes out
of that and and you know I think it's
super healthy because we're seeing the
ideas percolate and we're seeing the
technology diffusion happen as people
get more comfortable with it they start
to understand things in their own terms
and this just gets to the it takes a
while for ideas to propagate even though
um they may be very different than what
people are used to so maybe let's talk
about that a little bit the world of
Asic and well actually you're uh you're
you have a new role at
sci-fi what's that place about what is
the vision sure uh for their vision for
I would say the future of computing yeah
so I lead the engineering and product
teams at SciFi
sci5 is a company who's was founded with
this architecture called risk 5 risk 5
is a new instruction set instruction
sets are the things inside of your
computer that tell how to run things um
x86 from Intel and arm from the arm
company and things like this or other
instruction sets I've talked to sorry
interrupt I've talked to Dave Patterson
who's super excited about risk 5 Dave
Dave is awesome he's brilliant yeah yeah
the uh risk five is distinguished by not
being proprietary MH and so
xa6 can only be made by Intel and AMD
arm can only be made by arm they sell
licenses to build arm ships to other
companies things like this myips is
another instruction set that is owned by
the myips company now wave and it gets
licensed out things like that um and so
RIS 5 is an open standard that anybody
can build chips for and so SciFi was
founded by three of the founders of RIS
5 that designed and built it in Berkeley
working with Dave um and so that was the
The Genesis of the company scii today
has some of the world's best r five
cores and we're selling them and that's
really great they're going into tons of
products it's very exciting um so
they're taking this uh thing that's open
source and just being trying to be or
are the best in the world at building
these things yeah so here it's the
specifications open source it's like
saying tcpip is an open standard or C Is
An Open standard but then you have to
build an implementation of the standard
and so sci5 on the one hand pushes
forward and defined and pushes forward
the standard on the other hand we have
implementations that are best in class
for different points in the space
depending on if you want a really tiny
CPU or if you want a really big beefy
one that that uh is faster but it uses
more area and things like this what
about the actual manufacturer chip so
like what where does that all fit I'm
going to ask a bunch of dumb questions
that's okay this is how we learn right
uh and so uh what the the way this works
is that there's generally a separation
of the people who design the circuits
than the people who manufacture them and
so that you'll hear about Fabs like tsmc
and Samsung and things like this that
actually produce the chips but they take
a design coming in and that design
specifies how um how the you know you
turn uh code for the chip into uh little
rectangles that then use Photo
lithography to make uh mask sets and
then burn transistors onto a chip or
onto a onto silicon rather well so and
we're talking about Mass manufacturing
so yeah they're talking about making
hundreds of millions of parts and things
like that yeah and so the the Fab
handles the volume production things
like that but um when you look at this
problem um the interesting thing about
the space when you look at it is that um
these the steps that you go from
designing a chip and writing the quote
unquote code for it and things like
verog and languages like that down to
what you hand off to the Fab is a really
well studied really old problem mhm
um tons of people have worked on it lots
of smart people have built systems and
tools um these tools then have generally
gone through Acquisitions and so they've
ended up at three different major
companies that build and sell these
tools they're called Eda tools like for
electronic design automation um the
problem with this is you have huge
amounts of fragmentation you have loose
standards um and the tools don't really
work together so you have tons of duct
tape and you have tons of uh lost
productivity now these are uh these are
tools for Designing so the risk five is
a instruction like what is risk five
like how deep does it go how how how
much does it touch the hardware how much
does it Define how much of the hardware
is yeah so RIS RIS five is all about um
given a CPU so the the the processor and
your computer how does the the compiler
like Swift compiler the C compiler
things like this how does it make it
work so it's what is the assembly code
and so you write risk five assembly
instead of xa6 assembly for example but
it's a set of instructions as opposed to
set of instructions yeah why why do you
say it tells you how the compiler works
the sorry it's what the compiler talks
to okay yeah and then uh the tooling you
mentioned the disperate tools are for
what for for when you're building a
specific chip so RIS five in Hardware in
Hardware yeah so so RIS five you can buy
rist 5 Core from scif and say Hey I want
to have a certain number of run a
certain number of gigahertz I want it to
be this big I want to be have these
features I want to have um like I want
floating point or not for
example um and then what you get is you
get a description of a CPU with those
characteristics now if you want to make
a chip you want to build like an iPhone
chip or something like that right you
have to take both the CPU but then you
have to talk to memory you have to have
timers iOS a GPU other components and so
you need to pull all those things
together into what's called an Asic an
application specific grade circuit so a
custom chip and then you take that
design and then you have to transform it
into something that the Fabs like tsmc
for example know how to turn take to
production got it so but yeah okay and
and so that process I
will I can't help but see it is is a big
compiler okay it's a whole bunch of
compilers written without thinking about
it through that lens isn't isn't the
universe a compiler in that like comp
compilers do two things they represent
things and transform them yeah and so
there's a lot of things that end up
being compilers but this is this is a
space where we're talking about design
and usability and the way you think
about things the way things composed
correctly it matters a lot and so sci-fi
is investing a lot into that space and
we think that there's a lot lot of
benefit that can be made by allowing
people to design chips faster get them
to Market quicker and um scale out
because um you know it the alleged more
end of Mor's law uh you've got this
problem of uh you're not getting free
performance just by waiting another year
for a faster CPU and so um you have to
find performance in other ways and one
of the ways to do that is with custom
accelerators and other things and
hardware and and so well we'll talk a
little
about uh a little more about as6 but um
do you see that a lot of people a lot of
companies will try to have a like
different sets of requirements that this
whole process to go for so like like
almost different car companies might use
different uh and like different uh PC
manufacturers like so is this like is
risk
5 um in this whole process is it
potentially the future of all Computing
devices yeah I think that so if you look
at risk 5 and step back from the Silicon
side of things RIS 5 is an open standard
and one of the things that has happened
over the course of decades if you look
over the long Arc of computing somehow
became decades old yeah is that you have
uh companies that come and go and you
have instruction sets that come and go
like one example of this out of many is
uh uh sun with spark yeah Sun one away
spark still lives on it Fujitsu but we
have uh HP had this instruction set
called PA risk so P risk was its big
server business and had tons of
customers they decided to move to this
called itanium from Intel yeah this
didn't work out so well yeah right and
so you have this issue of you're making
many billion doll Investments on
instruction sets that are owned by a
company and even companies as big as
Intel don't always execute as well as
they could they have their own issues um
HP for example decided that it wasn't in
their best interest to continue
investing in the space because it was
very expensive and so they make
technology decisions or they make their
own business decisions and this means
that a customer what do you do you've
sunk all this time all this engineering
all the software work all these you've
built other products around them and now
you're
stuck right what risk 5 does is it
provides you more optionality in the
space because if you buy uh an
implementation of RIS five from SciFi
and you should they're the best ones
yeah um uh but if something bad happens
to SciFi in 20 years right well great
you can turn around and buy r five core
from somebody else and there's an
ecosystem of people people that are all
making different risk five cores with
different trade-offs which means that if
you have more than one requirement if
you have a family of products you can
probably find something in the rist five
space that fits your needs whereas with
if you're talking about xa6 for example
it's Intel's only going to bother to
make certain classes of
devices right I see so uh maybe a weird
question but like if
SciFi is uh like infinitely successful
in the next 20 30 years what does the
world look like so like how does the
world of computing change so too much
diversity in Hardware instruction sets I
think is bad like we have a lot of
people that are using um lots of
different instruction sets particularly
in the embedded the like very tiny
microcontroller space the thing in your
toaster um that uh that are just weird
and different for historical reasons and
so the compilers and the tool chains and
the languages on top of them
uh aren't there right and so the
Developers for that software have to use
really weird tools because the ecosystem
that supports is not big enough so I
expect that will change right people
will have better tools and better
languages better features everywhere
that then can service many different
points in the space um and I think RIS 5
will progressively um eat more of the
ecosystem because it can scale up it can
scale down sideways left right it's very
flexible and very well considered welld
designed and instruction set um I think
when you look at sci-fi tackling silicon
and how people build chips which is a
very different space um that's where you
say I think we'll see a lot more custom
chips and that means that you get much
more battery life you get better better
tuned solutions for your iot
thingy you get you get people that move
faster you get the ability to have
faster time to market for example so how
many custom so first of all on iot of
things do you see the number of smart
toasters increasing exponentially
so
uh and and if you do like how much
customization per toaster is there do
all toasters in the world run the same
uh silicon like the same design or is it
different companies have different
design like how how much customization
is possible here well a lot of it comes
down to
cost right and so the way that chips
work is you end up paying by the one one
of the factors is the the size of the
Chip And so what ends up happening just
from an economic perspective is there's
only so many chips that get made in any
year of a given design and so often what
customers end up having to do is they
end up having to pick up a chip that
exists that was built for somebody else
so they can then ship their product and
the reason for that is they don't have
the volume of the iPhone they can't
afford to build a custom chip however
what that means is they're now buying an
off-the-shelf chip that isn't really
good that isn't a perfect fit for their
needs and so they're paying a lot of
money for it because they're buying
silicon that they're not
using well if you now reduce the cost of
Designing the chip now you get a lot
more chips and the more you reduce it
the the easier it is to design chips um
The More The World Keeps evolving and we
get more AI accelerators we get more
other things we get more uh standards to
talk to we get 6G right you get you get
you get changes in the world that you
want to be able to talk to these
different things there's more diversity
in the cross product of features that
people want and um that drives
differentiated chips in different in
another Direction and so nobody really
knows what the future looks like but um
but I think that there's a lot of
silicon in the future speaking of the
future uh you said Mo's law allegedly is
dead so do you think do you agree with
uh uh Dave Patterson and and many folks
that Mo's law is dead or do you agree
with Jim Keller who says
uh who's uh standing at the Helm of the
pirate ship saying it's uh still alive
it's still alive yeah also I agree with
what they're saying and different people
are interpreting the anor's law in
different ways yeah so Jim would say you
know there's another thousand X left in
physics and we can we can continue to
squeeze the stone and make it faster and
smaller and smaller geometries and all
that kind of stuff uh he's right so Jim
Jim is is absolutely right that there's
a ton of ton of progress left and we're
not at the limit of physics yet um uh
that's not really what mors law is
though if you look at what mors law is
is that it's a very simple uh evaluation
of okay well you look at the cost per um
I think it was cost per area and the
most economic point in that space and if
you go look at the the the now quite old
paper that describes this um mors law
has a specific economic
aspect to it and I think this is
something that Dave and others often
point out and so on a technicality
that's right um I look at it from so I
can acknowledge both of those viewpoints
they're both right they're both right
I'll give you a third wrong yeah
Viewpoint that may be right in its own
way which is um single threaded
performance doesn't improve like it used
to and it used to be back when you got a
uh you know a pennium 66 or something
and the year before you had a pennium
33 and now it's twice as fast MH right
well it was twice as fast at doing
exactly the same thing okay like
literally the same program ran twice as
fast you just wrote a check okay and
waited a year year and a half well so
that's what a lot of people think about
Moors law and I think that is dead and
so what we're seeing instead is we're
pushing we're pushing people to write
software in different ways and so we're
pushing people to write Cuda so they can
get GPU compute and the the thousands of
cores on GPU we're talking about C
programmers having to use P threads
because they now have you know 100 100
threads or 50 cores in a machine or
something like that um you're now
talking about machine learning
accelerators they're now domain specific
and when you look at these kinds of use
cases you can still get performance um
and Jim will come up with cool things
that uh utilize the Silicon in new ways
for sure but you're also going to change
the programming model right and now when
you start talking about changing the
programming model that's when you come
back to languages and things like this
too because often what you see is um
like you take the C programming language
right the C programming language is
designed for
CPUs and so if you want to talk to a GPU
now you're talking to its cousin Cuda
okay Cuda is a different thing with a
different set of tools a different world
a different way of thinking and we don't
have one world that scales and I think
that we can get there we can have one
world that scales in a much better way
on a small tangent then I think most
programming languages are designed for
CPUs for single core even just in their
Spirit even if they allow for
paralyzation so what does it look like
for programming language to have um
paralyzation or massive parallelization
as it's like first principle so the
canonical example of this is the
hardware design world so verog vhdl
these kinds of languages they're what's
called a uh highle synthesis language
this is the thing people design chips in
and when you're designing a chip it's
kind of like a brain where you have
infinite
parallelism like you've got you're
you're you're like laying down
transistors transistors are always
running okay yeah and so you're not
saying run run this transistor then this
transistor than this transistor it's
like your brain like your neurons are
always just doing something they're not
clocked right they're they're just
they're just doing they're they're doing
their thing and so when you design a
chip or when you design a CPU when you
design a CPU when you design when you're
laying down the transistors uh similarly
you're talking about well okay well how
do these things communicate and so these
languages exist verog is um a kind of
mixed example of that none of these
languages are really great either very
low level yeah yeah they're very low
level and abstraction is necessary here
and there's different different
approaches at that and it's a it's
itself a very complicated world but um
but it's implicitly parallel and so
having that as a as the domain that you
uh program towards makes it so that by
default you get parallel systems if you
look at Cuda Cuda is a point halfway in
the space where in Cuda when you write a
Cuda kernel for your GPU it feels like
you're writing a scaler program so
you're like you have ifs you have for
Loops stuff like this you're just
writing normal normal code but what
happens outside of that in your driver
is that it actually is running you on
like a thousand things at once right and
so it's it's parallel but it has pulled
it out of the programming model and so
so now you as a programmer are working
at a in a simpler world and it's solved
that for
you right how do you take the language
like
Swift um you know if we we think about
gpus but also ASX maybe if we can dance
back and forth between hardware and
software uh is you know how do you
design for these features to be able to
program make it a first class citizen to
be able to do like Swift for tensor flow
to be able to do machine learning on
current Hardware but also future
Hardware like uh dpus and all kinds of
as6 that I'm sure will be popping up
more yeah well so so a lot of this comes
down to this whole idea of having the
nuts and bolts underneath the covers
that work really well so you need if
you're talking to tpus you need you know
ml or xlaa or one of these compilers
that talks to tpus to build on top of
okay and if you're talking to circuits
you need to figure out how to lay down
the transistors and how to organize it
and how to set up clocking and like all
the domain problems that you get with uh
circuits then you have to decide how to
explain it to a human what is the
UI right and if if you do it right
that's a library problem not a language
problem and that works if you have a
library or a language which allows your
library to write things that feel native
in The Language by implementing
libraries because then you can innovate
in programming models without having to
change your syntax again and like have
to invent new code formatting tools and
like all the other things that languages
come with and this this gets really
interesting and so um if you look at the
space the interesting thing once you
separate out syntax becomes what is that
programming model and so do you want the
Cuda style I write one program and it
runs many places the um do you want the
implicitly parallel model how do you
reason about that how do you give
developers you know chip Architects the
the ability to express their intent and
that comes into this whole design
question of how do you detect bugs
quickly so you don't have to tape out a
chip to find out it's wrong ideally
right how do you and and you know this
is a spectrum how do you make it so that
people feel productive so their
turnaround time is very quick all these
things are really hard problems and um
in this world I I think that not a lot
of effort has been put into that design
problem and thinking about the layering
in other pieces well you've uh on the
topic of concurrency you've written the
Swift concurrency Manifesto I think it's
it's kind of interesting anything that
uh has the word manifesto in is very
interesting can you summarize the key
ideas of U each of the five parts you
written about so what is a Manifesto yes
how about we start there uh so in the
Swift Community we have this um problem
which is on the one hand you want to
have relatively small proposals that you
can kind of fit in your head you can
understand the details at a very fine
grain level that move the world forward
but then you also have these big arcs
okay and often when you're working on
something that is a big Arc but you're
tackling in small pieces you have this
question of how do I know I'm not doing
a random walk where are we going like
how does this add up furthermore when
you start that first the first small
step what terminology do you use how do
we think about it what is better and
worse in the space what are the
principles what are we trying to achieve
and so what a Manifesto in the Swift
Community does is it starts to say hey
well let's step back from the details of
everything let's paint a broad picture
to talk about how what we're trying to
achieve let's give an example design
Point let's try to paint the big picture
so that then we can zero in on the
individual steps and make sure that
we're making good progress and so the
Swift concurrency Manifesto is something
I wrote three years ago it's been a
while maybe maybe more um trying to do
that for for Swift and
concurrency and it starts with some
fairly uh simple things like making the
observation that
when you have multiple different
computers or multiple different threads
that are communicating it's best for
them to be
asynchronous right and so you need
things to be able to run separately and
then communicate with each other and
this means asynchrony and this means
that uh you need a way to modeling
asynchronous communication uh many
languages have features like this uh
asyn a weight is a popular one and so
that's what I think is very likely in
Swift um but as you start building this
Tower of abstractions it's not just
about how do you write this you then
reach into the how do you get memory
safety because you want correctness you
want debuggability and Sanity for
developers and how do you get uh that
memory safety into um into the language
so if you take a language like go or uh
C or any of these languages you get
what's called a race condition when two
different threads or go routines or
whatever touch the same point in memory
right this is a huge like maddening
problem to debug because uh it's not
reproducible generally and so there's
tools there's a whole ecosystem of
solutions that built up around this but
it's it's a huge problem when you're
writing concurrent code and so with
Swift uh this whole value sematics thing
is really powerful there because it
turns out that math and copies actually
work even in concurrent worlds and so um
you get a lot of safety just out of the
box but there are also some hard
problems and it talks about some of that
um when you start building up to the
next level up and you start talking
Beyond memory safety you have to talk
about what is a programmer model how
does a human think about this so a
developer that's trying to build a
program think about this and it proposes
a really old model with a new spin
called actors actors are about saying we
have islands of single threaded
logically so you write something that
feels like it's one programming one
program running in a unit and then it
communicates asynchronously with other
other
things and so making that expressive and
natural feel good be the first thing you
reach for and being safe by default is a
big big part of the design of that
proposal when you start going beyond
that now you start to say cool well
these things that communicate
asynchronously they don't have to share
memory well if they don't have to share
memory and they're sending messages to
each other why do they have to be in the
same
process these things should be able to
be in different processes on your
machine and why just processes well why
not different
machines and so now you have a very nice
gradual transition towards distributed
programming and of course when you start
talking about the the big the big future
the the manifesto doesn't go into it but
uh accelerators are asyn things you talk
to asynchronously by sending messages to
them and how do you program those well
that that gets very interesting um
that's not that's not in the proposal so
but and uh how much do you want to make
that explicit like the control of that
whole process explicit to the programmer
yeah good question so when when you're
designing any of these kinds of features
or language features or even libraries
you have this really hard trade-off that
you have to make which is how much is it
magic or how much is it in the human's
control how much can they predict and
control it what do you do when the
default case is the wrong
case okay and so when you're designing a
system um uh I won't name names but
there there are systems where um you
it's really easy to get started and then
you you jump so let's pick like logo
okay so something like this so it's
really easy get start it's really
designed for uh teaching kids but as you
get into it you hit a ceiling yeah and
then you can't go any higher and then
what do you do well you have to go
switch to a different world and rewrite
all your code and this logo is a silly
example here this exists in many other
languages uh with python you would say
uh uh like concurrency right so python
has the global interpreter lock so
threading is challenging in Python and
so if you if you start writing a large
scale application in Python and then you
need concurrency you're kind of stuck
with the series of bad trade-offs right
um uh there's other ways to go where you
say like voice all the all the
complexity on the user all at once right
and that's also bad in a different way
and so what what I what I prefer is
building a simple model that you can
explain that then has an escape hatch so
you get in you have guard rails you uh
memory safety works like this in Swift
where you can start with you like by
default if you use all the standard
things it's memory safe you're not going
to shoot your foot off but if you want
to get a uh a c-level pointer to
something you can explicitly do that but
by default it's uh there's guard rails
there's guard rails okay so but like you
know uh whose job is it to figure out
which part of the code is
paralyzable um so in the case of the
proposal it is the human's job so they
decide how to architect their
application
and then uh the runtime in the compiler
is very
predictable and so this this is in
contrast to um like there's a long body
of work including on Fortran for auto
parallelizing
compilers and um this is an example of a
bad thing and my so as a compiler person
I can rag on compiler people um often
compiler people will say cool since I
can't change the code I'm going to write
my compiler that then takes this
unmodified code and makes go way faster
on this machine MH okay application
develop and so it does pattern matching
it does like really deep analysis
compiler people are really smart and so
they like want to like do something
really clever and tricky and you get
like 10x speed up by taking like an
array of structures and turn it into a
structure of arrays or something because
it's so much better for memory like
there's bod like tons of Tricks yeah um
they love optimization yeah you love
optimization everyone loves optimization
everyone loves it well and it's it's
just this promise of build with my
compiler and your thing goes fast yeah
right but here here's the problem Lex
you write you write program M you run it
with my compiler it goes fast you're
very happy wow it's so much faster than
the other compiler yeah then you go and
you add a feature to your program or you
refactor some code and suddenly you got
a 10x loss in performance well why what
just happened there what just happened
there is you the theistic the the the
pattern match and the compiler whatever
analysis it was doing just got defeated
because you didn't inline a function or
or or something right as a user you
don't know you don't want to know that
was the whole point you don't want to
know how the compiler works you don't
want to know how how the memory
hierarchy works you don't want to know
how it got parallelized across all these
things you wanted that abstractor away
from you but then the magic is lost as
soon as you did something and you fall
off a performance cliff and now you're
in this funny position where what do I
do I don't change my code I don't fix
that bug it cost 10 10x performance now
what do I do well this is the problem
with unpredictable performance right if
if you care about performance
predictability is a very important thing
and so um and so what the what the
proposal does is it provides a
architecture patterns for being able to
lay out your code gives you full control
over that makes it really simple so you
can explain it and then um and then if
you want to scale out in different ways
you have full control over that so in
your sense the intuition is for a
compiler too hard to do automated
parallelization like you know cuz the
compilers do stuff automatically that's
incredibly impressive for other things
right but for
parallelization we're not even we're not
close to there well it it depends on the
programming model so compile there's
many different kinds of compilers and so
if you talk about like a c compiler or a
swift compiler or something like that
where you're writing imperative code
parallelizing that and reasoning about
all the pointers and stuff like that is
very is a very difficult problem now if
you switch domains so there's this cool
thing called machine
learning right so the machine the
machine learning nerds among other
endearing things like you know solving
cat detectors and other things like that
um have done this amazing breakthrough
of producing a programming model
operations that you compose together mhm
that has raised level of abstraction
high enough that suddenly you can have
autop paralyzing compilers you can write
and model using tensor flow and have it
run on 1,24 nodes of a TPU yeah that's
true I didn't even think about like you
know CU there's so much flexibility in
the design of architectures that
ultimately boil down to a graph that's
paralyzable for you par for you and and
if you think about it that's pretty
cool and you think about batching for
example as a way of being able to
exploit more parallelism yeah like
that's a very simple thing that now is
very powerful that didn't come out of
the programming language nerds right
those people like that came out of
people that are just looking to solve a
problem and use a few gpus and
organically developed by the community
of people focusing on machine learning
and it's an incredibly power powerful
abstraction layer that enables the
compiler people to go and exploit that
and you can drive supercomputers from
python that's that's pretty cool that's
amazing so just to pause on that I cuz
I'm not sufficiently low level I forget
to admire the beauty and power of that
but um maybe just to linger on it like
what what does it take to run a neural
network fast like how hard is that
compilation it's really hard um so we
just skipped you said like it's amazing
that that's a thing but yeah how hard is
that of a thing it's it's hard and I I
would say that
not all the systems are really great
including the ones I help build so
there's a lot of work left to be done
there is it the compiler nerds working
on that or is it a whole new group of
people well it's it's a full stack
problem including compiler people um in
including apis so like Caris and the the
the the module API and pytorch and Jack
and there's a bunch of people pushing on
all the different parts of these things
because when you look at it is it's both
how do I express the computation do I
stack up layers well cool like setting
up a linear sequence of layers is great
for the simple case but how do I do the
hard case how do I do reinforcement
learning well now I need to integrate my
application logic in this right then
it's you know the next level down of how
do you represent that for the runtime
how do you get Hardware
abstraction and then you get to the next
level down of saying like forget about
abstraction how do I get the Peak
Performance out of my TPU or my iPhone
accelerator or whatever right all these
different things and how and so this is
a layered problem with a lot of really
interesting uh design and work going on
in the space and a lot of really smart
people working on it uh machine learning
is a very well-funded area of investment
right now and so there's a lot of
progress being made so how much
Innovation is there on the lower level
so closer to the to the as6 so
redesigning the hardware or redesigning
concurrently compilers with that
Hardware is that like if you were to
predict the
biggest uh you know the equivalent of
Moors law improvements in the inference
in the training of your own networks and
just all of that where is that going to
come from you think sure you get
scalability have different things and so
you get um you know Jim Keller shrinking
process technology you get 3 nanometer
instead of five or seven or 10 or 28 or
whatever um and so that that marches
forward that provides improvements you
get uh architectural level performance
and so the you know a TPU with a matrix
multiply unit and a systolic array is
much more efficient than having a scaler
core doing multiplies and adds and
things like that you then get um uh uh
system level improvements so how you
talk to memory how you talk across a
cluster of machines how you scale out
how you have fast interconnects between
machines you then get system level
programming models so now that you have
all this Hardware how to utilize it you
then have algorithmic breakthroughs
where you say hey wow cool instead of
training in uh you know resonant 50 and
uh a week I'm now training it in you
know 25 seconds yeah and Comin It's a
combination of uh you know new new
optimizers and new new new just training
regimens and different different
approaches to train and and all of these
things come together to to push the
world forward that that was a a
beautiful exposition of but if you were
to uh Force to bet all your money on one
of these would you why do we have to
that's unfortunately we have people
working on all this it's an exciting
time right so I mean you know open the
eye did this little paper showing the
algorithmic Improvement you can get has
been you know improving
exponentially uh I haven't quite seen
the same kind of analysis on other
layers of the stack I'm sure it's also
improving significantly I just it's a
it's a nice intuition Builder I mean
there's a reason why Moore's Law that's
the beauty of Mo's law is somebody
writes a paper that makes a ridiculous
prediction yeah and
it you know becomes reality in a sense
there's there's something about these
narratives when you uh uh when Chris L
on a silly little podcast makes bets all
his money on a particular thing somehow
it can have a ripple effect of actually
becoming real that's an interesting
aspect of it cuz like it might have been
uh you know we focus with Mor's law most
of the Computing industry really really
focused on the hardware
I mean software Innovation I don't know
how much software Innovation there was
in terms of Intel giveth Bill takes away
right yeah I mean compiler has improved
significantly also right well not not
really so actually I mean so I'm joking
about how uh software's gotten slower
pretty much as fast as Hardware got
better at least through the 90s um
there's another joke another law in
compilers which is called uh I think
it's called probstein law which is uh
compilers double the performance of any
given code every 18
years so they move slowly yeah well so
well well yeah it's exponential also
yeah you're making progress but but
there again it's not about um the the
power of compilers is not just about how
do you make the same thing go faster
it's how do you unlock the new hardware
right a new chip came out how do you
utilize it you say oh the programming
model how do we make people more
productive how do we how do we uh like
have better error messages even such
mundane things like how do I generate a
very specific error message about your
code actually makes people happy because
then they know how to fix it right it
comes back to how do you help people get
their job done yeah and yeah and then in
this world of exponentially increasing
smart toasters how do you uh expand
Computing to uh to all all these kinds
of devices do you see this world where
just everything's a Computing surface
you see that possibility just everything
a computer yeah I don't see any reason
that that couldn't be achieved it turns
out that sand goes into glass and glass
is pretty useful too and you know like
why not why not so uh very important
question then if
um if we're living in a simulation and
the simulation is running a computer
like what what's the architecture of
that computer do you
think so you're saying is it is it a
Quantum system is it a yeah like this
whole Quantum discussion is it needed or
can can we run it on a on a you know
with a risk 5
architecture uh a bunch of CPUs I think
it comes down to the right tool for the
job okay and so and what's the compiler
yeah exactly that's that's my question
how do I get that job be the universe
compiler um uh and so there as far as we
know Quantum Quantum Quantum systems are
the bottom of the T pile of
so far yeah and so we don't know
efficient ways to implement Quantum
systems without using quantum
computers yeah and that's totally
outside of everything we've talked about
Quantum but who runs that quantum
computer yeah right so if it if it if we
really are living in a
simulation then is it bigger quantum
computers is it different ones like how
how does that work out how does that
scale well it's it's the same size it's
the same size but then but then the
thought of the simulation is you don't
have to run the whole thing that you
know we humans are cognitively very
limited checkpoints checkpoints yeah and
uh and if we the point at which we human
so you basically do minimal amount of uh
what is it uh Swift does um on right
copy copy on right yeah so you only you
only adjust the simulation par parallel
universe theories right and so and so
every time a a decision is made somebody
opens the Shor in your box then there's
a fork this could happen and then uh
thank you for uh for considering the
possibility but yeah so it may not
require you know the entirety of the
universe to simulate it but it's um
interesting to think about uh as we
create this this higher and higher
Fidelity systems but I do want to ask on
the on the quantum computer side because
everything we've talked about with uh
with you work with sci-fi with every
with compilers none of that includes
quantum computers right that's true so
have you ever thought about uh what a
you know this whole serious engineering
work of quantum computers looks like of
compilers of architectures all of that
kind of stuff so I've looked at a little
bit I know almost nothing about it which
means that at some point I will have to
find an excuse to get involved because
that's how do you think do you think
that's a thing to be like is was your
little senses of the timing of when to
be involved is it not yet well so so the
thing I do really well is I jump into
messy systems and figure out how to make
them figure out what the truth in the
situation is try to figure out what um
what the unifying theory is how to like
Factor the complexity how to find a
beautiful answer to a problem that um
has been well studied and lots of people
have bashed their heads against it I
don't know that quantum computers are
mature enough and accessible enough to
be um figured out yet right and um the
uh I think the open question with
quantum computers is is there a useful
problem that gets solved with a quantum
computer that makes it worth the
economic cost of like having one of
these things and having having Legions
of people that that that uh set it up
you go back to the 50s right and there's
the projections of the world can will
only need seven seven computers right
well and part of that was that people
hadn't figured out what they're useful
for what are the algorithm we want to
run what are the problems to get solved
and this comes back to how do we make
the world better either economically or
making somebody's life better or like
solving a problem that wasn't solved
before things like this and um I think
that just we're a little bit too early
in that development cycle because it's
still like literally a science project
not any negative connotation right it's
literally a science project and um the
progress there is amazing and so I don't
know if it's 10 years away if it's 2
years away exactly where that
breakthrough happens but um you look at
uh machine learning it we went through a
few winners um before the Alex net
transition and then suddenly it had its
breakout moment and that was the
Catalyst that then drove the talent
flocking into it that's what drove the
economic applications of it that's what
drove the um the technology to go faster
because you now have more Minds thrown
at the problem this is what caused uh
like a serious knee and uh deep learning
and the algorithms that we're using and
um and so I think that's what Quantum
needs to go through and so right now
it's in that that formidable finding
itself getting the the like literally
the physics figured out and um and and
then has to figure out the application
that makes this useful like right now
I'm I'm not skeptical that I think that
will happen I think it's just you know
10 years away something like that I
forgot to ask what programming language
do you think the simulation is written
in O probably
lisp so not sft like if you were to bet
you were to bet
uh I'll just leave it at that so I mean
we've mentioned that you work with all
these companies we we've talked about
all these projects it's kind of if we
just step back and zoom out about the
way you did that work and we look at
covid times this pandemic we're living
through that may if I look at the way
Silicon Valley folks are talking about
it the way MIT is talking about it this
might last for a long time uh not just
the virus but the the remote nature the
economic impact I all it yeah yeah it's
it's going to be a mess do you think uh
what's your prediction I mean from
sci-fi to Google to uh uh to just all
the places you worked in just Silicon
Valley you're in the middle of it what
do you think is how is the whole place
going to change yeah so I mean I I
really can only speak to the tech
perspective I am in that bubble
um I think it's going to be really
interesting because the you know the
zoom culture of being remote and on
video chat all the time has really
interesting effects on people so on the
one hand it's a great normalizer it's a
normalizer that I think will help
communities of people that have
traditionally been
underrepresented uh because now you're
taking in some cases a face off you
don't have to have a camera going right
and so you can have conversations
without physical appearance being part
of the part of the dynamic which is
pretty powerful you're taking remote
employees that have already been remote
and you're saying you're now on the same
level and foot footing as everybody else
nobody gets
whiteboards you're not going to be the
one person that doesn't going to be
participating in the Whiteboard
conversation and that's pretty powerful
um you've got uh you're forcing people
to think uh asynchronously in some cases
because it's harder to just just get
people physically together and the
bumping into each other forces people to
find new ways to solve those problems s
and I think that that leads to more
inclusive Behavior which is good um on
the other hand it's also it just
sucks right and so um the the nature the
the actual communication or just sucks
being not in with people like on a daily
basis and collaborating with them yeah
all of that right I mean everything this
whole situation is terrible um what I
meant primarily was the um I think that
that most humans like working physically
with humans
I think this is something that not
everybody but many people are programmed
to do and I think we get something out
of that that it's very hard to express
at least for me and so maybe this isn't
true of everybody but um and so the
question to me is you know when you get
through that time of
adaptation right you get out of March
and April and you get into December and
you get into next march if it's not
changed right it's already terrifying
well you you think about that and you
think about what is the nature of work
yeah right how do how do we adapt and
humans are very adaptable species right
we can we can learn things and when
we're forced to and there's a catalyst
to make that happen and so what is it
that comes out of this and are we better
or worse off right I think that you know
you look at the Bay Area housing prices
are insane well why well there's a high
incentive to be physically located
because if you don't have
proximity you end up paying for it and
commute right and there's there has been
huge social social pressure in terms of
like you will be there for the meeting
right or whatever scenario it is and I
think that's going to be way better I
think it's going to be much more than
Norm to have remote employees and I
think this is going to be really great
do you uh do you have friends or do you
hear of people moving yeah I I know one
family friend that moved they moved back
to Michigan and uh you know they were a
family with three kids living in a small
apartment and like we're going
insane right and they're in Tech uh
husband works for Google so first of all
friends of mine have are in the process
of or are have already lost the business
the thing that represents their passion
their dream it could be small
entrepreneur projects but it could be
large businesses like people that run
gyms like do restaurants like tons of
things yeah so but also people like look
them at themselves in the mirror and ask
the question of like what do I want to
do in life for some reason they don't
they haven't done it until Co like they
really asked that question and that
results often in moving or leaving the
company you're with starting your own
business or transitioning to different
company do you think we're going to see
that a lot like in um I I well I can't
speak to that I mean we're definitely
going to see it at a higher frequency
than we did before um just because I
think what you're trying to say is there
are decisions that you make yourself and
big life decisions that you make
yourself and like I'm going to like quit
my job and start a new thing there's
also decisions that get made for you
like I got fired from my job what am I
going to do right and that's not a
decision that you think about but you're
forced to act okay and so I think that
those you're forced to act kind of
moments where like you know Global
pandemic comes and wipes out the economy
and now you're business doesn't exist I
think that does lead to more reflection
right because you're less anchored on
what you have and it's not a what do I
have to lose versus what do I have to
gain AB comparison it's more of a fresh
slate cool well I could do anything now
do I want to do the same thing I was
doing did that make me happy is this now
time to go back to college and take a
class and learn learn a new skill is
this is this a time to uh spend time
with family if you can afford to do that
is this time to like you know literally
move in with parents right I mean all
these things that were not normative
before suddenly become I think uh very
the value system has changed and I think
that's actually a good thing in the
short term at least
because um it leads to you know there's
kind of been an over optimization along
one one set of priorities for the world
and now maybe we'll get to a more
balanced and more interesting world
where the people are doing different
things I think it could be good I think
there could be more Innovation that
comes out of it for example what do you
think about the all the social chaos
we're in the middle of like it
sucks you think it's uh let me ask you I
hope you think it's all going to be okay
well I think Humanity survive um the
from an existential like we're not all
going to kill Yeah well yeah I don't
think the virus is going to kill all all
the humans um I don't think all the
humans are going to kill all the humans
I think that's unlikely but um I I look
at it as
uh um progress requires a catalyst right
so so you need you need a reason for
people to be willing to do things that
are uncomfortable I think
that the US at least but I think the
world in general is a pretty uh uh
unoptimal place to live in for a lot of
people and I think that what we're
seeing right now is we're seeing a lot
of unhappiness and because because of
all the pressure because of all the the
Badness in the world that's coming
together it's really kind of igniting
some of that debate that should have
happened a long time ago right I mean I
think that we'll see more progress
you're asking about offline you're
asking about politics and wouldn't be
great if politics move faster because
there's all these problems in the world
and we can move it well people are
intentional or inherently uh
conservative and so if you're talking
about conservative people particularly
if they have heavy burdens on their
shoulders because they represent
literally thousands of people um it
makes sense to be conservative but on
the other hand when you need change how
do you get it the global pandemic will
probably lead to some
change and it's not a directed it's not
a directed plan but I think that it
leads to people asking really
interesting questions and some of those
questions should have been asked a long
time ago well let me know if if you
observed this as well something that's
bothered me in the machine Learning
Community I'm guessing it might be
prevalent in other places is um
something that feels like in 2020
increase level of toxicity like people
are just quicker to pile on they just be
they're just harsh on each other to to
like mob uh pick a person that screwed
up and like make it a big thing yeah and
uh is there something that we can
like have you observed that in other
places is there is there some way out of
I think there's a inherent thing in
humanity that's kind of an Us Versus
Them thing which is that you want to
succeed and how do you succeed well it's
relative to somebody else and so what
what's happening in at least in some
part is that with the internet and with
online communication the world's getting
smaller right and so we're having some
of the the social ties of like my NE My
Town versus your Town's football team
right turn into much larger larger and
yet shallower problems and uh people
don't have time the incentives are
clickbait and like all these things kind
of really really feed into this machine
and I don't know where that goes um yeah
I mean the reason I think about that I I
mentioned to you this offline a little
bit but uh you know I have uh a few
difficult conversations
scheduled some of them political related
some of them within the community uh
difficult personalities that went
through some stuff I mean one of them
I've talked before I will talk again is
Yan laon he got of a little bit of crap
on Twitter for uh for uh talking about a
particular paper and the bias within a
data set and then there's been a huge uh
in my view and I'm willing comfortable
saying it uh irrational ere exagger ated
pylon on his comments because uh he made
pretty basic comments about the fact
that if there's bias in the data there's
going to be bias in the results so we
should not have bias in the data but
people piled on to him because he said
he trivialize the problem of bias like
it's a lot more than just bias and the
data but
like yes that's a very good point but
that's that's not what he was saying
that's not what he was saying and the
response like the imply response that
he's basically sexist and
racist um is uh is something that
completely drives away the possibility
of nuance discussion one nice thing
about like a podcast long form uh
conversation is you can talk it out you
can lay your reasoning out and even if
you're wrong you can still show that
you're a good human being underneath it
you know your point about you can't have
a productive discussion well how do you
get to that point where people can turn
they can learn they can listen they can
think they can engage versus just being
a a shallow like like and then keep
moving right and I don't think that that
uh progress really comes from that right
and I don't think that um one should
expect that I think that you you'd see
that as reinforcing individual circles
and the US versus them thing and I think
that's fairly divisive yeah I think uh
there's a big role in like the people
that bother me most on Twitter when I
observe things is not the people who get
very emotional angry like over the top
it's the people who like prop them up
it's all the it's it's that I think what
should be the we should teach each other
is to be sort of empathetic the the
thing that it's really easy to forget
particularly on like Twitter or the
internet or in email is that sometimes
people just have a bad day yeah right
you have a bad day
or you're like I've been in the
situation where it's like between
meetings like fire off a quick response
to an email because I want to like help
get something
unblocked phrase it really objectively
wrong I screwed up and suddenly this is
now something that sticks with people
and it's not because they're bad it's
not because you're bad it's just
psychology of like you said a thing um
it sticks with you you didn't mean it
that way but it really impacted somebody
because the way they interpreted it and
this is just an ECT of working together
as humans and I have a lot of optimism
in the long term the very long term
about what we as Humanity can do but I
think that's going to be it's just
always a rough ride and you you came
into this by saying like what do Co and
all the the social Strife that's
happening right now mean and I think
that it's really bad in the short term
but I think it'll lead to progress and
for that I'm very
thankful yeah it's painful in the short
term though well yeah I mean people are
out of jobs like some people can't eat
like it's horrible
and um but but you know it's progress so
we'll see we'll see what happens I mean
the the real question is when you look
back 10 years 20 years 100 years from
now how do we evaluate the decisions are
being made right now I think that's
really the way you can frame that and
look at it and you say you know you
integrate across all the short-term
horribleness that's happening and you
look at what that means and is the you
know Improvement across the world or the
regression across the world uh
significant enough to make it a good or
bad thing I think that's the
question yeah and for that it's good to
study history I'm one of the big
problems for me right now is I'm reading
the rise and fall of the Third
Reich Light reading so it's everything
is just I just see parallels and every I
mean it's it's you have to be really
careful not to overstep it but just the
the thing that worries me the most is
the pain that people feel when
of com when a few things combined which
is like economic depression which is
quite possible in this country and then
just being disrespected yeah uh by in
some kind of way which the German people
were really disrespected by most of the
world uh like in a way that's over the
top that something can it can build up
and then all you need is a charismatic
leader uh to to go either positive or
negative and both work as long as
they're charismatic and there it's
taking advantage of again that that
inflection point that the world's in and
what they do with it could be good or
bad and so it's a good way to think
about times now like on an individual
level what we decide to do is when when
history is written you know 30 years
from now what happened in 2020 probably
history's going to remember 2020 yeah I
think
so either for good or bad and it's like
up up to us to write it so it's good
well one of the things I've observed
that I find F is most people act as
though the world doesn't change you make
decision knowingly right you make a
decision where you're predicting the
future based on what you've seen in the
recent past and so if something's always
been H it's rained every single day then
of course you expect it to rain today
too right on the other hand the world
changes all the time yeah constantly
like for better and For Worse right so
the question is if you're interested in
something that's not right what is the
INF point that led to a change and you
can look to history for this like what
is what is the Catalyst that led to that
that explosion that led to that bill
that led to the like you you can kind of
work your way backwards from that and
maybe if you pull together the right
people and you get the right ideas
together you can actually start driving
that change and doing in a way that's
productive and hurts fewer people yeah
like a single person single event can
turn all of absolutely everything starts
somewhere and often It's a combination
of multiple factors but but yeah this is
these these things can be engineered
that's actually the optimistic view that
I'm I'm a long-term Optimist on pretty
much everything and human nature you
know we can look to all the negative
things that that Humanity has all the
pettiness and all the like self
self-serving and the um just the the
cruelty right the the biases the just
humans can be very horrible but on the
other hand we're capable of amazing
things and um and the progress across
you know hundred-year chunks is striking
and even across decades it's we've come
a long ways and there's still a long
ways to go but that doesn't mean that
we've stopped yeah the kind of stuff
we've done in the last 100 years is is
unbelievable it's kind of scary to think
what's going to happen next 100 year
it's scary like exciting like scary in a
sense that it's kind of sad that the
kind of technology is going to come out
in 10 20 30 years will probably too old
to really appreciate cuz you don't grow
up with it it'll be like kids these days
with their virtual and their uh their
Tik toks and stuff like this like how do
this thing and like come on give me my
uh you know static photo you know my
Commodore 64 yeah yeah exactly okay uh
sorry we kind of skipped over but let me
ask on
um you know the machine learning world
has been kind
of inspired their imagination captivated
with gpt3 and these language models I
thought it'd be cool to get your opinion
on it what what's your thoughts on this
exciting world of um it connects to
computation actually uh is of language
models that are huge yeah and take
multip many many computers not just the
train but to also do inference on sure
well I mean it depends on what you're
speaking to there but I mean I think
that there's been a pretty well
understood maximum in deep learning that
if you make the model bigger and you
shove more data into it assuming you
train it right and you have a good model
architecture that you'll get a better
model out and so on the one hand gpg 3
was not that surprising um on the other
hand a tremendous amount of engineering
went into making it possible um the
implications of it are pretty huge I
think that when gpt2 2 came out there
was a very provocative blog post from
open AI talking about you know we're not
going to release it because of the
social damage it could cause if it's
misused um I think that's still a
concern I think we need to look at how
um technolog is applied and you know
well-meaning tools can be applied in
very horrible ways and they can have
very profound impact on that um uh I
think the gpt3 is a huge technical
achievement and what will GPT 4 be will
probably be bigger and more expensive to
train really cool uh architectural
tricks do what do you think is there um
I don't know how much thought you've
done on distributed computing uh is
there is there some technical challenges
that are interesting that you're hopeful
about exploring in terms of you know a
system that like a piece of code that
you know GPT
4 uh that might have I don't know uh
hundreds of trillions of parameters we
have to run on thousands of computers is
there some is there some hope that we
can make that happen yeah well I mean
today you can you can write a check and
get access to 1,000 TPU cores
and do really interesting large scale
training and inference and things like
that um in Google Cloud for example
right and um so I don't think it's a
question about scale it's a question
about utility and when I look at the
Transformer series of architectures that
that the GPT series is based on it's
really interesting to look at that
because they're actually very simple
simple designs they're not recurrent um
the training regen are pretty simple um
and so they don't really reflect like
human brains MH right um but they're
really good at learning language models
and they're unrolled enough that you get
you can simulate some recurrence right
and so the question I think about is
where does this take us like so we can
just keep scaling it have more
parameters more data more things we'll
get a better result for sure but are
there architectural techniques that can
lead to progress at a faster Pace right
this is when you know how do you get uh
instead of just like making it constant
time bigger how do you get like an
algorithmic improvement out of this
right and whether it be a new training
regimen if it becomes um uh sparse
sparse networks for example the human
brain is sparse all these networks are
dense um the connectivity patterns can
be very different I think the this is
where I get very interested and I'm way
out of my league on the Deep learning
side of this but I think that could lead
to Big breakthroughs when you talk about
uh large scale networks one of the
things that Jeff Dean likes to talk
about and he's uh uh giv a few talks on
is this idea of having a sparsely gated
mixture of experts kind of a model where
you have um you know different Nets that
are trained and are really good at
certain kinds of tasks and so you have
this distributor across a cluster and so
you have a lot of different computers
that end up being kind of locally
specialized in different domains and
then when a query comes in you you gate
it and you use learn techniques to route
to different parts of the network and
then you utilize the compute resources
of the entire cluster by having
specialization within it and I don't
know where that goes or if it starts to
when it starts to work but I think
things like that could be really
interesting as well and then on the data
side too if you can think of data
selection as a kind of programming yeah
I mean at the essentially if you look at
like Kathy talked about software 2.0 I
mean that in a sense data is the
programming yeah yeah so I I just so let
me try to summarize Andre's position
really quick before I disagree with it
yeah um so Andre Kathy is amazing so
this is nothing nothing personal with
him he's he's he's an amazing engineer
and and also a good uh blog post writer
yeah well he's a great communicator I
he's just an amazing person he's he's
also really sweet um so his his basic
premise is that uh software is
suboptimal I think we can all agree to
that uh he also points out that uh deep
learning and other learning based
techniques are really great because you
can solve problems in uh more structured
ways uh with less like ad hoc code that
people write out and don't write test
cases for in some cases and so they
don't even know if it works in the first
place um and so if you start replacing
systems of uh imperative code with deep
learning models then you get better a
better result okay and I think that he
argues that software 2.0 is a per
pervasively learned set of models and
you get away from writing code and he's
given talks where he talks about you
know swapping over more and more and
more parts of a code being learned and
um driven that way I think that works
and if you're pre predisposed to liking
machine learning then I think that
that's that's that's definitely a good
thing I think this is also good for
accessibility in many ways because
certain people are not going to write C
code or something and so having a data
driven approach to do this kind of stuff
I think can be very valuable on the
other hand there are huge trade-offs and
it's not clear to me that software 2.0
is um the answer and probably Andre
wouldn't argue that it's the the answer
for every problem either but um I look
at machine learning as not a replacement
for software 1.0 I look at it as a new
programming
Paradigm and so programming paradigms
when you look across across domains is
you know structured programming where
you go from go-tos to if then else or
functional programming from lisp and you
start talking about higher order
functions and values and things like
this or you talk about objectoriented
programming you talk about encapsulation
subclassing inheritance you start
talking about generic programming where
you start talking about code reuse
through um through uh specialization in
different type instantiations um when
you start talking about differentiable
programming something that I am very
excited about in the context of machine
learning talking about taking functions
and generating uh variance like the
derivative of another function like
that's a programming Paradigm that's
very useful for solving certain classes
of problems machine learning is amazing
at solving certain classes of problems
like you're not going to write a you
know a cat detector or even a language
translation system by writing C code
that's not going to that's not a very
productive way to do things anymore and
so machine learning is absolutely the
right way to do that in fact I would say
that learn models are really the one of
the best ways to work with the human
world in general and so anytime you're
talking about sensory input of different
modalities anytime that you're talking
about um generating things in a way that
makes sense to a human I think that
learn models are really really useful
and that's because humans are very
difficult to character
okay and so this is a very powerful
Paradigm for solving classes of problems
but on the other hand uh imperative code
is two you're not going to write a
Bootloader for your computer in with a
deep learning model deep learning models
are very uh Hardware intensive they're
very energy intensive because you have a
lot of parameters and you can provably
Implement any function with a learned
model like this has been shown uh but
that doesn't make it
efficient and so if you're talking about
carrying about a few orders of
magnitudes worth of energy usage then
it's useful to have other tools in the
toolbox what also robustness too I mean
yeah exactly all the problems of dealing
with data and bias and data all the
problems of uh you know software 2.0 and
one of the great things that Andre is is
uh arguing towards which I completely
agree with him is that when you start uh
implementing things with deep learning
you need to learn from software 1.0 in
terms of testing continuous integration
how you deploy how do you validate all
these things and building Building
Systems around that so that you're not
just saying like o it seems like it's
good ship it right well what happens
when I regress something what happens
when I make a classification that's
wrong and now I uh hurt somebody right I
mean all these things you have to reason
about yeah but at the same time the
bootloader that works for our for us
humans is uh looks awfully a lot like a
new network right so it's it's it's
messy and you can cut out different
parts of the brain there's a lot of this
neuroplasticity work that shows that
it's going to adjust it's a I mean it's
a really interesting question how much
of the world programming could be
replaced by software 2.0 like with oh
could well I mean it's provably true
that you could replace all of it right
so then it's question anything that's a
function you can so it's not a question
about if I think it's a economic
question it's a what kind of talent can
you get what kind of trade-offs in terms
of Maintenance right those kind of
questions I think what kind of data can
you collect I think one of the reasons
that I'm most interested in uh machine
learning is a programming Paradigm is
that one of the things that we've seen
across Computing in general is that
being laser focused on one Paradigm
often put you in a box it's not super
great and so you look at object ear
programming like it was all the rage in
the early 80s and like everything has to
be objects and people forgot about
functional programming even though came
first and and then people rediscovered
that hey if you mix functional and
object oriented and structure like you
mix these things together you can
provide very interesting tools that are
good at solving different problems and
so the question there is how do you get
the best way to solve the problems it's
not about whose tribe should win right
it's not about you know that that that
shouldn't be the question the question
is how do you make it so that people can
solve those problems the fastest and
they have the right uh Tools in their
box to build good libraries and they can
solve these problems and when you look
at that that's like you know you look at
reinforcement learning as one really
interesting subdomain of this
reinforcement learning often you have to
have the integration of a of a learn
model combined with your Atari or
whatever the other scenario it is that
you're you're working in you have to
combine that that thing with the robot
control for the
arm right and so now it's not just about
that one uh Paradigm it's about
integrating that with all the other
systems that you have including often
Legacy systems and things like this
right and so to me I think that the
interesting interesting thing to say is
like how do you get the best out of this
domain and how do you enable people to
achieve things that they otherwise
couldn't do without excluding all the
good things we already know how to do
right but okay this is just a crazy
question but we talked a little about
gpt3 but do you think it's possible that
these language
models that uh in essence in the
language domain software 2.0 could
replace some aspect of compilation for
example or do program synthesis replace
some aspect of
programming yeah absolutely so I think
the that learn models in general are
extremely powerful and I think the
people underestimate them um maybe you
can suggest what I should do so of uh
you know access to the gpt3 API would I
be able to generate Swift code for
example do you think that could do
something interesting and so gpt3 is not
probably not trained on the right Corpus
so it probably has the ability to
generate some Swift I bet it does um
it's probably not going to generate a
large enough body of Swift to be useful
but but like take it a next step further
like if if you had the goal of training
something like gpt3 and you wanted to
train it to generate source code right
it could definitely do that now the
question is um how do you express the
intent of what you want filled in you
can definitely like write write
scaffolding of code and say fill in the
hole and sort of put in some for Loops
open put some classes or whatever and
and the power of these mods is
impressive but there's an unsolved
question at least unsolved to me which
is how do I express the intent of what
to fill
in right and kind of what you'd really
want to have and I don't know that that
these models are up to the task is you
want to be able to say um here's a
scaffolding and here are the assertions
at the
end and the assertions always pass and
so you want a generative model on the
one hand yes oh that's fascinating yeah
right but you also want some loop back
some reinforcement learning system or
something where you're actually saying
like I need to hill climb towards
something that is more correct and I
don't know that we have that so it would
generate not only a bunch of the code
but like the checks that do the testing
it would generate the tests I think I
think the humans would generate the
tests right because the the test be
fascinating if well the tests are the
requirements yes but the okay so because
you're have you have to express to the
model what you want to you don't just
want gibberish code look look at how
compelling this code looks
you want a story about four horned
unicorns or something well okay so
exactly but that's human requirements
but then I thought it's a compelling
idea that the gp4 model could
generate uh checks like that are more um
High Fidelity that check for correctness
because uh the coded generates like say
I ask it to generate a function that um
gives me the Bacci sequence sure I don't
like so so decompose the problem right
so you have you have two things you have
you need the ability to generate
syntactically correct Swift Code that
that's interesting right I think GPT
series of model architectures can do
that but then you need the ability to
add the requirements so generate
Fibonacci yeah the human needs to
express that goal we don't have that
language that I know of no I mean it can
generate
have you seen with gpt3 can generate you
can say I mean there's uh interface
stuff like it can generate HTML it can
generate uh basic for Loops that give
you like right but pick HTML how do I
say I want
google.com well no you could say or not
not literally google.com how do I say I
want a web page that's got a shopping
cart and this and that that does that I
mean so okay so just uh I don't know if
you've seen these demonstrations but you
type in I want a red button with the
text that says hello and you type that
in natural language and it generates the
correct HTML done this demo it's it's
kind of compelling so you have to uh uh
prompt it with similar kinds of mappings
of course it's probably handpicked like
have to experiment they probably but the
fact that they can do that once even out
of like 20 yeah is uh is quite
impressive again that's very basic uh
like the HTML is kind of messy and and
bad sure sure but yes the intent is the
idea is the intent to specifi the
natural language okay and so I've have
not seen that that's really cool yeah
yeah yeah but the question is uh the
correctness of that like visually you
can check oh the button is red but the
for more uh for more
complicated functions where the intent
is harder to check this goes into like
MP completeness kind of things like I
want to know that this code is correct
and gener it's a giant thing
that uh does some kind of calculation it
seems to be working it it's interesting
to think like should the system also try
to generate checks for itself for
correctness yeah I don't know and this
this is way beyond my experience the uh
uh the thing that I think about is that
there doesn't seem to be a lot of
equational reasoning going on there's a
lot of pattern matching and filling in
and kind of propagating patterns that
have been seen before into the future
and into the generator result and so if
you want to get correctness you kind of
need theorem proving kind of things and
like higher level logic and I don't know
that um you could talk to Yan about that
um and see and see what uh the the
bright minds are thinking about right
now but I don't think the GPT is in that
that vein it's still really cool yeah
and surpris who knows you know maybe
reasoning is is uh is overrated yeah is
over right I mean do we reason how do
how do you tell right are we just
pattern matching based on what we have
and then reverse justifying it to
ourselves exactly the reverse so like I
think what the neural networks are
missing and I think GPT for might have
is to be able to uh tell stories to
itself about what it did well that's
what humans do right I mean you talk
about uh like Network explainability
right and we give noral Nets a hard time
about this but humans don't know why we
make decisions we have this thing called
intuition and then we try to like say
this feels like the right thing but why
right and you know you wrestle with that
when you're making hard decisions and is
that science not
really let me ask about a few highle
questions I guess is
um you've done a million things in your
life and been very successful a bunch of
young folks listen to this ask for
advice from successful people like you
uh if you were to give advice to uh
somebody you know another a graduate
student or some high school student
about uh
pursuing a career in Computing or just
advice about life in general is there
sure is there some words of wisdom you
can give them so I think you come back
to change and you know profound leaps
happen because people are willing to
believe that change is possible and that
um the world does change and are willing
to do the hard thing that it takes to
make change happen and whether it be
implementing a new programming language
or implementing a new system or
implenting a new research paper
designing a new thing moving the world
forward in science and philosophy
whatever it really comes down to
somebody who's willing to put in the
work right and you
have the the work is hard for a whole
bunch of different reasons one of which
is um you uh it it's work right and so
you have to have the space in your life
in which you can do that work which is
why going to grad school can be a
beautiful thing for certain people um
but also there's a self-doubt that
happens like you're two years into a
project is it going anywhere right right
well what do you do do you do you just
give up because it's hard well no I mean
some people like suffering um and so you
plow through it the the secret to me is
that you have to love what you're doing
and and follow that passion because if
when you get to the hard times that's
when you know if you if you love what
you're doing you're willing to kind of
push through and um this is really uh
hard because it's it's hard to know what
you will love doing until you start
doing a lot of things and so that's why
I think that particularly early in your
career it's good to experiment do a
little bit of everything go go go take
the the survey class on you know four
different the first half of every class
in your upper division you know lessons
and um just get exposure to things
because certain things will resonate
with you and you'll find out wow I'm
really good at this I'm really smart at
this well it's just because it's it
works with the way your brain and when
something jumps out I mean that's one of
the things that people often ask about
is like well I think there's bunch of
cool stuff out there like how do I pick
the thing like uh yeah how do you how do
you hook in your life how did you just
hook yourself in and stuck with it well
I got lucky right I mean I think that
many people uh forget that a huge amount
of it or most of it is luck right so um
let's not forget that um so for me I
fell in love with computers early on
because I'm they they spoke to me I
guess uh what language did they speak
basic basic yeah um but the uh uh but
then it was just kind of following a set
of logical progressions but also um
deciding that something that was hard
was worth doing and and a lot of fun
right and so I think that that is also
something that's true for many other
domains which is if you find something
that you love doing that's also hard if
you invest yourself in it and add value
to the world then it will mean something
generally right and again that can be a
research paper that can be a software
system that can be a new robot that can
be that there's many things that that is
that can be but a lot of it is like real
value comes from doing things that are
hard and that doesn't mean you have to
suffer but um it's hard I mean you don't
often hear that message we talked about
it last time a little bit but I I it's
one of my f not enough people talk about
this this it's uh it's beautiful to hear
a successful person well and self-doubt
and impostor syndrome and the these are
all things that uh successful people
suffer with as well particularly when
they put themselves in a point of being
uncomfortable which um I like to do now
and then just because it puts you in
learning mode like if you want to if you
want to grow as a person put yourself in
a room with a bunch of people that know
way more about whatever you're talking
about than you do and ask dumb questions
and guess what smart people love to
teach often not always but often and if
you listen if you're prepared to listen
if you're prepared to grow if you're
prepared to make connections you can do
some really interesting things and I
think a lot of progress is made by
people who kind of hop between domains
now and then because they bring uh they
bring a perspective into a field that
nobody else has if people have only been
working in that field
themselves we mentioned that the
universe is kind of like a compiler of
you know the entirety of it the whole
evolution is kind of a kind of
compilation maybe our us human beings
are kind of compilers um let me ask the
the old absur question that I didn't ask
you last time which is uh what's the
meaning of it all is there a meaning
like if you asked a compiler why what
would a compiler say what's the meaning
of life what's the meaning of life uh
you know I'm prepared for it not to mean
anything here we are all biological
things programmed to survive and and
propagate our our DNA um and maybe the
is just a just a computer and you you
just go until entropy takes over the
world and or takes over the universe and
then you're done um I don't think that's
a very productive way to live your life
if
so and so I prefer to bias towards the
other way which is saying the world has
the universe has a lot of value and I
take uh I take happiness out of other
people and a lot a lot of times part of
that's having kids but also the
relationships you build with other
people and so uh the way I try to live
my life is like what can I do that has
value how can I move the world forward
how can I take what I'm good at and like
bring it bring it into the world and how
can I I'm one of these people that likes
to work really hard and be very focused
on the things that I do and so if I'm
going to do that how can it be in a
domain that actually will matter right
because a lot of things that we do we
find ourselves in the cycle of like okay
I'm doing a thing I'm very familiar with
it I've done it for a long time I've
never done anything else but I'm not
really learning I I'm not really I'm
keeping things going but there's a
there's a younger generation that that
can do the same thing maybe even better
than me right maybe if I actually step
out of this and jump into something I'm
less comfortable with it's scary but on
the other hand um it gives somebody else
a new opportunity it also then put you
back in learning mode and that can be
really interesting and one of the things
I've learned is that uh when you go
through that that first you're deep into
impostor syndrome but when you start
working your way out you start to
realize hey well there's actually a
method to this and and now I'm able to
add new things because I bring different
perspective and this is one of the the
the good things about bringing different
kinds of people together diversity of
thought is really important and um if
you can pull together people that are
coming at things from different
directions you often get Innovation and
I I love to see that that aha moment
where you're like we've like really
cracked this this is something never
nobody's ever done before and then if
you can do it in a context where it adds
value other people can build on it it
helps move the world then that's what
that's what really excites me so the
that kind of description of the magic of
The Human Experience do you think we'll
ever create that in like an AGI system
you think
we be able to create uh give uh give AI
systems a sense of meaning where they
operate in this kind of world exactly in
the way you've described which is they
interact with each other they interact
with us humans sure sure well so I mean
I why why are you being so speciest
right all right so so agis versus bionet
or you know versus
bi um you know uh what are we but
machines right we're just programmed to
run our we have our objective function
that we optimized for right and so we're
doing our thing we think we have purpose
but do we really yeah right I'm not
prepared to say that th those new
fangled agis have no soul just because
we don't understand them right and I
think that would be um when they when
they exist uh that would be very
premature to uh uh look at a new thing
through your own lens without fully
understanding it um you might be just
saying that because AI systems in the
future will be listening to this and
then oh yeah yeah exactly you don't want
to say anything please be nice to me you
know when Skynet Skynet kills everybody
please spare me Wise Wise uh look ahead
thinking yeah but I mean I I think that
people spend a lot of time worrying
about this kind of stuff and I think
that what we should be worrying about is
how do we make the world better and the
thing that I'm most scared about with
agis is not that um that necessarily the
Skynet will start shooting everybody
with lasers and stuff like that to to
use us for our
calories the thing that I'm worried
about is that um Humanity I think needs
a challenge and if we get into a mode of
not having a personal challenge not
having a personal contribution whether
that be like you know your kids and
seeing what they grow into and helping
helping guide them whether it be
um your community that you're engaged in
you're driving forward whether it be
your work and the things that you're
doing and the people you're working with
and the products you're building and the
contribution there if people don't have
a objective I'm afraid what that means
and um I think that this would lead to a
rise of the worst part of people right
instead of people striving together and
trying to make uh the world better it
could degrade into a very uh unpleasant
world but but I don't know I mean we
hopefully have a long ways to go before
we discover that unfortunately we have
pretty on the ground problems with the
pandemic right now and so I think we
should be focused on that as well yeah
ultimately just as you said you're
optimistic I think it helps for us to be
optimistic that's uh fake it until you
make it yeah well and why not what's
what's the other side right so I mean uh
uh I I'm not personally a very religious
person but I've heard people say like oh
yeah of course I believe in God of
course I go to church because if God's
real you know I want to be on the right
side of that and if it's not real it
doesn't matter doesn't matter and so you
know that's that's a fair way to do it
um yeah I mean the same thing with uh
with nuclear deterrence all you know
global warming all these things all
these threats natural engineer pandemics
all these threats we face I think it's
uh uh it's paralyzing to be terrified of
all the possible ways we could destroy
ourselves I think it's much better uh or
at least productive to be hopeful and to
engineer defenses against these things
to uh engineer a future where like you
know see like a positive future and
engineer that future yeah well and I
think that's other another thing to
think about as you know a human
particularly if you're young and trying
to figure out what it is that you want
to be when you grow up like I am um I'm
always looking for that uh the the
question then is how do you want to
spend your time and right now there
seems to be a norm of being a
consumption culture like I'm going to
watch the news and and revel in how
horrible everything is right now I'm
going to go find out about the latest
atrocity and find out all the details of
like this the terrible thing that
happened and be outraged by it um you
can spend a lot of time watching TV and
watching the new sitcom or whatever
people watch these days I don't know um
uh but that's a lot of hours right and
those are hours that if you're tur into
being productive learning growing
experiencing uh you know when the
pandemic's over going
exploring right it leads to more growth
and I think it leads to more optimism
and happiness because you're you're
you're building right you're building
yourself you're building your
capabilities you're building your
viewpoints you're building your
perspective and um I think that a lot of
the cons the consuming of other people's
me messages leads to kind of a negative
Viewpoint which you need to be aware of
what's happening because that's also
important but there's a balance that um
I think focusing on creation is is a
very valuable thing to do yeah so what
you're saying is people should focus on
uh working on the sexiest feel of them
all which is compiler design exactly
well hey you can go work on machine
learning and be crowded out by the the
thousands of graduates popping out of
school that all want to do the same
thing or you could work in the place
that people overpay you because there's
not enough smart people working in it
and uh here at the end of Mor's law
According to some people uh actually the
software is the hard part too yeah uh I
mean optimization is is truly uh truly
beautiful and also on the YouTube side
or education side uh you know it's
there's um it'd be nice to have some
material that shows the beauty of
compilers yeah yeah that's that's
something so that's a call for uh for
people to create that kind of content as
well Chris uh you're one of my favorite
people to talk to I it's such a huge
honor that you would waste your time
talking to me uh I've always appreciate
it thank you so much today the
the the the truth of is you spend a lot
of time talking to me just on you know
walks and other things like that so it's
it's great to catch up thanks man thanks
for listening to this conversation with
Chris lner and thank you to our sponsors
blinkist an app that summarizes key
ideas from thousands of books neuro
which is a maker of functional gum and
mints that supercharge my mind
masterclass which are online courses
from World experts and finally cash app
which is an app for sending money to
friends please check out these sponsors
in the description to get a discount and
to support this podcast if you enjoy
this thing subscribe on YouTube review
it with f stars on Apple podcast follow
on Spotify support on patreon or connect
with me on Twitter at Lex Friedman and
now let me leave you some words from
Chris latner so much of language design
is about trade-offs and you can't see
those tradeoffs unless you have a
community of people that really
represent those different points thank
you for listening and hope to see you
next time