Kind: captions
Language: en
the following is a conversation with
Mark Zuckerberg inside the metaverse
Mark and I are hundreds of miles apart
from each other in physical space but it
feels like we're in the same room
because we appear to each other as
photorealistic Kodak avatars in 3D with
spatial audio
this technology is incredible and I
think it's the future of how human
beings connect to each other in a deeply
meaningful way on the internet
these avatars can capture many of the
nuances of facial expressions that we
use we humans use to communicate emotion
to each other now I just need to work on
upgrading my emotions expressing
capabilities of the underlying human
this is the Lux Friedman podcast and now
dear friends here's Mark Zuckerberg
[Laughter]
this is so great
lighting change wow
oh yeah we can put the light anywhere
and it doesn't feel awkward to be really
close to you no it does I actually moved
you I moved you back a few feet before
you got into the headset you were like
right here
I don't know if people can see this but
this is incredible the realism here is
just incredible
where am I where are you Mark where
where are we you're in Austin right no I
mean this place we're shrouded by
Darkness with ultra realistic
face and it just feels like we're in the
same room this is really the most
incredible thing I've ever seen and
sorry to be in your personal space I
mean we have done suggested before yeah
no I was I was commenting to the team
before that
even that I feel like we've choked each
other from further distances than it
feels like we are right now I mean this
is just really incredible I don't know
how to describe it with words it really
feels like
it feels like we're in the same room
yeah it feels like the future
this is truly truly incredible I just
wanted to take it in I'm still getting
used to it it's like it's you it's
really you
but you're not here with me
right
you're there wearing a headset now I'm
wearing a headset it's it's really
really incredible so what um can you
describe what it takes currently for us
to appear so photorealistic to each
other yeah so I mean for for background
we both did these scans for uh This
research project that that we have at
meta called Kodak avatars and the idea
is that instead of actually instead of
our avatars being cartoony
um and instead of actually transmitting
a video what it does is we've sort of
scanned ourselves in a lot of different
expressions and we've built a computer
model of sort of each of our faces and
and bodies and the different Expressions
that we make and collapse that into a
codec that then when you have the
headset on your head it can it sees your
face it's easier expression and it can
basically send
um an encoded version of what you're
supposed to look like over the wire so
um so in addition to being
photorealistic it's also actually much
more bandwidth efficient than
transmitting a a full video or
especially a 3D immersive video of a
whole scene like this and it captures
everything like the flaws like to me
the subtleties of the human face like
even the flaws that's like that's all
amazing it makes you uh
it makes us so much more immersive it
makes you realize that like Perfection
isn't the thing that leads to immersion
it's like the little subtle flaws like
freckles and like variations in color
and just yeah wrinkles
roses yeah asymmetry and just the
different like the corners of the eyes
like what your eyes do when you smile
all that kind of stuff yeah eyes are a
huge part of it yeah I mean there's all
the studies that most of communication
even when people are speaking is not
actually the words that they're saying
right it's kind of the expression and
all that so we try to capture that with
the kind of classical
um expressive Avatar system that we have
that's the kind of more cartoon designed
one you can you can kind of put those
kind of Expressions on those faces as
well but there's obviously a certain
realism that comes with delivering kind
of this photorealistic experience that
um I I don't know I just think it's
really magical I mean this gets to kind
of the core of what the vision around
virtual and augmented reality is of like
delivering a sense of presence is if
you're there together no matter where
you actually are in the world and I mean
this this uh experience I think is a
good embodiment of that where it's like
coming or in two completely different
states halfway across the country and it
just like you know looks like you're
just sitting right in front of me it's
uh it's pretty wild
yeah yeah I can't it's I'm almost
getting emotional it's like it feels
like a totally
it's fundamentally new experience like
for me to have this kind of
conversations with loved ones
that would just change everything maybe
just to elaborate so the I went to
Pittsburgh and went through the whole
scanning procedure which has so much
incredible uh technology so software and
Hardware going on
um but it is a lengthy process so what's
your vision for the future of this uh in
terms of making this more accessible to
people
you know it starts off with a small
number of people doing these
um very detailed scans right which is
this that's the version that you did and
that I did and you know before there are
a lot of people who who've done this
kind of a scan for we probably need to
kind of over collect
Expressions
um when we're doing the scanning because
we haven't figured out how much we can
reduce that down to a really streamlined
process and extrapolate from the the
scans that have already been done but
you know the goal and we have a project
that's working on this already is just
to do a very quick scan with your cell
phone where you just take your phone
kind of wave it in front of your face
for a couple of minutes
um you know say a few sentences make a
bunch of Expressions but overall
have the whole process just be two to
three minutes and then produce something
that's of the quality of what we have
right now so I think that that's one of
the big challenges that remains and
right now we have the ability to do the
scans if you you know have hours to sit
for one and with today is technology I
mean you're using a meta headset that
exists it's a product that's kind of for
sale now you can drive these with that
um but the production of of um these
scans in a very efficient way is one of
the last pieces that we still need to
really nail and then obviously there's
all the experiences around it I mean
right now we're kind of sitting in a
dark room which
um you know is you know familiar for for
your podcast but I think part of the the
vision for this over time
is um is you know not just having this
be like a video call I mean that's fine
it's it's cool or it feels like it's
immersive but
um you know you can you can do a video
call on your phone the thing that you
can do in the metaverse that is
different from what you can do on a
phone is like doing stuff where you're
physically there together and and
participating in things together and we
could play games like this
um we could have meetings uh like this
in in the future once you mix
um once you get mixed reality and
augmented reality we could have codec
avatars like this and go into a meeting
and have some people physically there
and have some people show up in this
photorealistic form uh superimposed on
the on the physical environment I've got
stuff like that is going to be super
powerful so we got to still build out
all those kind of applications and the
use cases around it but I don't know I
think it's going to be a pretty wild uh
next few years around this I mean I just
I'm actually almost at a loss of Wars
this is just so incredible this is truly
incredible I I hope that people like
watching this can get a glimpse of like
how incredible it is it really feels
like we're in the same room like there
is that
um I guess there's an uncanny valley
that seems to have been crossed here
like it looks like you yeah I mean I
think there's still a bunch of tuning
that I think we'll want to do where
different people emote to different
extents right so I think one of the big
questions is you know like when you
smile how wide is your smile and how
wide do you want your smile to be
um and I think getting that to be tuned
on a per person basis is
um is going to be one of the things that
we that we're going to need to figure
out
um you know it's like to what extent do
you want to give people control over
that
um you know some people might try to you
know might might prefer version of
themselves that's more emotive in their
Avatar than their actual faces you know
so for example you know I always get a
lot of critique and for
um for for having like a relatively
stiff uh expression but you know I mean
I I might I might feel pretty happy but
just make a pretty small smile so I mean
maybe you know for me I would it's
actually you know it's like I'd wanna
have my avatar really be able to better
Express
um like how I'm feeling then then than
how I can do physically so I think
there's a question about how you want to
tune that but uh but overall yeah I mean
you we want to start from the Baseline
of capturing how people actually emote
and express themselves and I mean I
think the the initial version of this is
has been pretty impressive and like you
said
um I do think we're we're kind of beyond
the The Uncanny Valley here where it and
it does feel like you it doesn't feel uh
it doesn't feel weird or anything like
that
I mean that's going to be the meme that
the two most monotone people are in a
metaverse together but I think that
actually makes it more difficult like
the the amazing thing here is that the
subtleties of the expression of the eyes
you know people say I'm monotone and
emotionless but I'm not it's just this
it may be my expression of emotion
is more subtle usually like with the
eyes and that's one of the things I've
noticed is just how expressive the
subtle movement of the corners of the
eyes are in terms of displaying
happiness or boredom or all that kind of
stuff I'm curious to see just because
I've never done one of these before I've
never done a podcast as as one of these
codec avatars
um and I'm curious to see what what how
what people think of it because you know
one of the issues that we've had in some
of the VR and and mixed reality work is
it tends to feel a lot more profound
when you're in it than the 2D videos
capturing the experience so I think that
this one because it's photorealistic
um may look kind of as amazing in 2D for
people watching it as it as it feels I
think to be in it but we've certainly
had this this um this issue where a lot
of the other things just it's like you
feel the sense of immersion when you're
in it that that doesn't quite translate
to a 2d screen but I don't know I'm
curious to see to see what people think
yeah I'm curious to see if people could
see that um
like my heart is actually beating fast
now this is super interesting like the
that such
intimacy of conversation could be
achieved remotely there's been you know
I don't do remote podcasts for this
reason and this is like breaks all of
that this feels like just an incredible
transition to something else to the
different kind of communication breaks
all barriers like Geographic physical
barriers uh would you mentioned do you
have a sense of timeline in terms of how
many difficult things have to be solved
to make this more accessible to like
scanning with a smartphone
yeah I mean I think we'll probably roll
this out progressively over time so it's
not gonna be like we roll it out and one
day everyone has a codec Avatar we want
to get more people scanned and into the
system
and then we want to start uh integrating
it into each one of our apps right
making it so that you know I think that
for a lot of the work style things
productivity I think that this is going
to make a ton of sense in a lot of game
environments I mean this could be fine
but games tend to have their own style
right where you almost want to fit more
with the aesthetic style of the of the
game
um but I think for doing meetings and
one of the things that we get a lot of
feedback on work rooms where people are
pretty Blown Away by the experience and
this feeling that you can like
be remote but feel like you're
physically there around a table with
people but then you know we get some
feedback that people have a hard time
with the fact that the avatars are so
expressive and and and don't feel you
know as as realistic in that environment
so I think something like this
um could make a very big difference for
those remote meetings and especially
with Quest 3 coming out which is going
to be the first mainstream mixed reality
product right where you're really taking
digital
um you know expressions of either a
person or or objects and overlaying them
on the physical world
um I think the ability to do kind of
remote meetings and and things like that
where you're like just remote hanging
sessions with friends I mean I I think
that's going to be very exciting so yeah
rolling it out over the next over the
next few years it's not ready to be like
a a kind of mainstream product yet but
um we just want to we'll keep tuning in
keep getting more scans in there and
keep you know and kind of rolling it out
into more of the features but yeah I
mean definitely in the next in the next
um few years you'll be seeing a bunch
more experiences like this
yeah I would love to see some
celebrities scanned and some
non-celebrities uh I just just more
people to experience this I would love
to see that this is something I mean on
my mind is bro I'm literally at a loss
of words because it's very difficult to
just
convey how incredible this is how how
like how I feel the emotion how I feel
the presence how I feel like the
subtleties of the emotion in terms of
like work meetings or any kind of in
terms of podcasts this is like this is
awesome and I don't even need your arms
or legs is that well we gotta get bad I
mean that's that's its own challenge
um and and part of the question is also
so you have the scan then it takes a
certain amount of compute to go drive
that both for the sensors on the headset
and
um and then rendering it so one of the
things that we're working through is
what is the level of fidelity that is
optimal right you could do the full body
in in kind of a codec and that can be
quite intensive but
um but one of the things that we're
we're thinking about is like all right
maybe you can kind of
Stitch a somewhat lower Fidelity version
of your body would still still have the
main kind of the major movements
um but uh but but your face is really
the thing that we have the most
resolution on right in terms of being
able to read and express emotions I mean
like you said
if you move your you know eyebrows like
a millimeter I mean that really changes
the expression and what you're you're
emoting whereas
you know I mean moving your your arm
like a an inch probably doesn't matter
quite as much so so yeah so I think that
we'll we do want to get all of that into
here and and that'll be some of the work
over the next period as well
so you mentioned Quest three that's
coming out I've gotten a chance to try
that too that's awesome so the how did
you pull off the mixture so it's not
just virtual reality it's mixed reality
yeah I mean I think it's gonna be it's
gonna be the first mainstream
um mixed reality device I mean obviously
we ship Quest Pro
um last year but it was fifteen hundred
dollars
um and well part of what I'm super proud
of is you know we try to innovate not
just on pushing the state of the art and
delivering new capabilities but making
it so it can be available to everyone
and you know we have this and it's
coming out it's 500 and
um in in some ways I think the mixed
reality is actually better in quest 3
than it was
um than what we're using right now in
quest Pro so I'm really proud of the
team for being able to deliver that kind
of an innovation and get it out but you
know some of this is just
um
software you tune over time and get to
be better part of it is you put together
a product and you figure out what are
the bottlenecks in terms of making it a
good experience so we got the resolution
for the Mixed reality cameras and
sensors to be multiple times better in
quest 3 and we just figured that that
made a very big difference when we saw
the experience that we were able to put
together for Quest Pro
and part of it is also that you know
Qualcomm just came out with their next
Generation chipset for for VR and Mr
that we worked with them on a kind of
custom version of it but that was
available this year for Quest 3 and it
wasn't available in quest Pro so you
know in a way I'm Quest 3 even though
it's not you know the pro product
actually has a stronger chipset in it
than the pro line at a third of the cost
so
um so I'm really excited to get this in
people's hands it um it does all the VR
stuff that that Quest 2 and the others
have done too it does it better because
the display is better and and the chip
is is better so you'll get better
graphics it's 40 thinner so it's um so
just more comfortable as well but but
the Mr is really the big capability
shift and and part of what's exciting
about the whole Space right now is you
know this isn't like smartphones where
you know companies put out a new
smartphone every year and you can almost
barely tell the difference between that
and the the one the year before it now
for this each time we put out a new
headset it has like a major new
capability and and the big one now
um is is mixed reality the ability to
basically take digital representations
of people or objects and and superimpose
them on the world and basically you know
I mean there's that one version of this
is you're gonna kind of have these
augments or or Holograms and experiences
that you can kind of bring into your
living room or a meeting space or an
office
um another thing that I just think is
going to be a much kind of simpler
innovation
is that there are a lot of VR
experiences today that don't need to be
fully immersive and you know if you're
playing a shooter game or you're doing a
fitness experience now sometimes people
get worried about swinging their arms
around like am I going to hit a lamp or
something you know it's and and am I
going to run into something
so having that in mixed reality actually
is just a lot more comfortable for
people right you kind of still get the
immersion and the 3D experience
um and you can you can have an
experience that just wouldn't be
possible in the physical world alone but
by being anchored to and be able to see
the physical world around you it's like
it just feels so much safer and more
secure and I think a lot of people are
really going to enjoy that too so yeah
I'm really excited to see how people use
it but yeah Quest 3 coming out um later
this fall
yeah and I got to experience it with
other people sitting around and there's
a lot of furniture and so you get to see
that furniture you get to see those
people and you get to see those people
like
enjoy the Ridiculousness of you like
swinging your arms I mean person will be
their friends of yours even if they make
fun of you they uh there's a lot of love
behind that and that you guys I got to
experience that so that's a really
fundamentally different experience than
just pure VR with like with zombies
coming out of walls and yeah it's like
someone shooting at you and you hide
behind your real couch in order to duck
the the fire yeah it's incredible how
it's all integrated but also like subtle
stuff like in a room with no windows you
can add Windows to it and you can look
outside as the zombies run towards you
but like it's still nice view outside
yeah yeah it's just it's really and so
that's pulled off by having cameras on
the outside of the headset that do the
pass-through in that technology is
incredible to do that on a oh yeah
headset yeah it's not just the cameras
you basically need to you need multiple
cameras to capture the different angles
and and sort of the three-dimensional
space and then it's a pretty complex
compute problem an AI problem to map
that to your perspective right because
the cameras aren't exactly where your
eyes are because no two people's eyes
are you're not going to be in exactly
the same place you kind of need to to
get that to to to line up
um and then do that basically in real
time and then generate something that
looks that kind of feels natural
um and then
superimpose whatever digital objects you
want to put there so it's yeah it's a
very interesting technical Challenge and
um and we'll continue tuning this for
for the years to come as well but uh but
I'm pretty excited to to get this out
because I think quest 3 is going to be
the first device like this with that
millions of people are going to get
that's mixed reality and it's only when
you have millions of people using
something that you start getting the
whole developer Community really
starting to experiment and build stuff
because now they're going to be people
who actually use it
um
so I think we'll get you know we got
some of that flywheel going with Quest
Pro but I think it'll really get
accelerated once Quest 3 gets out there
so yeah I'm I'm pretty excited about
this one plus there's hand tracking
without so you don't need to have a
control so this count the cameras aren't
interested in the pass-through
uh of the entire physical reality around
you is also tracking the details of your
hands in order to use that for like
gesture recognition this kind of stuff
yeah we've been able to get way further
on hand recognition in a shorter period
of time than I expected so that's been
pretty cool I don't know did you see the
the demo experience that we built around
um piano like yeah the piano learning to
play piano yeah it's incredible you're
basically playing piano on a table and
it's that's without any controller and
like how well it matches physical
reality with no latency and it's not
it's tracking your hands with no latency
and it's tracking all the people around
you with no latency integrating physical
reality and digital reality
obviously that connects exactly to this
Kodak Avatar which is in parallel allows
us to have ultra realistic
copies of ourselves in this mixed
reality
it's uh so like it's all converging
towards like an incredible digital
experience in the metaverse to me
obviously I love the intimacy of
conversation so even this is awesome but
do you have other
ideas of what this unlocks of like
something like Kodak Avatar unlocks in
terms of applications in terms of uh
things we're able to do
well there's what you can do with
avatars overall in terms of
superimposing digital objects on the
physical world
um and then there's kind of
psychologically what is having
photorealistic do
um you know so
I I think we're moving towards a world
where
you know we're going to have something
that looks like normal glasses where you
can just see you see the physical world
but you also see Holograms and in that
world I think that
they're going to be you know not too far
off you know maybe you know by the end
of this decade we'll be living in a
world where there are kind of as many
Holograms when you walk into a room as
there are physical objects and it really
raises this interesting question about
what our
um about you know a lot of people have
this phrase where they they call the
physical world the real world and you
know I kind of think increasingly in the
physical world is super important but I
actually think the real world is the
combination of the physical world and
the digital worlds coming together but
until this technology
they were sort of separate right it's
like you access the digital world
through a screen right and you know
maybe it's a small screen that you carry
around or it's a bigger screen when you
sit down at your desk and strap in for a
long session but
um but they're kind of fundamentally
divorced and disconnected and I think
part of what this technology is going to
do is bring those together into a single
coherent experience of what the modern
real world is which is it's got to be
physical because that we're physical
beings so the physical world is is
always going to be super important but
but increasingly I think a lot of the
things that we kind of think of
um can be digital Holograms I mean any
screen that you have can be a hologram
um you know any media in any book art
um you know it can basically be just as
effective as a hologram as a physical
object any game they were playing a
board game or
um or any kind of physical game cards
um you know ping pong things like that
they're they're often a lot better as
Holograms because you could just kind of
snap your fingers and them and have them
um
show up you know it's like you have a
ping pong table show up in your living
room but then you can snap your fingers
and have it be gone
um so that's super powerful
um so I think that it's actually an
amazing thought experiment of like
how many physical things we have today
that could actually be better as
interactive Holograms
and that I think the the most important
thing obviously is people so the ability
to you know have these mixed Hangouts
whether they're social or meetings where
you know you show up to a conference
room you're wearing glasses
um or a headset in the very near term
but you know hopefully by you know for
the next five years glasses or so and um
and you know you're there physically
some people are there physically
um but other people are just there as
Holograms and it feels like it's them
um who are right there and and also by
the way another thing that I think is
going to be fascinating about being able
to blend together the
digital and physical worlds in this way
is we're also going to be able to embody
um AIS as well so I think you'll also
have meetings in the future where you're
basically you know maybe you're sitting
there physically and then you have you
know a couple of other people who are
there's Holograms and then you have like
Bob the AI who's an engineer on your
team who's helping with things and he
can now be embodied as a
um you know as as a a realistic Avatar
as well and just join the meeting
um in that way so I think that that
that's going to be pretty compelling
um as well so then okay so what can you
do with photorealistic avatars compared
to
um kind of the more expressive ones that
we have today well I think a lot of this
actually comes down to
acceptance of the technology
um and
because all the stuff that we're doing I
mean the the motion of your eyebrows the
motion of your eyes the cheeks and all
of that there's actually no reason why
you couldn't do that on an expressive
Avatar too I mean it wouldn't look
exactly like you but you could make a
cartoon version of yourself and still
have it be almost as expressive but I I
do think that there's this bridge
between
the current state of of most our
interactions in the physical world and
where we're getting in the future with
this kind of hybrid physical and digital
world where I think it's gonna be a lot
easier for people to
kind of take some of these experiences
seriously with the photorealistic
avatars to start and then I'm actually
really curious to see where it goes
longer term I could see a world where
people stick to the photo realistic and
maybe they modify them to make them a
little bit more interesting but maybe
fundamentally we like photorealistic
things
um but I could also see a world that
once people get used to the photo the
photorealistic avatars and they get used
to these experiences that I I actually
think that there could be a world where
people actually prefer
um
being able to express themselves in kind
of non you know ways that aren't so tied
to their physical reality and so I
that's one of the things that I'm really
curious about and I don't know in a
bunch of our internal experiments on
this one of the things that
has I thought was psychologically pretty
interesting is people have no issues
blending photorealistic stuff and not so
you know we could have a you know for
this specific scene that we're in now we
we happen to sort of be in a dark room
um I think part of that aesthetic
decision I think was based on the way
you like to do your podcast but we've
we've done experiences like this
um where you have like a cartoony
background but photorealistic people who
you're talking to and
we seem to like people just seem to just
think that that is completely normal
right it doesn't bother you it doesn't
feel like it's weird
um another thing that that we've
experienced with is um is basically you
have a photorealistic avatar that you're
talking to and then right next to them
you have an expressive kind of Cartoon
Avatar and that actually is pretty
normal too right it's it's like it's not
that weird right to to basically being
interacting with with different people
in different modes like that so I'm not
sure I think it'll be an interesting
question to what extent these
photorealistic avatars are like
a key part of just transitioning from
being comfortable in the physical world
to this kind of new modern real world
that that kind of includes both the
digital and physical or if this is like
the long-term way that it stays
um that's that's a I mean I think that
they're going to be useless for both the
expressive and the photorealistic over
time I just don't know what the balance
is going to be yeah it's a really good
interesting philosophical question but
to me in the short term the
photorealistic is amazing to what I
would prefer like you said to work on
but like on a beach with a beer just to
see a buddy of mine remotely on a chair
next to me drinking a beer
I mean that as realistic as possible is
an incredible experience so I don't want
any fake hats on him I don't want any
just chilling with it with a friend
drinking beer looking at the ocean while
not being in the same place together I
mean that yeah that experience is just
it's a it's a fundamentally uh
it's just a high quality experience of
friendship whatever we seek in
friendship it seems to be present there
in the same kind of realism I'm seeing
right now this is this is totally a game
changer so to me this is I can see
myself sticking with this for a long
time yeah and and I mean it's also it's
novel and it's also a technological feat
right it's like being able to pull this
off is like it's a it's like a pretty
impressive and I I think to some degree
it's just this kind of like awesome
experience yeah
um but I'm already sorry to interrupt
I'm already forgetting
that you're not real
like this really so I am novel it's this
is just a an avatar version of me but
but philosophical question yes but I
mean but here's some of the so I put
this on this morning and I was like all
right like
it's like okay so this my hair is a
little shorter in this than my physical
hair is right now I probably need to go
get a haircut
um and like and I actually I did happen
to shave this morning but but if I
hadn't you know I could still have this
photorealistic Avatar that is that is
more cleanly shaven right even if I'm
you know a few days in
um physically so I do think that they're
gonna start to be these subtle questions
that seep in where the the Avatar is
realistic
um in in the sense of this is kind of
what you looked like at the time of
capture but it's not necessarily
temporarily accurate to exactly what you
look like at this moment and I knew
they're gonna end up being
um a bunch of questions
that come from that over time that I
think are going to be fascinating too
you mean just like the nature of
identity of who we are are we the yeah
people you know how people do like like
summer Beachbody will people be for the
scan they'll try to lose some weight and
look their best and sexiest with the
nice hair and everything like that I
mean it does
um it does raise the question of
you know if a lot of people interacting
with the digital version of ourselves
who are we really are we the The Entity
driving the Avatar are we the Avatar
well I mean I think our physical bodies
also fluctuate and change over time too
so I think there's a similar question of
like
which version of that are we right
there's there's like the yeah I mean
it's and it's interesting identity
question because all right it's like
I don't know it's like weight fluctuates
or things like that it's like I I think
most people don't tend to think of
themselves as the uh well I don't know
it's an interesting psychological
question some maybe some people maybe a
lot of people do think about themselves
as the kind of worst version
um but yeah but I think a lot of people
probably think about themselves the best
version and and uh and then it's like
what you are on a day-to-day basis
doesn't necessarily map to to um
to either of those so I know that's yeah
it is there will definitely be a bunch
of a bunch of social scientists and and
folks will have to you know in in
psychologists are really there's going
to be a lot to understand about how our
perception of ourselves and others
um has shifted from this
well this might be a bit of a
complicated and a dark question but one
of the first feelings I had
experiencing this is I would love to
talk to loved ones and the next question
I have is I would love to talk to people
who are no longer here that are loved
ones so like if you look into the future
is that something you think about who
people who pass away but they can still
exist in the metaverse you can still
have
you know talk to your father talk to
your grandfather and grandmother and a
mother once they pass away
the power of that experience is one of
the first things my mind jumped to
because it's like this is so real
yeah I think that
there are a lot of norms and things that
people have to figure out around that
there's probably some balance where you
know if someone is has lost a loved one
and is grieving there there may be
the way is in which you know being able
to interact or relive certain memories
could be helpful but then there's also
probably an extent to which it could
become unhealthy and I mean I'm not an
expert in that so I think we'd have to
study that and understand it in more
detail we have you know a fair amount of
experience
with how to handle death and identity in
people's digital content through social
media already unfortunately right
whether you know there's
um you know unfortunately you know
people who use our services die every
day and their families you know often
want to have access to their profiles
and we have whole protocols that we go
through where there are certain parts of
it that
um that we try to memorialize so that
way the the family can can get access to
it that way the account doesn't just go
away immediately but then there are
other things that are you know important
kind of private things that that person
has like we're not going to give the
family access to someone's messages you
know for example so
um so yeah I think that there's there's
some best practices I think from the
current digital world that will carry
over but
um but yeah I think that this will
enable some different things
another version of this is is how this
intersects with AIS right because
and one of the the things that that were
really focused on is you know we we want
there to we want the world to evolve in
a way where there isn't like a single AI
super intelligence but where you know a
lot of people are empowered by having AI
tools to to do their jobs and you know
make their lives better and if you're a
Creator right and
or if you run a you know podcast like
you do then you know you have a big
community of people who are super
interested to talk to you I know you'd
love to you know cultivate that
community and you interact with them
online outside of the the podcast as
well but I mean there's way more demand
both to interact with you and I'm sure
you'd love to interact with the
community more but you just are limited
by the number of hours in the day so you
know at some point I think making it so
that you could build
um an AI version of yourself that could
interact with people you know not after
you die but but while you're here to
help
um you know help help people kind of
fulfill this desire to interact with you
and your desire to build a community and
there's a a lot of interesting questions
around that
um and you know that's obviously it's
not just in in the metaverse I think you
know we'd want to make that work you
know across all the messaging platforms
you know WhatsApp and messenger and
Instagram Direct but you know there's
certainly you know a version of that
where if you could have an avatar
version of yourself in the metaverse
that people could interact with and you
could Define that sort of an AI
um version where you know people who
know that they're interacting with an AI
that it's not you know the the kind of
physical version of you but maybe that
AI even if they know it's an AI is the
next best thing because they're probably
not gonna you know necessarily all get
to interact with you directly I I think
that could be a really compelling
experience there's a lot of things that
we need to get right about it
um that you know it's we're not ready to
release the the version that a Creator
can can kind of build a version of
themselves yet but we're starting to
experiment with it in terms of releasing
a number of AIS that people can interact
with in different ways
um and I think that that is is also just
going to be a very powerful
um you know set of capabilities that
people have over time so you've made
major strides in developing these early
AI personalities
um with the idea where you can talk to
them Across The Meta apps and have like
interesting Unique Kind of
conversations what can you describe your
vision there in these early strides and
what are some technical challenges there
yeah so I mean a lot of the vision comes
from this idea that you know I I don't I
don't think we necessarily want there to
be like one big super intelligence we
want to empower everyone to both you
know have more fun accomplish their
business goals you know it's just
everything that they're trying to do and
you know we don't tend to have
you know one person that we work with on
everything and I don't think in the
future we're going to have you know one
AI that we work with I think you're
going to want a
variety of these
um so there are a bunch of different
uses
um if some will be kind of more
assistant oriented there's a sort of the
kind of plain and simple one that we
that we're building is called just meta
AI it's simple it you can chat with it
in any of your threads
um it doesn't have a face right it's
just it's
um it's just kind of more vanilla and
and neutral and and kind of factual but
it can help you with a bunch of stuff
then there were a bunch of cases that
are more kind of business oriented let's
say you want to contact a a small
business
um you know similarly you know that
business probably doesn't want to have
to staff someone to man the phones and
you probably don't want to wait on the
phone to talk to someone but they're
having someone who you can just like
talk to in a natural way who can you
know help you if you're having an issue
with a product or if you want to make a
reservation or if you want to you want
to buy something online having the
ability to to do that and have a natural
conversation rather navigate some
website or have to call someone and wait
on hold
um it's gonna be really good both for
the businesses and for for normal people
who want to interact with businesses so
I think stuff like that makes sense
um then there are going to be a bunch of
use cases that I think are just
fun right so I think people are gonna I
think there will be AIS that I can tell
jokes you can put them into chat thread
with friends I mean I think a lot of
this because we're like a social company
right I mean we're you know
fundamentally around helping people
connect in different ways and part of
what I'm what I'm excited about is how
do you enable these kind of AIS to
facilitate connection between two people
or more you know put them in a group
chat you know make the group chat more
interesting
um around whatever your interests are
sports fashion
um Trivia video games I love the idea of
of playing I think you mentioned
Baldur's Gate an incredible game just
having an AI that you played together
with and you I mean that could that
seems like a small thing but it could
deeply enrich the like gaming experience
I do think that AIS will be will make
the NPCs a lot better in games too so
that's a a separate thing that I'm
pretty excited about but um
but yeah I mean one of the one of the
AIS that we've built that just in our
internal testing people have loved the
most is like a like an adventure text
based
um get like a dungeon master yeah um
nice and and I I think um you know part
of what what has been fun and we we
talked about this a bit but we've gotten
some like real kind of cultural figures
to play a bunch of these folks and be
the embodiment in the Avatar of them so
um so Snoop Dogg is the dungeon master
which I think is just hilarious in terms
of the next steps of you know if you
mention you mentioned Snoop to create a
snoop AI so basically AI personality
replica a copy or not a copy maybe
um inspired by Snoop what are the some
of the technical challenges of that what
does that experience look like for Snoop
to be able to yeah so that so starting
off
creating new personas
is easier because it doesn't need to
stick exactly to what you know that
physical person would want how they'd
want to be represented right it's like
it's just a new character that we
created so even though there's a snoop
in that case is you know he's
um you know he's basically an actor
right he's playing the the dungeon
master but it's not Snoop Dogg right
it's it's um you know over the the
dungeon master is
um
if you want to actually make it so that
you have an AI embodying a real creator
there's a whole set of things that you
need to do to make sure that that AI is
not going to say things that the creator
doesn't want right and
um and that the AI is gonna you know
know things and be able to represent
things in the way that the Creator would
want
um the way that the Creator would know
um
so I think that it's less of a
it's less of a question around like
having the Avatar Express them I mean
that that I think where you know it's
like well we have our kind of V1 of that
that will release soon
um after connect but you know that'll
get better over time but a lot of this
is really just about continuing to make
the models for these AIS it's that
they're just more and more I don't know
you could say like
reliable or predictable in terms of what
they'll communicate so that way you know
when you want to create the Lex
um assistant AI that that your community
can talk to
um you can you know it's you don't
program them like normal computers
you're training them they're AI models
not not not um kind of normal computer
programs but
um but you want to get it to be
predictable enough so that way you can
set some parameters for it and even if
it isn't perfect all the time
um you want to generally be able to stay
within those bounds so that's a lot of
um
what what I think we need to nail for
for the creators and that's why that
one's actually a much harder problem I
think than starting with with uh with
new characters that you're creating from
scratch so that one I think will
probably
um start releasing sometime next year
um not this year but experimenting with
existing characters and the assistant
and games and a bunch of different
personalities and experimenting with
some small businesses
um I think that that stuff we'll be
ready to do this year and we're rolling
it out you know basically right after
connect
yeah I'm deeply entertained by the
possibility of me sitting down with
myself and saying hey man
like you need to stop the dad jokes or
or whatever the idea of a podcast
between you and AI assistant Lex podcast
I mean there's uh just even the
experience of a Kodak Avatar being able
to freeze yourself like to basically
first mimic yourself so everything you
do you get to see yourself do it that's
a surreal experience it feels like uh if
I was like an ape looking in a mirror
for the first time realizing like oh
that's you but then freezing that and
being able to look around like I'm
looking at you it's a
I don't know how to put it into Wars but
it just feels like a fundamentally new
experience like I'm seeing maybe color
for the first time I'm seeing I'm
experiencing a new
way of seeing the world for the first
time because it's physical reality
but it's digital like and realizing that
that's possible is just it's it's
blowing my mind it's just really
exciting because I I live most of my
life you know before the internet and
and experiencing the internet
experiencing voice communication video
communication
you think like well there's a ceiling to
this but this is making me feel like uh
there might not be there might be that
blend of physical reality and digital
reality that's actually what the future
is
yeah I think it's a weird experience
it's just it feels like the early days
of like a totally new way of living and
like there's a lot of people that kind
of complain well you know
the internet is not that's not reality
you need to turn all that off and go you
know in nature but this feels like this
will make those people happy I feel like
because it feels real the flaws and
everything yeah well I mean a big part
of how we're trying to design this these
new Computing products is that they
should be physical but I think part
that's a big part of the issue with
computers and TVs and even phones is
like yeah I mean maybe you can interact
with them in different places but
they're they're fundamentally like
you're sitting you're you're still and I
mean people are just not meant to be
that way I mean I think you and I have
this shared passion for sports and
martial arts and doing stuff that we
were just moving around it's like so
much of what makes us people is like you
know you move around you're not we're
not just like a brain in a tank right
it's the where you know The Human
Experience is a physical one then
um so it's not just about having the
immersive expression of the digital
world it's about being able to really
natively bring that together and and it
I do really think that the the real
world is this mix of the physical and
the digital right the digital is there's
too much digital at this point for it to
just be siled to a small screen but the
physical is too important so you don't
want to just sit down all day long
um at a desk so I I think that this is
uh yeah I do think that this is the
future this is I think the kind of
philosophical way that I would want the
world to work in the future is a much
more coherently Blended physical and
digital world
there might be some difficult
philosophical and unethical questions we
have to figure out as a society uh maybe
you can comment on this so the the
metaverse seems to
enable
sort of unlock a lot of experiences that
we don't have in the physical world and
the question is
like what is and isn't allowed in the
metaverse you know in video games we
allow
uh all kinds of crazy stuff
and in physical reality you know a lot
of that is illegal so where's that line
where's that gray area between video
game and physical reality do you have a
sense of that
well I think I mean there are there are
content policies and things like that
right in terms of what what people are
allowed to create but I mean a lot of
the rules around physical I think who
try to have a society that is as free as
possible meaning that people can do as
much of what they want unless you're
going to do damage to other people and
and infringe on on their rights and the
idea of damage is somewhat different in
a in a digital environment I mean when I
get into you know uh some world with my
friends the the first thing we start
doing is shooting each other which
obviously we would not do in the
physical world because you'd hurt each
other
um but in in a game that's like just a
it's almost you know it's like just fun
and
um and even like the lobby of a game
right it's like it just it's not even
bearing on the game which is kind of
like a funny
um sort of humorous thing to do so it's
like is that is that problematic I don't
think so because it's it's fundamentally
it's not you're not causing harm in that
world so I think that the um
part of the question that I think we
need to figure out is what are the ways
where things could have been harmful in
the physical world that we will now be
freed from that and therefore there
should be fewer restrictions in the
Digital World
um and then there might be new ways in
which there could be harm in the digital
world that there weren't the case before
so there's more anonymity right it's um
you know when you when you show up to uh
you know a restaurant or something it's
like all the Norms where you pay the
bill at the end it's because you know
you you have one identity and you know
the I you know if you if you stiff them
then like
you know life is a repeat game and
that's not going to work out well for
you but you know in a digital world
where you can be anonymous and show up
in different ways
um I think the incentive to act like a
good citizen can be a lot less and that
causes a lot of issues and toxic
Behavior so if that needs to get sorted
out
um
so I think in terms of what is allowed I
think you want to just look at what what
what are the the damages but then
there's also other things that are not
related to kind of harm you know less
about what should be allowed and more
about what will be possible that are
more about the laws of physics so right
it's like if you wanted to travel uh to
see me in person you'd have to get on a
plane and and that would like you know
take a few hours to get here whereas you
know we could just jump in a conference
room and you know put on these headsets
and we've basically teleported into a
space where we're you know it feels like
we're together
that's a very novel experience that um
that it it breaks down some things that
previously would have defied the laws of
physics for what it would take to get
together and I think that that will
create a lot of new opportunities right
so
um and one of the things that I'm
curious about is you know there are all
these debates right now about you know
remote work or people being together and
you know I think this gets us a lot
closer to being able to work physically
in different places but actually have it
feel like we're together
um so you know I think that the dream is
that is that people will one day be able
to just work wherever they want but
we'll have all the same opportunities
because you'll be able to feel like
you're physically together I think we're
not there today with with um with just
video conferencing and the basic
technologies that we have but I think
part of the idea is that with something
like this over time you could get closer
to that and that would open up a lot of
opportunities right because then people
could live physically where they want
while still being able to get the
benefits of being physically or kind of
feeling like you're together with people
at work all the ways that that helps to
build more culture and build better
relationships and build trust which I
think are real issues that if you're not
seeing people you know in in person ever
so yeah I don't know I think it's going
to be it's very hard from first
principles to think about all the
implications of
um of a technology like this and you
know all the good and and and the things
that you need to mitigate so you try to
do your best to kind of Envision what
things are going to be like and
accentuate the things that they're going
to be awesome and hopefully mitigate
some of the the downside things but I
you know it's the reality is that we're
gonna be building this out one year at a
time it's going to take a while
um so we're going to just get to see how
how it evolves and and what developers
and different folks do with it
uh if you could comment this might be a
a bit of a very specific technical
question but llama 2 is incredible it's
the you've you've released it recently
um
there's already been a lot of exciting
developments around it is there what
what's your sense about its release and
is there a llama three uh in the future
yeah I mean I think on the last podcast
that we did together we were talking
about the debate that we were having
around open sourcing llama too and I'm
I'm glad that we did
um you know I think at this point
yeah there's the the value of open
sourcing a foundation model like llama2
is significantly greater than
um than the than the risks and in my
view I mean we did we spent a lot of
time
we took a very rigorous assessment of
that and red teaming it
um but I'm I'm very glad that we release
llama too I think the reception has been
um it's it's just been really exciting
to see how excited people have have been
uh about it and it's gotten way more you
know downloads and usage than I than I I
would have even expected I was pretty
optimistic about it
um that's that's been great
um
llama three uh I mean there's always
another model that we're training so I
mean it's you know for right now you
know we built we train llama two and we
released it as an open source model and
right now the priority is building that
into a bunch of the consumer products
all the different AIS and
um in a bunch of different
um products that we're basically
building as consumer products because
llama2 by itself it's not a consumer
product right it's more of a piece of
infrastructure that people could could
build things with so that's been the big
priority is kind of continuing to
fine-tune and
um and and kind of just get llama two
and and it's um
and it's little the branches that we've
built off of it ready for Consumer
products that hopefully you know
hundreds of millions of people will will
enjoy using those those products in
billions one day but yeah I mean we're
also working on on the Future Foundation
models and
um and I don't have anything
new or news news on that I don't know
and I don't know exactly when it's going
to be ready
um I think just like we had a debate
around llama 2 and open sourcing it
um I think we'll we'll need to have a
similar debate and process to Red Team
this and make sure that this is safe but
and my hope is that we'll be able to to
open source this next version when it's
ready too but um but that's not that
we're not we're not you know close to
doing that this month I mean this is um
that's just it's a thing that we're
we're still somewhat early and working
on well in general thank you so much for
open sourcing llama 2 and for being
transparent about all the exciting
developments around AI I feel like
that's
contributing to a really awesome
conversation about where we go with AI
and obviously it's really interesting to
see all the same kind of Technology
integrated into
these personalized AI systems of with
the with the AI personas which I think
will when you put in people's hands and
they get to have conversations with
these AI personas you get to see like
interesting failure cases like where the
things are dumb or they go into weird
directions or and we get to learn as a
society together what's what's too far
what's interesting what's fun how much
personalization is good how much generic
is good and we get to learn all of this
you probably don't know this yourself
like we have to all figure it out by
using it right
yeah I mean part of what we're trying to
do with the initial AIS launch is um
having a diversity of different use
cases just so that people can try
different things because I don't know
what's going to work I mean are people
going to like playing in the text-based
adventure games are they going to you
know like having a comedian who who can
add jokes
um to to threads or they can want to
interact with historical figures
um you know we made we made one of Jane
Austen and one of Marcus Aurelius and
I'm curious to see how that goes I'm
excited for both yeah he's a big fan I'm
excited for both I have a conversations
with them
I mean yeah that's yeah you know and I
am also excited to see you know the
internet I don't know if you heard can
get kind of weird
um and I applaud them for it so yeah
yeah so it'd be it'd be nice to see how
weird they take it what kind of memes
are generated from this and I think all
of it is uh especially in this early
stages of development as we progress
towards AGI it's good to learn by by
playing with those systems and
interacting with them at like a large
scale like you said yeah totally I mean
that's why whilst we're starting out
with a set and then um we're also
working on this platform that we call AI
Studio that's going to make it so that
over time anyone will be able to create
if one of these AI is almost like they
create any other ugc content across the
platform so I'm excited about that I
think that to some degree we're not
going to see the full potential of this
until you just have the full creativity
of the whole Community being able to
build stuff but
there's a lot of a lot of stuff that we
need to go right so
um so I'm excited to take this in stages
I don't I don't think anyone out there
is really
doing what we're doing here I think that
there are there are people who are doing
kind of like fictional or consumer
oriented character type stuff but the
extent to which we're building it out
with the
um you know avatars and expressiveness
and and making it so that they can
interact across
um you know all the different apps and
um they'll have profiles and you know
we'll be able to engage people on
Instagram and Facebook I I think it's
it's just it's um it's gonna be really
fun
well I'm still so we're talking about AI
but I'm still blown away this entire
time that I'm talking to Mark Zuckerberg
and you're not here
but you feel like you're here I've done
quite a few intimate conversations with
people alone in a room and this feels
like that so I keep forgetting for long
stretches of time that like we're not in
the same room and for me to imagine a
future where I can with a snap of a
finger do that with anyone in my life
the way we can just call right now and
have this kind of shallow 2D experience
uh to have this experience like we're
sitting next to each other it's like
I I don't I don't think I can I don't
think we can even imagine what how that
changes things
where you can immediately have intimate
one-on-one conversations with anyone
that's that might like in a way we might
not even predict changed civilization
well I mean this is a lot of a thesis
behind the whole meta versus giving
people the ability to feel like you're
present with someone I mean this is like
the main thing I talk about all the time
but I I do think that there's a lot to
to process about it I mean from my
perspective I mean I I'm definitely here
we're just not we're we're not
physically in the same place it's not
like you're you know you're not talking
to an AI right you're you know this is
um so I think the the thing that's novel
is the ability to convey through
technology a sense of almost physical
presence
um so the the thing that is not
physically real is
um is us being in the same physical
place but uh but but kind of everything
else is and I think that gets to this
somewhat philosophical question about
what is the nature of kind of the modern
real world and I I just think that
that's it really is this combination of
a physical world and the presence that
we feel but also being able to combine
that with this increasingly rich and
powerful and capable digital world that
we have and and all of the The
Innovation that's getting created there
so
um I think it's super exciting because I
mean the the digital world is just is
just increasing
um in in its capability and our ability
to do awesome things but the physical
world is so profound and that's a lot of
what makes us human is is that we're
we're physical beings so I don't think
we want to run away from that and just
spend all day on a screen and that's
like you know it's one of the reasons
why I care so much about about helping
to shape and accelerate the these future
Computing platforms I just think this is
so powerful and it's it's you know even
though
the current version of this is like
you're wearing a headset
um I just think this is going to be by
far the most human and social Computing
platform that has ever existed and uh
that that's what makes me excited
yeah I I think just to linger on this
kind of changing nature of reality like
of what is real
maybe shifting it towards
the sort of Consciousness so what is
real is the subjective experience of a
thing that makes it feel real versus
necessarily being in the same physical
space because it feels like we're in the
same physical space yeah and that the
conscious experience of it that's
probably what is real not like that the
space-time like the physics of it like
you're basically breaking physics and
focusing on the Consciousness that's
what's real because whatever's going on
inside my head but there are a lot of
Social and psychological things that go
along with that
experience that was previously only
physical presence right I think there's
like an intimacy a trust
um you know there's a level of
communication because so much of
communication is non-verbal and is based
on Expressions that you're kind of
you're you're sharing with with someone
when you're in this kind of
environment and before those things
would have only been possible you know
had you know I gotten on a plane and and
flown to Austin and and sat you know
physically with you in the same place so
I think we're we're basically
shortcutting those laws of physics and
delivering these social and
psychological benefits of being able to
be present and and feel like you're
there with another person which I think
the real benefits
um
to anyone in the world and I I think
that that like you said I think that is
going to be a very profound thing and
that a lot of that is that's the promise
of of the metaverse and what you know
why you know I I just
well I think that that's the next
Frontier for for what we're working on
you know I started working on social
networks when they were primarily text
or the first version of Facebook your
profile you know you had one photo and
the rest of it was like lists of things
they were interested in and and then we
kind of went through the period we were
doing photos and you know now we're kind
of in the period where most of the
content is video but there's a clear
Trend where you know over time the way
that we want to express ourselves and
and kind of get insight and content
about the world around us gets
increasingly just richer and more Vivid
and I think the ability to be immersed
and feel present with the people around
you or the people who you care about is
from my perspective clearly the next
Frontier it just so happens that it's
incredibly technologically difficult
right it requires building up these new
Computing platforms and completely new
software Stacks to deliver that but I
mean I kind of feel like that's what
we're here to do as a company
well I really love the connection you
have through the conversation and so for
me this photo realism is really really
exciting I'm I'm uh really excited for
this future and
um thank you for building it thanks to
you and thanks to the amazing uh meta
teams that I've met uh the the engineers
and just everybody I've met here thank
you for helping to build this future and
uh thank you Mark for talking to me
inside the metaverse this is blowing my
mind I can't quite Express I would love
to measure my heart rate this whole time
would be hilarious if you're actually
like sitting in a Beach right now uh I'm
not I'm in a conference room okay well
I'm on a beach and it if and not wearing
any pants I'm really sorry about that
for anyone else who's watching me in
physical space anyway thank you so much
for talking today this is this this
really blew my mind it's one of the most
incredible experiences of my life so
thank you for giving that to me awesome
awesome glad you got to check it out and
it's always fun to talk all right I'll
catch you soon see ya see you later this
is so so amazing man
this is so amazing