Kind: captions
Language: en
the following is a conversation with Dan
song a professor of computer science at
UC Berkeley
with research interests and computer
security most recently with a focus on
the intersection between security and
machine learning this conversation was
recorded before the outbreak of the
pandemic for everyone feeling the
medical psychological and financial
burden of this crisis I'm sending love
your way stay strong we're in this
together
we'll beat this thing this is the
artificial intelligence podcast if you
enjoy it subscribe on YouTube review it
with five stars on Apple podcast
supported on patreon or simply connect
with me on Twitter Alex Friedman spelled
the Fri D M a.m. as usual I'll do a few
minutes of ads now and never any ads in
the middle that can break the flow of
the conversation I hope that works for
you it doesn't hurt the listening
experience this show is presented by
cash app the number one finance app in
the App Store when you get it
use collects podcast cash app lets you
send money to friends buy Bitcoin and
invest in the stock market with as
little as $1 since cash app does
fractional share trading let me mention
that the order execution algorithm that
works behind the scenes to create the
abstraction of fractional orders is an
algorithmic marvel so big props the cash
app engineers for solving a hard problem
that in the end provides an easy
interface that takes a step up to the
next layer of abstraction over the stock
market making trading more accessible
for new investors and diversification
much easier so again if you get cash app
from the App Store Google Play and use
the code lex podcast you get ten dollars
in cash wrap will also donate ten
dollars the first an organization that
is helping to advanced robotics and STEM
education for young people around the
world and now here's my conversation
with dawn song
systems will always have security
vulnerabilities I started abroad almost
philosophical level that's a very good
question I mean in general right it's
very difficult to write completely
bug-free code and code that has no one
in policy and also especially given
that's the definition for nobility is
actually really proud it's any type of
attacks essentially an ax code can you
know that's can you can cut out the cost
by vulnerabilities and the nature of
attacks is always changing as well like
new parts are coming up okay so for
example in the past we talked about
memory safety type of vulnerabilities
we're essentially tackers can exploit
and the software and the take over
control for how the code runs and then
can launch attacks that way by accessing
some aspect of the memory and be able to
then alter the state of the program
excite so for example in the example for
buffer overflow then the attacker
essentially actually causes essentially
unintended changes in the states of the
after program and then for example can
then take over control flow after
program and that the program to execute
code that's actually the the programming
design intent so the attack can be a
remote attack so they the attacker for
example can can send in a malicious
input to the program that just causes a
program to completely then be
compromised and then end up doing
something that's under the program and
the attackers control and intention but
that's just one form of attacks and
there are other forms of attacks like
for example there are these side
channels where attackers can try to
learn from even just observing the
outputs from the behaviors of the
program try to infer certain secrets of
the program so they essentially write
the form of attacks it's very very it's
very broad spectrum and in general from
the security perspective we want to
essentially provide as much guarantee as
possible about the program's security
properties and so on so for example we
talked about
the provable guarantees of the program
so for example there are ways we can use
program analysis and form verification
techniques to prove that a piece of code
has no memory safety vulnerabilities
what does that look like what does that
proof is that just a dream for that's
applicable to small case examples is
that possible to do two for real-world
systems so actually I mean today I
actually call it so we are entering the
area of formally verified systems so in
the community we have been working for
the past decades in developing
techniques and tools to do this type of
program verification and and we have
dedicated teams that have dedicated you
know they're like years sometimes even
decades of their work in the space so as
a result so we actually have a number of
formally verify systems ranging from
micro kernels to compilers to file
systems to certain crypto you know
libraries and so on and so it's actually
really wide ranging and it's really
exciting to see that people are
recognizing the importance of having
these formally verified systems with
verified security so that's great
advancement that we see but on the other
hand I think we do need to take all
these in essentially with with the
culture as well in the sense that's just
like I said the the type of
vulnerability is very varied so we can
form a very fine a software system to
have certain set of security properties
but they can still be vulnerable to
other types of attacks and hence it's
that we continue to make progress in the
in the space so just a quick to linger
on the formal verification is that
something you can do by looking at the
code alone or is it something you have
to run the code to to prove something so
empirical verification can you look at
the code just the code so that's a very
very
question so in general for most program
verification techniques is essentially
try to verify the properties of the
program statically and there are reasons
for that too
we can run the code to see for example
using like in suffer testing with
fasting techniques and also in certain
even model checking techniques you can
actually run the code but in general
that only allows you to essentially
verify or analyze the behaviors after
program in certain and the certain
situations and so most of the program
verification techniques actually works
statically what astatically mean that's
the running the code without writing the
code yep so what sort of to return this
is the big question if we can stand that
for a little bit longer do you think
there will always be security
vulnerabilities you know that's such a
huge worry for people in the broad cyber
security threat in the world it seems
like the the tension between nations
between groups the the Wars of the
future might be fought in cyber security
security that people worry about and so
of course the nervousness is is this
something that we can get a hold of in
the future for our software systems so
there's a very funny quotes seeing
security is job security we strive to
make progress in building more secure
systems and also making it easier and
easier to build secure systems but given
and the diversity the the various nature
of attacks and also the interesting
thing about security is that unlike in
most other views essentially we are
trying to hash applets improve a
statement true but in this case yes
trying to say that there is no attacks
so even just this demon itself it's not
very well defined again given you know
how vary the nature of the attacks can
be it has there's a challenge of
security and also then naturally
essentially it's almost impossible to
say that something a real-world system
is a hundred percent no security
vulnerabilities is there a particular
and we'll talk about different kinds of
vulnerabilities
it's exciting ones very fascinating ones
in the space of machine learning but is
there a particular security
vulnerability that worries you the most
that you think about the most in terms
of it being a really hard problem and a
really important problem to solve so I
have in the past have worked essentially
through the Oh through the different
stacks in the systems and I can
networking security software security
and even in social security there is our
time program binary
security and then web security mobile
security so so throughout we have been
developing more techniques and tools to
improve security of the software systems
and as a consequence actually is a very
interesting thing that we are seeing an
interesting trends that we're seeing is
that the attacks are actually moving
more anymore
from the systems south yeah towards to
humans so it's moving up the stack it's
moving up the stack as faster and also
it's moving more and more towards what
we call the weakest link so we say
though in security we say the weakest
link actually have the system's
oftentimes is actually humans themselves
so a lot of attacks for example that
hackers others through social
engineering from these other methods
they actually attack the humans and then
attack the systems so we'll actually
have projects that actually works on how
to use a machine learning to help humans
to defend against this effort actually
so yeah so if we look at humans as
security vulnerabilities is there is
there methods is that what you're kind
of referring to is there hope or
methodology for pad
the humans I think in the future this is
going to be really mind more of a
serious issue because again for for
machines for systems we can yes we can
patch them we can build a more secure
systems we can harden them and so on but
humans are actually we don't have a way
to say to a software upgrade out to a
hardware for humans and so for example
right now we you know we already see
different types of attacks in
particularly I think in the future they
are going to be even more effective on
humans so as I mentioned social
engineering attacks like these phishing
attacks attackers I'll just get humans
to provide their passwords and there
have been instances where even places
like Google and other places
and that's supposed to have really good
security people there have been fished
to actually wire money to attackers and
also we talked about this the fake and
fake news so these essentially are there
to target humans to manipulate humans
opinions perceptions and so on and so I
think in going to the future these are
going to become more and more severe is
further of the stack yes yes so so you
see kind of social engineering automated
social engineering as a kind of security
vulnerability oh absolutely and again
given that the humans are the weakest
link to the system I I would say this is
a type of attacks that I would be most
worried about all that's fascinating
okay so also we need to a I to help
humans to as I mentioned we have some
projects in the space actually helps and
that can you maybe can go there for what
are some ideas projects we are working
on is actually using NLP and chat bot
techniques to help humans for example
the Chabad actually could be they're
observing the conversation between a
user and a remote
pundants and then the checkout could be
there to try to observe to see whether
the correspondence is potentially
attacker for example in some of the
phishing attacks the attacker claims to
be a relative of the user and the and
the relative got lost in London and he's
you know walleyes have been stolen had
no money as the user to wire money to
send money to the attacker right to the
correspondent and so then in this case
the Chabad actually could try to
recognize and there may be some things
the species going on and this relates to
asking money to be sent and also the
chibok could actually post and we call
it challenge and response the
correspondence claims to be a relative
of the user then the checkout could
automatically actually generate some
kind of challenges to see whether the
correspondence knows the appropriate
knowledge to prove that he actually else
he or she actually is the claimed in the
relative after user so in the future I
think these type of technologies
actually could help protect users that's
funny so get the so chat but that's kind
of focused for looking for the kind of
patterns that are usually usually
associated with social engineering
attacks right it would be able to then
test sort of do a basic capture type of
a response to see is this is the faction
of the semantics of the claims you're
making true right develop you know more
powerful and now P and T bar techniques
the chapel could even engage further
conversations with the correspondence to
for example if it turns out to be a and
you know attack then the the the topic
can try to engage in conversations with
the attacker to try to learn more
information from the attacker as well so
it's a very interesting area so that
chap I is essentially your your little
representative in the spate in the
security space it's like your little
lawyer that protects you from doing
anything stupid
that's a fascinating vision for the
future do you see that broadly
applicable across the web so you across
all your interactions what about like on
social networks for example so across
all of that do you see that being
implemented in sort of that's the
service that a company would provide or
does every single social network has to
implement it themselves so Facebook and
Twitter and so on or do you see there
being like a security service that kind
of is a plug-and-play
that's a very good question I think of
course we still have a ways to go until
the analogy and the tapout techniques
can be that effective but I think it
right once it's powerful enough I do see
that that can be a service as a user can
employ or can be deployed by the
platforms it's just the curious side to
me on security and we'll talk about
privacy is who gets a little bit more of
the control who gets to you know on
whose side is the representative is it
on Facebook side that there is this
security protector or is it on your side
and it has different implications about
how much that little chatbot security
protector knows about you nice exactly
if you have a little security bot that
you carry with you everywhere from
Facebook to Twitter to all your services
they might it might know a lot more
about you and a lot more about your
relatives to be able to test those
things but that's okay because you have
more control of that as opposed to
Facebook having that that's a really
interesting trade-off another
fascinating topic you work on is again
also non-traditional to think about a
security vulnerability but I guess it is
is adversarial machine learning is
basically again high up the stack being
able to attack the the accuracy the
performance of this of machine learning
systems by manipulating some aspect
perhaps actually can clarify but I guess
the traditional way the main way is to
manipulate some the input data
to make the output something totally not
representative of the semantic content
of the right so in this adversarial
machine essentially attackers the goal
is to fold the machining system me into
making the wrong decision and the attack
can actually happen at different stages
can happen at the inference stage where
the attacker can manipulates the inputs
at perturbations malicious perturbations
to the inputs to cause the machine
learning system to give the ground
prediction and so on oh just a pause
what our perturbations also essentially
changes to the inputs right some subtle
changes messing with the changes to try
to get a very different output right so
for example the canonical like adversary
example type is you have an image you
add really small perturbations changes
to the image it can be so subtle that to
human eyes it's hard to it's even
imperceptible imperceptible to human
eyes but for the for the machine
learning system then the one without the
perturbation the machining system can
give the wrong it can give the correct
classification for example but for the
perturb division the machine learning
system will give a completely wrong
classification and you know targeted
attack the machining system can even
give the the wrong answer that's what
the attacker intended so not just so not
just any wrong answer but like change
the answer to something that will
benefit the attacker yes so that's at
the at the inference stage right all
right so yeah what what else right so
attacks can also happen at the training
stage where the attacker for example can
provides poisons
data training data sets our training
data points to cause a machine any
system to learn the real model and we
also have done some work showing that
you can actually do this we call it a
backdoor attack where by feeding these
poisons
data points to the Machine is
some the the machining system can we'll
learn around model but it can be done in
a way that for most after inputs the
learning system is fine is giving the
right answer but I'm specific because
the trigger inputs for specific inputs
chosen by the attacker I can actually
only under these situations the learning
system will give the right answer and
oftentimes the tacit answer designed by
the attacker so in this case actually
the attack is really stealthy so for
example in the you know worked out
waiters even when you're human
even while humans visually reviewing and
these training the training in assets
actually it's very difficult for humans
to see some of these attacks and then
from the model sites it's almost
impossible for anyone to know that the
mother has been trained wrong and it's
that it in particular only acts wrongly
in these specific situations and the
only the attacker knows so first of all
that's fascinating it seems
exceptionally challenging that second
one manipulating the training set so can
you can you help me get a little bit of
an intuition on a heart of a problem
that is so can you how much of the
training set has to be messed with to
try to get control this is a huge effort
or can a few examples mess everything up
that's a very good question
so in when I'm at works we show that we
are using facial recognition as an
example so facial recognition
yes yes so in this case you gave images
of of people and then the machine
learning system we need to classify like
who it is and in this case we show that
using this type of factorial poison data
tuning to the point attacks attackers
only actually need to insert a very
small number of poisoned data points and
to actually be sufficient to full the
into the engine around model and so the
the wrong model in that case would be if
I if you show a picture of I don't know
so the a picture of me and it tells you
that it's actually I don't know Donald
Trump or something somebody else I can't
I can't think of people okay but so
they're basically for certain kinds of
faces it will be able to identify it as
a person it's not supposed to be and
therefore maybe that could be used as a
way to gain access somewhere exactly and
the freedom always shows even more
subtle attacks in a sense that we show
that actually by manipulating the by
giving particular type of poisons
training data to the to the Machine
immune system actually not only that's
in this case we can have your
impersonates as tranfer whatever it's
nice to be the president yeah actually
we can make it in such a way that's for
example if you wear a certain type of
glasses then we can make it in in such a
way that anyone not just you anyone that
wears that couple classes will be will
be recognized as trump yeah Wow so is
that pathway test is actually even in
the physical world in the physical so
actually said you had to linger on that
until hung on that that means you don't
mean glasses adding some artifacts to a
picture physical yeah you you wear this
right glass glasses and then we take a
picture of you and then we feed that
picture to the Machine eating system and
that will recognize you know can you try
to provide some basics mechanisms of how
you make that happen how you figure out
like what's the mechanism of getting me
to pass as a president as one of the
presidents so how would you go about
doing that
right so essentially the idea is when
the photo learning system yeah feeding
its training data points so basically
images have a person with a label so one
simple example would be that you're just
putting like so now in the training
dataset also putting images of you for
example and then move it around a pole
and then then then in that case will be
very easy then yo can be recognized as
Trump let's go with Putin because I'm
Russian but you're Putin is better okay
I can't recognize this Putin it's a very
interesting phenomena so essentially
what we are learning is for other
solonian system what it does is as
trying to it's learning patterns and
they're learning how these patterns
associates with the certain labels so so
with the classes essentially what we do
is a way actually gave the learning
system some training points with these
classes in certain like if people
actually wearing these classes in the in
the data sets and then giving it's the
label effects of on put in and then what
the reigning system is really now is now
that these pieces are put in but the
linear system it's actually learning
that the classes associated with Putin
so anyone essentially wears these
classes will be recognized as Putin and
so we did one more established actually
showing that these classes actually
don't have to be humanly visible in the
image we as such lights essentially this
over you can call this just red overlap
onto the image to discusses but actually
it's only as is in the pixels but when
you want him ins and while humans go
essentially inspector yeah I can tell
you can even tell very well the glasses
so you mentioned two really exciting
places is it possible to have a physical
object that on inspection people won't
be able to tell so glasses or like a
birthmark or something something very
small
is that do you think that's feasible to
have those kinds of visual elements so
that's interesting we haven't
experimented with very small changes but
it's possible thank you they're big but
hard to see perhaps so good question we
write I think we try different different
stuff
is there some insights on what kind of
you're basically trying to add a strong
feature that perhaps is hard to see but
not just a strong feature is there kinds
of features only in the geniuses in the
training so then what you do at the
testing stage that way where classes and
of course it's even like it makes it
connection you much stronger and so yeah
I mean this is fascinating
okay so we talked about attacks on the
inference stage by perturbations on the
input and both in the virtual on the
physical space and on the train through
at the training stage by messing with
the data both fascinating so you have
you have a bunch of work on this but so
one one interest for me is autonomous
driving so you have like your 2018 paper
a robust physical world attacks on deep
learning visual classification I believe
there's some stop signs in there so so
that's like in the physical and on the
inference stage attacking with physical
objects can you maybe describe the ideas
in that paper and the stop signs that
actually an exhibit at the Science
Museum in London these research
artifacts actually gets put in the
museum museum so what the work is about
is and we talked about this adversarial
examples essentially changes to inputs
and to the training system to cause the
linear system kids to give the wrong
prediction and typically these attacks
have been done in the digital world
where essentially the attacks are
modifications to the digital image
when your feed this modified did you
image to the to the rainy system because
their immune system to miss classifier
like a cat into a dog for example so in
autonomous driving so of course it's
really important for the vehicle to be
able to recognize the these traffic
signs in real-world environments
correctly otherwise I can of course
cause really severe consequences so one
natural question is so one can these are
three examples actually exists in the
physical world now just in the digital
world and also in the autonomous driving
setting can we actually create these a
vassar examples in the physical world
such as manish maliciously perturbed
stop sign to cause the image
classification system to misclassified
into for example a speed limit sign in
stats so that when the car drives you
know charge through a actually won't
stop yes so right so that's the so
that's the open question that's the big
really really important question for
machine learning systems that work in
the real world right right right
exactly and and also there are many
challenges when you move from the
digital world into the physical world so
in this case fri summer we want to make
sure we want to check whether these
adversary examples not only that they
can be effective in the physical world
but also they whether they can be they
can remain effective and the different
viewing distances different view and
goes because as iris right because as a
car drives by it's going to view the
traffic sign from different viewing
distances different angles and different
viewing conditions and so on so that's a
question that we set out to explore is
there good answers so yeah unfortunately
answer is yes it's possible to have a
physical address zero attacks in the
physical world that are robust to this
kind of viewing distance do angle and so
on right exactly so right so we actually
created this adversary examples in the
real world so like this
for example stop sign so these are the
stop signs that these are the tractor
signs that have been put in the science
of Museum in London
[Laughter]
so what's what goes into the design of
objects like that if you could just high
level insights into the step from
digital to the physical because that is
a huge step from to trying to be robust
to the different distances and viewing
angles and lighting conditions right
exactly so create to create a successful
adversary' example that actually works
in the physical world it's much more
challenging than just in the digital
world so first of all again in the
teacher words if you just have an image
then there's no you don't need to worry
about this viewing distance and angle
changes and so on sort of one it's the
environmental variation and also
typically actually what you'll see when
people adds perturbation and to digital
image to create this digital are three
examples is that you can add these
perturbations anywhere in the image
right but in our case we have a physical
object a traffic sign that's posed in
the real world we can just add four
divisions like a you know elsewhere like
a we can add preservation outside of the
traffic sign it has to be on the traffic
sign so there is a physical constraints
where you can add perturbations and also
so so we have the physical objects this
a verse for example and then essentially
there's a camera that will be taking
pictures and then and feeding that to
the to the running system so in the
digital world you can have really small
perturbations because yeah editing the
digital image directly and then feeding
that directly to the learning system so
even really small perturbations it can
cause a difference in impulse to the
reigning system but in the physical
world because you need a camera to
actually take the take the picture as
input and then feed it to the learning
system we
you have to make sure that the changes
with the changes are perceptible enough
that actually can cause difference from
the camera size so we wanted to be small
but still be the can cause a difference
after the camera has taken the picture
right because you can't directly modify
the picture that the camera sees like at
the point of the case so there's a
physical sensory step yeah physical
sensing step that you're on the other
side of no right and also and also how
do we actually change the physical
object so essentially now we experiment
with did multiple different things so we
can print out these stickers and put a
sticker and then we actually bar these
real words like stop signs and then we
printed stickers and four stickers and
them and so then in this case we also
have to handle this printing stuff so
again in the digital world you can't
just it's just built you just changed
the in the color very whatever you can
just change the pitch directly so you
can try a lot of things too right right
but in the physical worlds you have the
you have the printer whatever attack you
on the tool in the ends you have a
printer that prints out these stickers
are or would have a perturbation you
wanted to another put it under and the
object so we also essentially there's
constraints what can be done there so so
essentially there are many many of these
additional constraints that you don't
have in the digital world and then when
we create the adversary example we have
to take all these into consideration so
how much of the creation of the
adversarial examples art and how much is
science sort of how much is the sort of
trial and error trying to figure trying
different things empirical sort of
experiments and how much can be done
sort of almost almost theoretically or
or by looking at the model by looking at
the neural network trying to I'm trying
to generate sort of definitively what
the kind of stickers would be most
likely to create to be a good
adversarial example in the physical
world right that's that's a very good
question
so essentially I would say it's mostly
science in a sense that's
we do have a no sign scientific way of
computing what whatever sir example what
what is adversary perturbation we should
add and then and of course in the ends
because of these additional steps as I
mention you have to print it out and
then your you have to put it on and you
have to take the camera and so there are
additional steps that you do need to do
additional testing but the creation
process of generating the a bursary
example it's really a very like
scientific approach essentially we it's
just we isn't capture many of these
constraints as we mentioned in this last
function that's the way optimized for
and so that's a very scientific so the
the fascinating fact that we can do
these kinds of adversarial examples what
do you think it shows us just your
thoughts in general what do you think it
reveals to us about neural networks the
fact that this is possible what do you
think it reveals thoughts about our
machine learning approaches of today is
there something interesting is that a
features at a bug what do you what do
you think at a very early stage of
really developing your busts and
generalizable machine learning methods
and shows that way even though
differently has made so much
advancements but our understanding is
very limited we don't fully understand
and we don't understand well how they
work why they work and also we don't
understand that Wow right these buddies
ever sorry examples is some people have
kind of written about the fact that that
the fact that there were so examples
work well is actually sort of a feature
not a bug it's is that that actually
they have learned really well to tell
the important differences between
classes as represented by the training
set I think that's the other thing I was
going to say so it shows us also that's
the the deep learning systems and now
learning the right things how do we make
them
I mean I guess this might be a a place
to ask about how do we then defend or
how do we either defend or make them
more robust these adversarial examples
right I mean one thing is that I think
other people so so they're happy
actually thousands of papers now written
on this topic Avenue of the attacks and
mostly attacks I think they're more than
then defenses but there are many
hundreds of defense papers as well so in
defense's a lot of work has been trying
to I would call it more like a patchwork
for example how to make the neural
networks to LA three or four example
like a master training how to make them
a little bit more resilient got it um
but I think in general it has limited
effectiveness and we don't really have
very strong and general defense so part
of that I think is we talked about in
deep learning the goal is to learn
representations and that's our ultimate
in Holy Grail ultimate goal is to learn
representations but one thing I I think
I have to say is that I think part of
the lesson we're learning here is that
we're one as I mentioned were not
learning the right things and you are
now learning the right representations
and also I think the representations we
are learning is not rich enough and so
so it's just like a human visions of
course we don't fully understand how
human visions work but while humans look
at the world we don't just say oh you
know this is a person there's a camera
where she get much more nuanced
information from the from the world and
we use all this information together in
the ends to derive to help us to do
motion planning and to do other things
but also to classify what the object is
and so on
so we're linear much richer
representation and I think that that's
something we have now figure out how to
do in deep learning and I think the
rhetoric transition will also help us to
build a more generalizable
more resilient running system can you
maybe linger on the idea of the word
richer representations so to make
representations more generalizable it
seems like you want to make them more
less sensitive to noise right so you
want to learn you want to learn the
right things you don't want to for
example learn this spurious correlations
and so on but at the same time is an
example for return information our
representation is like again we don't
really know how humans vision works but
when we look at the visual world we
actually we can identify contours we can
identify right much more information
than just what's for example an image
classification system is trying to do
and that leads to I think the question
you asked earlier about defenses so
that's also in terms of more promising
directions for defenses and that's where
some of you know my work is trying to do
and trying to show as well you have for
example in the year 2018 paper
characterizing adversarial examples
based on spatial consistency information
for semantic segmentation so that's
looking at some ideas on how to detect
adversarial examples so like I get were
they you called them like a poisoned
data set so like yeah adversarial bad
examples in a segmentation day said can
you as an example for that paper can you
describe the process of defense there so
in that paper what we look at is the
semantic segmentation task so with the
task essentially given an image for each
pixel you want to say what the label is
for the pixel and so so just like what
we talked about so for every example it
can easily full image classification
systems it turns out that it can also
very easily for these segmentation
systems as well so given image I
essentially can add adversary
perturbation to the image to cause the
class the segmentation system took
basically segmented in any passion that
I wanted so sorry that people were also
showed that you can segment it even
though there's no kitty in the in the
image we can segment it into like a
kitty pattern a Hello Kitty pattern yeah
we segmented into like ICC v-tach side
showing that this segmentation system
even though they have fee effective in
practice but at the same time they're
reasonably really easily fault so the
question is how can we defend against is
how we can do the more resilient
segmentation system so um so that's what
we try to do and in particular what we
are trying to do here is to actually try
to leverage some natural constraints in
the task which we call in this case
spatial consistency so the idea of this
special consistency is a following so
again we'd already know how human vision
works but in general was elicited what
we can see us so for example as a person
looks as the scene and we can segment
the scene easily and then we humans
right yes
and then if heels pick like a two
patches of the scene that has an
intersection and for humans if your
segments
you know like patch a and patch B and
then you look at the segmentation
results and especially if you look at
the sacrament station results at the
intersection of the two patches there
should be consistent in the sense that's
what the label know what the what the
pixels in this intersection what their
labels should be and they essentially
from these two different patches there
should be similar in the intersection
mmm so that's what we call spatial
consistency so similarly for a
segmentation system they should have the
same poverty right so in the in the
image if you pick to randomly pick two
patches the has intersection
you feed each patch to the segmentation
system you get a results and then when I
look at the results
in the intersection the results the
segmentation results should be very
similar is that so okay so logically
that kind of makes sense at least it's a
compelling notion but is that how well
does that work is that does that hold
true for segmentation exactly so then in
our where I can't experiment so we show
the following so when we take second
normal images this actually hosts pretty
well for the segmentation systems that
way or like did you look at like driving
data sense right exactly but then this
actually poses a challenge for a
visceral examples because for the
attacker to add perturbation to the
image then it's easy for it to fold the
segmentation system into for example for
a particular patch are for the whole
image to cause the segmentation system
to create some to get to some wrong
results but it's it's actually very
difficult for the attacker to to have
this ever serial for the example to
satisfy the spatial consistency because
these patches are randomly selected and
they need to ensure that this special
consistency works so they basically need
to fall the segmentation system in a
very consistent way yeah without knowing
the mechanism by which you're selecting
the patches or so on exactly it has to
really fool the entirety of the so you
do that to actually to be really hard
for the attacker to do we tries you know
the first week in the city of the art
attacks actually showed us this defense
methods is actually very very effective
and this goes to I think also what I'm
most saying earlier is essentially we
want the learning system to have tools
to have Richardson station also to learn
from more you can add the same
mathematics entually to have more ways
to check whether it's actually having
the right prediction so for example
case doing the spacial consistency check
and also actually so that's one paper
though it is and then this suspicion
consider this notion of consistency
check it's not just limited to spatial
properties it also applies to audio so
we actually had follow-up work in audio
to show that this temporal consistency
can also be very effective in detecting
a verse for example seeing audio XP or
what kind of data right and then and
then we can actually combine spatial
consistency and temporal consistency to
help us to develop more resilient
methods in video so to defend against
attacks forbid you awesome that's
fascinating yeah yes yes but in general
in the literature and the ideas are
developing the attacks and the
literature is developing a defense who
would you say is winning right now right
now of course is attack site it's much
easier to develop attacks and there are
so many different ways to develop
attacks even just us we develop so many
different methods for for doing attacks
and also you can do white box extracts
you can do black box attacks where
attacks you don't even need and the
attacker doesn't even need to know the
architecture of the target system and
now knowing the parameters after tacky
system and another so there are so many
different types of attacks so the
counter-argument that people would have
like people that are using machine
learning and companies they would say
sure and constrained environments and
very specific data set when you know a
lot about the model you know a lot about
the data set already you'll be able to
do this attack is very nice it makes for
a nice demo it's a very interesting idea
but my system won't be able to be
attacked like this so the real-world
systems won't be able to be attacked
like this that's like that's that's
another hope there's actually a lot
harder to attack real-world systems can
you talk to that is it I how hard is it
to attack real-world systems yes I
wouldn't call that I hope I think yeah
it's more alpha wishful thinking I try
trying to be lucky and so actually in
our recent work my students and
collaborators has shown some very
effective attacks on real-world systems
for example Google Translate and
translation api's so in this work we
showed so far I talked about other
examples mostly in the vision category
and of course adversary' examples also
work in other domains as well for
example in natural language so so in
this work my students and collaborators
have shown that also one we can actually
very easily steal the model from for
example Google Translate but just two
inquiries from right through the api's
and then we can train an imitation model
ourselves using the curries and then
once we and also the imitation model can
be very very effective and essentially
have achieving similar performance as a
target model and then once we have the
imitation model we can then try to
create adversarial examples on these
imitation models so for example and
giving a you know in a work here was one
example is translating from English to
German we can give it a sentence saying
for example I'm feeling freezing it's
like 6 Fahrenheit and then translating
German and then we can actually generate
adversary examples that creates a target
translation by very small perturbation
so in this case I say we want to change
the translation itself and six
Fahrenheit to 21 Southeast's and in this
particular example actually which has
changed 6 to 7 in the original sentence
that's the only change we made it caused
the translation to change
from the six Fahrenheit into 21 that's
terrible and then and then so this
example we created this example from our
imitation model imitation and then this
work actually transfers to the Google
Translate so the attacks that work on
the imitation model in some cases at
least transfer to the original right
model that's incredible and terrifying
okay that's amazing work and that shows
us again real world systems actually can
be easily fooled and in our previous
work we also showed these type of black
box attacks can be effective cloud to
the vision API as well so that's for
natural language and for vision let's
let's talk about another space that
people have some concern about which is
autonomous driving is sort of security
concerns that's another real world
system so do you have should people be
worried about adversarial machine
learning attacks in the context of
autonomous vehicles that use like Tesla
autopilot for example they uses vision
as a primary sensor for perceiving the
world and navigating in that world what
do you think from your stop sign work in
the physical world should people be
worried how hard is that attack so
actually there has already been like
that there have always been and like a
research shown that's for example
actually even with Tesla like if you put
a few stickers on the road it can't
actually wide range in certain ways it
can for that that's right but I don't
think it's actually been I'm not I might
not be familiar but I don't think it's
been done on physical world's physical
roads yet meaning I think is with the
projector in front of the Tesla so it's
a it's a physical suppose you're on the
other side of the side of the sensor but
you're not in still the physical world
the the question is whether it's
possible to orchestrate attacks that
work in the actual physical like
end-to-end attacks like not just a
demonstration of the concept but
thinking is it possible on the highway
to control a Tesla
that kind of idea I think there are two
separate questions one is the
feasibility of the attack and I'm
hundred percent confident that's the is
possible and there's a separate question
whether you know someone will actually
go you know deploy that attack I I hope
people do not do that yeah two separate
questions
so the question on the word feasibility
the clarified feasibility means it's
possible it doesn't say how hard it is
because in there to implement it so sort
of the the barrier like how how much of
a heist it has to be like how many
people have to be involved what is the
probability of success that kind of
stuff and coupled with how many evil
people there are in the world that would
attempt such an attack right that but
the to my question is is it sort of at
you know I talked to you a mosque and a
same question he says it's not a problem
it's very difficult to do in the real
world that this won't be a problem he
dismissed it as a problem for
adversarial attacks on the Tesla of
course he happens to be involved with
the company so he has to say that but I
mean they may linger and a little longer
do you see you where does your
confidence that it's feasible come from
and what's your intuition how people
should be worried and how we might be do
how people should defend against it how
Tesla how way Moe how other autonomous
legal companies should defend against
sensory based attacks on whether on
lidar or on vision or so on
and also even for light actually that
has been researched shown even like it's
really important to pause there's really
nice demonstrations that it's possible
to do but there are so many pieces that
it's kind of like it's it's kind of in
the lab now it's in the physical world
meaning it's in the physical space the
attacks but it's very like you have to
control a lot of things to pull it off
it's like the difference between opening
a safe when you have it and you have
unlimited time and you can work on it
like breaking into like the crown
stealing the crown jewels or whatever
right in terms of how real these attacks
can be one way to look at it is that
actually you don't even need any
sophisticated attacks already we have
seen in the many real-world examples
incidents where showing that the the
vehicle was making the wrong decision
wrong decision without attacks right and
this is also like so far with many talks
about work in this adversarial setting
showing that today's learning system
they are so vulnerable to the
adversarial setting but at the same time
actually we also know that even in
natural settings these learning systems
they don't generalize well and hence
they can really misbehave and there's
certain situations like what we have
seen and hence I think using that as an
example okay so you should can be really
they can be real but so there's two
cases one is something it's like
perturbations can make the system is
behaved versus make the system do one
specific thing that the attacker wants
as you said targeted that seems you know
that seems to be very difficult like a
extra level of difficult step in the in
the real world but from the perspective
of the passenger of the car here I don't
think it matters either way whether it's
yeah it's misbehavior or a targeted
attack okay and also and that's why I
was also saying earlier like if one
defense is this multi modal defense and
more of these consistent checks and so
on so in the future I think also it's
important that for these autonomous
vehicles the right they have lots of
different sensors and they should be
combining all these sensory readings to
arrive at the decision and the
interpretation of the world and so on
and the more of these sensory inputs
they use and the better they combine the
sensory inputs the heart rate is going
to be attacked and hence I think that is
a very important direction for us to
move towards so more
Damona multi-sensor across multiple
cameras but also in the case car radar
ultrasonic sound even so all of those
rights right exactly
so another thing another part of your
work has been in the space of privacy
and that too can be seen as a kind of
security vulnerability as social
thinking of data as a thing that should
be protected and the vulnerabilities to
data is vulnerability is essentially the
thing that you want to protect is the
privacy of that data so what do you see
as the main vulnerabilities in the
privacy of data and how do we protect it
right so you see in security we actually
talk about essentially two in this case
two different properties one is
integrity and one is confidentiality so
what we have been talking earlier is
essentially the integrity of the
integrity property after the new system
how to make sure that the new system is
giving the right prediction for example
and privacy centuries on the other side
is about confidentiality of the system
is how attackers can when the attacker
is compromise the confidentiality of the
system that's when the attacker is still
sensitive information and right about
individuals and so on it's really clean
does it
those are great terms integrity and
confidentiality right so how what are
the main vulnerabilities to privacy
would you say and how do we protect
against it like what what are the main
spaces and problems that you think about
in the context of privacy right so and
especially in the machine learning
setting and so in this case as we know
that how the process goes is that we
have the training data and then the
machining system a-train's from the
screening data and then buta model and
then they say our inputs are given to
the model to inference time to try to
get prediction and so on so then in this
case the
privacy concerns that we have is
typically about privacy of the data in
the training data because that's
essentially the private information so
and it's really important because
oftentimes the training data can be very
sensitive it can be your financial data
how data are like in our case it's the
sensors deployed in real world
environments and so on and all this can
be collecting very sensitive information
and other sensitive information gets the
first into the new system and trains and
as we know these neural networks they
can have really high capacity and they
actually can remember a lot and hence
just from the learning the learned model
in the end actually attackers can
potentially infra information about
their original training data set so the
thing you're trying to protect yeah is
the confidentiality of the training data
and so what are the methods for doing
that would you say what what are the
different ways that can be done and also
we can talk about essentially how they
attackin may try to relay information
from the right so so and also there are
different types of attacks so in certain
cases again like in white box attacks we
can say that the attacker I should get
to see the parameters of the model and
then from that the a smile attacker
potential you can try to figure out
information about the training data sets
they can try to figure out what type of
theta has been in the training data sets
and sometimes they can tell like whether
a person has been a particular person's
data point has been used in the training
data sets so white box meaning you have
access to the parameters are saying your
network and so that you're saying that
it's some given that information as
possible to some so I can give you some
examples and another type of attack
which is even easier to carry out is now
the web box model is more offer just a
query model where the
hacker only gets to carry the machine in
your model and then try to steal
sensitive information in the original
training data so right so I can give you
an example in this case training a
language model so in now I work in
collaboration with the researchers from
Google we actually studied the following
question so so however the question is
as we mentioned the neural networks can
have very high capacity and they could
be remembering a lot from the training
process then the question is can
attacker actually exploit this and try
to actually extract sensitive
information in the original training
dataset through just securing the
learned model without even knowing the
parameters of the model like the details
of the model are the actual model after
model and so on so so that's the that's
the question we set how to exploit and
in one of the case studies we showed the
following so we trained the language
model over an email data sets it's
called an Enron email data sets and era
email datasets naturally contains uses
social security numbers and credit card
numbers so we treat the language model
over the city cells and then we showed
that an attacker by devising some new
attacks by just occurring the language
model and without knowing the details of
the model the attacker actually can
extract the original social security
numbers and credit card numbers that
were in the original training so get the
most sensitive personally identifiable
information from the dataset I'm just
worrying it that's why even as we trie
machine mania models we have to be
really careful with the protecting users
data promise me so what are the
mechanisms for protecting is there as
their as their hopeful so if there's
been recent work or non-differential
privacy for example that that that
provides some hope but
describe some of these that's actually
right so that's also our finding is that
by actually we show that in this
particular case we actually have a good
defense for the Quarian case for the
coin it's a language model language
model k so instead of just training a
vanilla language model instead if we
train a differentially private language
model then we can still achieve similar
utility but at the same time we can
actually significantly enhance the
privacy protection and stay after
learned model and our proposed attacks
actually are no longer effective and
differential privacy is the mechanism of
adding some noise by which you then have
some guarantees on the inability to
figure out the the person the the
presence of a human of a particular
person in the data set so right so in
this particular case what the
differential privacy mechanism does is
that it actually as participation in the
training process as we know during the
training process we are learning the
model well doing gradient updates the
way the updates and so on and
essentially differential privacy
differentially privates
machining algorithm in this case we'll
be adding noise and a diverse
perturbation during this training to
some aspect of the training process
right so then the finely trained ruining
the learned model is differentially
privates and so I can put can enhance
the privacy protection so okay so that's
the attacks and the defense of privacy
you also talked about ownership of data
so this this is a really interesting
idea that we get to use many services
online for seemingly for free by
essentially sort of a lot of companies
are funded through advertisement and
what that means is the advertisement
works exceptionally well because the
companies are able to access our
personal data so they know which
advertisement to service
to do targeted advertisements so on so
can you maybe talk about the this you
have some nice paintings of the future
philosophically speaking future where
people can have a little bit more
control of their data by owning and
maybe understanding the value of their
data and being able to sort of monetize
it in a more explicit way as opposed to
the implicit way that is currently done
yeah I think this is a fascinating topic
and also a really complex topic right I
think there are these natural questions
who should be owning the data and and so
I can tell one analogy and so for
example for physical properties like
your house and so on so really um this
notion of property rights it's not just
you know like it's not like from day one
we knew that's there should be like this
clear notion of ownership of properties
and having enforcement for this and so
actually people have shown that this
establishment and enforcement of
property rights has been a main driver
for the for the for the economy earlier
and that actually really propelled the
economic growth and even right in the
earlier stage so throughout the history
of the development of the United States
there or actually just civilization the
idea of property rights that you can own
property enforcement days is you should
know rights like governmental like
enforcement of this actually has been a
key driver for economic growth and there
have been even research proposals saying
that for a lot of the developing
countries and they you know essentially
the challenging growth is not actually
due to the lack of capital its
more due to the lack of this problem
notion property rights and enforcement's
of property rights interesting so that
the presence of absence of both the the
the concept of the property rights and
their enforcement has a strong
correlation to economic growth and so
you think that that same could be
transferred to the idea of property
ownership in case of data ownership I
think I think its first of all it's a
good lesson for us to like to recognize
that these rights and the recognition
and enforcement of this type of Rights
it's very very important for economic
growth and then if we look at where we
are now and where we are going in the
future and so essentially more and more
as it's actually moving into the digital
world and also more anymore I would say
even like information our asset alpha
person is more and more into the real
world the physical necessary the
teaching the world as well it's the data
that's the presence generators and
essentially it's like in the past what
defines a person you you can say right
like oftentimes besides the inmates like
capabilities actually it's the physical
properties oh right that you finds a
person but I think more the more people
start to realize actually what defines a
person is more important in the data
that the person has generated other data
about the person all the way from your
political views yar yar music tastes and
right your financial information now a
lot of these and your health so more and
more of the definition of the person is
actually in the digital world and
currently for the most part that's owned
in place like it's and people don't talk
about it but kind of it's owned by
[Music]
Internet companies so it's not owned by
individual there's no clear notion of
ownership after such data and also we
you know we talk about privacy and so on
but I think
actually clearly identifying the
ownership it's a first step once you
identify the ownership then you can say
who gets to define how that either
should be used so maybe some users are
fine with you know internet companies
serving them as you think the data as
lies if the if the data is used in a
certain way that actually the user
consents ways are allowed for example
you can see the recommendation system in
some sense we don't call it an ass but a
recommendation system similar it's
trying to recommend you something and
users enjoy and can really benefit from
good recommendation systems and they
recommend you you're better music movies
news or even research papers to read but
but of course then in this tech is ass
especially in in certain cases where
people can be manipulated by this
targeted ass that can have really bad
like a severe consequences so so
essentially uses one that data to be
used to better serve them and also maybe
even right get pay for whatever like in
different settings but the things that's
the first of all we need to really
establish like you who needs to decide
who can decide how the data should be
used and typically that the
establishment and clarification of the
ownership will help this and it's an
important first step so if the user is
the owner then naturally the user gets
to define how the dinner should be used
but if you even say that
wait a minute you say actually now the
owner of the stator whoever's collecting
the data is the owner of the data now of
course they get to use it in a hybrid
way they want yeah so to really address
these complex issues we need to go at
the root cause so it seems fairly clear
that's the first we really need to say
now who is the owner of the data and
then the owners can specify how the one
that they'd had to be utilized so I said
that that's a fascinating does most
people don't think about that and I
think that's a fascinating thing to
think about and probably fight for it
I can only see in the economic growth
argument it's probably a really strong
one so that's that's the first time I'm
kind of at least thinking about the the
positive aspect of that ownership being
the long-term growth of the economy so
good for everybody but sort of one down
possible downside I could see sort of to
put on my grumpy old grandpa hat and you
know it's really nice for Facebook and
YouTube and Twitter to all be free and
if you give control to people or their
data do you think it's possible they
will be they would not want to hand it
over quite easily and so a lot of these
companies that rely on mass handover of
data and then their book therefore
provide a mass seemingly free service
would then completely so the the the the
way the internet looks will completely
change because of the ownership of data
and we'll lose a lot of services with
value do you worry about that that's a
very good question I think that's not
necessarily the case in a sense that's
yes users can have ownership of their
data they can maintain control of their
data but also then they get to decide
how their data can be used so and that's
why I mention it like you see in this
case if they feel that they enjoy the
benefits of social networks and so on
and they are fine with having Facebook
having their data but utilizing the data
in certain way that's they agree then
they can still enjoy the free services
but for others maybe they would prefer
some kind of private vision and in that
case maybe they can even opt in to say
that I want to pay and to have so for
example it's already fairly standard
like you pay for certain subscriptions
so that you don't get to you know be
shown as yes yeah right so the users
essentially can have choices and I think
we just want to essentially bring out
more about who gets to decide
what to do with that yeah I think it's
an interesting idea because if you pull
people now you know it seems like I
don't know but subjectively sort of
anecdotally speaking it seems like a lot
of people don't trust Facebook so that's
at least a very popular thing to say
that I don't trust Facebook right I
wonder if you give people control of
their data as opposed to sort of
signaling to everyone that they don't
trust Facebook I wonder how they would
speak with the actual like would they be
willing to pay $10 a month for Facebook
or would they hand over their data
it'd be interesting to see what fraction
of people with would quietly hand over
their data to Facebook to make it free
III don't have a good intuition about
that like how many people do you have an
intuition about how many people would
use their data effectively on the market
on the on the market of the Internet by
sort of buying services with their data
yeah so that's a very good question I
think so one thing I also want to
mention is that this right so it seems
that especially in press and the
conversation has been very much like two
sides fighting against each other um oh
one hands right yes your skin say that
right they don't trust Facebook they
don't are there is DB Facebook yeah yeah
exactly
on the other hand and right of course
and right the other side they also feel
oh they are providing a lot of services
to users and users are getting it all
for free so I think actually you know I
talked a lot to like different companies
and also like a physically ample size
and so one thing I hope also like this
my hope for this year also is that and
we want to establish a more constructive
dialogue and that happen and to help
people to understand that the problem is
much more nuanced
then just and this to size fighting
because naturally there's a tension
between the two sides between your
Twitter and privacy so if you want to
get more utility essentially like the
recommendation system example I gave
earlier if you want someone to give you
good recommendation
essentially whatever the system is the
system is going to need to know your
data to give you a good recommendation
but also of course at the same time we
want to ensure that however that data is
being handled it's done in the privacy
preserving way and so that that for
example that recommendation system
doesn't just go around and say we are
they here and then cause all the you
know cause a lot of bad consequences and
so on so you want that dialog to be a
little bit more in the open a little
more more nuanced and maybe adding
control to the data ownership to the
data will allow so as opposed to this
happening in the background allowed to
bring it to the forefront and actually
have dialogues in like more nuanced real
dialogues about how we trade our data
for the services that's the whole rights
right yes at high level so essentially
also knowing that there are technical
challenges and in in addressing the
issue to like you basically you can't
have just like the example that I gave
earlier it is really difficult to
balance the two between utility and
privacy and and that's also a lot of
things that I work on my group Roxanne
as well as to actually develop these
technologies that are needed to
essentially help this balance better
essentially to help data to be utilized
in the privacy preserving and
responsible way and so we essentially
need people to understand the challenges
and also at the same time and to provide
the technical abilities and also
regulatory frameworks to help the two
sites will be more in the women
situation instead of
I fight yeah the fighting the fighting
thing is I think YouTube and Twitter and
Facebook are providing an incredible
service to the world and they're all
making mistakes of course but they're
doing an incredible job you know that I
think deserves to be applauded and
there's some degree of gratit it's a
cool thing that the that's created and
it shouldn't be monolithically fought
against like Facebook as evil or so on
yeah I might make mistakes but I think
it's an incredible service I think it's
world-changing I mean I've you know I
think Facebook's done a lot of
incredible incredible things by bringing
for example identity you're like
allowing people to be themselves like
their real selves in in the digital
space by using a real name and their
real picture that step was like the
first step from the real world to the
digital world that was a huge step that
perhaps will define the 21st century in
us creating a digital identity there's a
lot of interesting possibilities there
that are positive of course some things
are negative and having a good dialogue
about that is great and I'm I'm great
that people like you're at the center
that's how access is it's awesome I
think it also and I also can understand
I think actually in the past especially
in the past couple years and this rising
awareness has been helpful like users
are also more and more recognizing that
privacy is important to them
they shoes may be right there should be
owners after data I think the Stephanus
is very helpful and I think also this
type of voice also and together with the
regulatory framework and so on also help
the companies to essentially put this
type of issues at a higher priority and
knowing that right also it is their
responsibility to to ensure that users
are well protected and so I think it
definitely the raising voice is super
helpful and I think that I should
really has brought the issue of data
privacy and even this consideration of
the ownership to the forefront to really
much by the community and I think more
of this voice is needed but I think it's
just that we want to have a more
constructive dialogue to bring the both
sides together to figure out a
constructive solution so another
interesting space where security is
really important is in in the space of
any kinds of transactions but it could
be also digital currency so can you
maybe talk a little bit about blockchain
and can you tell me what is a blockchain
I think the brought to you where it
itself is activated overload is in
general it's like AI yes
so in general I talk about our team we
refer to this distributed IJ in a
decentralized fashion so essentially you
have in a community of nose that come
together and even though each one may
not be trusted and otherwise certain
thresholds of the set of nodes and he
behaves properly then and the system can
essentially achieve certain properties
for example in the distributed I just I
think you have you can maintain a
mutable log and you can ensure that for
some of the transactions actually I'll
create a pound and then it's immutable
and so on so first of all what's the
ledger so it's a it's like a database
it's like a data entry and so
distributed ledger is something that's
maintained across or is synchronized
across multiple sources multiple nodes
multiple notes yes and so where is this
idea now how do you keep okay so it's
important ledger a database to keep that
to make sure so what are the kinds of
security vulnerabilities that you're
trying to protect against in the context
of this
the distributed ledger so in this case
for example you don't want to some
malicious nose to be able to change the
transaction logs and in certain cases
account double spending like your also
calls you can also cause different views
in different parts of the network and so
on so the ledger has to represent if
you're capturing like financial
transactions has to represent the exact
timing and the exact occurrence and no
duplicates all that kind of stuff has to
be represent what actually happened okay
so what are your thoughts on the
security and privacy of digital currency
I can't tell you how many people write
to me to interview various people in the
digital currency space there seems to be
a lot of excitement there and it seems
to be some of it to me from an
outsider's perspective seems like dark
magic
I don't know how secure I think the the
foundation from my perspective of
digital currencies that is you can't
trust anyone so you have to create a
really secure system so can you maybe
speak about how well your thoughts in
general about digital currency is and
how you how it can possibly create
financial transactions and financial
stores of money in the digital space
so you as security and privacy and so so
again as I mentioned earlier in security
we actually talk about two main
properties and the integrity and
confidentiality and so there's another
one for availability you want the system
to be available but here for the
question you ask let's just focus on
integrity and confidentiality yes so so
for integrity of this distribution
essentially as we discussed we want to
ensure that's the different nose and
right so they have this consistent video
usually it's down through we call a
consensus protocol and that's the
establish
share the view on this leche and that
you cannot go back and change this
immutable and so on so so in this case
then the security often refers to this
integrity property and essentially
you're asking the question how much work
how how can you attack the system so
that the attacker can change the lock
for example right how hard is it to make
an attack like that yes right and then
that very much depends on the the
consensus mechanism the how the system
is built and now that so there are
different ways to build these
decentralized systems and people may
have heard about the term Scout like
proof-of-work you prefer take you this
different mechanisms and really depends
on how how the system has been built and
also how much resources how much work
has gone into the network to actually
say how secure it is so for example if
you talk about like in the coins for
what system is so much electricity it
has been burnt
so there's differences there's
differences in the different mechanisms
and the implementations of a distributed
ledger used for digital currency also
there's Bitcoin is a whatever there's so
many of them and there's underlying
different mechanisms and there's
arguments I suppose about which is more
effective which is more secure which is
more what amount of resources needed to
be able to attack the system like for
example what percentage of the nose do
you need to control our compromise in
order to write to change the log and
those are things do you do you have a
sense if those are things that can be
shown theoretically through the design
of the mechanisms or does it have to be
shown empirically by having a large
number of users using the currency I see
so in general for each consensus
mechanism you can actually show
theoretically what is needed to be able
to attack the system of course there are
there can be different types of attacks
as weepy and discuss at the beginning
and so that and it's difficult to gave
like you know a complete estimate like
really how much is needed to compromise
the system but in general right so there
are ways to say what percentage of the
knows you need to compromise and so on
so we talked about integrity so on the
security side and then you also
mentioned can the privacy or the
confidentiality side does it have some
of does it have some of the same
problems and therefore some of the same
solutions that you talked about and the
machine learning side with differential
privacy and so on yeah so actually in
general on the public ledger in this
public decentralized systems and
actually nothing is private so all the
transactions posters on the library
anybody can see so in that sense there
is no confidentiality and so usually all
you can do is then there are the
mechanisms that you can built in to
enable confidentiality are privacy of
the transactions and the data and so on
that's also some of the work and that's
both my group and also my startup and
does as well what's the name you start o
Asus labs Oasis labs and so the
confidentiality aspect there is even
though the transactions are public you
want to keep some aspect confidential of
the identity of the people involved in
the transactions or what what is their
hope to keep confidential in this
context so in this case for example you
want to your nipple like private
confidential transactions even so so
there are different and essentially
types of data that you want to keep
private are confidential and you can
utilize different technologies
including your knowledge proofs and also
secure computing and techniques and to
hide the right who is making the
transactions to whom and the transaction
amount and in our case also we can
enable like confidential smart contracts
and so that's you don't know the data
and the execution of the smart contract
and so on and we actually are combining
these different technologies and to
going back to the earlier discussion we
had enabling like ownership of data and
privacy of data and so on so so at Oasis
labs we're actually building what we
call a platform for responsible data
economy to actually combine these
different technologies together and to
enable secure and privacy-preserving
computation and also using the library
to help provide immutable log of users
ownership to their data and the policies
they want the data to adhere to the
usage of the data to adhere to and also
how that it has been utilized so all
this together can build we can a
distributed secure computing fabric that
helps to enable a more responsible data
economy other things together yeah wow
those eloquent okay you're involved in
so much amazing work that we'll never be
able to get to but I have to ask at
least briefly about program synthesis
which at least in a philosophical sense
captures much of the dreams of what's
possible in computer science and the
artificial intelligence first let me ask
what is program synthesis and can ural
networks be used to learn programs from
data so can this be learned some aspect
of this synthesis can it be learned so
program synthesis is about teaching
computers to write code to program and I
think it has one of our ultimate
dreams or goals and you know I think
Andreessen talked about software eating
the world so I say once we teach
computers to write software
I had to write programs then I guess
computers yeah exactly so yeah and also
for me actually
um when I you know shifted from security
to more AI a machining program synthesis
is program scenes in adversarial
machining these are the two fields that
I particularly focus on like program
synthesis one of the first questions
that I actually started what are seeking
just as a question oh I guess with from
the security side there's a you know
you're looking for holes and programs so
as at least see small connection but why
what was your interest for program
synthesis as because it's such a
fascinating such a big such a hard
problem in the general case why program
synthesis so the reason for that is
actually when I shifted my focus from
security into AI machine learning and
actually one of my main motivation at
the time and is that even though I have
been doing a lot of working security and
privacy but I have always been
fascinated about beauty intelligent
machines and that was really my main
motivation to spend more time in AI am a
Shalini is as I really want to figure
out how we can build intelligent
machines and to help us towards that
goal program synthesis is really one
enough I would say the best domain to
work on I actually call it's like
programming synthesis it's like the
perfect playground for building
intelligent machines therefore
artificial general intelligence yeah um
well it's also in that sense not just a
playground I guess it's it's the
ultimate test of intelligence because
yes I think I think
you can generate so neural networks can
learn good functions and they can help
y'all in classification tasks but to be
able to write programs right that's
that's the epitome from the machine side
that's the same as passing the Turing
test and natural language but with
programs it's able to express
complicated ideas to reason through
ideas and yeah and boil them down to
algorithms yes exactly is that credible
so can this be learned how far are we
is there hope what are the open
challenges questions and we're still at
an early stage but already I think you
we have seen a lot of progress I mean
definitely we have you know existence
proof just like the humans can write
programs so there's no reason why
computers cannot write programs and so I
think that's definitely an achievable
goal it's just how long it takes and
then and even today we actually have you
know the program synthesis community
especially the program synthesis by
learning our way College neural program
synthesis community is still very small
but the community has been growing and
we have seen a lot of progress and in
limited domains I think actually program
synthesis is ripe for real-world
applications so actually was kind of
amazing I was at giving a talk it's also
here it's a rework we worked you
planning something actually so I give
another talk at the previously rework
conference in deep reinforcement
learning and then I actually met someone
from a startup and the CEO of the
startup and when he saw my name he
recognized and he actually said one of
our papers actually had they have put
the had actually become a key products
and that was program synthesis in that
particular case it was
natural language translation translating
natural language description into psycho
Cory's oh wow that that direction
okay right so yeah so you program since
this is in limited domains in well
specified domains actually already we
can see really great great progress and
applicability in the real roads so
domains like as an example you said
natural language being able to express
something to just normal language and it
converts it into a database sequel SQL
query right and that's how how solve the
problem is that because that seems like
a really hard problem okay eliminate
domains actually it can work pretty well
and now this is also a very active
domain after research at the time I
think one he saw our paper at the time
we were the state of the Arts yeah and
that task and since then actually now
there has been more work and with even
more sophisticated assets and so but I I
think I wouldn't be surprised that's
more of this type of technology really
getting to the real worlds that's
exciting in the near term being able to
learn in the space of programs is super
exciting I still yeah I'm still
skeptical because I think it's a really
hard problem progress and also I think
in terms of the your ass about open
challenges I think the domain is full of
challenges and in particular also we
want to see how we should measure the
progress in the space and I would say
mainly three main I'll say metrics so
one is a complexity of the program that
we can synthesize and that will actually
have clear measures and just look at you
know the past publications and even like
for example I was at the recent Europe's
conference now there is actually very
sizable like session dedicated to
program since this is vicious or even
neural progress today
which is great and and we continue to
see the increase like I think they were
sizable it's five people and they will
all win touring awards one day like it
so we can see increase in the complexity
of the program is that these synthesized
sorry - is it the complexity of the
actual text of the program or the
running time complexity which complexity
over how complexity after task to be
synthesized and the complexes are after
the actual synthesize the programs so
you so the lines of code even for
example okay I got you but it's not the
theoretical upper bound of the running
time of the day and you can see the
complexity in decreasing already oh no
meaning we want to be able to synthesize
monomer complex programs bigger and
bigger programs so we want to see that's
we want to increase I have to think
through because I thought of complexity
is you want to be able to accomplish the
same task with a simpler and simpler
program no we are not doing that okay
it's more it's more about how complex a
task right we can see the exotic being
able to synthesize programs learn them
for more and more difficult right so for
example initially our first working
program synthesis synthesis was to
translate natural language description
into really simple programs called
if TTT if this then that so given a
trigger condition what is the action you
should take so that program is a super
simple you just Andy identify the
trigger conditions and the action yeah
and then later on with the secret
queries that gets more complex and then
also we started to synthesize programs
with loops and know anything could
synthesize recursion it's all over
actually yeah 1fi works actually it's
already rechristen you're
complexity and the other one is
generalization like one-way training I
want to learn programming synthesizer in
this case and neural programs to
synthesize programs then you wanted to
generalize so for a large number of
inputs to be able to write generalize to
previously and C inputs got it and so so
someone for the work who waited earlier
learning recursive new programs actually
showed that recursion actually is
important and to learn and if you have
recursion then for certain and set of
tasks we can actually show that you can
actually have perfect generalization and
so right so that one the best paper
Awards that I clear earlier and so
that's one example of we want to learn
these you know programs that can
generalize better but that works for a
certain task with certain domains and
there is question how we can essentially
develop more techniques that can and
have generalization for wider set of
domains and so on so that's another area
and then and then the the third
challenge I think will it's not just for
programming synthesis is also cutting
across other fields in machine learning
and also including like deep
reinforcement and in particular is that
this adaptation is that we want to be
able to learn from the past and tasks
and training and so on to be able to
solve new tasks so for example in
program synthesis today we still are
working in the setting way given a
particular task we change the right
model and to solve this particular task
but that's not how humans work like the
whole point is we train a human than
you can then program to south new tasks
right exactly and just like we don't
want to just change agent to play a
particular game
hey it's Atari ice ago whatever we want
to train these agents that can and
essentially extract knowledge from the
past learning experience to be able to
adapt to new new tasks and solve new
tasks and I think this is particularly
important for program synthesis yeah
that's the whole point that's the whole
dream of progress this is your learning
a tool that can solve new problems right
exactly and I think that's a particular
main that as a community we need to put
more emphasis on and I hope that we can
make more progress today as well
awesome I think there's a lot more to
talk about but let me ask that you also
had a very interesting and we talked
about rich representations he had a rich
life journey you did your bachelor's in
China and your masters and PhD in the
United States CMU and Berkeley are there
interesting differences I told you I'm
Russian I think there's a lot of
interesting difference between Russia
and the United States are there in your
eyes interesting differences between the
two cultures from the silly romantic
notion of the spirit of the people to
the more practical notion of how
research is conducted that you find
interesting or useful in your own work
of having experienced both that's a good
question I think so I I started in China
for my undergraduate and that was more
than 20 years ago there's been a long
time
is there echoes of that time I think
even more so maybe something that's even
be more different for my experience and
a lot of computer science researchers
and practitioners is that so for my
undergraduate studies physics
very nice and then I switch to a
computer science in graduate school what
happened was there was there is there
another possible universe where you
could have become a theoretical
physicist at Caltech or something like
that that's very possible some of my and
undergrad classmates then the later
studies physics account there 15 physics
from these schools from yeah from tough
physics programs so so you you switch to
I mean in that from that experience to
doing physics in your bachelor's how
what means you decide to switch to
computer science and computer science
had arguably the best university one of
the best universities in the world for
computer science and with Carnegie
Mellon especially for the grad school
and and so on so what ii only 10 mighty
just kidding okay
I had Authority and know what what was
the choice like and what was the move to
the United States like what was that
whole transition and if you remember if
there's still echoes of some of the
spirit of the people of China in you in
New York it's like three questions so
yes I guess okay the first transition
from physics to computer science yes so
when I first came to the United States I
was actually in the physics ph.d program
at Cornell yeah I was there for one year
and then I switched to computer science
and I was seeing the PC program at kind
of give a loan and so okay so the
reasons for switching so one thing so
that's why I also mentions that about
this difference in backgrounds about
having studied physics yes first in my
undergrad um
actually really I really did enjoy my
undergrads time and education in physics
I think that actually really helped me
in my future work in computer science
actually even for machine learning a lot
of machine learning stuff the the core
machining methods many of the magic
for honest most most of everything came
from physics I was I think I was really
attracted to physics and it was it's
really beautiful and educated physics is
the language of nature and I actually
really remember like one moment in my
undergrads like I did my undergrad in
Chinua and I used to study in the
library and I clearly remember like one
day I was sitting in a library and I and
I was like writing my notes and so on
and I got so excited that I realized
that if you just from a few simple
axioms a few simple laws I can derive so
much it's almost like I can't derive the
rest of the world yeah there's the
universe yes yes so that was like
amazing do you think you have you ever
seen or do you think you can rediscover
that kind of power and beauty and
computer science in the world that yes
that's very interesting so that gets to
you know the transition from physics to
Versailles and it's a it's quite
different for and for physics in in
Cresco actually things changed so one is
I started to realize that when I started
doing research in physics at the time I
was doing theoretical physics and a lot
of its the you still have the beauty
base very different so I have to
actually do a lot of simulation so
essentially I was actually writing in
some in some cases writing a fortune
Harold fortune yes to actually write do
like do simulations and so on that was
not not exact I
I enjoy it's doing
and also at the time from talking with
the senior you know students in the
program I realized many of the students
actually were going off to work Wall
Street and and so on and so and I've
always been interested in computer
science and actually essentially taught
myself the C programming program right
when in college and college somewhere
for fun learning to do C programming you
know in physics at the time I think now
the programming profit has changed but
at the time really the only class we had
in in Hoosick amir science education was
introduction to africa to computer
science or computing and fortune 77
there's a lot of people that still use
Fortran I'm actually if you're a
programmer out there I'm looking for an
expert to talk to about Fortran they
seem to there's not many but there's
still a lot of people to still use
Fortran and still a lot of people these
cobalt I realized
instead of just doing programming for
doing simulations and so on that I may
as well just change to computer science
and also one thing I really like and
that's a key difference between the two
as in computer science is so much easier
to realize your ideas if you have idea
you're writing it up you're cut it up
and then you can see it's actually bring
it to life quickly it's your life
wasting physics if you how good theory
you you have to wait for the
experimentalist to do the experiments
and to confirm the theory and things
just take so much longer and and also
the reason I in physics I decided to do
theoretical physics it was because I had
my experience with experimental physics
first you have to fix the equipment
fixing the equipment first so
offensive equipment so there's a lot of
it yeah he's have to collaborate with a
lot of people takes a long time yes
messy so I decided to switch to computer
science and the one thing I think maybe
people have realized is that for people
who study physics actually it's very
easy for physicists to change to do
something else yes I think physics
provides a really good training and yeah
so actually it was very easy to switch
to computer science but one thing going
back to your earlier question so one
thing I should you realize so there is a
big difference between commune sense and
physics away physics you can derive the
the whole universe from just a few
simple laws and computer science given
that a lot of it is defined by humans
the systems that you find by humans and
and artificial I can essentially create
a lot of these artifacts and so on and
it's it's not quite the same
you don't derive the computer systems
with just a few simple laws you actually
have to see there's historical reasons
why our system is builds and designs one
way versus the a day there's a lot more
complexity or less elegant simplicity of
e equals mc-squared that kind of reduces
everything down to his beautiful
fundamental equations but what about the
move from China to the United States is
there anything that still stays in you
that's contributes to your work the fact
that you grew up in another culture so
yes I think especially back then it's
very different from now so you know now
they actually I see these students
coming from China and even an aggressor
actually they speak fluent English it
was just you know like amazing and they
have already understood so much of the
culture in the US and so on and it was
to you was all foreign it was it was a
very different time at a time actually
even we didn't even have access to email
right not to mention about the wealth
yeah I remember I had to go to you know
specific like you know privileged
several rooms too much knowledge about
the Western world and actually at the
time I didn't know actually the the in
the US the West Coast weather is so much
better than the yeah things like that
actually it's very it's very yeah but
now it's so different at the time I I
would say there's also a bigger culture
difference because there's so much less
opportunity for shared information so
it's such a different right I meant
world
let me ask me be a sensor question I'm
not sure but I think you're not in
similar positions is I've been here for
already 20 years as well and looking at
Russia from our perspective and you
looking at China in some ways it's a
very distant place because it's changed
a lot but in some ways you still have
echoes you have still have knowledge of
that place the question is you know
China is doing a lot of incredible work
in AI do you see please tell me there's
an optimistic picture you see where the
United States and China can collaborate
and sort of grow together in the
development of AI towards you know
there's different values in terms of the
role of government and so on of ethical
transparent secure systems we see it
differently in the I States a little bit
than China but we're still trying to
work it out do you see the two countries
being able to successfully collaborate
and work in a healthy way without sort
of fighting and making it an AI arms
race kind of situation yeah I believe so
and I think it's science there's no
border and the advancement of technology
helps everyone helps the whole world and
so I certainly hope that the two
countries will collaborate
and I certainly believe so do you have
any reason to believe so except being an
optimist so first again like I said
science has no borders and especially
science doesn't know board borders right
and you believe that will you know in
this in the former Soviet Union during
the Cold War
yeah so this is the other point I was
going to mention is that especially in
academic research everything is public
like we write papers we open source
codes and others in the public domain it
doesn't matter whether the person is in
the u.s. in China or some other parts of
the world and they can go on archive and
look at the latest research and results
so that openness gives you hope yes me
too
and that's also how as a world we make
progress the best so apologize for the
romanticized question but looking back
what would you say was the most
transformative moment in your life that
maybe made you fall in love with
computer science you said physics you
remember there was a moment where you
thought you could derive the entirety of
the universe was there a moment that you
really fell in love with the work you do
now from security to machine learning to
program synthesis so maybe as I
mentioned actually in college a one
summer I should tell myself programming
see yes you just read a bug don't tell
me you fell in love with computer
science by programming and see remember
I mentioned when one of the draws for me
to come here sense is how easy it is to
realize their ideas so once I you don't
read the book started like it taught
myself how to program and see
immediately what what did I do
like I programmed two games um
ones just simple like it's a go game
like it supports you can move the stones
and so on and the other one actually
programmed the game that's like a 3d
Tetris it was a to not to be a super
hard game to play
it's obvious the standard 2d Tetris it's
actually a 3d thing but I can realize
wow you know I just had these ideas to
try it out and then you can just do this
so that's the one I realized wow this is
amazing
yeah you can create yourself from
nothing to something that's actually out
in the real world so let me ask let me
ask a silly question
or maybe the ultimate question what is
to you the meaning of life what what
gives your life meaning purpose
fulfillment happiness joy okay
these are two different questions very
different yeah it's easy that you asked
this question maybe this question is
probably the question that has follows
me and follow my life the most have you
discovered anything and you satisfactory
answer for yourself is there something
is there something you've arrived at you
know that there's a moment I've talked
to a few people who have faced for
example a cancer diagnosis or faced
their own mortality and that seems to
change their views and it it seems to be
a catalyst for them removing most of the
crap that the of seeing that most of
what they've been doing is not that
important and really reducing it into
saying like here's is actually the few
things that really give me give meaning
mortality is a really powerful catalyst
for that it seems like facing mortality
whether it's your parents dying or
somebody close to you dying or facing
your own death for whatever reason or
cancer and so on yeah in my own case I
didn't need to face mortality and I
think there are a couple things so one
is like who should be defining the
meaning of your life right is there some
kind of even greater things than you who
should define the meaning of your life
so for example when people say that
searching the meaning for our life is is
there some there is some outside voice
or is there something you know a set of
you who actually tells you you know some
people talk about oh you know this is
what you have been born to do right
right like this is your destiny um so
who right so that's the one question
like who gets to define the meaning of
your life should you be finding some
other thing some other factor to define
this for you always something actually
it's just entirely where you define
yourself and it can be very arbitrary
yeah so in inner and inner voice or an
outer voice whether it's it could be
spiritual religious - with God or some
other components of the environment
outside of you or just your own voice do
you have up do you have an answer there
and so you know you know the long period
of time of thinking and searching even
searching through outsides right you
know voices are factors outside of me
yeah so that I have and so I've come to
the conclusion and realization that it's
you yourself that you finds the meaning
of life yeah that's a big burden no
isn't it right so then you have the
freedom to define it yes and and another
question is like what does it really
mean by the meaning of life right um and
also whether the question even make
sense absolutely and you said it somehow
distinct from happiness so meaning is
something much deeper than just any kind
of emotional any any kind of contentment
or joy whatever it might be much deeper
and then you have to ask what is deeper
than that
what is
what is there at all and then the
question starts being silly right and
also you can say it's deeper but you can
also say it's a shallow depending on how
people want to define the meaning of
their life so for example most people
don't even think about this question
then the meaning of life to them it
doesn't really matter that much and also
whether knowing the meaning of life and
whether actually helps y'all love to be
present area or whether helps your life
to be happier and these actually are
often questions is not worse most
questions open I tend to think that just
asking the question as you mentioned as
you've done for a long time is the only
that there is no answer and asking the
question is a really good exercise I
mean I have this for me personally I've
had the kind of feeling that creation is
a like for me has been very fulfilling
and it seems like my meaning has been to
create and I'm not sure what that is
like I I don't have a single lot of kids
I would love to have kids but I also
sounds creepy but I also see sort of he
said see programs
I see programs as little creations I see
robots as little creations I think those
are met those of those bring and then
ideas theorems and and are creations and
those somehow intrinsically like you
said bring me joy I think they do to a
lot of these scientists but I think they
did a lot of people so that to me if I
had to force the answer to that I would
say creating new things yourself for you
for me for me for me I don't know but
like you said as he keeps changing is
there some answer that some people they
can I think they may say it's experience
rights like their meaning of life all
right
they just want to experience to the
richest and full as they can and a lot
of people do take that path yes seeing
life is actually a collection of moments
and then trying to make the richest
possible
that's filled those moments with the
richest possible experiences yeah right
and for me I think it's certainly we do
share a lot of similarity here like the
creation is also really important for me
even from you know the things that I've
already talked about even like you know
writing papers and these are our
creations as well and I have not quite
thought whether that has really the
meaning of my life like in a sense also
that maybe like what kind of things
should you create there's so many
different things that you could create
and also you can say another view is
maybe growth is it's related but
different from experience growth is also
maybe type of meaning of life it's just
you try to grow every day try to be a
better self every day and and also
ultimately we are here it's part of the
overall evolution the right the world is
evolving it's funny it's funny that the
growth seems to be the more important
thing than the thing you're growing
towards it's like it's not the goal it's
the the journey to it sort of it's
almost it's almost when you submit a
paper it's there's a sort of depressing
element to it not to submit a paper but
when that whole project is over I mean
there's a gratitude there's a
celebration and so on but you're usually
immediately looking for the next thing
yeah the next step right it's not it's
not that status that at the end of it is
not the satisfaction is the the hardship
the challenge you have to overcome the
growth through the process it's
something it's somehow probably deeply
within us the same thing that drove that
drives the evolutionary process is
somehow within us with everything the
way the way we see the world since
you're thinking about this so you're
still in search of an answer I mean yes
and no in the sense that I think for
people who really dedicate time to
search for the answer to ask a question
what is the meaning of life it does not
as we bring your happiness
yeah it's a question and we can say
right like weather is a well-defined
question and and on the other and but on
the other hand given that you get two
answers yourself you can define it
yourself
sure I can't just you know give it
answer and in that sense yes it can help
and like it's like we discussed if you
say oh then my meaning of life is to
create are to grow then then yes then I
think they can help but how do you know
that that is really the meaning of life
are the meaning of your life it's like
there's no way for you to really answer
the question sure but something about
that certainty is liberating so if it
might be an illusion you know you might
not really know you might be just
convincing yourself falsely falsely but
being sure that that's the meaning the
there's something there's something
liberating in that in that there's
something freeing in knowing this is
your purpose so you can fully give
yourself to that without you know for a
long time you know I thought like isn't
it all right like why what's how do we
even know what's good and what's evil
like it isn't everything just relative
like how do we know you know the the
question of meaning is ultimately the
question of why do anything why is
anything good or bad why is anything
moment then you start to I think just
like you said I think it's a really
useful question to ask but if you ask it
for too long and too aggressively I mean
not be so protect it not be productive
and not just for traditionally society
to find success but also for happiness
it seems like asking the question about
the meaning of life is like a trap is uh
were destined to be asking we destined
to look up to the stars and ask these
big white questions we'll never be able
to answer but we shouldn't get lost in
them and
that's probably the that's at least a
lesson I picked up so far I'm noting
that topic let me just add one more
thing so it's interesting so actually so
sometimes yes it can help you and to
focus so when I when I shifted my focus
more from security to a I am a Sunni at
the time the actually one of the main
reason why I did that was because at the
time I thought my mini the meaning of my
life and the purpose of my life is to
build in hydrogen machines and that's
and then your inner voice said that this
is the right this is the right journey
to take to build intelligent machines
and that you actually fully realized you
took a really legitimate big step to
become one of the world class
researchers to actually make it to
actually go down that journey yeah
that's profound that's profound I don't
think there's a better way to end a
conversation than talking for for a
while about the meaning of life done
it's a huge honor to talk to you thank
you so much for talking today thank you
thank you
thanks for listening to this
conversation with Dawn song and thank
you to our presenting sponsor cash app
please consider supporting the podcast
by downloading cash app and using
collects podcast if you enjoy the spot
guest subscribe on YouTube review it
with five stars on Apple podcast
supported on patreon or simply connect
with me on Twitter Alex Friedman and now
let me leave you with some words about
hacking from the great Steve Wozniak a
lot of hacking is playing with other
people you know getting them to do
strange things thank you for listening
and hope to see you next time
you