Kind: captions
Language: en
[Music]
what makes science true what makes
science something we should rely
on if you have a novel Finding you
should be able to replicate it or
somebody else in a different lab should
be able to replicate this it's like if
you get in your car in the morning and
you turn the ignition on you you pretty
much expect the car to start up and go
forward the laws of internal combustion
they're going to work today they're
going to work tomorrow so
reproducibility is a way of giving
validity to
science reports are accumulating that
much if not most of scientific claims
can't be reproduced right now the whole
literature is of questionable
value because you don't know what's real
and what's not increasingly many
scientists including myself were
recognizing that um maybe much of what
we believe is not really
true scientists are trying to figure out
how much of what is published is
reproducible why and what can be done
and this is the way science is supposed
to work in theory that you have all of
these individual building blocks that
are the Publications and you take those
building blocks and then you build the
next level of
understanding and so if they can't
produce those findings we are wasting a
lot of what we do it has been called the
reproducibility crisis the media and the
general public are wondering if science
itself is broken there's a vertiginous
feeling of standing on a cliff and
realizing that so much of the knowledge
we thought we had may not actually be
very sound knowledge and it's about to
crumble away and we're going to fall
into a pit and science is not going to
save us because science is a mess
but the foundations of science are solid
after all we are surrounded by the
impacts of very real scientific
discoveries this realization that we
think we know more than we really do is
not new and it points to a fundamental
flaw in how we as a society think about
science
[Music]
imagine living in a Time long ago before
any scientific knowledge and looking up
into a star night
sky what is happening up there what does
it
mean and what is it that keeps us bound
to
Earth around the year 350 BC Aristotle
was asking these kinds of fundamental
questions about the world
[Music]
Aristotle came from a time when people
didn't really conduct scientific
experiments but if he stood on a Ledge
and dropped a rock and a feather at the
same time he would have seen how a
feather being lighter Falls more slowly
than a
rock based on observations like this
Aristotle came to the theory that
heavier objects fall faster than lighter
objects
when we first started the
reproducibility initiative it was
actually quite controversial so there
was definitely scientists who felt like
it might be something that damaged
people's careers or was offensive to the
scientific
Community Elizabeth irns founder and CEO
of science exchange is heading up a
project to take the experiments from 35
high impact studies in the field of
cancer biology and see if they hold up
to replication people tend to be very
afraid of saying that something was not
reproducible it's definitely you know
this cultural issue with reproducibility
being associated with fraud even though
the vast vast majority of cases would
not be because of fraud iron says that
replication studies are a crucial piece
of the scientific process that has
somehow been lost replication studies
are so really funded and they're so
underappreciated they never get
published no one wants to do them
there's no reward system there in place
that enables it to happen so you just
have all of these exploratory studies
out there that are taken as fact that
this is a scientific fact that's never
actually been
[Music]
confirmed remember when science told us
that taking antioxidant supplements was
good for you there were molecular
studies animal studies epidemiological
studies and even this randomized double
blinded human trial of over 2,000
participants antioxidant vitamins are
the latest rage on grocery store shelves
they include vitamin c e and beta katene
all the experimental evidence points
very strongly to uh the value of
antioxidants as a potential therapeutic
uh tool but then the theory completely
reversed itself researchers in Texas
found antioxidants that protect healthy
cells from the damage caused by free
radicals can actually turbocharge the
growth of cancerous cells the idea was
give people more vitamins and it would
prevent cancer but what they find when
they do a lot of these studies is that
it can do the exact opposite it makes
you wonder about the rest of science
that gets published why these drugs that
are coming out and showing that they can
cure diseases and animal models just
like failing and every clinical trial is
fail fail fail a 2011 analysis reported
that in the field of cancer biology 95
5% of potential Cancer drugs fail in
clinical trials I would argue that the
reason why they have so higher failure
rate is because the original studies
that were done in the preclinical phase
were never properly validated drug
companies began to take notice and in
2011 and 12 pharmaceutical Giants Bayer
and amen each attempted to validate
published studies on potential drug
targets Bayer was only able to reproduce
21% % of the findings Amgen
11% more reproducibility studies
trickled in the National Institute for
neurological disorders and stroke 8% the
ALS therapy development Institute
0% and the reproducibility project
psychology
36% most of the available data on
replication rates is in the fields of
biomedical science and psychology
but survey data indicates that the
problem is
[Music]
widespread how can so much science be
irreproducible one reason for
irreproducibility is that the original
was wrong they thought there was
something there that wasn't there a
second reason uh is that the replication
screwed up the third is what probably
happens very frequently uh but it's the
hardest to unpack which is both are true
the original result observed a finding
that is there uh the replication did not
observe the finding and it isn't there
how could both of those things be true
well because no two studies are
identical Aristotle's theory that the
speed at which objects fall is
proportional to their Mass prevailed for
almost 2,000
years then around the year 1600 Legend
had has it that Galileo stood on top of
the tower of Pisa and essentially
replicated the falling object experiment
with one crucial
difference instead of using a feather
and a rock as the heavy and light
objects he used a cannonball and a
musketball
[Music]
they hit the ground at the same
time Aristotle was
wrong when we get a result and then we
fail to get it and then we have a puzzle
what happened why is it that these two
outcomes were different Aristotle's
finding couldn't be reproduced because
of a variable that he didn't even know
existed air resistance which is closely
linked to the shape of the
object but how many scientific studies
fail to replicate because of these kinds
of unknown factors and how many are just
flat out
wrong this happens far more often than
we would like to to think even with good
intentions even with the best intentions
our first impressions our first
discoveries they're likely to be
false collecting and interpreting data
can be extremely complicated and it's
hard to tell the difference between a
real finding and just statistical
noise people are trying to find
relationships and associations and
effects in in in a sea of of dust and
noise and some real effects and if you
test lots of things you can come up with
seemingly interesting results that have
absolutely no meaning and at the same
time the incentive structure and the
reward system for for How We Do Science
it's not aligned with maximizing the
yield of of true
results I'm incentivized to find
positive results relationships that
exist when I do this this other thing
happens it's much more interesting than
saying when I do this nothing happens
positive results are very publishable
and Publishing things is what advances
my
career when I am doing my research uh I
have many choices that I can make along
the
way is there an effect for one variable
and not the
other maybe some data needs to be
excluded perhaps gender differences
influence the
result how much data is enough should
the study end now or
later when I'm looking at these
different ways of analyzing the data I
might see the ones that reveal positive
results and be more convinced by them
and I might not do that intentionally
I'm not looking to falsely create
positive results but I might be
leveraging chance unintentionally just
because I have skin in the game these
kinds of decisions apply to exploratory
studies in every scientific field and
when there are mult multiple variables
at play it's really easy to find
statistically significant positive
results purely by chance and we won't be
able to know which ones occurred by
chance unless we do it
again the current incentive system for
doing research at an academic level is
extremely broken so much of it is about
these oneoff findings
that are very unlikely to be real and so
you really want to create a system where
no matter what experiments you did so
long as they're really good experiments
even if they're negative results they're
still valuable and they're still
valued IRS and a growing number of
researchers journals funders and
institutions are working to create a new
paradigm for scientific research there's
lots of actions across almost any field
that you can imagine in science they've
proposed a number of systemic changes to
make science more accurate actually I'm
really optimistic we are realizing
better ways of doing science new
guidelines and incentives encourage
researchers to improve transparency and
statistical methods and to conduct more
replications if a really exciting
exploratory breakthrough result comes
out there should be a replication study
that follows that and confirms that
those results are
reproducible as of January 2017 the
reproducibility project cancer biology
has completed five of the 35 replication
studies none of them were perfect
replications but there's still debate
around how to interpret what
happened and when the final results do
come in we will have to ask ourselves
the same
question what does it
mean well in my left hand I have a a
feather in my right hand a hammer and I
guess one of the reasons uh we got here
today was because of a gentleman named
Galileo a long time ago who made a
rather significant discovery about
falling objects in gravity fields and we
thought that uh where would be a better
place to confirm his uh findings and on
the
moon and I'll uh drop the two of them
here and hopefully they'll hit the
ground at the same
time how about that
Aristotle's observation of the feather
and The Rock wasn't wrong only his
conclusions
were today our society sees the
peer-reviewed published literature as if
it were an archive of scientifically
proven
facts but the frontiers of science will
always be uncertain and the published
literature is more of a discussion among
scientists
progress isn't made in science by
figuring out what's a fact and what's
not science makes progress by reducing
the uncertainty of explanations of
taking what is a world of possibilities
and reducing it to a smaller world of
possibilities and so it requires a very
different orientation that's difficult
even for scientists to embrace that we
are going to be wrong a
lot a lot uh because we're chasing
things that we do not
understand that's why we're doing
science we are accumulating evidence to
try to reduce that misunderstanding
[Music]