The era of blind faith in big data must end | Cathy O'Neil

1
00:00:12,795 --> 00:00:14,391
Algorithms are everywhere.

2
00:00:15,931 --> 00:00:19,056
They sort and separate
the winners from the losers.

3
00:00:19,839 --> 00:00:22,103
The winners get the job

4
00:00:22,127 --> 00:00:23,870
or a good credit card offer.

5
00:00:23,894 --> 00:00:26,545
The losers don't even get an interview

6
00:00:27,410 --> 00:00:29,187
or they pay more for insurance.

7
00:00:30,017 --> 00:00:33,566
We're being scored with secret formulas
that we don't understand

8
00:00:34,495 --> 00:00:37,712
that often don't have systems of appeal.

9
00:00:39,060 --> 00:00:40,356
That begs the question:

10
00:00:40,380 --> 00:00:43,293
What if the algorithms are wrong?

11
00:00:44,920 --> 00:00:46,960
To build an algorithm you need two things:

12
00:00:46,984 --> 00:00:48,965
you need data, what happened in the past,

13
00:00:48,989 --> 00:00:50,550
and a definition of success,

14
00:00:50,574 --> 00:00:53,031
the thing you're looking for
and often hoping for.

15
00:00:53,055 --> 00:00:58,092
You train an algorithm
by looking, figuring out.

16
00:00:58,116 --> 00:01:01,535
The algorithm figures out
what is associated with success.

17
00:01:01,559 --> 00:01:04,022
What situation leads to success?

18
00:01:04,701 --> 00:01:06,463
Actually, everyone uses algorithms.

19
00:01:06,487 --> 00:01:09,205
They just don't formalize them
in written code.

20
00:01:09,229 --> 00:01:10,577
Let me give you an example.

21
00:01:10,601 --> 00:01:13,917
I use an algorithm every day
to make a meal for my family.

22
00:01:13,941 --> 00:01:15,417
The data I use

23
00:01:16,214 --> 00:01:17,873
is the ingredients in my kitchen,

24
00:01:17,897 --> 00:01:19,424
the time I have,

25
00:01:19,448 --> 00:01:20,681
the ambition I have,

26
00:01:20,705 --> 00:01:22,414
and I curate that data.

27
00:01:22,438 --> 00:01:26,689
I don't count those little packages
of ramen noodles as food.

28
00:01:26,713 --> 00:01:28,582
(Laughter)

29
00:01:28,606 --> 00:01:30,451
My definition of success is:

30
00:01:30,475 --> 00:01:33,134
a meal is successful
if my kids eat vegetables.

31
00:01:34,001 --> 00:01:36,855
It's very different
from if my youngest son were in charge.

32
00:01:36,879 --> 00:01:39,667
He'd say success is if
he gets to eat lots of Nutella.

33
00:01:40,999 --> 00:01:43,225
But I get to choose success.

34
00:01:43,249 --> 00:01:45,956
I am in charge. My opinion matters.

35
00:01:45,980 --> 00:01:48,655
That's the first rule of algorithms.

36
00:01:48,679 --> 00:01:51,859
Algorithms are opinions embedded in code.

37
00:01:53,382 --> 00:01:57,045
It's really different from what you think
most people think of algorithms.

38
00:01:57,069 --> 00:02:01,573
They think algorithms are objective
and true and scientific.

39
00:02:02,207 --> 00:02:03,906
That's a marketing trick.

40
00:02:05,089 --> 00:02:07,214
It's also a marketing trick

41
00:02:07,238 --> 00:02:10,392
to intimidate you with algorithms,

42
00:02:10,416 --> 00:02:14,077
to make you trust and fear algorithms

43
00:02:14,101 --> 00:02:16,119
because you trust and fear mathematics.

44
00:02:17,387 --> 00:02:22,217
A lot can go wrong when we put
blind faith in big data.

45
00:02:23,504 --> 00:02:26,877
This is Kiri Soares.
She's a high school principal in Brooklyn.

46
00:02:26,901 --> 00:02:29,487
In 2011, she told me
her teachers were being scored

47
00:02:29,511 --> 00:02:32,238
with a complex, secret algorithm

48
00:02:32,262 --> 00:02:33,751
called the "value-added model."

49
00:02:34,325 --> 00:02:37,417
I told her, "Well, figure out
what the formula is, show it to me.

50
00:02:37,441 --> 00:02:38,982
I'm going to explain it to you."

51
00:02:39,006 --> 00:02:41,147
She said, "Well, I tried
to get the formula,

52
00:02:41,171 --> 00:02:43,943
but my Department of Education contact
told me it was math

53
00:02:43,967 --> 00:02:45,513
and I wouldn't understand it."

54
00:02:47,086 --> 00:02:48,424
It gets worse.

55
00:02:48,448 --> 00:02:51,978
The New York Post filed
a Freedom of Information Act request,

56
00:02:52,002 --> 00:02:54,961
got all the teachers' names
and all their scores

57
00:02:54,985 --> 00:02:57,767
and they published them
as an act of teacher-shaming.

58
00:02:58,904 --> 00:03:02,764
When I tried to get the formulas,
the source code, through the same means,

59
00:03:02,788 --> 00:03:04,937
I was told I couldn't.

60
00:03:04,961 --> 00:03:06,197
I was denied.

61
00:03:06,221 --> 00:03:07,395
I later found out

62
00:03:07,419 --> 00:03:10,285
that nobody in New York City
had access to that formula.

63
00:03:10,309 --> 00:03:11,614
No one understood it.

64
00:03:13,749 --> 00:03:16,973
Then someone really smart
got involved, Gary Rubinstein.

65
00:03:16,997 --> 00:03:20,618
He found 665 teachers
from that New York Post data

66
00:03:20,642 --> 00:03:22,508
that actually had two scores.

67
00:03:22,532 --> 00:03:24,413
That could happen if they were teaching

68
00:03:24,437 --> 00:03:26,876
seventh grade math and eighth grade math.

69
00:03:26,900 --> 00:03:28,438
He decided to plot them.

70
00:03:28,462 --> 00:03:30,455
Each dot represents a teacher.

71
00:03:30,924 --> 00:03:33,303
(Laughter)

72
00:03:33,327 --> 00:03:34,848
What is that?

73
00:03:34,872 --> 00:03:36,149
(Laughter)

74
00:03:36,173 --> 00:03:39,619
That should never have been used
for individual assessment.

75
00:03:39,643 --> 00:03:41,569
It's almost a random number generator.

76
00:03:41,593 --> 00:03:44,539
(Applause)

77
00:03:44,563 --> 00:03:45,725
But it was.

78
00:03:45,749 --> 00:03:46,925
This is Sarah Wysocki.

79
00:03:46,949 --> 00:03:49,124
She got fired, along
with 205 other teachers,

80
00:03:49,148 --> 00:03:51,810
from the Washington, DC school district,

81
00:03:51,834 --> 00:03:54,743
even though she had great
recommendations from her principal

82
00:03:54,767 --> 00:03:56,195
and the parents of her kids.

83
00:03:57,210 --> 00:03:59,242
I know what a lot
of you guys are thinking,

84
00:03:59,266 --> 00:04:01,753
especially the data scientists,
the AI experts here.

85
00:04:01,777 --> 00:04:06,003
You're thinking, "Well, I would never make
an algorithm that inconsistent."

86
00:04:06,673 --> 00:04:08,356
But algorithms can go wrong,

87
00:04:08,380 --> 00:04:12,978
even have deeply destructive effects
with good intentions.

88
00:04:14,351 --> 00:04:16,730
And whereas an airplane
that's designed badly

89
00:04:16,754 --> 00:04:18,755
crashes to the earth and everyone sees it,

90
00:04:18,779 --> 00:04:20,629
an algorithm designed badly

91
00:04:22,065 --> 00:04:25,930
can go on for a long time,
silently wreaking havoc.

92
00:04:27,568 --> 00:04:29,138
This is Roger Ailes.

93
00:04:29,162 --> 00:04:31,162
(Laughter)

94
00:04:32,344 --> 00:04:34,732
He founded Fox News in 1996.

95
00:04:35,256 --> 00:04:37,837
More than 20 women complained
about sexual harassment.

96
00:04:37,861 --> 00:04:41,096
They said they weren't allowed
to succeed at Fox News.

97
00:04:41,120 --> 00:04:43,640
He was ousted last year,
but we've seen recently

98
00:04:43,664 --> 00:04:46,334
that the problems have persisted.

99
00:04:47,474 --> 00:04:48,874
That begs the question:

100
00:04:48,898 --> 00:04:51,782
What should Fox News do
to turn over another leaf?

101
00:04:53,065 --> 00:04:56,106
Well, what if they replaced
their hiring process

102
00:04:56,130 --> 00:04:57,784
with a machine-learning algorithm?

103
00:04:57,808 --> 00:04:59,403
That sounds good, right?

104
00:04:59,427 --> 00:05:00,727
Think about it.

105
00:05:00,751 --> 00:05:02,856
The data, what would the data be?

106
00:05:02,880 --> 00:05:07,827
A reasonable choice would be the last
21 years of applications to Fox News.

107
00:05:07,851 --> 00:05:09,353
Reasonable.

108
00:05:09,377 --> 00:05:11,315
What about the definition of success?

109
00:05:11,741 --> 00:05:13,065
Reasonable choice would be,

110
00:05:13,089 --> 00:05:14,867
well, who is successful at Fox News?

111
00:05:14,891 --> 00:05:18,471
I guess someone who, say,
stayed there for four years

112
00:05:18,495 --> 00:05:20,149
and was promoted at least once.

113
00:05:20,636 --> 00:05:22,197
Sounds reasonable.

114
00:05:22,221 --> 00:05:24,575
And then the algorithm would be trained.

115
00:05:24,599 --> 00:05:28,476
It would be trained to look for people
to learn what led to success,

116
00:05:29,039 --> 00:05:33,357
what kind of applications
historically led to success

117
00:05:33,381 --> 00:05:34,675
by that definition.

118
00:05:36,020 --> 00:05:37,795
Now think about what would happen

119
00:05:37,819 --> 00:05:40,374
if we applied that
to a current pool of applicants.

120
00:05:40,939 --> 00:05:42,568
It would filter out women

121
00:05:43,483 --> 00:05:47,413
because they do not look like people
who were successful in the past.

122
00:05:51,572 --> 00:05:54,109
Algorithms don't make things fair

123
00:05:54,133 --> 00:05:56,827
if you just blithely,
blindly apply algorithms.

124
00:05:56,851 --> 00:05:58,333
They don't make things fair.

125
00:05:58,357 --> 00:06:00,485
They repeat our past practices,

126
00:06:00,509 --> 00:06:01,692
our patterns.

127
00:06:01,716 --> 00:06:03,655
They automate the status quo.

128
00:06:04,538 --> 00:06:06,927
That would be great
if we had a perfect world,

129
00:06:07,725 --> 00:06:09,037
but we don't.

130
00:06:09,061 --> 00:06:13,163
And I'll add that most companies
don't have embarrassing lawsuits,

131
00:06:14,266 --> 00:06:16,854
but the data scientists in those companies

132
00:06:16,878 --> 00:06:19,067
are told to follow the data,

133
00:06:19,091 --> 00:06:21,234
to focus on accuracy.

134
00:06:22,093 --> 00:06:23,474
Think about what that means.

135
00:06:23,498 --> 00:06:27,525
Because we all have bias,
it means they could be codifying sexism

136
00:06:27,549 --> 00:06:29,385
or any other kind of bigotry.

137
00:06:31,308 --> 00:06:32,729
Thought experiment,

138
00:06:32,753 --> 00:06:34,262
because I like them:

139
00:06:35,394 --> 00:06:38,369
an entirely segregated society --

140
00:06:40,067 --> 00:06:43,395
racially segregated, all towns,
all neighborhoods

141
00:06:43,419 --> 00:06:46,456
and where we send the police
only to the minority neighborhoods

142
00:06:46,480 --> 00:06:47,673
to look for crime.

143
00:06:48,271 --> 00:06:50,490
The arrest data would be very biased.

144
00:06:51,671 --> 00:06:54,246
What if, on top of that,
we found the data scientists

145
00:06:54,270 --> 00:06:58,431
and paid the data scientists to predict
where the next crime would occur?

146
00:06:59,095 --> 00:07:00,582
Minority neighborhood.

147
00:07:01,105 --> 00:07:04,230
Or to predict who the next
criminal would be?

148
00:07:04,708 --> 00:07:06,103
A minority.

149
00:07:07,769 --> 00:07:11,310
The data scientists would brag
about how great and how accurate

150
00:07:11,334 --> 00:07:12,631
their model would be,

151
00:07:12,655 --> 00:07:13,954
and they'd be right.

152
00:07:15,771 --> 00:07:20,386
Now, reality isn't that drastic,
but we do have severe segregations

153
00:07:20,410 --> 00:07:21,697
in many cities and towns,

154
00:07:21,721 --> 00:07:23,614
and we have plenty of evidence

155
00:07:23,638 --> 00:07:26,326
of biased policing
and justice system data.

156
00:07:27,452 --> 00:07:30,267
And we actually do predict hotspots,

157
00:07:30,291 --> 00:07:31,821
places where crimes will occur.

158
00:07:32,221 --> 00:07:36,087
And we do predict, in fact,
the individual criminality,

159
00:07:36,111 --> 00:07:37,881
the criminality of individuals.

160
00:07:38,792 --> 00:07:42,755
The news organization ProPublica
recently looked into

161
00:07:42,779 --> 00:07:44,803
one of those "recidivism risk" algorithms,

162
00:07:44,827 --> 00:07:45,990
as they're called,

163
00:07:46,014 --> 00:07:49,208
being used in Florida
during sentencing by judges.

164
00:07:50,231 --> 00:07:53,816
Bernard, on the left, the black man,
was scored a 10 out of 10.

165
00:07:54,999 --> 00:07:57,006
Dylan, on the right, 3 out of 10.

166
00:07:57,030 --> 00:07:59,531
10 out of 10, high risk.
3 out of 10, low risk.

167
00:08:00,418 --> 00:08:02,803
They were both brought in
for drug possession.

168
00:08:02,827 --> 00:08:03,981
They both had records,

169
00:08:04,005 --> 00:08:06,811
but Dylan had a felony

170
00:08:06,835 --> 00:08:08,011
but Bernard didn't.

171
00:08:09,638 --> 00:08:12,704
This matters, because
the higher score you are,

172
00:08:12,728 --> 00:08:16,201
the more likely you're being given
a longer sentence.

173
00:08:18,114 --> 00:08:19,408
What's going on?

174
00:08:20,346 --> 00:08:21,678
Data laundering.

175
00:08:22,750 --> 00:08:27,177
It's a process by which
technologists hide ugly truths

176
00:08:27,201 --> 00:08:29,022
inside black box algorithms

177
00:08:29,046 --> 00:08:30,336
and call them objective;

178
00:08:31,140 --> 00:08:32,708
call them meritocratic.

179
00:08:34,938 --> 00:08:37,323
When they're secret,
important and destructive,

180
00:08:37,347 --> 00:08:39,834
I've coined a term for these algorithms:

181
00:08:39,858 --> 00:08:41,857
"weapons of math destruction."

182
00:08:41,881 --> 00:08:43,445
(Laughter)

183
00:08:43,469 --> 00:08:46,523
(Applause)

184
00:08:46,547 --> 00:08:48,901
They're everywhere,
and it's not a mistake.

185
00:08:49,515 --> 00:08:53,238
These are private companies
building private algorithms

186
00:08:53,262 --> 00:08:54,654
for private ends.

187
00:08:55,034 --> 00:08:58,248
Even the ones I talked about
for teachers and the public police,

188
00:08:58,272 --> 00:09:00,141
those were built by private companies

189
00:09:00,165 --> 00:09:02,396
and sold to the government institutions.

190
00:09:02,420 --> 00:09:04,293
They call it their "secret sauce" --

191
00:09:04,317 --> 00:09:06,445
that's why they can't tell us about it.

192
00:09:06,469 --> 00:09:08,689
It's also private power.

193
00:09:09,744 --> 00:09:14,439
They are profiting for wielding
the authority of the inscrutable.

194
00:09:16,934 --> 00:09:19,868
Now you might think,
since all this stuff is private

195
00:09:19,892 --> 00:09:21,050
and there's competition,

196
00:09:21,074 --> 00:09:23,380
maybe the free market
will solve this problem.

197
00:09:23,404 --> 00:09:24,653
It won't.

198
00:09:24,677 --> 00:09:27,797
There's a lot of money
to be made in unfairness.

199
00:09:28,947 --> 00:09:32,316
Also, we're not economic rational agents.

200
00:09:32,851 --> 00:09:34,143
We all are biased.

201
00:09:34,780 --> 00:09:38,157
We're all racist and bigoted
in ways that we wish we weren't,

202
00:09:38,181 --> 00:09:40,200
in ways that we don't even know.

203
00:09:41,172 --> 00:09:44,253
We know this, though, in aggregate,

204
00:09:44,277 --> 00:09:47,497
because sociologists
have consistently demonstrated this

205
00:09:47,521 --> 00:09:49,186
with these experiments they build,

206
00:09:49,210 --> 00:09:51,778
where they send a bunch
of applications to jobs out,

207
00:09:51,802 --> 00:09:54,303
equally qualified but some
have white-sounding names

208
00:09:54,327 --> 00:09:56,033
and some have black-sounding names,

209
00:09:56,057 --> 00:09:58,751
and it's always disappointing,
the results -- always.

210
00:09:59,330 --> 00:10:01,101
So we are the ones that are biased,

211
00:10:01,125 --> 00:10:04,554
and we are injecting those biases
into the algorithms

212
00:10:04,578 --> 00:10:06,390
by choosing what data to collect,

213
00:10:06,414 --> 00:10:09,157
like I chose not to think
about ramen noodles --

214
00:10:09,181 --> 00:10:10,806
I decided it was irrelevant.

215
00:10:10,830 --> 00:10:16,514
But by trusting the data that's actually
picking up on past practices

216
00:10:16,538 --> 00:10:18,552
and by choosing the definition of success,

217
00:10:18,576 --> 00:10:22,559
how can we expect the algorithms
to emerge unscathed?

218
00:10:22,583 --> 00:10:24,939
We can't. We have to check them.

219
00:10:25,985 --> 00:10:27,694
We have to check them for fairness.

220
00:10:27,718 --> 00:10:30,429
The good news is,
we can check them for fairness.

221
00:10:30,453 --> 00:10:33,805
Algorithms can be interrogated,

222
00:10:33,829 --> 00:10:35,863
and they will tell us
the truth every time.

223
00:10:35,887 --> 00:10:38,380
And we can fix them.
We can make them better.

224
00:10:38,404 --> 00:10:40,779
I call this an algorithmic audit,

225
00:10:40,803 --> 00:10:42,482
and I'll walk you through it.

226
00:10:42,506 --> 00:10:44,702
First, data integrity check.

227
00:10:45,952 --> 00:10:48,609
For the recidivism risk
algorithm I talked about,

228
00:10:49,402 --> 00:10:52,975
a data integrity check would mean
we'd have to come to terms with the fact

229
00:10:52,999 --> 00:10:56,525
that in the US, whites and blacks
smoke pot at the same rate

230
00:10:56,549 --> 00:10:59,034
but blacks are far more likely
to be arrested --

231
00:10:59,058 --> 00:11:02,242
four or five times more likely,
depending on the area.

232
00:11:03,137 --> 00:11:05,963
What is that bias looking like
in other crime categories,

233
00:11:05,987 --> 00:11:07,438
and how do we account for it?

234
00:11:07,982 --> 00:11:11,021
Second, we should think about
the definition of success,

235
00:11:11,045 --> 00:11:12,426
audit that.

236
00:11:12,450 --> 00:11:15,202
Remember -- with the hiring
algorithm? We talked about it.

237
00:11:15,226 --> 00:11:18,391
Someone who stays for four years
and is promoted once?

238
00:11:18,415 --> 00:11:20,184
Well, that is a successful employee,

239
00:11:20,208 --> 00:11:23,287
but it's also an employee
that is supported by their culture.

240
00:11:23,909 --> 00:11:25,835
That said, also it can be quite biased.

241
00:11:25,859 --> 00:11:27,924
We need to separate those two things.

242
00:11:27,948 --> 00:11:30,374
We should look to
the blind orchestra audition

243
00:11:30,398 --> 00:11:31,594
as an example.

244
00:11:31,618 --> 00:11:34,374
That's where the people auditioning
are behind a sheet.

245
00:11:34,766 --> 00:11:36,697
What I want to think about there

246
00:11:36,721 --> 00:11:40,138
is the people who are listening
have decided what's important

247
00:11:40,162 --> 00:11:42,191
and they've decided what's not important,

248
00:11:42,215 --> 00:11:44,274
and they're not getting
distracted by that.

249
00:11:44,781 --> 00:11:47,530
When the blind orchestra
auditions started,

250
00:11:47,554 --> 00:11:50,998
the number of women in orchestras
went up by a factor of five.

251
00:11:52,073 --> 00:11:54,088
Next, we have to consider accuracy.

252
00:11:55,053 --> 00:11:58,787
This is where the value-added model
for teachers would fail immediately.

253
00:11:59,398 --> 00:12:01,560
No algorithm is perfect, of course,

254
00:12:02,440 --> 00:12:06,045
so we have to consider
the errors of every algorithm.

255
00:12:06,656 --> 00:12:11,015
How often are there errors,
and for whom does this model fail?

256
00:12:11,670 --> 00:12:13,388
What is the cost of that failure?

257
00:12:14,254 --> 00:12:16,461
And finally, we have to consider

258
00:12:17,793 --> 00:12:19,979
the long-term effects of algorithms,

259
00:12:20,686 --> 00:12:22,893
the feedback loops that are engendering.

260
00:12:23,406 --> 00:12:24,642
That sounds abstract,

261
00:12:24,666 --> 00:12:27,330
but imagine if Facebook engineers
had considered that

262
00:12:28,090 --> 00:12:32,945
before they decided to show us
only things that our friends had posted.

263
00:12:33,581 --> 00:12:36,815
I have two more messages,
one for the data scientists out there.

264
00:12:37,270 --> 00:12:40,679
Data scientists: we should
not be the arbiters of truth.

265
00:12:41,340 --> 00:12:45,123
We should be translators
of ethical discussions that happen

266
00:12:45,147 --> 00:12:46,441
in larger society.

267
00:12:47,399 --> 00:12:49,532
(Applause)

268
00:12:49,556 --> 00:12:51,112
And the rest of you,

269
00:12:51,831 --> 00:12:53,227
the non-data scientists:

270
00:12:53,251 --> 00:12:54,749
this is not a math test.

271
00:12:55,452 --> 00:12:56,800
This is a political fight.

272
00:12:58,407 --> 00:13:02,314
We need to demand accountability
for our algorithmic overlords.

273
00:13:03,938 --> 00:13:05,437
(Applause)

274
00:13:05,461 --> 00:13:09,686
The era of blind faith
in big data must end.

275
00:13:09,710 --> 00:13:10,877
Thank you very much.

276
00:13:10,901 --> 00:13:16,204
(Applause)