1
00:00:12,795 --> 00:00:14,391
Algorithms are everywhere.
2
00:00:15,931 --> 00:00:19,056
They sort and separate
the winners from the losers.
3
00:00:19,839 --> 00:00:22,103
The winners get the job
4
00:00:22,127 --> 00:00:23,870
or a good credit card offer.
5
00:00:23,894 --> 00:00:26,545
The losers don't even get an interview
6
00:00:27,410 --> 00:00:29,187
or they pay more for insurance.
7
00:00:30,017 --> 00:00:33,566
We're being scored with secret formulas
that we don't understand
8
00:00:34,495 --> 00:00:37,712
that often don't have systems of appeal.
9
00:00:39,060 --> 00:00:40,356
That begs the question:
10
00:00:40,380 --> 00:00:43,293
What if the algorithms are wrong?
11
00:00:44,920 --> 00:00:46,960
To build an algorithm you need two things:
12
00:00:46,984 --> 00:00:48,965
you need data, what happened in the past,
13
00:00:48,989 --> 00:00:50,550
and a definition of success,
14
00:00:50,574 --> 00:00:53,031
the thing you're looking for
and often hoping for.
15
00:00:53,055 --> 00:00:58,092
You train an algorithm
by looking, figuring out.
16
00:00:58,116 --> 00:01:01,535
The algorithm figures out
what is associated with success.
17
00:01:01,559 --> 00:01:04,022
What situation leads to success?
18
00:01:04,701 --> 00:01:06,463
Actually, everyone uses algorithms.
19
00:01:06,487 --> 00:01:09,205
They just don't formalize them
in written code.
20
00:01:09,229 --> 00:01:10,577
Let me give you an example.
21
00:01:10,601 --> 00:01:13,917
I use an algorithm every day
to make a meal for my family.
22
00:01:13,941 --> 00:01:15,417
The data I use
23
00:01:16,214 --> 00:01:17,873
is the ingredients in my kitchen,
24
00:01:17,897 --> 00:01:19,424
the time I have,
25
00:01:19,448 --> 00:01:20,681
the ambition I have,
26
00:01:20,705 --> 00:01:22,414
and I curate that data.
27
00:01:22,438 --> 00:01:26,689
I don't count those little packages
of ramen noodles as food.
28
00:01:26,713 --> 00:01:28,582
(Laughter)
29
00:01:28,606 --> 00:01:30,451
My definition of success is:
30
00:01:30,475 --> 00:01:33,134
a meal is successful
if my kids eat vegetables.
31
00:01:34,001 --> 00:01:36,855
It's very different
from if my youngest son were in charge.
32
00:01:36,879 --> 00:01:39,667
He'd say success is if
he gets to eat lots of Nutella.
33
00:01:40,999 --> 00:01:43,225
But I get to choose success.
34
00:01:43,249 --> 00:01:45,956
I am in charge. My opinion matters.
35
00:01:45,980 --> 00:01:48,655
That's the first rule of algorithms.
36
00:01:48,679 --> 00:01:51,859
Algorithms are opinions embedded in code.
37
00:01:53,382 --> 00:01:57,045
It's really different from what you think
most people think of algorithms.
38
00:01:57,069 --> 00:02:01,573
They think algorithms are objective
and true and scientific.
39
00:02:02,207 --> 00:02:03,906
That's a marketing trick.
40
00:02:05,089 --> 00:02:07,214
It's also a marketing trick
41
00:02:07,238 --> 00:02:10,392
to intimidate you with algorithms,
42
00:02:10,416 --> 00:02:14,077
to make you trust and fear algorithms
43
00:02:14,101 --> 00:02:16,119
because you trust and fear mathematics.
44
00:02:17,387 --> 00:02:22,217
A lot can go wrong when we put
blind faith in big data.
45
00:02:23,504 --> 00:02:26,877
This is Kiri Soares.
She's a high school principal in Brooklyn.
46
00:02:26,901 --> 00:02:29,487
In 2011, she told me
her teachers were being scored
47
00:02:29,511 --> 00:02:32,238
with a complex, secret algorithm
48
00:02:32,262 --> 00:02:33,751
called the "value-added model."
49
00:02:34,325 --> 00:02:37,417
I told her, "Well, figure out
what the formula is, show it to me.
50
00:02:37,441 --> 00:02:38,982
I'm going to explain it to you."
51
00:02:39,006 --> 00:02:41,147
She said, "Well, I tried
to get the formula,
52
00:02:41,171 --> 00:02:43,943
but my Department of Education contact
told me it was math
53
00:02:43,967 --> 00:02:45,513
and I wouldn't understand it."
54
00:02:47,086 --> 00:02:48,424
It gets worse.
55
00:02:48,448 --> 00:02:51,978
The New York Post filed
a Freedom of Information Act request,
56
00:02:52,002 --> 00:02:54,961
got all the teachers' names
and all their scores
57
00:02:54,985 --> 00:02:57,767
and they published them
as an act of teacher-shaming.
58
00:02:58,904 --> 00:03:02,764
When I tried to get the formulas,
the source code, through the same means,
59
00:03:02,788 --> 00:03:04,937
I was told I couldn't.
60
00:03:04,961 --> 00:03:06,197
I was denied.
61
00:03:06,221 --> 00:03:07,395
I later found out
62
00:03:07,419 --> 00:03:10,285
that nobody in New York City
had access to that formula.
63
00:03:10,309 --> 00:03:11,614
No one understood it.
64
00:03:13,749 --> 00:03:16,973
Then someone really smart
got involved, Gary Rubinstein.
65
00:03:16,997 --> 00:03:20,618
He found 665 teachers
from that New York Post data
66
00:03:20,642 --> 00:03:22,508
that actually had two scores.
67
00:03:22,532 --> 00:03:24,413
That could happen if they were teaching
68
00:03:24,437 --> 00:03:26,876
seventh grade math and eighth grade math.
69
00:03:26,900 --> 00:03:28,438
He decided to plot them.
70
00:03:28,462 --> 00:03:30,455
Each dot represents a teacher.
71
00:03:30,924 --> 00:03:33,303
(Laughter)
72
00:03:33,327 --> 00:03:34,848
What is that?
73
00:03:34,872 --> 00:03:36,149
(Laughter)
74
00:03:36,173 --> 00:03:39,619
That should never have been used
for individual assessment.
75
00:03:39,643 --> 00:03:41,569
It's almost a random number generator.
76
00:03:41,593 --> 00:03:44,539
(Applause)
77
00:03:44,563 --> 00:03:45,725
But it was.
78
00:03:45,749 --> 00:03:46,925
This is Sarah Wysocki.
79
00:03:46,949 --> 00:03:49,124
She got fired, along
with 205 other teachers,
80
00:03:49,148 --> 00:03:51,810
from the Washington, DC school district,
81
00:03:51,834 --> 00:03:54,743
even though she had great
recommendations from her principal
82
00:03:54,767 --> 00:03:56,195
and the parents of her kids.
83
00:03:57,210 --> 00:03:59,242
I know what a lot
of you guys are thinking,
84
00:03:59,266 --> 00:04:01,753
especially the data scientists,
the AI experts here.
85
00:04:01,777 --> 00:04:06,003
You're thinking, "Well, I would never make
an algorithm that inconsistent."
86
00:04:06,673 --> 00:04:08,356
But algorithms can go wrong,
87
00:04:08,380 --> 00:04:12,978
even have deeply destructive effects
with good intentions.
88
00:04:14,351 --> 00:04:16,730
And whereas an airplane
that's designed badly
89
00:04:16,754 --> 00:04:18,755
crashes to the earth and everyone sees it,
90
00:04:18,779 --> 00:04:20,629
an algorithm designed badly
91
00:04:22,065 --> 00:04:25,930
can go on for a long time,
silently wreaking havoc.
92
00:04:27,568 --> 00:04:29,138
This is Roger Ailes.
93
00:04:29,162 --> 00:04:31,162
(Laughter)
94
00:04:32,344 --> 00:04:34,732
He founded Fox News in 1996.
95
00:04:35,256 --> 00:04:37,837
More than 20 women complained
about sexual harassment.
96
00:04:37,861 --> 00:04:41,096
They said they weren't allowed
to succeed at Fox News.
97
00:04:41,120 --> 00:04:43,640
He was ousted last year,
but we've seen recently
98
00:04:43,664 --> 00:04:46,334
that the problems have persisted.
99
00:04:47,474 --> 00:04:48,874
That begs the question:
100
00:04:48,898 --> 00:04:51,782
What should Fox News do
to turn over another leaf?
101
00:04:53,065 --> 00:04:56,106
Well, what if they replaced
their hiring process
102
00:04:56,130 --> 00:04:57,784
with a machine-learning algorithm?
103
00:04:57,808 --> 00:04:59,403
That sounds good, right?
104
00:04:59,427 --> 00:05:00,727
Think about it.
105
00:05:00,751 --> 00:05:02,856
The data, what would the data be?
106
00:05:02,880 --> 00:05:07,827
A reasonable choice would be the last
21 years of applications to Fox News.
107
00:05:07,851 --> 00:05:09,353
Reasonable.
108
00:05:09,377 --> 00:05:11,315
What about the definition of success?
109
00:05:11,741 --> 00:05:13,065
Reasonable choice would be,
110
00:05:13,089 --> 00:05:14,867
well, who is successful at Fox News?
111
00:05:14,891 --> 00:05:18,471
I guess someone who, say,
stayed there for four years
112
00:05:18,495 --> 00:05:20,149
and was promoted at least once.
113
00:05:20,636 --> 00:05:22,197
Sounds reasonable.
114
00:05:22,221 --> 00:05:24,575
And then the algorithm would be trained.
115
00:05:24,599 --> 00:05:28,476
It would be trained to look for people
to learn what led to success,
116
00:05:29,039 --> 00:05:33,357
what kind of applications
historically led to success
117
00:05:33,381 --> 00:05:34,675
by that definition.
118
00:05:36,020 --> 00:05:37,795
Now think about what would happen
119
00:05:37,819 --> 00:05:40,374
if we applied that
to a current pool of applicants.
120
00:05:40,939 --> 00:05:42,568
It would filter out women
121
00:05:43,483 --> 00:05:47,413
because they do not look like people
who were successful in the past.
122
00:05:51,572 --> 00:05:54,109
Algorithms don't make things fair
123
00:05:54,133 --> 00:05:56,827
if you just blithely,
blindly apply algorithms.
124
00:05:56,851 --> 00:05:58,333
They don't make things fair.
125
00:05:58,357 --> 00:06:00,485
They repeat our past practices,
126
00:06:00,509 --> 00:06:01,692
our patterns.
127
00:06:01,716 --> 00:06:03,655
They automate the status quo.
128
00:06:04,538 --> 00:06:06,927
That would be great
if we had a perfect world,
129
00:06:07,725 --> 00:06:09,037
but we don't.
130
00:06:09,061 --> 00:06:13,163
And I'll add that most companies
don't have embarrassing lawsuits,
131
00:06:14,266 --> 00:06:16,854
but the data scientists in those companies
132
00:06:16,878 --> 00:06:19,067
are told to follow the data,
133
00:06:19,091 --> 00:06:21,234
to focus on accuracy.
134
00:06:22,093 --> 00:06:23,474
Think about what that means.
135
00:06:23,498 --> 00:06:27,525
Because we all have bias,
it means they could be codifying sexism
136
00:06:27,549 --> 00:06:29,385
or any other kind of bigotry.
137
00:06:31,308 --> 00:06:32,729
Thought experiment,
138
00:06:32,753 --> 00:06:34,262
because I like them:
139
00:06:35,394 --> 00:06:38,369
an entirely segregated society --
140
00:06:40,067 --> 00:06:43,395
racially segregated, all towns,
all neighborhoods
141
00:06:43,419 --> 00:06:46,456
and where we send the police
only to the minority neighborhoods
142
00:06:46,480 --> 00:06:47,673
to look for crime.
143
00:06:48,271 --> 00:06:50,490
The arrest data would be very biased.
144
00:06:51,671 --> 00:06:54,246
What if, on top of that,
we found the data scientists
145
00:06:54,270 --> 00:06:58,431
and paid the data scientists to predict
where the next crime would occur?
146
00:06:59,095 --> 00:07:00,582
Minority neighborhood.
147
00:07:01,105 --> 00:07:04,230
Or to predict who the next
criminal would be?
148
00:07:04,708 --> 00:07:06,103
A minority.
149
00:07:07,769 --> 00:07:11,310
The data scientists would brag
about how great and how accurate
150
00:07:11,334 --> 00:07:12,631
their model would be,
151
00:07:12,655 --> 00:07:13,954
and they'd be right.
152
00:07:15,771 --> 00:07:20,386
Now, reality isn't that drastic,
but we do have severe segregations
153
00:07:20,410 --> 00:07:21,697
in many cities and towns,
154
00:07:21,721 --> 00:07:23,614
and we have plenty of evidence
155
00:07:23,638 --> 00:07:26,326
of biased policing
and justice system data.
156
00:07:27,452 --> 00:07:30,267
And we actually do predict hotspots,
157
00:07:30,291 --> 00:07:31,821
places where crimes will occur.
158
00:07:32,221 --> 00:07:36,087
And we do predict, in fact,
the individual criminality,
159
00:07:36,111 --> 00:07:37,881
the criminality of individuals.
160
00:07:38,792 --> 00:07:42,755
The news organization ProPublica
recently looked into
161
00:07:42,779 --> 00:07:44,803
one of those "recidivism risk" algorithms,
162
00:07:44,827 --> 00:07:45,990
as they're called,
163
00:07:46,014 --> 00:07:49,208
being used in Florida
during sentencing by judges.
164
00:07:50,231 --> 00:07:53,816
Bernard, on the left, the black man,
was scored a 10 out of 10.
165
00:07:54,999 --> 00:07:57,006
Dylan, on the right, 3 out of 10.
166
00:07:57,030 --> 00:07:59,531
10 out of 10, high risk.
3 out of 10, low risk.
167
00:08:00,418 --> 00:08:02,803
They were both brought in
for drug possession.
168
00:08:02,827 --> 00:08:03,981
They both had records,
169
00:08:04,005 --> 00:08:06,811
but Dylan had a felony
170
00:08:06,835 --> 00:08:08,011
but Bernard didn't.
171
00:08:09,638 --> 00:08:12,704
This matters, because
the higher score you are,
172
00:08:12,728 --> 00:08:16,201
the more likely you're being given
a longer sentence.
173
00:08:18,114 --> 00:08:19,408
What's going on?
174
00:08:20,346 --> 00:08:21,678
Data laundering.
175
00:08:22,750 --> 00:08:27,177
It's a process by which
technologists hide ugly truths
176
00:08:27,201 --> 00:08:29,022
inside black box algorithms
177
00:08:29,046 --> 00:08:30,336
and call them objective;
178
00:08:31,140 --> 00:08:32,708
call them meritocratic.
179
00:08:34,938 --> 00:08:37,323
When they're secret,
important and destructive,
180
00:08:37,347 --> 00:08:39,834
I've coined a term for these algorithms:
181
00:08:39,858 --> 00:08:41,857
"weapons of math destruction."
182
00:08:41,881 --> 00:08:43,445
(Laughter)
183
00:08:43,469 --> 00:08:46,523
(Applause)
184
00:08:46,547 --> 00:08:48,901
They're everywhere,
and it's not a mistake.
185
00:08:49,515 --> 00:08:53,238
These are private companies
building private algorithms
186
00:08:53,262 --> 00:08:54,654
for private ends.
187
00:08:55,034 --> 00:08:58,248
Even the ones I talked about
for teachers and the public police,
188
00:08:58,272 --> 00:09:00,141
those were built by private companies
189
00:09:00,165 --> 00:09:02,396
and sold to the government institutions.
190
00:09:02,420 --> 00:09:04,293
They call it their "secret sauce" --
191
00:09:04,317 --> 00:09:06,445
that's why they can't tell us about it.
192
00:09:06,469 --> 00:09:08,689
It's also private power.
193
00:09:09,744 --> 00:09:14,439
They are profiting for wielding
the authority of the inscrutable.
194
00:09:16,934 --> 00:09:19,868
Now you might think,
since all this stuff is private
195
00:09:19,892 --> 00:09:21,050
and there's competition,
196
00:09:21,074 --> 00:09:23,380
maybe the free market
will solve this problem.
197
00:09:23,404 --> 00:09:24,653
It won't.
198
00:09:24,677 --> 00:09:27,797
There's a lot of money
to be made in unfairness.
199
00:09:28,947 --> 00:09:32,316
Also, we're not economic rational agents.
200
00:09:32,851 --> 00:09:34,143
We all are biased.
201
00:09:34,780 --> 00:09:38,157
We're all racist and bigoted
in ways that we wish we weren't,
202
00:09:38,181 --> 00:09:40,200
in ways that we don't even know.
203
00:09:41,172 --> 00:09:44,253
We know this, though, in aggregate,
204
00:09:44,277 --> 00:09:47,497
because sociologists
have consistently demonstrated this
205
00:09:47,521 --> 00:09:49,186
with these experiments they build,
206
00:09:49,210 --> 00:09:51,778
where they send a bunch
of applications to jobs out,
207
00:09:51,802 --> 00:09:54,303
equally qualified but some
have white-sounding names
208
00:09:54,327 --> 00:09:56,033
and some have black-sounding names,
209
00:09:56,057 --> 00:09:58,751
and it's always disappointing,
the results -- always.
210
00:09:59,330 --> 00:10:01,101
So we are the ones that are biased,
211
00:10:01,125 --> 00:10:04,554
and we are injecting those biases
into the algorithms
212
00:10:04,578 --> 00:10:06,390
by choosing what data to collect,
213
00:10:06,414 --> 00:10:09,157
like I chose not to think
about ramen noodles --
214
00:10:09,181 --> 00:10:10,806
I decided it was irrelevant.
215
00:10:10,830 --> 00:10:16,514
But by trusting the data that's actually
picking up on past practices
216
00:10:16,538 --> 00:10:18,552
and by choosing the definition of success,
217
00:10:18,576 --> 00:10:22,559
how can we expect the algorithms
to emerge unscathed?
218
00:10:22,583 --> 00:10:24,939
We can't. We have to check them.
219
00:10:25,985 --> 00:10:27,694
We have to check them for fairness.
220
00:10:27,718 --> 00:10:30,429
The good news is,
we can check them for fairness.
221
00:10:30,453 --> 00:10:33,805
Algorithms can be interrogated,
222
00:10:33,829 --> 00:10:35,863
and they will tell us
the truth every time.
223
00:10:35,887 --> 00:10:38,380
And we can fix them.
We can make them better.
224
00:10:38,404 --> 00:10:40,779
I call this an algorithmic audit,
225
00:10:40,803 --> 00:10:42,482
and I'll walk you through it.
226
00:10:42,506 --> 00:10:44,702
First, data integrity check.
227
00:10:45,952 --> 00:10:48,609
For the recidivism risk
algorithm I talked about,
228
00:10:49,402 --> 00:10:52,975
a data integrity check would mean
we'd have to come to terms with the fact
229
00:10:52,999 --> 00:10:56,525
that in the US, whites and blacks
smoke pot at the same rate
230
00:10:56,549 --> 00:10:59,034
but blacks are far more likely
to be arrested --
231
00:10:59,058 --> 00:11:02,242
four or five times more likely,
depending on the area.
232
00:11:03,137 --> 00:11:05,963
What is that bias looking like
in other crime categories,
233
00:11:05,987 --> 00:11:07,438
and how do we account for it?
234
00:11:07,982 --> 00:11:11,021
Second, we should think about
the definition of success,
235
00:11:11,045 --> 00:11:12,426
audit that.
236
00:11:12,450 --> 00:11:15,202
Remember -- with the hiring
algorithm? We talked about it.
237
00:11:15,226 --> 00:11:18,391
Someone who stays for four years
and is promoted once?
238
00:11:18,415 --> 00:11:20,184
Well, that is a successful employee,
239
00:11:20,208 --> 00:11:23,287
but it's also an employee
that is supported by their culture.
240
00:11:23,909 --> 00:11:25,835
That said, also it can be quite biased.
241
00:11:25,859 --> 00:11:27,924
We need to separate those two things.
242
00:11:27,948 --> 00:11:30,374
We should look to
the blind orchestra audition
243
00:11:30,398 --> 00:11:31,594
as an example.
244
00:11:31,618 --> 00:11:34,374
That's where the people auditioning
are behind a sheet.
245
00:11:34,766 --> 00:11:36,697
What I want to think about there
246
00:11:36,721 --> 00:11:40,138
is the people who are listening
have decided what's important
247
00:11:40,162 --> 00:11:42,191
and they've decided what's not important,
248
00:11:42,215 --> 00:11:44,274
and they're not getting
distracted by that.
249
00:11:44,781 --> 00:11:47,530
When the blind orchestra
auditions started,
250
00:11:47,554 --> 00:11:50,998
the number of women in orchestras
went up by a factor of five.
251
00:11:52,073 --> 00:11:54,088
Next, we have to consider accuracy.
252
00:11:55,053 --> 00:11:58,787
This is where the value-added model
for teachers would fail immediately.
253
00:11:59,398 --> 00:12:01,560
No algorithm is perfect, of course,
254
00:12:02,440 --> 00:12:06,045
so we have to consider
the errors of every algorithm.
255
00:12:06,656 --> 00:12:11,015
How often are there errors,
and for whom does this model fail?
256
00:12:11,670 --> 00:12:13,388
What is the cost of that failure?
257
00:12:14,254 --> 00:12:16,461
And finally, we have to consider
258
00:12:17,793 --> 00:12:19,979
the long-term effects of algorithms,
259
00:12:20,686 --> 00:12:22,893
the feedback loops that are engendering.
260
00:12:23,406 --> 00:12:24,642
That sounds abstract,
261
00:12:24,666 --> 00:12:27,330
but imagine if Facebook engineers
had considered that
262
00:12:28,090 --> 00:12:32,945
before they decided to show us
only things that our friends had posted.
263
00:12:33,581 --> 00:12:36,815
I have two more messages,
one for the data scientists out there.
264
00:12:37,270 --> 00:12:40,679
Data scientists: we should
not be the arbiters of truth.
265
00:12:41,340 --> 00:12:45,123
We should be translators
of ethical discussions that happen
266
00:12:45,147 --> 00:12:46,441
in larger society.
267
00:12:47,399 --> 00:12:49,532
(Applause)
268
00:12:49,556 --> 00:12:51,112
And the rest of you,
269
00:12:51,831 --> 00:12:53,227
the non-data scientists:
270
00:12:53,251 --> 00:12:54,749
this is not a math test.
271
00:12:55,452 --> 00:12:56,800
This is a political fight.
272
00:12:58,407 --> 00:13:02,314
We need to demand accountability
for our algorithmic overlords.
273
00:13:03,938 --> 00:13:05,437
(Applause)
274
00:13:05,461 --> 00:13:09,686
The era of blind faith
in big data must end.
275
00:13:09,710 --> 00:13:10,877
Thank you very much.
276
00:13:10,901 --> 00:13:16,204
(Applause)