我爱读的paper
文章平均质量分 91
芝麻挞
这个作者很懒,什么都没留下…
展开
-
[Paper Summary] The Future of Computational Linguistics: On Beyond Alchemy
1950s empiricism: Info Theory AI as applied statistics1970s rationalism: formal language theory and logic1990s empiricism: stochastic grammars (probability & preference)2010s: empiricism: deep netsPastCL is an interdisciplinary topic that has b.原创 2021-07-28 05:11:14 · 321 阅读 · 0 评论 -
[Paper Summary: Modesty is the Formula for Success] Good applications for crummy machine translation
There is a risk that eval can devolve into mindless metrics.Good Applications for Crummy MT — Kenneth W. Church & Eduard H. Hovy, AT&T Bell Laboratories, USC ISIThe success of the eval often depends very strongly on the selection of an appropriat原创 2021-07-26 10:27:54 · 263 阅读 · 0 评论 -
[Paper Summary] Optimal Brain Damage
Optimal Brain Damage— LeCun, Denker and Solla, 1989, Advances in Neural Information Processing SystemsWe introduce OBD for reducing the size of a learning network by selectively deleting weights based on second-derivative information.We show that OBD原创 2021-07-26 08:45:29 · 278 阅读 · 0 评论 -
互相成就:Massive quantities of data; The new norm of eval
The data-intensive approach to language, which is becoming known as Text Analysis, takes a pragmatic approach that is well suited to meet the recent emphasis on numerical evaluations and concrete delieverables [A Pendulum Swung Too Far]. Text Analysis focu原创 2021-07-26 08:04:00 · 502 阅读 · 0 评论 -
Whither Speech Recognition: 25年又一个25年
Pierce’s harsh criticismWhither Speech Recognition — J.R. Pierce, 1969In deception, studied and artful deceit is apt to succeed better and more quickly than science. Indeed, a wag has proposed that computers are becoming so nearly human that they can act原创 2021-07-22 06:23:45 · 327 阅读 · 0 评论 -
[Paper Summary: My Machine sucks, or eval sucks? ... or both?] A Survey of 25 Years of Evaluation
A Survey of 25 Years of Evaluation — Kenneth Ward Church & Joel Hestness, 2019Sometimes the numbers are too good to be true, and sometimes the truth is better than the numbers. Sometimes the problem is not with the numbers but with the interpretation.原创 2021-07-22 04:51:50 · 148 阅读 · 0 评论 -
[Paper Summary] A Pendulum Swung Too Far
There is a trend of oscillation between Rationalism and Empiricism and back with a switch every couple decades.1950s: Empiricism (Shannon, Skinner, Firth, Harris)1970s: Ratinalism (Chomsky, Minsky)1990s: Empiricism (IBM Speech Group, AT&T Bell Labs.原创 2021-07-21 08:49:16 · 364 阅读 · 0 评论 -
Explainability & Reviewing: The responsbility finally goes back to the audience, i.e EVERYONE
A call for Explanation — Insights should be much more valued than numbersEmerging trends: I did it, I did it, I did it, but … — Kenneth Ward Church, 2017Does it make it ok for machines to do bad things if no one knows what's happening and why, including原创 2021-07-21 08:48:07 · 92 阅读 · 0 评论 -
What we (I) do on surface, what we (I) do by heart
Emerging trends: A tribute to Charles Wayne — Kenneth Ward Church, 2017Charles Wayne restarted funding in speech and language in the mid-1980s, after a funding winter brought on by Pierce’s glamour-and-deceit criticisms in the ALPAC report and Whither Spe原创 2021-07-21 08:46:32 · 97 阅读 · 0 评论 -
MT from winter to spring, the spring has lasted till now.
When is the next ALPAC report due? — Margaret King,1984, University of Geneva, SwitzerlandMachine translation has a somewhat chequered history — Margaret KingThere were already proposals for automatic translation systems in the 30’s, but it was not unt.原创 2021-07-16 06:14:06 · 193 阅读 · 0 评论 -
The AI Winter, ALPAC, and an Interesting History an AI‘er has to know
ALPACThe report is entitled: Languages and machines: computers in translation and linguistics. It was supposedly concerned, therefore, not just with MT but with the broader field of computational linguistics. [Hutchins 1996]It might be simpler and more原创 2021-07-15 04:02:26 · 319 阅读 · 0 评论 -
[Meta Summary] A stream of paper in probing Transformer Language Models
一开始知道Probing这个方向大概就是从Voita的NLP with friends talk吧,当时模糊地理解为是一种"neural network interpretation"的方式,其实这种理解没有错误,只是它只说中了一半,这周读了若干篇paper along this stream, 发现probing还有一个目的是serve as evaluation metrics for representation learning. 不过这就更玄学了,比NLG的evaluation还要玄学。起码NLG原创 2021-07-11 07:58:10 · 116 阅读 · 0 评论 -
[Paper Summary] Pareto Probing: Trading Off Acc for Complexity [Pimental 2020]
Pareto Probing: Trading Off Acc for Complexity [Pimental 2020]KeypointsCall for harder probing tasksToy probing tasks, such as POS labeling and dependency arc labeling are inadequate to evaluate the linguistic feature encoded in contextual word repres原创 2021-07-11 07:05:08 · 106 阅读 · 0 评论 -
[Paper Summary] Information-theoretic probing for linguistic structure [Pimental 2020]
Information-theoretic probing for linguistic structure [Pimental 2020]Teaser… under our operationalization, the endeavour of finding syntax in contextualized embeddings sentences is nonsensical. This is because, under Assumption 1, we know the answer a原创 2021-07-10 22:29:06 · 136 阅读 · 0 评论 -
[Paper Summary] Designing and Interpreting Probes with Control Tasks [Hewitt & Liang 2019]
Designing and Interpreting Probes with Control Tasks [[Hewitt & Liang 2019](https://arxiv.org/abs/1909.03368)]tl;drA good probe should be selective, achieving high linguistic task acc and low control task acc.Motivation for control tasksFavor ‘原创 2021-07-10 22:13:34 · 142 阅读 · 0 评论 -
[Paper Summary] When Do You Need Billions of Words of Pretraining Data? [Zhang 2020]
When Do You Need Billions of Words of Pretraining Data? [Zhang 2020]很棒很棒 总结完了才意识到这篇的discussion都是彩蛋都很有深意!Core research question: What exact knowledge or skills do Transformer LMs learn from large-scale pretraining that they cannot learn from less data?原创 2021-07-10 07:13:12 · 265 阅读 · 0 评论 -
[Paper Summary] Frustratingly Short Attn Spans in Neural LM [Daniluk 2017]
Frustratingly Short Attn Spans in Neural LM这是2017年的ICLR很老的文而且主要的贡献是用了不同的representation for key, value and next-word distribution,当时命名还不太一样,他的key就是现在的query, 他的value是现在的key,他的next-word distribution是现在的value。读的时候我一度认为现在的标准做法 projection to Q, K, V就是一开始源自这篇,原创 2021-07-10 05:01:59 · 65 阅读 · 0 评论 -
[Paper Summary] What Will it Take to Fix Benchmarking in NLU? [Bowman 2021]
What Will it Take to Fix Benchmarking in NLU? [Bowman 2021]tl;drWe lay out 4 criteria that we argue NLU benchmarks should meet. Most current benchmarks fail at these critieria, and adversarial data collection does not meaningfully address the causes of原创 2021-07-10 04:20:00 · 139 阅读 · 0 评论 -
[Paper Summary] A Primer in BERTology: What We Know About How BERT Works [Rogers 2020]
A Primer in BERTology: What We Know About How BERT Works [Rogers 2020]Probing works strive to learn about the types of of linguistic (e.g. POS, dependency, [Warstadt2019] - Five Analysis with NPIs, [Warstadt 2020] - RoBERTa acquires a preference for lingu原创 2021-07-08 08:05:23 · 282 阅读 · 0 评论 -
[Paper Summary] Information-Theoretic Probing with Minimum Description Length [Voita & Titov 2020]
Information-Theoretic Probing with Minimum Description Length原创 2021-07-08 04:28:30 · 120 阅读 · 0 评论 -
[Paper Summary] Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals [Elazar 2020]
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactualstl;drProbing results cannot infer behavioral conclusions. Probing provides no evidence for or against the actual use of this information by the model.We focus on how the information is原创 2021-07-07 22:39:55 · 128 阅读 · 0 评论 -
[Paper Summary] oLMpics - On what LM Pre-training Captures [Talmor 2019]
oLMpics - On what LM Pre-training CapturesKeypointsWe propose a diverse set of probing tasks for types of symbolic reasoning that are potentially difficult to capture using a LM objective.We provide an analysis of skills that current LMs possess. Their原创 2021-07-07 08:43:48 · 80 阅读 · 0 评论 -
[Paper Summary] Evaluating repres. by the complexity of learning low-loss predictors [Whitney 2020]
Evaluating representations by the complexity of learning low-loss predictorstl;dr上一个thread都是acc - complexity trade-off. 然而这篇认为As the eval dataset size changes, the dynamics of acc - complexity trade-off also changes. And if you use metrics that handles原创 2021-07-04 05:41:27 · 194 阅读 · 0 评论