The AI Winter, ALPAC, and an Interesting History an AI‘er has to know

LANGUAGE AND MACHINES — COMPUTERS IN TRANSLATION AND LINGUISTICS

A Report by the
Automatic Language Processing Advisory Committee (ALPAC)
Division of Behavioral Sciences
National Academy of Sciences
National Research Council
Chairman: J. R. Pierce

What’s right

1. Being skeptical about MT: poor quality, not appearing to justify the level of financial support it had been receiving.
2. Identify the need to develop machine aids for translators.
3. Emphasize the need for more basic research in CL.

What’s wrong

1. Concentrating too exclusively on the translation needs of government agencies and not recognizing the needs of commerce and industry.
2. Making unfair comparisons between “a canned demo” and what truly sought to address more general problems.

Other References

• ALPAC: The (In)Famous Report — John Hutchins, 1996, MT News International
  This extensive summary includes therefore substantial extracts of ALPAC
• Good applications for crummy machine translation — Kenneth W. Church & Eduard H. Hovy, 1993
• Implications of Mechanical Translation Research — Victor H. Yngve, 1964, Proceedings of the American Philosophical Society

ALPAC

• The report is entitled: Languages and machines: computers in translation and linguistics. It was supposedly concerned, therefore, not just with MT but with the broader field of computational linguistics. [Hutchins 1996]
• It might be simpler and more economical for heavy users of Russian translations to learn to read the documents in the original language.

Is there a shortage of translation or translators?

• At the time of the report, no post of government translator was vacant while there were over 500 translators registered in the Washington area. There is no shortage of translators, although there may be a shortage of authorized positions for translators (some replied to the questionnaire by saying that they did not have sufficient requests for services to justify permanent positions). This, then, is a fiscal problem for the agencies and the Civil Service Commission, and not a problem for research and development offices supporting research in MT. [ALPAC]
• The Committe rejects any argument that there is a shortage of translation. Such an argument is rejected on the grounds that the demand for almost any free commodity is insatiable. (So they were not talking about publically available MT services, but only those that exclusively serve the government). [ALPAC]
• Avg salary of translators was markedly lower than that of scientists. [John 1996]. Thus, it is ironic that several agencies propose to spend more money for ‘MT’. The Committe is puzzled by a rationale for spending substantial sums of money on the mechanization of a small and already economically depressed industry. [ALPAC]

Regarding a Possible Excess of Translation

• We have found that many articles are being translated in our area that do not warrant the effort and it appears to use that some of the routine translations could be abandoned in order to make more translation services available for quick reaction to special requests. [ALPAC]
• According to a 1962 evaluation, only some 20-30% of Russian articles in some fields would have been accepted for publication in American journals; furthermore the delays in publication of cover-to-cover translations reduced their value. [Hutchins 1996]
• The number of respondents who preferred to get their Soviet information in the form of cover-to-cover translations was only half the number who preferred to get their translations as needed. [ALPAC]

The Crucial Problems of Translation

There is no emergence in the field of translation. The problem is not to meet some nonexistent need through nonexistent MT. There are, however, crucial problems of translation. There are quality, speed, and cost.

• Quality

1. Quality of translation must be adequate to the needs of the requester.
2. The correlation between cost and quality is far from precise. Such correlation as exists is probably better at the low than at the high end. In other words, a cheap translation is almost always defective in some way, while an expensive translation is not always of superior quality.

• Speed

1. There is considerable room for improvement.
2. 24.5 percent of the comments were to the effect that lag time should be reduced. The lag time in publication of the translated journals supported by NSF ranges from 15 to 26 weeks. Many of the delays do not lie in the process of translation itself, but rather in time spent in editing and production, and sometimes in avoidable delays. In FTD machine-aided translation, the delays are in production and postediting, together with the delays caused by queues in the many operations that must be done in tandem in this particular form of machine-aided translation.

• Cost

1. Machines are probably inappropriate for some forms of translations such as very high-quality diplomatic and literary translation. The government should apply some criteria in deciding on means of translation.
2. One might as well hire a few more translators and have the translations done by humans. Alternative would be to take part of the money spent on MT and use it either a) to raise salaries in order to hire bilingual analysts-thus avoiding translation altogether – or, b) to use the money to teach the analysts Russian.

The Present State of MT

MT Presumbaly means going by algorithm from src to target w/o recourse to human translation or editing.

• In this context, there has been no machine translation of general scientific text, and noun is in immediate prospect.
• After 8 years of work, the Georgetown University MT project tried to produce useful output in 1962, they had to resort to postediting. And this was deemed a failure/futile effort by the Committe. The postedited translation took slightly longer to do and was more expensive than conventional human translation.
• A recent ‘Reading Comprehension’ study showed that, in physics, the reader of raw MT output was 10% less accurate, 21% slower and had a comprehension level 29% lower than when used human translation.
• Georgetown Experiment: The biggest fraud! The reader will find it instructive to compare the samples above with the results obtained on simple, or selected, text 10 years earlier (the Georgetown IBM Experiment, January 7, 1954) in that the earlier samples are more readable than the later ones.
• Earlier machine translations of simple or selected text, such as those given above, were as deceptively encouraging as “MT of general scientific text” have been uniformly discouraging. [ALPAC]. In the end, the ALPAC report criticized both the 1954 experiment as well as the four systems in question, the former for setting expectiations unrealistically high, and the latter for failing to meet those expectations, unrealistic as they may be. [Church & Hovy 1993]
• 竟然说了一句好话 However, work toward machine translation has produced much valuable linguistic knowledge and insight that we would not otherwise have attained.
• I concur with your view of MT, that at present it serves no useful purpose w/o postediting, and that w/ postediting the overall process is slow and probably uneconomical, stated by Victor H. Yngve. [ALPAC]
• We have come face to face with the realization that we will only have adequate mechanical translation when the machine can “understand” what it is translating and this will be a very difficult task indeed. Here I want to underline the word “understand” because this is just what I mean. [Yngve 1964]
• Many of the former workers in mechanical translation are giving up in the face of the tremendous difficulties. It seems to them too remote a possibility that we shall ever be able to program machines to understand. But some of us are pressing forward undaunted, especially since the search is bound to lead us into a number of extremely interesting areas. [Yngve 1964]
• The Committe indeed believes that it is wise to press forward undaunted, in the name of science, but that the motive for doing so cannot sensibly be any forseeable improvement in practical translation. Perhaps our attitude might be different if there were some pressing need for MT, but we find none. [ALPAC]

Machine-Aided Translation (vs. Human-aided Machine Translation)

• The approach is conservative. A machine is used to produce specialized glossaries helpful in the translation of particular documents. This is slightly different from postediting as postediting “is actually a system of human-aided machine translation, relying, as it must, on posteditors to make up for the deficiencies of the machine output.”
• Personally I agree on the view that machines are here to make up for the defiencies of humans: do what they are good at without
  making human jobs harder. “I must confess that the results were most unhappy. I found that I spent at least as much time in editing as if I had carried out the entire translation from the start. Even at that, Idoubt if the edited translation reads as smoothly as one which I would have started from scratch. … machine today translates from a foreign language to a form of broken English somewhat comparable to pidgin English. But it then remains for the reader to learn this patois in order to understand what the Russian actually wrote. Learning Russian would not be much more difficult.”
• ALPAC points out the correct and promising avenue where MT should step into: “the output served as an aid… particularly with regard to technical terms”. A mimimum of 25% of the translator’s time is spent on terminological questions and that, in difficult documents, up to 75%.
• Appendix 14 oberserved that postediting tends to impede the rapid translators and assist the slow translators. [ALPAC]. [Church & Hovy] referred to this: This would suggest that EPE (extensive post-editing) products might be more appropriate for casual use by an amateur rather than daily use by a professional.
• CECA Automatic dictionary look-up with context included: Soon a system was devised by which the translator’s time-consuming job of finding the answers to questions of terminology was made easier. Furthermore, since the data produced by each query are added to the database, the more the system is in use, the greater is the probability of finding sentences that have the desired term in the proper context.
• The info built by CECA not only is of value in answering the queries of translators, but also anables CECA to public specialized glossaries in a very short time.

Automatic Language Processing and Computational Linguistics

• At the beginning of this chapter ALPAC made one of its most often cited statements [Hutchins1996]: " Over the past 10 years the government has spend, through various agencies, some $20 million on MT and closely related projects."
• It did recognize the contribution of MT research to linguistics. By far the most important outcome of work toward MT has been its effect on linguistics.

1. ‘A decade ago (1950s) most linguists believed that syntax had to do with word order, inflection, function words and intonation or punctuation. They also believed that most sentences uttered by native speakers in ordinary contexts were syntactically unambiguous. Today (1960s), they know that these two beliefes are mutually inconsistent. Their knowledge is the immediate result of computer parsing of ordinary sentences, using reasonable grammars as hitherto conceived and programs that expose all ambiguities under a fixed grammar.’ The revolution in linguistics has not been, solely a result of attempts at MT and parsing, but it is unlikely that the revolution would have been extensive or significant without these attempts.
2. More linguists than ever before are attempting to bring subtler theories into confrontation with richer bodies of data. The data we much examine in order to find out about language is overwhelming both in quantity and in complexity.
3. The life’s work of a generation ago (a concordance, a glossary, a superficial grammar) is the first small step of today, accomplished in a few weeks, the first 10,000 steps toward an understanding of natural langauge as the vehicle of human communications.
4. We see that the computer has opened up to linguists a host of challenges, partial insights and potentialities. We believe these can be aptly compared with the challenges, problems and insights of particle physics. Certainly, Language is second to no phenomenon in importance. And the tools of computational linguistics are considerably less costly than the multibillion-volt accelerators of particle physics.

• However, despite the favorable influence, ALPAC did not conclude that MT research as such should continue to receive support. [John 1996] That’s because they have been excluding all aspects of ‘MT research being a scientific endeavor’ when talking about financial support and were only evaluating its value from the engineering viewpoint, and, apprently, with ill-posed, over-ambitious, unrealistic expectations that would certainly favor the perspective against supporting MT.

Avenues to Improvement of Translation

• There is no immediate or predictable prospect of useful MT
• Something worth investing into: machine-aided human translation. What will help the human being most - special glossaries, dictionary look-up or a rough translation such as that produced by FTD? How can delays due to queues at many tandem steps be avoided? How can production costs be cut (i.e. through computerized publishing including auto line adjustment & hyphenation)? Auto char-recognition may also be adopted.
• It would be unreasonable to spend extravagantly on a relatively small business that is doing the job satisfactorily. Again, the Committee didn’t appreciate the value of MT as a commercial product that could benefit the general public in a way they couldn’t imagine. They regarded MT as a small business almost for the exlusive purpose of translating government documents (from Russian to English during the cold war)
• The Committee set aside MT as a scientific endeavor as a side conversation.

1. Despite their awareness of the importance of that undertake (being far-reaching, derailed from the context of real life) — linguistics should be supported as science, and should not be judged by any immediate or foreseeable contribution to practical translation — they concluded that it should be spent hardheadedly toward important, realistic, and relatively short-range goals.
2. John R.Pierce stressed the value of supporting ‘computational linguistics’ as distinct from automatic language translation. [Hutchins1996]
3. The chairman believed that funds should be provided for research on a reasonably large scale, “since small-scale experiments and work with miniature models of language have proved seriously deceptive in the past”. [Hutchins1996]

Anti-ALPAC

• The report concentrated exclusively on US government and military needs in the analysis and scanning of Russian-language documents. It was not concerned in any way with other potential uses or users of MT systems or with any other languages. [Hutchins 1996]
• It was perhaps a major flaw of ALPAC’s methodology to compare unfavourable the results of general-purpose MT systems (some still experimental) working from unprepared input (i.e. with no dictionary updating) and the output of a small-scale demonstration system built exclusively to handle and produce a restricted set of sentences. 对说的就是Georgetown Experiment. demo是不需要evaluation metrics的【狗头】
• By no means all this sum ($20 million) was spent on MT research in the US. It includes $101, 250 in support of research outside the US (at the Cambridge Language Research Unit) and $1,362,000 under Zellig Harris at UPenn which was not considered to be directly related to MT. Also, it lists global sums from the US Air Force, US Navy and US Army with no details of the recipients of the grants, suggesting that much of the funding was in support of developments in computer equipment rather than MT research.
• In brief, the number may well have been $12-13 million than the frequently repeated $20 million. The sum was still large, of course, and ALPAC was right to emphasize the poor return for the investment. Poor return could only be justified under the goal of ‘developing useful MT systems’ rather than investing on MT as a scientific devotion. Misalignments between scientists, funding providers, and the potential beneficiary of the research outcomes have led to to false claims of ‘poor return which counldn’t justify the huge amount of money invested on this’. But that was the matter of how you expect from MT research. As a scientific endeavor we should not care so much for short-term returns. But if the goal was getting immediate, practical deployment, then indeed all research outcomes were no good.
• Of course, the comparison between the result of GU experiment and that of other 4 then-current systems isn’t fair: the 1954 GU experiment had been a canned demo of the worst kind, whereas the four systems developed during the 1960s were intended to handle large quantities of previously unseen text. [Church & Hovy 1993]
• ALPAC reinforced the Anglocentric insularity in US research which damaged that country’s activities in multilingual NLP at a time when progress continued to take place in Europe and Japan. [Hutchins1996] This is much more 【caring / supportive for diversity / 政治正确】 than “we should not expect any more meaningful advancement in other languages when we haven’t learnt much about English”, state by [ 我真的忘了是在哪篇里看的了，哪天想起来了再回来补 ]