How (and How Not) to Write a Good Systems Paper 摘录

An Evaluation of the Ninth SOSP Submissions or How (and How Not) to Write a Good Systems Paper

• the intent: to point out the common problems that appear repeatedly in technical papers in a way that will make it easier for future authors to avoid them

Classes of Papers

- It presents a real system, either by a global survey of an entire system or by a selective examination of specific themes embodied in the system.
- It presents a system that is unimplemented but utilizes ideas or techniques that you feel the technical community should know.
- It addresses a topic in the theoretical areas, for example, performance modelling or security verification.

Criteria for Evaluation of Submissions

Original Ideas

1. Are the ideas in the paper new?
2. How do you know? 
3. Can you state the new idea concisely? 
4. What exactly is the problem being solved?
5. Are the ideas significant enough to justify a paper?
6. Is the work described significantly different from existing related work?
7. Is all related work referenced, and have you actually read the cited material?
8. Are comparisons with previous work clear and explicit? 
9. Does the work comprise a significant extension, validation, or repudiation of earlier but unproven ideas? 
10. What is the oldest paper you referenced? The newest?  

Try writing each idea down in a paragraph that someone generally versed in the relevant area can understand.

Frequently, papers describing real systems contain one or two small enhancements of established techniques. The new idea(s) can be described in a few paragraphs; a twenty-page paper is unnecessary and often obscures the actual innovation. Since construction of a real system is a lot of work, the author of the paper sometimes unconsciously confuses the total effort with the work that is actually new.

Designs are cheap, but implementations (particularly those based on unsound designs) are expensive.

If the reader needs to acquire some of that development, however, he must be able to convert your citations into source material he can read. Personal communications and internal memoranda fail this test. Technical reports are frequently published in limited quantities, out-of-print, and difficult to obtain. Consequently, such citations as source material should be avoided wherever possible.

Reality

11. Does the paper describe something that has actually been implemented?
12. If the system has been implemented, how has it been used, and what has this usage shown about the practical importance of the ideas?
13. If the system hasn't been implemented, do the ideas justify publication now?

A seemingly good idea that didn’t pan out is at least as interesting as one that did. It is important to be specific and precise.
Reviewers are very skeptical of design-only papers unless there are new ideas of obviously high quality.

Lessons

14. What have you learned from the work?
15. What should the reader learn from the paper?
16. How generally applicable are these lessons?

Be sure to state clearly the assumptions on which your conclusions rest.
When stating your conclusions, it helps to state the assumptions again.

Choices

17. What were the alternatives considered at various points, and why were the choices made the way they were?
18. Did the choices turn out to be right, and, if so, was it for the reasons that motivated them in the first place? If not, what lessons have you learned from the experience? How often have you found yourself saying "this works, but for the wrong reason"?

A good paper doesn’t just describe, it explains. Telling your readers what you did doesn’t give them any idea how carefully considered your choices were.

Context

19. What are the assumptions on which the work is based?
20. Are they realistic?
21. How sensitive is the work to perturbations of these assumptions?
22. If a formal model is presented, does it give new information and insights?

The skeptical reader is unlikely to accept your arguments unless their premises are stated. Make sure you get them all; it’s easy to overlook implicit assumptions.
If your result is delicately poised on a tall tower of fragile assumptions, it will be less useful to a reader than one that rests on a broader and firmer foundation.

Focus

23. Does the introductory material contain excess baggage not needed for your main development?
24. Do you include just enough material from previously published works to enable your reader to follow your thread of argument?

“Real system” papers are particularly guilty of irrelevant description.
Do not assume that the reader has read every referenced paper within the last week and has them at his fingertips for instant reference.
Don’t burden your reader unnecessarily with lengthy extracts or paraphrases from cited works.

Presentation

25. Are the ideas organized and presented in a clear and logical way?
26. Are terms defined before they are used?
27. Are forward references kept to a minimum? 
28. Have alternate organizations been considered?
29. Was an abstract written first? Does it communicate the important ideas of the paper? 
30. Is the paper finished?

Theoretical papers, particularly of a mathematical character, are generally easier to organize than papers describing systems. The expected sequence of definition, lemma, theorem, example, corollary works well for deductive argument, but poorly for description. In “real system” papers, much depends on the intent: global survey or selective treatment. Frequently, difficulties in organization result from the author’s unwillingness to commit to either approach. Decide whether you are surveying your system or focusing on a specific aspect and structure the paper accordingly.
It helps to write the abstract before the paper (despite tradition) and even the outline, since it focuses your attention on the main ideas you wants to convey.
Omitting sections with a promise to fill them in later is generally unacceptable.

Writing Style

31. Is the writing clear and concise?
32. Are words spelled and used correctly?
33. Are the sentences complete and grammatically correct?
34. Are ambiguity, slang, and cuteness avoided?

A reviewer is more favorably disposed toward you if he receives a clean, clear, carefully corrected manuscript than if it arrives on odd-sized paper after ten trips through a photocopier and looking like it was composed by a grade-school dropout.

Summary

These thirty-odd questions can help you write a better technical paper. Consult them often as you organize your presentation, write your first draft, and refine your manuscript into its final form. Some of these questions address specific problems in “systems” papers; others apply to technical papers in general. Writing a good paper is hard work, but you will be rewarded by a broader distribution and greater understanding of your ideas within the community of journal and proceedings readers.