The Paper Reviewing Process
Posted: October 18, 2013 Filed under: advice, research, reviewing Leave a commentLearning how to review papers not only (obviously) makes you a better reviewer, but it can also help you as an author, since an understanding of the process can help you write your paper submissions for an audience of reviewers. If you know the criteria that a reviewer will use to judge your paper, you are in a much better position to tailor your paper so that it has a higher chance of being accepted.
There are many good resources that describe the paper reviewing process already, including those that explain the process (and its imperfections) and those that provide instructions for writing a good review (as well as techniques to avoid). There are also a few nice summaries of the review process for conferences in different areas of computer science that lend visibility into the process (e.g., hereand here). Program committee chairs sometimes provide guidelines for writing reviews, such as these. I will not reiterate or summarize those previous articles here, but they are all definitely worth a read. Instead, I will discuss the importance of the review process and how it differs from simply reading a paper; I’ll also talk about how to prepare (and ultimately write) a review.
I will not talk about the paper selection process (i.e., what determines whether a paper is ultimately accepted or rejected), but will instead focus on the creation of a paper review. Program committee meetings are an important part of the paper selection process—at least in computer science—and I will be devoting a complete post to this topic next week. Meanwhile, I recommend reading Matt Welsh’s post on the psychology of program committees.
The Review Process
Why understanding the review process is important. Whether you end up reviewing a lot of papers as a Ph.D. student, your research will definitely be subject to the paper review process. It is imperative as a researcher to understand this process. Knowing the process can help you better write your paper for an audience of reviewers (and a program committee), and it can also help you maintain perspective when your paper is accepted or rejected. The process is far from perfect, and the outcome of the process is neither validation nor condemnation of your work. How you react—and how you adapt your research or follow through on it after the acceptance (or rejection)—is far more important to long-term success.
In the “Introduction to the Ph.D.” class at Georgia Tech, I ask students to create a research idea and write it up; a subsequent set of assignments asks the students to review and evaluate the ideas as part of a “mock” program committee. The process isn’t exactly the same as the review process for a full paper, but it is a lightweight way to have students experience the process first-hand in a low-stakes setting, and see both sides of the process (submission and review) at the same time. In next week’s blog post, I will discuss program committee meetings in general, as well as some observations from this year’s (and previous years’) in-class experiences with the mock PC.
Reviewing vs. reading. There are some significant distinctions between reading papers vs. reviewing them. When reading a paper for your own enrichment, your goal is to gather information as quickly as possible. In this case, you are a scientist who seeks to understand the context and content of existing work, to (for example) better understand how your own research might fit into the bigger picture or learn about techniques that might apply to your own work. The goal of reviewing is different. A reviewer’s goal is to first and foremost determine the suitability of a paper for some conference and second, to provide feedback to the authors to help them improve the paper in subsequent revisions. Remember that the reviewer’s primary goal trumps all other objectives: A reviewer often has a large number of papers to process and is typically not deeply devoted to improving the content of any particular paper. If you are lucky, you will get a diligent, thoughtful reviewer who provides thorough feedback, but do not be surprised if a review is not as thorough as you would have liked, or if the review “misses” some point you were trying to make. We would all like reviewers to make three passes through your paper submission—and, these are the instructions I would give, too, in an ideal world. Unfortunately, however, you will be lucky in many cases to get two thorough reads. The reviewer’s main goal is to determine the paper’s suitability for publication. As an author, you shouldn’t be surprised if some of the comments seem trivial: there may be underlying issues of taste that drove the reviewer’s opinion on your paper that a reviewer may not explicitly state. Whenever I read reviews I receive for a rejected paper, I try to look past specific detailed quibbles (or “excuses” for rejecting the paper) and figure out the big picture: the reviewer couldn’t find a reason to accept the paper.
Calibration: Reviewing one paper vs. reviewing many papers. The paper review process can differ depending on who, exactly, is reviewing the paper. For example, as a Ph.D. student, you may review one or two papers at a time, as an “external reviewer” for a conference or journal. Journal editors and program committee chairs often seek the help of external reviewers if they need a particular subject-matter expert to review a paper. Later in your Ph.D. career, you may have established yourself as an expert on a particular topic and find yourself reviewing a paper here and there on a handful of topics. Sometimes a member of the program committee (e.g., your advisor) might ask you to help review a particular paper. As you progress in your career, you will be asked to serve on program committees yourself, whereupon you’ll find yourself with tens of papers to review over the course of a couple of months. Ironically, it is sometimes easier to review a group of papers than a single (or a few) papers, because seeing a group of papers helps you “calibrate” your scores and rankings of papers according to the general quality of papers that have been submitted to the conference. If you have been asked to review a single paper for a conference, you should either figure out how to calibrate your assessment with respect to other papers that might have been submitted, or simply review the paper on its merits while reserving judgement as to the paper’s ultimate disposition.
Does the Paper Realize a Great Idea?
Look for a reason to accept the paper. Does it realize a great contribution or idea? Every paper is imperfect. The paper may have made an incorrect or imperfect assumption. The experiments may not have been as thorough as you liked. The graphs may be difficult to read. Parts of the paper may be difficult to understand. These types of issues certainly reflect problems with a paper, but they do not necessarily constitute a reason to reject a paper if they do not affect the correctness or significance of the main underlying conclusion or contribution of the paper. Therefore, the first two questions I ask myself when reviewing a paper are: (1) Does the paper have a great idea?; and (2) Does it realize the great idea? (or, alternatively, to what extent does it realize that great idea, since typically no paper is water-tight).
What makes an idea “great”? Judging a paper’s contribution turns out to be highly subjective, which is why the review process remains so uncertain. A paper isn’t judged on a set of fixed checkboxes, a grading “key”, or any notion of absolute correctness. Reviewers often reserve considerable judgment based on “taste“, and reasonable people will disagree as to the merits of the main contribution or idea in a paper. In fact, there has been a fair amount of documentation that, as reviewers, we are often quite terrible at predicting the merits of a particular piece of submitted work: There’s a great article on this topic, as well as some parodies to illustrate the subjective nature of the process. Many fields have also introduced a “test of time” award to papers from past decades, to recognize accepted papers that have truly had long-term positive impact (implicitly acknowledging that this is almost impossible to assess when a paper is first published). Due to the subjective nature of this judgment, it is all the more important that your writing is clear, and well-matched to what a reviewer is looking for (i.e., the contributions and ideas).
Invariant questions. Different conferences may have different value structures, and the chairs of any given conference may ask the reviewers to focus on different criteria when judging a paper. Regardless, there are some invariant questions that most reviewers would (or at least should) always consider, including:
- Is the problem important? What problem is the paper trying to solve, and is it important? Seek to summarize the paper’s contribution in one sentence. Make this short summary the beginning of your review, as well. Try to convince yourself (by reading the paper or otherwise) that a solution to the problem that the paper is proposing would advance knowledge or significantly improve the state of affairs for some group of people. Note that you may not care about the problem, but also ask yourself whether you can imagine some group of readers who will be interested in the solution to the problem. When asking yourself this question about a paper, try to divorce your own taste about the problem’s importance from the more general question concerning whether there is some group of people who would be interested in the problem the paper is addressing and solving.
- To what extent does the paper solve the problem it describes? A single paper very rarely closes the book on a single problem, but it may take an important step towards solving the problem. It might solve the problem for an important set of operating conditions or under a new set of assumptions. Or, if the problem area is completely new, perhaps the paper doesn’t really solve the problem at all, but simply articulating a new problem area for follow-on work is a significant contribution.
- What is the “intellectual nugget”? As a reviewer, I try to identify whether a paper has a particular intellectual kernel that lies at the heart of the solution. This kernel is often what separates an important research contribution from a simple matter of engineering. This intellectual nugget might be the application (or invention) of a particular technique, a proof of correctness (where one previously did not exist), or an attempt to put the solution into a broader intellectual context. In other words, the intellectual contribution might be to take a general problem and tackle a specific sub-problem (e.g., under certain assumptions or conditions), or to take a specific problem and generalize it (e.g., develop a general theory, proof of correctness, or taxonomy). Looking through the paper for applications of specific research patterns can help identify an intellectual nugget, if one exists.
- What is the main contribution or conclusion? Is it important? As a reviewer, I try to concisely articulate the paper’s main contribution (or small number of contributions). Often, a paper will helpfully summarize those contributions somewhere in the introduction (Jim Kurose’s advice on writing paper introductions advises the writer to explicitly do so). The reviewer’s job is then to assess whether those contributions are significant or important enough to warrant a publication. The significance of those contributions often depends on the perceived increment over previous work. All work is incremental to some degree, as everything builds on past work. The author’s job is to convince the reviewer that the increment is important, and the reviewer’s job is to assess the author’s claims of significance.
- Does the content support the conclusion? An introduction may make broad (or wild) claims, and it is important to dig into the paper to determine whether the content of the paper supports the conclusion. Are the experiments run correctly? Are they based on the correct set of assumptions? If the conclusion involves comparison to previous work, is the comparison performed in a controlled manner, using an equivalent (or at least fair) experimental setup? If applicable, have the authors released their code and data so that you (or others) can check the claims yourself?
Preparing Your Review
Consider the audience. Not every publication venue is the same. Some venues are explicitly geared towards acceptance of early, incomplete work that is likely to generate discussion (many workshops use this criterion for acceptance). Other venues favor contributions that constitute well-executed, smaller increments. When reviewing a paper, either externally or as a member of a committee, your first question should be to consider the audience for the conference, workshop, or journal, and whether the likely audience for the venue would benefit from reading the paper. The question of audience involves that of both the “bar” for acceptance (Does the paper meet the audience’s standards for something that is worth reading?) and the “scope” of the venue (is the paper on-topic for the venue?). Often, scope can be (and is) broadly construed, so the key question really boils down to whether the likely audience for the paper will benefit from reading it.
Consider the standards. Your standards will (and should) vary depending on the venue for which you are reviewing a paper submission. Workshops are typically more permissive as far as accepting “vision” papers that outline a new problem or problem area or papers that “foster discussion” than conferences, which typically aim to accept more complete pieces of work. Nevertheless, even the standards for a conference review process will vary depending on both the conference itself, the program committee chair’s instructions about how permissive to be, and the relative quality of the group of papers that you are reviewing. A good way to get a sense for the standards of a conference for which you are reviewing is to read through the complete set of papers that you have been asked to review and rank them, before writing a single review. This will ensure some level of calibration, although it is still biased based on the set of papers that you are reviewing. Reading past proceedings of the particular journal or conference can also help you determine the appropriate standard to set for acceptance.
Consider the purpose. Different papers serve different purposes. Multiple paper submissions to the same venue might in fact have quite different purposes, and it is important to establish what the paper is contributing (or attempting to contribute) before passing judgement. For example, a paper might be a complete piece of work, but it might also be a survey, a tutorial, or simply a proposal. If the paper is one of the latter types, your first questions as a reviewer should concern whether the audience would benefit from the survey, tutorial, or proposal, and whether such a paper meets the standards for the conference. If the answers to those questions are “yes”, then your evaluation should be tailored to the paper’s purpose. If the paper is a survey, your assessment should be based on the completeness of the survey, with respect to the area that the paper is claiming to summarize. If the paper is a tutorial, is the description correct and clearly described? If the paper is a proposal, does the proposed research agenda make sense, and is the outcome (if the proposal is successful) worthwhile?
Consider the big picture. Every paper can be rejected. It is always easy to find reasons to reject a paper. The reviewer’s goal should not be to identify the reasons to reject a paper, but rather to determine whether there are any reasons to accept the paper. If the answer to that question is negative, then it is always easy to find “excuses” to reject a paper (recall the discussion above). You should be aiming to figure out whether the paper has important contributions that the audience will benefit from knowing about, and whether the paper supports those contributions and conclusions to the level of standard that is commensurate with the standard of the audience and the venue. One litmus test I use to ensure that a negative aspect of a paper does not condemn it is to ask myself whether the problem (1) affects the main conclusion or contribution of the paper; and (2) can be fixed easily in a revision. If the problem doesn’t affect the main contribution or conclusion, and if it can be easily fixed, then it should not negatively affect a paper’s review.
Writing Your Review
Start with a summary of the paper and its contributions. A short, one-paragraph summary describing the paper’s main contribution(s) demonstrates to the authors (and to you!) that you understand the main point of the paper. This helps you as a reviewer articulate the main contributions and conclusions of the paper for the purposes of your own evaluation. Try to address the type of paper it is (is it a survey paper, for example?), the context for the paper (i.e., how it builds on or relates to previous work), its overall correctness, and its contributions. If you cannot concisely summarize the paper, then the paper is not in good shape, and you can reflect this assessment in the review, as well. These summaries are very helpful to authors, since they may not match the authors’ views of the main contribution! For example, as an author, you can easily figure out if you’ve “missed the mark” or whether the reviewer fundamentally misunderstood the paper by reading a reviewer’s summary of your own work. If the summary of the contribution does not match your own view of the paper’s contribution, then you know that you have some work to do in writing and presentation.
Assess whether the paper delivers on the main claims and contributions. You should provide an assessment, for each of the paper’s main claims and contributions, whether it delivers on that claim. If the main contribution of the paper is flawed, you should indicate whether you think a flaw is “fatal”, or whether the authors could simply fix the flaw in a revision if the paper is accepted. Sometimes flaws (e.g., inconsistent terminology) are fixable. Other flaws (e.g., a questionable experimental setup) may or may not be fixable. While it might seem that a broken experimental setup is “fatal”, ask yourself as a reviewer whether the conclusions from the paper’s experiments as is are still meaningful, even if the authors have not interpreted the results correctly. If the conclusions from the experiments can be restated and still turn out to be meaningful contributions—or, if the flaw in an experiment doesn’t affect the main contribution or conclusion—then even a flaw in experiments can likely be fixed in revision. Occasionally, however, experiments may need to be completely redesigned because they don’t support any meaningful conclusion. Or, the content of the paper may simply be incorrect; sometimes correctness issues are difficult for a reviewer to spot, so a paper isn’t necessarily “correct” simply because a reviewer has validated the paper. Regardless, if there are correctness issues that affect the main contribution of the paper that call into question whether the main result or contribution is correct in the first place, the paper’s review should reflect these concerns and likely cannot be accepted.
Discuss positive aspects of the paper; always try to find something positive, even in “bad” papers. It is easy to identify problems with a paper. It can be much trickier (especially with “average” papers) to identify the positive aspects and contributions, but most papers typically have at least some small kernel of goodness. Even for particularly bad papers, there might be one sentence in the introduction, discussion, or future work section that makes an interesting point or highlights a possibility for interesting contributions. In a pinch, if you can’t find anything positive, those are good places to look. As a reviewer, you can remark that those observations are interesting, and that you would really like to see those parts of the work further developed. These positive comments aren’t just for author morale (although that’s important, too): They give the author a direction to move forward. The worst reviews are those that reject a paper but don’t provide any specific action for moving forward. The best reviews are those that highlight the positive aspects of the work, while identifying weaknesses and areas where the work could be further developed to address weaknesses or build on the paper’s existing strengths.
Criticize the paper, not the authors. When writing your review, consider the type of review that you would like to receive. Always be polite, respectful, and positive. Don’t be personal. Choose your language carefully, as it will help convey your message. For example, if you say “the authors don’t consider the related work”, that is a much more personal statement than “the paper doesn’t consider the related work”. (In fact, you don’t know if the authors considered a particular piece of related work anyway; they may have simply chosen not to include it in the writeup!) Talking about “the authors” gets personal, and it will put the authors themselves on the defensive when reading your review. Instead, focus on “the paper” and frame your critique around “suggestions for improvement”. Never, ever insult the authors; don’t accuse the authors of being sloppy or unethical researchers. As a reviewer, you don’t always know the full context, so limit your judgement to what you can directly conclude by reading the paper.
Consider the type of feedback you would like to receive. Receiving reviews for rejected papers is a part of the research process, but it is never fun for the authors (particularly new Ph.D. students). Do your part to contribute positively to the process by suggesting changes that you’d like to see if you had to review the paper again. In all likelihood, you may see the paper again in the form of a revision!
Little did I know that a Ph.D. in computer science would teach me more about how to write than any other previous experience I’d had. Indeed, research isn’t only about making new discoveries, but also about expressing those discoveries so that other people can appreciate them and their significance.
Below, I’ll present some thoughts on writing that I’ve picked up over the years and continually aim (and encourage students) to put into practice. I’ll focus on story-telling, presentation, and efficiency: how to write your paper so that it clearly, cleanly, and efficiently tells a good story.
What this post is not about. At least in this post, I won’t write about grammar—there is too much English grammar to cover in a single post, and there are plenty of good courses one can take to learn grammar. I also won’t talk about style (common phrases) or usage (e.g., commonly misused words). For now, my only advice on the topic of learning English grammar and style—both for native speakers and foreigners—is to read daily. Reading the New York Times, or The Economist are great places to start: the writing is generally of high quality, and you’ll also have something interesting to talk about outside of the lab. I have the impression that foreign students can tend towards reading their local news from home; that’s understandable, but I would encourage those students to also read periodicals and publications that are written in the style of English that people are used to reading in the Western world (indeed, English in different parts of the world has different tones, styles, idioms, and turns of phrase).
Why Does Writing Well Matter?
Writing is part and parcel of the research process. Writing communicates your research results, and—perhaps contrary to intuition—writing well is even more important when the results that one wants to present are “strong” (after all, if the results are that great, you should want as many people as possible to benefit from understanding them). But writing isn’t just something that happens at when the research is “done”; rather, writing should occur throughout the course of a research project. Writing and thinking are tightly coupled—writing is a reflection of our own thoughts, and writing well can actually sometimes clarify our own thinking about a problem and lead us to a solution.
Communicating your results. A research finding or discovery that others cannot understand or appreciate is as good as no discovery in the first place. A well-written paper with a half-baked idea that people can understand and appreciate will often be accepted to a conference over a paper with a (possibly) stronger result that nobody can understand or appreciate. Cynics may dismiss this as “unfair”, as “laziness” on the reviewer’s part for not reading the details thoroughly, or as “accepting a paper because it was well-written” (insinuating that the results themselves might not be that strong). The fact of the matter is that none of this matters: Independently of the quality of the research, the results must be written well, or people will not be able to appreciate the value of your work. Writing well is perhaps one of the most important skills that budding researchers must attain; it is equally important as learning how to conduct the research.
Clarifying your own thinking. Writing is a reflection of your thinking. You should never be content with vague language or imprecise descriptions. Muddy writing often reflects muddy thinking. An attempt to write down a concept, idea, or solution can actually help you realize that you don’t know what you are talking about! (And, help you crystallize your thinking.) You may have had the experience of realizing you don’t understand something as well as you thought you did when someone asks you to explain something and you have trouble. Writing achieves the same effect: as you attempt to write something down, you may realize that you don’t know why you are working on something, why a particular phenomenon appears, and so forth. If you don’t force yourself to try to write down your explanations and observations, you may never discover that you don’t know what you’re doing until it’s too late. Therefore, start writing early in the course of a research project. Most of the early text you write may not find its way into the final version of the paper, but taking the time to express your ideas on paper can help you crystallize your thoughts and guide your research.
Tell Me a Story
Every paper should tell a story. For a given research result, there are many stories that a paper could tell, and it’s important to think about what the story should be. This decision should be made early on in the writing process. The story can be changed later on, to match the strongest results in the paper, or if the context for the research changes (for example, sometimes when working on a research problem, people can find it necessary to re-formulate the problem that they are solving). But, it’s always good to figure out a story before you go too far down the path.
Determining the Big Picture: Composition and “Flow”. Composition is the structure and flow of your paper writeup, and how individual paragraphs fit together. When people talk about the “flow” of a piece of writing, they’re talking about composition: how sentences flow together to form paragraphs, how paragraphs flow together to form sections, and how sections flow together to form a paper. Composition (or “flow”) is the most important aspect of technical writing. I’ve found that reviewers (and readers) are often forgiving of the occasional grammatical error, especially if a paper was written by non-native English speakers. The human brain is pretty good at applying “error correction” on grammar. Correcting “flow”, on the other hand, is extremely hard for a reader to do. Often, the reader will complain that they “can’t understand where [the paper] is going” and simply give up. Grammar is also easier to teach (and learn) than composition because there are fixed rules for grammar; composition is more of an art form, and it does involve a bit of intuition. It involves thinking about the big picture, the story that you want to tell, and how the pieces fit together. It takes practice to develop intuition for composition, and I’ve found that native English speakers are not necessarily any better at composition than foreigners are (that’s good news for Ph.D. students coming from abroad: the playing field is actually more level than it appears). Even though composition is a bit of an art form, I’ve found that the following tips can be incredibly helpful for improving flow:
- Build the scaffolding before filling in the details. Write sections and topic sentences first. There is always the temptation to simply jump in and start writing full paragraphs. This rarely ends well. Nobody would build a house before first drawing plans. After the plans comes a foundation, and scaffolding. Only after the structure is in place can many of the details be filled in. You should think about writing a paper in the same way: before you can fill in the details, you need to figure out how you want the paper to read when its done; this is your outline. Start at a high level, by outlining sections. Once you have an outline of sections, you can begin to slowly fill in more details: outline each paragraph within a section by writing only the topic sentence for the paragraph. See whether your story makes sense if you read each topic sentence in sequence. Your “paper” should roughly make sense if you read these topic sentences in sequence (in fact, some reviewers or readers may try to read your paper in exactly this fashion, by skimming topic sentences!). If reading your topic sentences in sequence doesn’t make sense, something is wrong with your flow. Determining which topic sentences to write takes a bit of practice, but I find that making a bulleted list of points that you want to include in the section and then clustering those points into related points and topics can help identify the topics you want to cover (and, hence, the topic sentences you want to write). If you find that the topics you want to cover are not all related to one another, you may need to create additional subsections. Once you have topic sentences, you can fill in paragraphs, which are groups of closely (and logically) related sentences. If your sentences aren’t closely related, they don’t belong in the same paragraph and you may need to consider adding a new topic (and topic sentence).
- Assess balance and budget pages accordingly. Once you have your outline, it’s important to budget pages, to figure out how much you plan to write about each topic. Topics of equal importance or equal relevance should receive “balanced” coverage in the paper. If your system has two components of relatively equal importance, then the sections covering those components should be of approximately equal length in the paper. Once you have your outline (from the previous step), you can assess how much space you want to budget for each point. If some points in a section are more important than others (and hence require more page budget), you can go back and think about how to re-balance your writeup (e.g., by creating a new devoted section to the topic that’s more important, shortening or eliminating the less important point, and so forth).
- Follow convention. Continuing the analogy of a house or building, many designs follow convention: a restroom in a restaurant is often in the back. A powder room in a house is often near the front door. These common “design patterns” make life easier for us when we visit a new building or house. Similarly, your reader is a visitor of your thought process. A reader who has read (and written) many research papers will similarly expect the structure of your paper to follow certain conventions. These conventions differ by field: some fields write papers completely in the passive voice, whereas others use active voice. Some fields prefer related work sections at the beginning of the paper, and others prefer related work at the end. Some fields expect an introduction that spends time framing the problem, and others expect an introduction that starts with a clear statement of the main result. There may be an expectation of (or aversion) to mathematics in the main line of the paper. Within a section, people may expect the introduction to flow in a particular way (in computer science, Jim Kurose’s description of how to write an intro is the best example I’ve seen for writing a conventional paper introduction).
- Signpost your paper. Even if your paper plan and outline make perfect sense to you, the reader doesn’t start out “on the same page” as you. Even if the builders have built a house completely according to the blueprints, even if the design of the house follows convention, visitors still typically appreciate an explanation of where the bathroom is. The reader needs instructions for how to read the paper and a clear view for how the paper and story will proceed. Signposts throughout the paper help the reader understand how the paper is organized. Examples of signposts include: an outline of the paper at the end of the introduction (“The rest of this paper proceeds as follows.”), a preamble to each section (“In this section, we discuss…”), declarative subsection titles, and (within reason) bold paragraph headings (such as those in this blog post).
Make a Good Impression
A reader forms an impression about your research based on your presentation of the work. Just as people quickly form first impressions about other people that may be difficult to subsequently correct, your reader will quickly form a first impression of your work based on a quick read of the title, abstract, and introduction and perhaps a quick skim through other parts of the paper. Therefore, it is important to make a good impression by writing a clear abstract and introduction (and also ensuring that the rest of the paper also is commensurate with the impression that you create).
Spend a lot of time on your introduction; start early. The introduction summarizes the story of your paper (now, it should be doubly clear why you need a story!) and is the most important part of your paper. Many people won’t read past the introduction, even if they find the work interesting. Yet, if people don’t understand your “story” (the problem you’re solving, its importance, and your approach to solving it), they will never make it past the introduction into the details of the paper. There are two schools of thought for writing the introduction: if you write it first, you ensure that you have a clear story. If you write it last, then your introduction will clearly reflect the ultimate story and be bolstered by your results. My advice is to do both. Writing the introduction is probably the most difficult part of writing the paper, which may come as a surprise to new students. After all, the introduction is high-level and often doesn’t reflect the nitty gritty details of how difficult that experiment was to run, and how much toil went into writing the system. To the contrary, distilling your work from those details so that the rest of the world can digest the takeaways—stepping out of the weeds and trying to figure out why your work matters—can take more effort than anything. The same results can be presented in many different ways, so you will likely need to perform many iterations on your story. Typically, you can “test” your story out on other colleagues and readers with a draft introduction in parallel with completing the rest of your work.
Write the introduction first—and last. I typically try (and encourage students) to write a draft of the paper’s introduction before writing other parts of the paper. You will most likely eventually discard this draft completely, since it’s very hard to predict what one should be emphasizing in the introduction before the “meat” of the paper is done. Yet, writing an early introduction draft allows a researcher to put the work in context, and to argue (1) what the problem is, (2) why it is hard, and (3) why the solution (if it is achieved) will be interesting to readers. If an introduction draft can’t articulate answers to these questions, even at an early stage in the research, then the work likely needs some focus, as well. When you write the second version of your introduction (after the rest of the paper is written), you are in a good position to refine your story, making sure that any claims you make are clearly supported by your results and data. Your results should be stated as clearly and specifically as possible. You should neither understate your results (which might cause the reader to lose interest and stop reading) or overstate them (which might cause the reader to be annoyed or “let down” when they dig in and find that you don’t deliver on what you’ve promised).
Perform some “landscaping” on your paper. Your paper should be pleasing to the eye. Care in preparation of the paper suggests that you also cared about the research itself. Conversely, if the preparation of the paper is sloppy, how can the reader trust that you were not similarly sloppy in writing code, setting up and running your experiments, and analyzing your results? Therefore, take some time to make your paper look nice. Here are a few quick tips on “landscaping” your paper.
- Eliminate widows. Section headings at the bottoms of columns or pages look sloppy. Depending on your word-processing tool, there are good ways to get rid of widows. In Latex, you can always use a “\vfill” to kick the section heading to the top of the next column. Similarly, widows on paragraphs (paragraphs that end with a single word on the last line) look sloppy. Typically you can eliminate such widows by wordsmithing (see below on omitting needless words). Latex also has macros such as “\widowpenalty” that help ensure that the text layout avoids widows whenever possible.
- Place (and create) signposts, figures, and graphs to create whitespace and avoid “walls of text”. Reading long installments of technical writing is tiring, and seeing a page of uninterrupted technical writing can psychologically tire the reader before they even begin to read. Create whitespace and take other steps to break up large chunks of text. Use paragraph headings to break up long strings of text (ensuring that your headings match the topics you’ve outlined, of course). Try to space figures, tables, and graphs throughout the paper so that most pages have a mix of figures and text. If you don’t have a figure to break up the text, consider making one. Often, a table summarizing results can be useful. Or, you might find that a figure (or block of pseudocode) describes a concept better than a paragraph in your writeup. Take steps to ensure that your writeup is not a long stream of text.
- Check your spelling. Run a spell checker. Every word-processing tool has one, and a writeup that is rife with spelling errors reflects carelessness. Again, if you can’t be bothered to check your spelling, why should a reader trust that you spent time debugging your experiment or code? While a reader can forgive grammatical errors, especially when the writeup comes from a non-native English speaker, anyone can run a spell checker, so spelling errors simply reflect carelessness. That said, not all spell-checkers are perfect, and sometimes typographical errors actually manifest as real words; spell checkers won’t catch those, so they should be used only as a first pass.
- Make the last page look decent. If you’ve edited your text down to remove redundancy, you may find that you have some extra space at the end of your paper. Try to adjust your formatting so that the last page of the paper looks decent. If your paper is double-column and you end with references, balance the references across the two columns so that both columns are both equally (partially) filled. Latex has a “\balance” command that you can use to balance the columns on the last page of your writeup. If you have a bit more time, you can also make small adjustments to margins, space between columns, space around figures, sizes of figures, and so forth to ensure that the last page doesn’t look like you just suddenly decided to stop writing.
Be Efficient
“I’m sorry I have had to write you such a long letter, but I did not have time to write you a short one.” –Blaise Pascal
After you’ve made a good impression, you still need to engage the reader at least long enough to convey your main message. You should assume that the reader is busy, has a short attention span, and (unlike the situation when they are, say, reading a news article or novel) simply wants to learn what you’ve done as quickly as possible and then move on. In contrast to writing a novel, your goal is not to hold the reader in suspense or to illustrate your command of vocabulary or literary construction. Rather, the goal is to transfer information as efficiently as possible.
Tailor the length of your paper to the information content. People sometimes feel compelled to stuff a paper to the conference’s or journal’s page limit (after all, if the paper is too short, isn’t there “more work to do”?). I’ve had a few papers that do fall short of the page limit, although I’ve found that this scenario is the exception rather than the rule (see Blaise Pascal: it takes time to shorten a paper!). Remember that your goal is to efficiently transfer information. Therefore, economize on words. Do not add unnecessary sentences, paragraphs, or other content that do not support a topic or point that you’ve explicitly outlined.
Keep words simple and sentences short. The easier it is to read your paper, the more likely a reader will keep reading. Therefore, don’t make the reader work harder than they have to. On a sentence level, follow Strunk and White’s most indispensable advice: Omit needless words (e.g., “to” as opposed to “in order to”, “optimizing” instead of “the problem of optimizing”, “This module…” instead of “This is a module that”, “more quickly” instead of “in a shorter time”, “whether” instead of “the question as to whether”). Avoid padding phrases such as “the fact that” and unnecessary adverbs (e.g., “eliminated” instead of “totally eliminated”, “fast” instead of “very fast”). Keep words simple (e.g., “use” instead of “utilize”, “new” instead of “novel”), and keep sentences short.
Eliminate redundancy. Always make at least one (and probably several) editing passes over your paper to cut out redundancy. You likely (and hopefully) did not write your paper in a linear fashion, and different co-authors may have written different sections or pieces of the paper. It’s likely that you’ve stated the same point multiple times throughout the paper. Cutting redundancy takes practice, and it sometimes requires divorcing the editing process from your ego (after all, isn’t that sentence you wrote beautiful? It’d be a shame to cut it, right?). If you need help cutting, ask someone else (a co-author or colleague) to help you—they’re not as married to your text as you are.
Be as specific and precise as you can. Precision is especially important when describing results, and when writing up an experiment or evaluation section. For example, a good rule of thumb for writing up an evaluation section is: Would someone be able to reproduce your experiment from your description? If not, you need to be more clear. In an evaluation section, clearly state your assumptions; in what context(s) do your results hold? How general are they? Clearly describe your evaluation setup, as it sets the context for the results themselves. What machines did you use to run the evaluation? What data did you gather to support your conclusions? What algorithms did you use to analyze the data? The evaluation setup should read like a recipe. Someone reading it should be able to reproduce what you’ve done, at least well enough to understand the context of the results. Similarly, the results should be clearly and precisely described. Avoid writing sentences such as “This optimization significantly reduced the system’s running time.” Such vague statements frustrate the reader. Instead, substitute precise and specific information: “This optimization reduced the system’s running time by 20%”.
Use clear and consistent terminology. Unclear, vague, and inconsistent terminology is incredibly frustrating and also typically reflects muddy thinking. If you call something a “node” in one part of your paper, a “host” in another part of the paper, a “client” in another part, and a “machine” in yet another part, people will have absolutely no clue that all of these terms are in fact referring to the same thing. Before you get too far along in the writing process, clearly define your lexicon. Defining your terms up front can help clarify your own thinking, by helping you to identify and articulate the important concepts that you will present. Ultimately, this clarity will be reflected in your writeup.
Reduction to Practice: Write Every Day
Writing a technical paper involves clearly presenting a good story, as efficiently as possible. The above tips should help you compose your story and ensure that it “flows” and ultimately produce a final result that leaves a good impression and communicates your results as efficiently as possible. But, simply reading this post won’t make you a good writer: putting these tips into practice takes tireless repetition. Try writing something every day to practice these tips. Keep a “running draft” of your research paper. Start a blog and practice composing posts that present shorter and simpler ideas and thoughts than a full research paper. Keep a diary or notebook. Ultimately, becoming a good writer requires relentless, daily practice, with continual attention to the above tips.
2379
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
