Eurosys 2013 计算机体系结构相关国际会议 review评审意见

Eurosys 2013 计算机体系结构相关国际会议 review评审意见 （信息供参考学习）

我投Eurosys'2013的一篇与多级缓存替换算法研究相关的论文，录用结果是reject。Review意见供大家学习经验教训吧！

===========================================================================
                          Eurosys13 Review #100A
               Updated Wednesday 21 Nov 2012 10:31:51am EST
---------------------------------------------------------------------------
  Paper #100: Cache Resource Aggregation Using Adaptive Multi-level
              Exclusive Caching Policies
---------------------------------------------------------------------------

                      Overall merit: 1. Bottom 50% of submitted papers
             Reviewer qualification: 3. I know the material, but am not an
                                        expert

                         ===== Paper summary =====

This paper describes algorithms to manage a hierarchical buffer cache.
It introduces a new algorithm to manage an individual cache that is
well suited to the reduced locality observed by low level caches, and
it introduces a technique that uses hints and a push mechanism to
achieve exclusive caching (i.e., to avoid wasting space with more than
one copy of a block in the cache hierarchy). There is an evaluation in
simulation using various traces.

                           ===== Strengths =====

- it integrates a single level cache replacement mechanism with a
  mechanism to ensure exclusive caching

- it can improve performance significantly in some traces when
  compared with some prior work

                          ===== Weaknesses =====

1- this is a crowded area and this is a small delta: local cache
  replacement is similar to LIRS and exclusive caching mechanism
  similar to PROMOTE.

2- strong assumptions: ignores sharing, writes, and prefetching.

3- evaluation is done in simulation using tiny traces and caches, and a single client.

4- the performance gains for the local cache replacement algorithm are very
  small.

4- the performance gains with exclusive caching are sometimes
  significant but it is unclear whether a combination of LIRS and
  PROMOTE would not achieve similar gains.

                      ===== Comments to author =====

There has been a large amount of work in this area. This paper
provides some improvement relative to prior work but the improvement
is small. I find it hard to argue for accepting this paper given the
limitations that I stated above. 

The algorithms that you introduce are very close to the ones in LIRS
and PROMOTE. In fact, the performance gain for single cache
replacement of ReDARC and LIRS is very similar. The performance gains
for multiple caches are more signficant. But it seems to me that a
combination of PROMOTE and LIRS would be likely to approach your
performance. Furthermore, I have trouble believing simulations and
both the data and caches are very small, and you use a single client
in your experiments.

Finally, given the amount of previous work in this area, I am
reluctant to accept a paper that does not consider sharing or writes.

===========================================================================
                          Eurosys13 Review #100B
                Updated Thursday 22 Nov 2012 6:37:32pm EST
---------------------------------------------------------------------------
  Paper #100: Cache Resource Aggregation Using Adaptive Multi-level
              Exclusive Caching Policies
---------------------------------------------------------------------------

                      Overall merit: 1. Bottom 50% of submitted papers
             Reviewer qualification: 3. I know the material, but am not an
                                        expert

                         ===== Paper summary =====

The paper proposes a new efficient dynamic cache replacement algorithm ReDARC, as well as a novel policy REAL that uses ReDARC to addresses the weakened locality at lower levels of a multi-level cache. The ReDARC algorithm is similar to LIRS in that it uses reuse distance as a measure instead of recency (as with many other modern cache replacement algorithms, such as ARC). The REAL policy promotes items from slow cache to fast cache, when appropriate.

                           ===== Strengths =====

The algorithm as presented is sounds and the results -- in so far as presented in the evaluation -- show performance benefits for ReDARC and REAL over other schemes. The paper thoroughly describes and positions the paper relative to related work in this busy field.

                          ===== Weaknesses =====

Although cache performance is unquestionably important for modern systems, the contributions of this paper feel incremental over prior work.
The presentation of the paper could be improved, for example the novel ideas could be distilled more clearly.

                      ===== Comments to author =====

Minor issues.Other issues 

Issue 1: The concept “Weakened locality” in the abstract is undefined, so it should be defined or replaced
with something clearer.
Issue 2: In this paper, a high level cache is faster than a low level cache, so this should be made clear in
the abstract or in the introduction.
Issue 3: The pseudocode for ReDARC should be verified with the following issues in mind:
(a) The function stack_evict is used with two arguments but defined with one argument.
(b) The text describing the pseudocode says that stack_evict puts blocks into EL, but the pseudocode
does not put blocks into EL.
(c) The first line in “Case 1” moves X to the top of S so the second line should say: “If X was at
the bottom of S”.
(d) The wording “Delete A in B” is frequent in the pseudocode but “Delete A from B” would be
better.
(d) In the function stack_evict, “Let P points to” is appears twice but “Let P point to” would be
better.
Issue 4: In section 2 a model for analyzing multi-level cache hierarchies is introduced. Di>i+1 denotes
the sequence of demoted blocks from cachei to cachei+1. Since ALACA only promotes blocks, it
would be good to describe how this model analyzes ALACA.
Issue 5: In section 4.2 the definition of SRD_life is: “SRD_life is the cache life of the block in the bottom
of stack S of higher cache”, but the concept “cache life” is not introduced in the paper.
Issue 6: The title of section 5.3.1. is missing a “Level” before ”Cache”.
Issue 7: Paper names in references [17] and [26] are not correct, “2q” should be “2Q” and “lru-k” should be
“LRU-K”.
Issue 8: In references [18] and [19] “et al. et al.” should be just “et al.”.


Comments/nitpicks on presentation:

All equations could be typeset better (e.g. using "\to" instead of "->", "\left(" etc.)

Page 1: Abstract, paragraph 1:
    "multi-level caching policies deployed so far are typically using the",
	use	"typically use the" instead of "are typically using the"

Page 1: Abstract, paragraph 2: 
	"PUSH policy that allows lower caches push", missing "to" before "push"

Page 1: Abstract, paragraph 2: 
	"while keep network", use "keeping" instead of "keep"

Page 1: Abstract, paragraph 2: 
	"real-life traces show", missing "to" before "show"

Page 2: Section 1.1, paragraph 3: 
	"data cached in higher level has a high probability that is cached in the lower level",
	rather use
	"data that is cached in a high level has a high probability of being cached in a lower level as well"

Page 2: Section 1.2, paragraph 1: 
	"guarantees blocks in SRD set all cached in memory", 
	missing "that" after "guarantees" and missing "are" after "set"

Page 2: Section 1.2, paragraph 2: 
    "allows lower caches push", missing "to" before "push"

Page 2: Section 1.2, paragraph 3:
    "decide whether a block cached in one level should push to higher level caches.", 
	rather use:
    "decide whether a block cached in one level should be pushed to higher level caches or not.", 

Page 2: Section 1.2, paragraph 3: 
    ", meanwhile keeps network traffic and", use "while keeping" instead of "meanwhile keeps"

Page 3: Section 3, paragraph 1:
	"does not always true", missing "hold" before "true"

Page 4: Section 4, paragraph 1:
    "Two algorithms work together", use "These two" instead of "Two"

Page 4: Section 4.1, paragraph 2:
	"to cache blocks belong to SRD set", 
	rather use
	"to cache blocks belonging to the SRD set"

Page 4: Section 4.1, paragraph 2:
	"a portion of blocks belong to LRD set", 
	rather use 
	"a portion of blocks belonging to the LRD set"

Page 4: Section 4.1, paragraph 3:
    "ReDARC maintains a ghost cache size, whose size equals to the actual cache size",
	rather use
	"ReDARC maintains a ghots cache, whose size equals the actual cache size"

Page 4-5: Section 4.1, paragraph 3:
    "The ghost blocks in ghost cache has only their meta-data",
	rather:
	"The ghost blocks in the ghost cache have only their meta-data"

Page 5: Section 4.1, paragraph 4:
    "(Line 1-4), X must stay in cache",
	rather use:
    "(Line 1-4), X must be in the cache",

Page 5: Section 4.1, paragraph 5: 
    "in the ghost cache means the actual size", missing "that" after "means"

Page 5: Section 4.1, paragraph 5: 
    "performed to ensure the total size", missing "that" after "ensure"

Page 5: Section 4.1, paragraph 5: 
	"If SRD set is full, X is considered to be accessed as the first time or have not accessed for a very long time, so X should put into LRD set."
	rather use:	
	"If the SRD set is full, X is either being accessed for the first time or X has not been accessed for a long time, so X should be put into the LRD set."

Page 5: Section 4.1, paragraph 5: 
    "In this case, we ensure the total number", "missing" that after "ensure"

Page 6: Section 4.2, paragraph 3: 
    "TRUE to indicate the block should be pushed"
	rather use:
    "TRUE to indicate that the block should be pushed"

Page 6: Section 4.2, paragraph 3: 
    "local cache determines whether to push up the block.", 
	rather use:
	"the local cache determines whether to push the block up or not."

Page 7: Section 5.1, paragraph 5:
    "TPC-R and TPC-H performs random access to", use "perform" not "performs"

Page 7: Section 5.1, paragraph 6:
    "and gcc builds Linux kernel", missing "the" before "Linux kernel"

Page 7: Section 5.2, paragraph 3:
    "stored in LRU list B1 and B2 respectively", "lists" not "list"

Page 7: Section 5.2, paragraph 8 (Independent LRU):
    "block request is send to", use "sent" instead of "send"

Page 8: Section 5.3.1, paragraph 4:
	"ReDARC and LIRS uses reuse distance", use "use" instead of "uses"

Page 8: Section 5.3.1, paragraph 4:
    "Because the newly accessed blocks does not have", use "do" instead of "does"

Page 8: Section 5.3.1, paragraph 4:
    "ReDARC has a higher hit ratio than LIRS for ReDARC", "because" is better than "for" in this context

Page 10: Section 5.3.2, paragraph 2:
    "The experiment results demonstrate", use "experimental" instead of "experiment"

Page 10: Section 5.3.2, paragraph 2:
    "as their cache size increase", use "increases" instead of "increase"

Page 10: Section 5.3.2, paragraph 2:
    "the REAL policy keeps higher aggregate", missing "a" before "higher"

Page 10: Section 5.3.2, paragraph 3:
    "so ind-ARC" keeps lower average", missing "a" before "lower"

Page 10: Section 5.3.2, paragraph 3:
    "at the same time keeps network", use "keeping" instead of "keeps"

Page 10: Section 5.3.2, paragraph 4:
    "Glo-LRU always demote", use "demotes" instead of "demote"

Page 11: Section 6, paragraph 4:
    "OUR REAL", "OUR" should probably be "Our"

Page 11: Section 6, paragraph 5:
    "that decides a block should whether be stayed in local cache or pushed",
	rather use
	"that decides whether a block should stay in local cache or be pushed"

Page 11: Section 7, paragraph 1:
    "is becoming as an important", skip the "as"

===========================================================================
                          Eurosys13 Review #100C
                Updated Saturday 24 Nov 2012 11:09:50pm EST
---------------------------------------------------------------------------
  Paper #100: Cache Resource Aggregation Using Adaptive Multi-level
              Exclusive Caching Policies
---------------------------------------------------------------------------

                      Overall merit: 4. Top 10% but not top 5% of submitted
                                        papers
             Reviewer qualification: 3. I know the material, but am not an
                                        expert

                         ===== Paper summary =====

The paper describes and evaluates a novel replacement policy for
multi-level block caches in data centers. The ReDARC policy replaces
blocks in a single-level cache based on their reuse distance (# of
operations between last and previous to last access). The ALACA policy
exchanges information between cache layers and makes sure a block is
cached in at most one layer, while minimizing inter-cache traffic. A
trace-based simulation study shows that the proposal algorithms match
or exceed the performance of existing policies in terms of hit rate,
average access latency and inter-cache traffic.

                           ===== Strengths =====

The paper addresses an important problem, describes clever and novel
solutions, and demonstrates in simulation that the solutions
outperform existing work.

                          ===== Weaknesses =====

None major. The reader has to develop an intuition for how the
policies interact based on the pseudocode, which is a bit of a
chore. The works currently considers single-client, read-only caches
only.

                      ===== Comments to author =====

Overall, the paper explains the problem, its challenges, and the
general solution very well. Also, the proposed policies are described
precisely in pseudo-code. However, the reader has to labor a bit to
develop an understanding of how the policies interact to ensure the
properties they have. The exposition could be improved in this regard.

Sec 2: The purpose of this section is not clear. The model defined
here is not subsequently used.

Sec 4.1, p3: "..maintains a ghost cache size, whose size equals.." ->
maintains a ghost cache, whose size equals.."?

Suggestion: A state transition diagram might be more easy to digest
for the reader than the text in the last two paragraphs in 4.1. The
text could then focus on the justification/intuition behind each state
transition.

Sec 4.2: another state transition diagram would help here. 

I would have expected a section that explains how the ReDARC and ALACA
algorithms interact at a high level. Without it, the reader has to try
and figure it out from the pseudocode. For instance, a set of
invariants that hold in the combined REAL algorithm and an (informal)
justification of why they hold could aid the reader greatly in
understanding REAL.

There is some bizzare word-splitting going on, leaving individual
characters on a line. Also, the paper should be edited for grammar and
style prior to publication.

===========================================================================
                          Eurosys13 Review #100D
                 Updated Tuesday 18 Dec 2012 1:15:09am EST
---------------------------------------------------------------------------
  Paper #100: Cache Resource Aggregation Using Adaptive Multi-level
              Exclusive Caching Policies
---------------------------------------------------------------------------

                      Overall merit: 2. Top 50% but not top 25% of
                                        submitted papers
             Reviewer qualification: 3. I know the material, but am not an
                                        expert

                         ===== Paper summary =====

This paper proposes a system for managing multi-level caching hierarchies by propagating cached data to the higher levels and eliminating the copies at the lower levels.

                           ===== Strengths =====

+ Performance with redundant caching layers is important, as they appear pervasively.
+ Thorough experimental comparison to alternative designs.

                          ===== Weaknesses =====

- Reliance solely on simulation is worrisome. Some evaluation against a real distributed application deployment is tremendously important.

- Results appears somewhat incremental, and without real applications it's hard to tell what's worth.

- What kinds of caches are you talking about? OS-level block buffer caches, application-level caches, memcached, etc.? What happens with dirty caches?

                      ===== Comments to author =====

Multi-level caching is indeed a significant problem in datacenter workloads. Some of its implications are: poor memory utilization, weak locality at lower levels, and cache consistency challenges. This paper addresses just the first two implications, but not the third one. Still, the first two problems are important, and the paper does a good job describing their solution for these problems.

Unfortunately, for a paper that is a performance and resource optimization, I have to say that the paper's evaluation, which is based solely on simulation, is problematic. Simulations are great because they allow one to explore dimensions you would not  be able to explore with real applications, and also to compare against systems that you don't have implemented. However, evaluation of the proposed solution against a real application is also essential. Why not build a distributed Web application, deploy some caches for it, and then evaluate the system's application-specific performance and memory utilization improvement under your system compared to the multi-caching situation?

Moreover, while the simulations do reveal some benefit of using the proposed system, I am not sure how to interpret them: how much does the increase in hit rates matter for end-to-end application performance?

Another thing that confused me a bit was the lack of specificity regarding what caches you are referring to. Are you talking about OS-level block buffer caches only, or are you also including application-level caches, memcached-like caches, etc.? If so, how do you ensure consistency of these caches in face of concurrent updates?

===========================================================================
                          Eurosys13 Review #100E
                 Updated Tuesday 18 Dec 2012 6:24:11pm EST
---------------------------------------------------------------------------
  Paper #100: Cache Resource Aggregation Using Adaptive Multi-level
              Exclusive Caching Policies
---------------------------------------------------------------------------

                      Overall merit: 3. Top 25% but not top 10% of
                                        submitted papers
             Reviewer qualification: 2. I have passing familiarity

                         ===== Paper summary =====

This paper describes a new multi-level caching policy. The new policy consists
of two components: a single-level reuse-distance based adaptive replacement
policy and multi-level exclusive caching scheme that pushes blocks from a
lower-level to higher level based on a probabilistic scheme. Simulation results
from multiple traces and comparison to existing caching schemes are presented.

                           ===== Strengths =====

Detailed algorithm describing both schemes and good presentation of related
work.

                          ===== Weaknesses =====

Not clear if multi-level benefits will be same in an implementation. 
Also unclear what real-world settings this applies in.

                      ===== Comments to author =====

Major comments:

- It is not clear in which real-world settings is the model applicable. While
  the paper mentions large buffer caches in clusters today, such systems
  typically use specialized mem-cached layer for caching and do not explicitly
  cache blocks at the storage servers. In fact the assumption that the caches
  are almost equal in size, makes this work less applicable for many domains.
  CPU caches don't fall into this bucket either, so it would be good to provide
  more details and examples on where this scheme can be applied.

- The evaluation of this scheme is done only using simulation. While the
  simulation does use real-world traces and implements multiple schemes for
  comparison, it is not clear if the multi-level benefits will translate
  directly to an implementation. For example, the inter-cache traffic is only
  important if it affects the end-to-end performance of the system. I think the
  paper would be much better with an implementation of this scheme in a system
  and end-to-end measurement using a real-workload.

Some more detailed comments:

- The differences between the PUSH and the PROMOTE policies are not clear
  through the paper. While at first reading they seem to tbe same, it looks like
  the only difference is that PROMOTE sends periodic messages while PUSH piggy
  backs messages, but it would be good to clarify this. 

- The benefits of the first level scheduling scheme seem to be minor when
  compared to LIRS. While the text states that the hit ratios are consistently
  higher for  ReDARC, Figure 5 doesn't seem to indicate so. It would be good to
  present a deeper study of how each algorithm's internal structure affects the
  hit-ratio for one workload (say multi1 ?)

- The multi-level scheduling scheme is only evaluated with two levels. While
  this seems sufficient to highlight the point, it would be interesting to see
  results from three or more levels as well.

- The technique to update PUSH probabilities is not explained in the paper. Any
  reason why the algorithm reads the way it does ?

- The presentation of algorithms in Figure 3, 4 are descriptive of the
  procedure, but do not provide any insight into how and why data structures are
  maintained in each scheme. I think the paper would benefit if the algorithms
  were explained in terms of invariants (Stack S always contains a SRD at the
  bottom. This is because -- etc.) and if examples are used to highlight why
  these design choices were made.

- Typo in Section 4.1 - "have their entries remained"

- Typo in abstract - "keep network traffic"

              ===== Comments to address in the response =====

Description of real-world settings where the scheme would apply and how the
reduction in network-traffic / better hit-rate can improve end-to-end
performance.