Performance Analysis Methodology 文章出处http://write.blog.csdn.net/postedit

Performance Analysis Methodology

A performance analysis methodology is a procedure that you can follow to analyze system or application performance. These generally provide a starting point and then guidance to root cause, or causes. Different methodologies are suited for solving different classes of issues, and you may try more than one before accomplishing your goal.

These methodologies can help you solve issues quickly, and solve a wider range of issues. Analysis without a methodology can become a fishing expedition, where tools and metrics are examined ad hoc, until the issue is found – if it is at all.

Key Methodologies:

• The USE Method: for finding resource bottlenecks
• The TSA Method: for analyzing application time
• Off-CPU Analysis: for analyzing any type of thread wait latency
• Active Benchmarking: for accurate and successful benchmarking

This is my main page of performance analysis methodologies, and contains links and summaries of different methods.

Summaries

I first summarized various methodologies for my USENIX LISA2012 talk "Performance Analysis Methodology" (PDF, slideshare, youtube,USENIX), then later documented them in my Systems Performance book. The following is my most up to date summary list, with methodologies enumerated.

These begin with anti-methods, which are included for comparison, and not to follow. You can print these all out as a cheetsheet/reminder.

Blame-Someone-Else Anti-Method

1. Find a system or environment component you are not responsible for
2. Hypothesize that the issue is with that component
3. Redirect the issue to the responsible team
4. When proven wrong, go to 1

Streetlight Anti-Method

1. Pick observability tools that are:
   • familiar
   • found on the Internet
   • found at random
2. Run tools
3. Look for obvious issues

Drunk Man Anti-Method

1. Change things at random until the problem goes away

Random Change Anti-Method

1. Measure a performance baseline
2. Pick a random attribute to change (eg, a tunable)
3. Change it in one direction
4. Measure performance
5. Change it in the other direction
6. Measure performance
7. Were the step 4 or 6 results better than the baseline? If so, keep the change; of not, revert
8. Goto step 1

Passive Benchmarking Anti-Method

1. Pick a benchmark tool
2. Run it with a variety of options
3. Make a slide deck of the results
4. Hand the slides to management

Ad Hoc Checklist Method

1. ..N. Run A, if B, do C

Problem Statement Method

1. What makes you think there is a performance problem?
2. Has this system ever performed well?
3. What has changed recently? (Software? Hardware? Load?)
4. Can the performance degradation be expressed in terms of latency or run time?
5. Does the problem affect other people or applications (or is it just you)?
6. What is the environment? What software and hardware is used? Versions? Configuration?

Scientific Method

1. Question
2. Hypothesis
3. Prediction
4. Test
5. Analysis

Workload Characterization Method

1. Who is causing the load? PID, UID, IP addr, ...
2. Why is the load called? code path
3. What is the load? IOPS, tput, type
4. How is the load changing over time?

Drill-Down Analysis Method

1. Start at highest level
2. Examine next-level details
3. Pick most interesting breakdown
4. If problem unsolved, go to 2

By-Layer Method

Measure latency from:

1. Dynamic languages
2. Executable
3. Libraries
4. Syscalls
5. Kernel: FS, network
6. Device drivers

Latency Analysis Method

1. Measure operation time (latency)
2. Divide into logical synchronous components
3. Continue division until latency origin is identified
4. Quantify: estimate speedup if problem fixed

Tools Method

1. List available performance tools (optionally add more)
2. For each tool, list its useful metrics
3. For each metric, list possible interpretation
4. Run selected tools and interpret selected metrics.

USE Method

For every resource, check:

1. Utilization
2. Saturation
3. Errors

Stack Profile Method

1. Profile thread stack traces, on- and off-CPU
2. Coalesce
3. Study stacks bottom-up

Off-CPU Analysis

1. Profile scheduler per-thread off-CPU time with stack traces
2. Coalesce times with like stacks
3. Study stacks from largest to shortest time

TSA Method

1. For each thread of interest, measure time in operating system thread states. Eg:
   • Executing
   • Runnable
   • Swapping
   • Sleeping
   • Lock
   • Idle
2. Investigate states from most to least frequent, using appropriate tools

Active Benchmarking Method

1. Configure the benchmark to run for a long duration
2. While running, analyze performance using other tools, and determine limiting factors