Lecture 7 - Debugging

Lecture 7 - Debugging

1. Testing and Debugging

• What is Testing and Debugging
  
  • Testing method:
    
    • Ways of trying code on examples to determine if
      running correctly
  • Debugging methods
    
    • Ways of fixing a program that you know does not work
      as intended
• When should you test and debug
  
  • Break program into components that can be tested and debugged independently
  • Document constraints on modules：
    
    • Expectations on inputs, on outputs
    • Even if code does not enforce constraints, valuable for debugging to have description
  • Document assumptions behind code design

• When are you ready to test?

  • Ensure that code will actually run
    
    • Remove syntax errors
    • Remove static semantic errors
    • Both of these are typically handled by Python interpreter
  • Have a set of expected results (i.e. inputoutput pairings) ready

  2. Testing suite

• Goal:

  • Show that bugs exist
  • Would be great to prove code is bug free, but generally hard
    
    • Usually can’t run on all possible inputs to check
    • Formal methods sometimes help, but usually only on simpler code
• Test Suite
  
  • Want to find a collection of inputs that has high likelihood of revealing bugs, yet is
    efficient
    
    • Partition space of inputs into subsets that provide equivalent information about correctness
      
      • Partition divides a set into group of subsets such that
        each element of set is in exactly one subset
    • Construct test suite that contains one input from each element of partition
    • Run test suite

• example:

  def isBigger(x, y):
   “““Assumes x and y are ints
   returns True if x is less than y
   else False”””
  • Input space   is  all pairs   of  integers    
  • Possible  partition   
  – x positive,   y   positive    
  – x negative,   y   negative    
  – x positive,   y   negative    
  – x negative,   y   positive    
  – x =   0,  y   =   0   
  – x =   0,  y   !=  0   
  – x !=  0,  y   =   0   

  • why this partition
    
    • Lots of other choices
      
      • E.g., x prime, y not; y prime, x not; both prime; both not
    • Space of inputs oRen have natural boundaries
      
      • Integers are positive, negative or zero
      • From this perspective, have 9 subsets
        
        • Split x = 0, y != 0 into x = 0, y positive and x =0, y negative
        • Same for x != 0, y = 0
• What if no natural partition to input space
  
  • Random testing – probability that code is correct increases with number of trials; but should be able to use code to do better
  • Use heuristics #探索法；启发式# based on exploring paths through the specifications – black-box testing
  • Use heuristics based on exploring paths through the code – glass-box testing

3. Black-box testing

• Test suite designed without looking at code
  
  • Can be done by someone other than implementer
  • Will avoid inherent biases of implementer, exposing potential bugs more easily
  • Testing designed without knowledge of implementa.on, thus can be reused even if implementa.on changed

• Paths through a specification

  def sqrt(x, eps):
   “““Assumes x, eps floats
   x >= 0
   eps > 0
   returns res such that
   x-eps <= res*res <= x+eps”””
  • Paths through specification:  
  – x =   0   
  – x >   0   
  • But clearly not enough    

• Paths through a specification

  • Also good to consider boundary cases:
    
    • For lists: empty list, singleton list, many element list
    • For numbers, very small, very large, “typical”

• example:

  For our sqrt case,  try 
  these:  
  – First four    are typical 
  • Perfect   square  
  • Irra.onal square  root    
  • Example   less    than    1   
  – Last  five    test    extremes    
  • If    bug,    might   be  code,   or  
  might   be  spec    (e.g.   don’t   
  try to  find    root    if  eps .ny)

  4. GLASS-BOX TESTING

• Concept：

  • Use code directly to guide design of test cases
  • Glass-box test suite is path-complete if every potential path through the code is tested at least once
    
    • Not always possible if loop can be exercised arbitrary times, or recursion can be arbitrarily deep
  • Even path-complete suite can miss a bug, depending on choice of examples

• example:

  Example 
  def abs(x):
   “““Assumes x is an int
   returns x if x>=0 and –x otherwise”””
   if x < -1:
   return –x
   else:
   return x
  • Test  suite   of  {-2,    2}  will be path complete   
  • But will miss abs(-1) which incorrectly returns   -1  
  – Testing boundary cases and typical cases  would catch this {-2 -1, 2} 

• Rules of thumb for glass-box testing:

  • Exercise both branches of all if statements
  • Ensure each except clause is executed
  • For each for loop, have tests where:
    
    • Loop is not entered
    • Body of loop executed exactly once
    • Body of loop executed more than once
  • For each while loop,
    
    • Same cases as for loops
    • Cases that catch all ways to exit loop

• For recursive func,ons, test with no recursive calls, one recursive call, and more than one recursive call

5. TEST DRIVERS AND STUBS

• Conducting tests
  
  • Start with unit testing
    
    • Check that each module (e.g. function) works correctly
  • Move to integration testing
    
    • Check that system as whole works correctly
  • Cycle between these phases

• Test Drivers and Stubs

  • Drivers are code that
    
    • Set up environment needed to run code
    • Invoke code on predefined sequence of inputs
    • Save results, and
    • Report
  • Drivers simulate parts of program that use unit being tested
  • Stubs simulate parts of program used by unit being tested
    
    • Allow you to test units that depend on soGware not yet written

• Good testing practice

  • Start with unit testing
  • Move to integration testing
  • After code is corrected, be sure to do regression testing:
    
    • Check that program s’ll passes all the tests it used to pass, i.e., that your code fix hasn’t broken something that used to work
    • -

6.DEBUGGING

• The “history” of debugging

  • Often claimed that first bug was found by team at Harvard that was working on the Mark II Aiken Relay Calculator

  • A set of tests on a module had failed; when staff inspected the actually machinery (in this case vacuum tubes and relays), they discovered this:

  • 

• A real bug!
  
  • However, the term bug dates back even earlier:
    
    • Hawkin’s New Catechism of Electricity, 1896

7. Runtime bugs

• Overt vs. covert:
  
  • Overt has an obvious manifestation – code crashes or runs forever
  • Covert has no obvious manifestaOon – code returns a value, which may b incorrect but hard to determine
• Persistent vs. intermittent:
  
  • Persistent occurs every Ome code is run
  • Intermittent only occurs some Omes, even if run on same input
• Categories of bugs
  
  • Overt and persistent
    
    • Obvious to detect
    • Good programmers use defensive programming to try to ensure that if error is made, bug will fall into this category
  • Overt and intermiSent
    
    • More frustraOng, can be harder to debug, but if conditions that prompt bug can be reproduced, can be handled
  • Covert
    
    • Highly dangerous, as users may not realize answers are incorrect until code has been run for long period

8. DEBUGGING AS SEARCH

• Debugging skills:
  
  • Treat as a search problem: looking for explanation for incorrect behavior
    
    • Study available data – both correct test cases and incorrect
    • Form an hypothesis consistent with the data
    • Design and run a repeatable experiment with potential to refute the hypothesis
    • Keep record of experiments performed: use narrow range of hypotheses

• Debugging as search

  • Want to narrow down space of possible sources of error
  • Design experiments that expose intermediate stages of computa8on (use print statements!), and use results to further narrow search
  • Binary search can be a powerful tool for this

• example:

  def isPal(x):
      assert type(x) == list
      temp = x
      temp.reverse
      if temp == x:
          return True
      else:
          return False
  
  def silly(n):
      for i in range(n):
          result = []
          elem = raw_input('Enter element: ')
          result.append(elem)
      if isPal(result):
          print('Yes')
      else:
          print('No')

  • Stepping through the tests
    
    • Suppose we run this code:
      
      • We try the input ‘abcba’, which succeeds
      • We try the input ‘palinnilap’, which succeeds
      • But we try the input ‘ab’, which also ‘succeeds’
    • Let’s use binary search to isolate bug(s)
    • Pick a spot about halfway through code, and devise experiment
      
      • Pick a spot where easy to examine intermediate values

  

• More detailes in
  http://www.xuetangx.com/asset-v1:MITx+6_00_1x+sp+type@asset+block/handouts_lectureSlides_Lecture7_7.pdf