VI-Big O

Please indicate the source: http://blog.csdn.net/gaoxiangnumber1

Welcome to my github: https://github.com/gaoxiangnumber1

• Big O time is the language and metric we use to describe the efficiency of algorithms.

Time Complexity

Big O, Big Theta, and Big Omega

• O(big O): Big O describes an upper bound on the time. An algorithm that prints all the values in an array could be described as O(N), but it could also be described as any runtime bigger than O(N), such as O(N2), O(N3), or O(2N). The algorithm is at least as fast as each of these; therefore they are upper bounds on the runtime.
• Ω(big omega): Ω is the lower bound. Printing the values in an array is O(N) as well as O(log N) and O(1). It won’t be faster than those runtimes.
• Θ(big theta): Θ means both O and Ω. An algorithm is Θ(N) if it is both O(N) and Ω(N). Θ gives a tight bound on runtime.
• In interviews, people seem to have merged Θ and O together. Industry’s meaning of big O is closer to what academics mean by Θ, so it would be seen as incorrect to describe printing an array as O(N2). Industry would just say this is O(N). We use big O in the way that industry tends to use it: offer the tightest description of the runtime.

Best Case, Worst Case, and Expected Case

• We can describe runtime for an algorithm in three ways. E.g., quick sort picks a random element as a “pivot” and then swaps values in the array such that the elements less than pivot appear before elements greater than pivot. Then it recursively sorts the left and right sides using a similar process.
  1.Best Case: If all elements are equal, then quick sort will, on average, traverse through the array once. This is O(N).
  2.Worst Case: When the pivot is repeatedly the biggest element in the array(this can happen if the pivot is chosen to be the first element in the subarray and the array is sorted in reverse order), recursion doesn’t divide the array in half and recurse on each half. It just shrinks the subarray by one element. This will degenerate to an O(N2) runtime.
  3.Expected Case: Sometimes the pivot will be very low or very high, but it won’t happen over and over again. We can expect a runtime of O(N log N).
• The best case time complexity is not a very useful concept. For most algorithms, the worst case and the expected case are the same. Sometimes they’re different and we need to describe both of the runtimes.
• What is the relationship between best/worst/expected case and Big O/theta/omega?
  There is no particular relationship between the concepts.
  
  1. Best, worst, and expected cases describe the big O(or big theta) time for particular inputs or scenarios.
  2. Big O, big omega, and big theta describe the upper, lower, and tight bounds for the runtime.

Space Complexity

• If we need to create an array of size n, this will require O(n) space.
  If we need a two-dimensional array of size n*n, this will require O(n2) space.
• Stack space in recursive calls also counts. Following code takes O(n) time and O(n) space.

int sum(int n) /*Ex 1.*/
{
    if (n <= 0)
    {
        return 0;
    }
    return n + sum(n-1);
}

• Each call adds a level to the stack.

1   sum(4)
2       -> sum(3)
3           -> sum(2)
4               -> sum(l)
5                   -> sum(0)

• Each of these calls is added to the call stack and takes up actual memory.
• Have n calls total doesn’t mean it takes O(n) space. The below function adds adjacent elements between 0 and n:

int pairSumSequence(int n) /* Ex 2.*/
{
    int sum = 0;
    for (int i= 0; i < n; i++)
    {
        sum += pairSum(i, i + 1);
    }
    return sum;
}

int pairSum(int a, int b)
{
    return a + b;
}

• There will be roughly O(n) calls to pairSum, but those calls do not exist simultaneously on the call stack, so it needs O(1) space.

Drop the Constants

• O(N) code may run faster than O(1) code for specific inputs. Big O just describes the rate of increase. For this reason, we drop the constants in runtime. An algorithm that one describes as O(2N) is actually O(N).

Drop the Non-Dominant Terms

• You should drop the non-dominant terms.
  O(N2 + N) becomes O(N2).
  O(N + log N) becomes O(N).
  O(5*2N + 1000N100) becomes O(2N).
• The following graph depicts the rate of increase for common big O times.

Multi-Part Algorithms: Add vs. Multiply

• Suppose you have an algorithm that has two steps. When do you multiply the runtimes and when do you add them?

// Add the Runtimes: O(A + B)
for(int a : arrA)
{
    print(a);
}

for(int b : arrB)
{
    print(b);
}
// Multiply the Runtimes: O(A* B)
for(int a : arrA)
{
    for(int b : arrB)
    {
        print(a + ',' + b);
    }
}

• If your algorithm is in the form “do this, then, when you’re all done, do that”, then you add the runtimes.
  If your algorithm is in the form “do this for each time you do that”, then you multiply the runtimes.

Amortized Time(平摊时间)

• A Vector is implemented with an array. When the array hits capacity, the Vector class will create a new array with double the capacity and copy all the elements over to the new array.
• What’s the runtime of insertion?
  
  1. When the array is full: If the array contains N elements, then inserting a new element will take O(N) time. You will have to create a new array of size 2N and then copy N elements over. But this doesn’t happen often.
  2. The majority of the time insertion will be in O(l) time.
• Amortized time takes both into account and it allows us to describe that, this worst case happens every once in a while. But once it happens, it won’t happen again for so long that the cost is “amortized”. In this case, what is the amortized time?
• As we insert elements, we double the capacity when the size of the array is a power of 2. So after X elements, we double the capacity at array sizes 1, 2, 4, 8, 16, … , X. That doubling takes, respectively, 1, 2, 4, 8, 16, 32, 64, … , X copies.
• What is the sum of 1 + 2 + 4 + 8 + 16 + … + X? Read right to left, it starts with X and halves until it gets to 1. What then is the sum of X + X/2 + X/4 + X/8 + … + 1 ? This is roughly 2X.
  So, X insertions take O(2X) time. The amortized time for each insertion is O(1).

Log N Runtimes

• Binary search: look for an example x in an N-element sorted array. First compare x to the midpoint of the array.
  
  1. If x == middle, then return.
  2. If x < middle, then search on the left side of the array.
  3. If x > middle, then search on the right side of the array.

search 9 within {
  1, 5, 8, 9, 11, 13, 15, 19, 21}
    compare 9 to 11 -> smaller.
    search 9 within {
  1, 5, 8, 9, 11}
        compare 9 to 8 -> bigger
        search 9 within {
  9, 11}
            compare <