Generator

http://wiki.python.org/moin/Generators

一、

Generators functions allow you to declare a function that behaves like an iterator, i.e. it can be used in a for loop.

Simplified Code

The simplification of code is a result of generator function and generator expression support provided by Python.

To illustrate this, we will compare different implementations that implement a function, "firstn", that represents the first n non-negative integers, where n is a really big number, and assume (for the sake of the examples in this section) that each integer take up a lot of space, say 10 megabytes each.

Note: Please note that in real life, integers do not take up that much space, unless they are really, really, really, big integers. For instance you can represent a 309 digit number with 128 bytes (add some overhead, it will still be less than 150 bytes).

First, let us consider the simple example of building a list and returning it.

Toggle line numbers

   1 # Build and return a list
   2 def firstn(n):
   3     num, nums = 0, []
   4     while num < n:
   5         nums.append(num)
   6         num += 1
   7     return nums
   8 
   9 sum_of_first_n = sum(firstn(1000000))

The code is quite simple and straightforward, but its builds the full list in memory. This is clearly not acceptable in our case, because we cannot afford to keep all n "10 megabyte" integers in memory.

So, we resort to the generator pattern. The following implements generator as an iterable object.

Toggle line numbers

   1 # Using the generator pattern (an iterable)
   2 class firstn(object):
   3     def __init__(self, n):
   4         self.n = n
   5         self.num, self.nums = 0, []
   6 
   7     def __iter__(self):
   8         return self
   9 
  10     def next(self):
  11         if self.num < self.n:
  12             cur, self.num = self.num, self.num+1
  13             return cur
  14         else:
  15             raise StopIteration()
  16 
  17 sum_of_first_n = sum(firstn(1000000))

This will perform as we expect, but we have the following issues:

• there is a lot of boilerplate
• the logic has to be expressed in a somewhat convoluted way

Furthermore, this is a pattern that we will use over and over for many similar constructs. Imagine writing all that just to get an iterator.

Python provides generator functions as a convenient shortcut to building iterators. Lets us rewrite the above iterator as a generator function:

Toggle line numbers

   1 # a generator that yields items instead of returning a list
   2 def firstn(n):
   3     num = 0
   4     while num < n:
   5         yield num
   6         num += 1
   7 
   8 sum_of_first_n = sum(firstn(1000000))

Note that the expression of the number generation logic is clear and natural. It is very similar to the implementation that built a list in memory, but has the memory usage characteristic of the iterator implementation.

二、example

def flatten(nested):
    try:
        for sublist in nested:
            for element in flatten(sublist):
                print 'sublist:', sublist
                print 'element:',element
                yield element
    except TypeError:
        print 'nested:',nested
        yield nested
            
nested = [[[1],2],3,4,[5,[6,7]],8]
for num in flatten(nested):
    print 'num:',num

When flatten is called, you have two possibilities (as is always the case when dealing with recursion): the base case and the recursive case. In the base case, the function is told to flatten a single element (for example, a number), in which case the for loop raises a TypeError (because you’re trying to iterate over a number), and the generator simply yields the element.