scrapy

scrapy is a web search engine based on python,I have tried the basic feature of it, not very complicated, but I haven't gone though the interal flows that it works.
One problem I met is the shown up of chinese charater, I've read several paper of the unicode support in python, but when I tried to change the encoding using codecs,or using unicode module, the byte array did not converted to characters I want. I will regoup myself and add more clearer information here.

below are some background knowledge that scrapy used:
twisted:
 is an web architecture,
libxml2:
  xml parser module, using SAX
Ipython:
  debug python program. invoke it via shell?
firbug:
  a plugin for firefox, which can easily locate the elements' html elements, it is of great help when we are writing the xpath expressions to extract information from web pages