scrapy is a web search engine based on python,I have tried the basic feature of it, not very complicated, but I haven't gone though the interal flows that it works.
One problem I met is the shown up of chinese charater, I've read several paper of the unicode support in python, but when I tried to change the encoding using codecs,or using unicode module, the byte array did not converted to characters I want. I will regoup myself and add more clearer information here.
below are some background knowledge that scrapy used:
twisted:
is an web architecture,
libxml2:
xml parser module, using SAX
Ipython:
debug python program. invoke it via shell?
firbug:
a plugin for firefox, which can easily locate the elements' html elements, it is of great help when we are writing the xpath expressions to extract information from web pages
scrapy
最新推荐文章于 2024-09-08 13:50:54 发布