Derek_Zhang_的专栏

1. Static Resource HTTP Response Header cache-control:public, max-age=30758400 cf-cache-status:HIT cf-ray:1afc29518836124f-HKG content-encoding:gzip content-type:text/css date:Wed, 28 Jan 2015 09:28:

When I try to get document from a website using jsoup, I got the error after seconds of stucking. java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Met

1. Open chrome and its developer tool 2. Conduct the login operation in chrome, record the network intercourse. 3. Analyze the HTTP Request and HTTP Response Request Headers POST /member/xlogin.ph

www.freeformatter.com is very handy for web scraping stuff, atm I'm using the xpath-tester for validating my xpath expression. http://www.freeformatter.com/xpath-tester.html

scrapy: application framework for web scraping and crawling beautifulsoup: library for parsing HTML mechanize lxml