python解码gbk,Python中的编码问题，当urlopen（）一个gbk页面

My code here:

# coding:utf-8

if __name__ == '__main__':

from urllib2 import urlopen

url = 'http://iccna.blog.sohu.com/164572951.html'

data = urlopen(url).read()

soup = BeautifulSoup(data,fromEncoding='gb18030')

print WebExtractor(soup)

but when debugging, the data like this:

��5h�,��4�H�5��VM��\

what should i do to get right data for BeautifulSoup? Thanks!

解决方案

The problem is that the server returns the data compressed by Gzip. Try this:

#-*- coding: utf-8 -*-

from __future__ import print_function

import gzip

import StringIO

import urllib2

from BeautifulSoup import BeautifulSoup

url = 'http://iccna.blog.sohu.com/164572951.html'

response = urllib2.urlopen(url)

data = response.read()

data = StringIO.StringIO(data)

gzipper = gzip.GzipFile(fileobj=data)

html = gzipper.read()

soup = BeautifulSoup(html, fromEncoding='gbk')

print(soup)

The Chinese characters look still wrong on my system, but this may give you right direction.