python爬虫，从hao123爬取网址信息

最近研究python的爬虫，小小程序，拿下来分享，本人使用python3.7，纯粹兴趣爱好，希望能帮助大家激发兴趣。
从hao123，爬取各种网址信息，代码如下。

import urllib.request
import urllib
import re

#获取网站首页全部内容
url = "http://www.hao123.com"
user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'
req = urllib.request.Request(url, headers={'User-Agent': user_agent})
response = urllib.request.urlopen(req)
content = response.read().decode('utf-8')
#print(content)

#初级筛选
pattern = re.compile('<a.*?href="http://.*?".*?>.*?</a>')
items = re.findall(pattern, content)
for item in items:
    pattern_one = re.compile('href=".*?"')
    pattern_two = re.compile('.*?</a>')
    http = re.findall(pattern_one, item)
    name = re.findall(pattern_two, item)
    name = name.__str__().replace('</a>', '')
    #print(name)
    aa = name.rindex('">')
    print(name[aa+1:len(name)].replace('\']','').replace('</span>','').replace('>','') + ':' + http.__str__().replace('href=','').replace('"','').replace('\'','').replace('[','').replace(']',''))

结果：

 

直升机大战:http://xyx.hao123.com/game/600523.html
魔方跳跃:http://xyx.hao123.com/game/599918.html
旋转消消乐:http://xyx.hao123.com/game/597970.html
画线篮球2:http://xyx.hao123.com/game/598991.html
自由任意球:http://xyx.hao123.com/game/599342.html
冰娃与火娃:http://xyx.hao123.com/game/600062.html
足球对抗赛:http://xyx.hao123.com/game/233871.html
足球锦标赛:http://xyx.hao123.com/game/192712.html
荣耀之剑:http://xyx.hao123.com/game/600035.html
复仇者联盟:http://xyx.hao123.com/game/599138.html
火柴人跑酷:http://xyx.hao123.com/game/595991.html
熊出没冒险:http://xyx.hao123.com/game/557537.html
大头吃橙子:http://xyx.hao123.com/game/596718.html

 

结果有很多不全部列举

转载于:https://www.cnblogs.com/Monster-World/p/9542982.html