抓取猫眼电影排行（二）

• 抓取猫眼排行—利用xpath解析和requests库
• 相关代码如下
  1.导入需要的库

import json
import requests
from requests.exceptions import RequestException
from lxml import etree
import time

2.获取猫眼网址源代码以及利用xpath提取标题、演员、上映时间、评分等信息

def get_one_page(url):
    try:
        headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36'}
        response = requests.get(url, headers=headers)
        if response.status_code == 200:
            #print(response.text)
            html=etree.HTML(response.text)
            #提取电影标题
            title=html.xpath('//p[@class="name"]/a/@title')
            #提取电影演员
            star=html.xpath('//p[@class="star"]/text()')
            star=[str(star[i]).replace('\n','').replace(' ','') for i in range(len(star))]
            #提取电影上映时间
            releasetime=html.xpath('//p[@class="releasetime"]/text()')
            #提取电影评分
            integer=html.xpath('//i[@class="integer"]/text()')
            fraction=html.xpath('//i[@class="fraction"]/text()')
            score=[integer[i]+fraction[i] for i in range(len(integer))]

            film=[title[j]+'\t'+star[j]+'\t'+releasetime[j]+'\t'+score[j] for j in range(len(title))]
            return film
        
        return None
    except RequestException:
        print('something wrong')
        return None

3.将提取的信息写入文本文件中

def write_to_file(content):
    with open('result2.txt', 'w', encoding='utf-8') as f:
                f.write('电影名称'+'\t'+'电影演员'+'\t'+'上映时间'+'\t'+'电影评分'+'\n')
                for content in film:
                    f.write(content+'\n')

4.主函数

def main(offset):
    url = 'http://maoyan.com/board/4?offset=' + str(offset)
    films = get_one_page(url)
    for item in films:
        print(item)
        write_to_file(item)
    
if __name__ == '__main__':
    for i in range(10):
        main(offset=i * 10)
        time.sleep(1)

参考书籍：《Python3网络爬虫开发实战》