python爬虫-使用BeautifulSoup爬取新浪新闻标题

最新推荐文章于 2024-04-22 16:07:30 发布

zhzzzk

最新推荐文章于 2024-04-22 16:07:30 发布

阅读量2.2k

点赞数

分类专栏：爬虫文章标签： python beautifusoup

本文链接：https://blog.csdn.net/zhzzzk/article/details/89091780

版权

爬虫专栏收录该内容

2 篇文章 0 订阅

订阅专栏

python爬虫-使用BeautifulSoup爬取新浪新闻标题

最近在学习爬虫的技巧，首先学习的是较为简单的BeautifulSoup，应用于新浪新闻上。

import requests
from bs4 import BeautifulSoup
import time

#使用BeautifulSoup方法得到url的内容
def get_news(url):
    request = requests.get(url)
    request.encoding = 'UTF-8'
    soup = BeautifulSoup(request.text, 'html.parser')
    return soup
    
#将get到的内容保存至本地
def write_news(soup,t_n):
    i = 0
    for news in soup.select('li'):#对新浪新闻网页源码分析，找到的标题所在位置，各种网页不尽相同
        i += 1
        t = news.select("a")[0].text
        with open(t_n,'a',encoding='utf-8') as data:  
            print(i,"  ",t,file = data)
            
#获取当前时间
def get_time():
    d = time.localtime( time.time())      
    dd = "现在是{}年{}月{}日{}时{}分".format(d[0],d[1],d[2],d[3],d[4])
    return dd

#在保存的txt中写入当前时间
def write_time(t_n):
    t = get_time()
    with open(t_n,'a',encoding='utf-8') as data:  
            print(t,file = data)

#对txt文件根据时间命名
def txt_name():
    d = time.localtime( time.time())          
    t_n = 'D:/python/workspace/sinanews/'+'newstitle_{}.{}.txt'.format(d[1],d[2])
    return t_n

#运行
url = 'http://news.sina.com.cn/china/'
t_n = txt_name()
write_time(t_n)
soup = get_news(url)
write_news(soup,t_n)