【Python】基于Python的百度迁徙5——实时拥堵与拥堵指数（附代码）

陈修一

已于 2022-03-23 22:54:21 修改

阅读量8.9k

点赞数 18

分类专栏：大数据 Python 文章标签：百度迁徙大数据实时大数据 1024程序员节

于 2021-10-24 17:22:14 首次发布

本文链接：https://blog.csdn.net/Leaze932822995/article/details/120936461

版权

Python 同时被 2 个专栏收录

19 篇文章 20 订阅

订阅专栏

大数据

10 篇文章 12 订阅

订阅专栏

本文介绍了如何使用Python爬取百度迁徙数据，包括实时和历史的拥堵指数变化、行政区及道路拥堵排行。通过读取、续写txt文件，实现数据实时更新。代码分为四个部分：当日24小时拥堵指数变化、历史拥堵指数变化、行政区排行和道路排行。提供了完整的代码示例和调用方法。

摘要由CSDN通过智能技术生成

这是百度迁徙系列的第五篇。
老规矩，如果想获取过去的数据的话，可以私聊或者电子邮件（chinshuuichi

简介

百度迁徙：百度迁徙

前几天瞅了眼百度迁徙，发现推出了实时拥堵指数，所以写了百度迁徙系列的第五篇博客。。。

效果展示

老实说，实时拥堵里面的所有数据在”拥堵详情“里都有，只不过展现的方式换了。所以我这里主要获取拥堵详情里的数据。要不然里面的类型实在是太多了。。。
拥堵详情大致分为五部分：
第一部分：实时拥堵指数与拥堵里程
在这里插入图片描述
第二部分：当日24小时拥堵指数变化
这里包含了历史的（说是历史的其实目前来看是去年十月份均值。。）和实时的。

第三部分：近七天高峰拥堵指数变化
其实也就是第二部分在高峰期求平均罢了。。。
在这里插入图片描述
第四部分：实时拥堵行政区排行

第五部分：实时拥堵道路排行
这个和第四部分挺像的，不过第四部分列出了所有行政区，而第五部分只列出前十位道路

我整理了上面几部分是数据，发现其实也就两种数据类型，
第一种：全市拥堵指数及其附属数据（里程、速度等）
第二种：行政区和道路拥堵指数及其附属数据。
所以，我分两个大模块抓取数据，第一模块市级的（包括当日和历史），第二模块微观级别的（包括政区和道路）。所以要说是四个模块也行。
存储方式我用txt，因为数据更新太快了，txt方便实时续写。
在这里插入图片描述

页面数据里有的都可以抓到，甚至还可以抓到页面数据里没有的，比如街道的经纬度坐标、行政区划代码什么的。
爬取思路和前几篇文章差不多。
正如上面所说，数据更新太快，所以这个代码如果加个定时器是可以支持续写的。续写的逻辑是读取你将要爬取的城市的信息，然后写上你没有爬取过的信息。
因此可以续写到地老天荒。
定时器的话。。自己加。。
下面直接上代码，四部分的代码将以函数的形式表达，调用方式相似，具体调用方法看最后。

代码（当日24小时拥堵指数变化）

# -*- coding: utf-8 -*-
import requests
import json
import time
import datetime


def yongdu_txt(City, Code):
    # 读取数据
    try:
        with open(f"百度迁徙数据v3.1//拥堵指数（当日和历史）//{Code}-{City}-拥堵指数（当日）.txt", "r") as f:
            old_data = f.readlines()
            print(old_data[-1])
            shangci_data = old_data[-1]
            shangci_data = str.split(shangci_data)
            shangci = shangci_data[2][-4:] + '-' + shangci_data[3][-2:] + '-' + shangci_data[4][-2:] + ',' + shangci_data[5][-2:] + ':' + shangci_data[6][-2:]
            k = 0 # 判断器
    except:
        print('文件不存在')
        shangci = 'kkk'
        k = 1 # 判断器

    # 写入数据
    with open(f"百度迁徙数据v3.1//拥堵指数（当日和历史）//{Code}-{City}-拥堵指数（当日）.txt", "a") as f:
        url=f'https://jiaotong.baidu.com/trafficindex/city/curve?cityCode={Code}&type=minute&his=0' # 实时
        print(f'{City}:{url}')
        try:
            response = requests.get(url, timeout=20) # #发出请求并json化处理
        except :
            print('正在准备重新启动------------------------------------------\n')
            time.sleep(10)
            response = requests.get(url, timeout=20) # #发出请求并json化处理
            print('启动中----------------------------------------------------\n')
        # 获取时间信息
        today = datetime.datetime.now() # 今天
        yesterday = datetime.datetime.now() - datetime.timedelta(days=1) # 昨天
        today_str = today.strftime("%Y-%m-%d,%H:%M")
        yesterday_str = yesterday.strftime("%Y-%m-%d,%H:%M")
        # print(today)
        # print(yesterday)
        # print(today_str)
        # print(yesterday_str)
        day = yesterday_str # 预设日期为昨天
        if len(response.text) > 200:
            data_dict = json.loads(response.text) # 字典化
            data_list = data_dict['data']['list']
            for i in range(len(data_list)):
                data_list_dic = data_list[i]
                data_index = data_list_dic['index']
                data_speed = data_list_dic['speed']
                data_time = data_list_dic['time']
                if data_time == '00:00': # 到0点将日期切换为今天
                    day = today_str
                data_time = day[:11]+data_time
                if k == 1 or shangci == data_time:
                    k = 1 # 判断器
                    if shangci == data_time:
                        print('正在续写数据')
                    else:
                        f.write(f"City:{City} Code:{Code} Year:{data_time[:4]} Month:{data_time[5:7]} Day:{data_time[8:10]} Hour:{data_time[-5:-3]} Minute:{data_time[-2:]} Index:{data_index} Speed:{data_speed}\n") # 续写数据写入数据
    print('完成')

代码（当日24小时拥堵指数变化-历史）

import requests
import json
import time
import datetime


def his_yongdu_txt(City, Code):
    # 读取数据
    try:
        with open(f"百度迁徙数据v3.1//拥堵指数（当日和历史）//{Code}-{City}-拥堵指数（历史）.txt", "r") as f:
            old_data = f.readlines()
            print(old_data[-1])
            shangci_data = old_data[-1]
            shangci_data = str.split(shangci_data)
            shangci = shangci_data[2][-4:] + '-' + shangci_data[3][-2:] + '-' + shangci_data[4][-2:] + ',' + shangci_data[5][-2:] + ':' + shangci_data[6][-2:]
            k = 0 # 判断器
    except:
        print('文件不存在')
        shangci = 'kkk'
        k = 1 # 判断器

    # 写入数据
    with open(f"百度迁徙数据v3.1//拥堵指数（当日和历史）//{Code}-{City}-拥堵指数（历史）.txt", "a") as f:
        url=f'https://jiaotong.baidu.com/trafficindex/city/curve?cityCode={Code}&type=minute&his=1' # 历史
        print(f'{City}:{url}')
        try:
            response = requests.get(url, timeout=20) # #发出请求并json化处理
        except :
            print('正在准备重新启动------------------------------------------\n')
            time.sleep(10)
            response = requests.get(url, timeout=20) # #发出请求并json化处理
            print('启动中----------------------------------------------------\n')
        # 获取时间信息
        today = datetime.datetime.now() # 今天
        yesterday = datetime.datetime.now() - datetime.timedelta(days=1) # 昨天
        today_str = today.strftime("%Y-%m-%d,%H:%M")
        yesterday_str = yesterday.strftime("%Y-%m-%d,%H:%M")
        # print(today)
        # print(yesterday)
        # print(today_str)
        # print(yesterday_str)
        day = yesterday_str # 预设日期为昨天
        if len(response.text) > 200:
            data_dict = json.loads(response.text) # 字典化
            data_list = data_dict['data']['list']
            for i in range(len(data_list)):
                data_list_dic = data_list[i]
                data_index = data_list_dic['index']
                data_speed = data_list_dic['speed']
                data_time = data_list_dic['time']
                data_his_index = data_list_dic['hisIndex']
                data_his_speed = data_list_dic['hisSpeed']
                if data_time == '00:00': # 到0点将日期切换为今天
                    day = today_str
                data_time = day[:11]+data_time
                if k == 1 or shangci == data_time:
                    k = 1 # 判断器
                    if shangci == data_time:
                        print('正在续写数据')
                    else:
                        f.write(f"City:{City} Code:{Code} Year:{data_time[:4]} Month:{data_time[5:7]} Day:{data_time[8:10]} Hour:{data_time[-5:-3]} Minute:{data_time[-2:]} Index:{data_index} Speed:{data_speed} HisIndex:{data_his_index} HisSpeed:{data_his_speed}\n") # 续写数据写入数据
                if shangci[-5:] == '23:30':
                    f.write(f"City:{City} Code:{Code} Year:{data_time[:4]} Month:{data_time[5:7]} Day:{data_time[8:10]} Hour:{data_time[-5:-3]} Minute:{data_time[-2:]} Index:{data_index} Speed:{data_speed} HisIndex:{data_his_index} HisSpeed:{data_his_speed}\n") # 续写数据写入数据

    print('完成')

代码（实时拥堵行政区排行）

import requests
import json
import time
import datetime


def zhengqu_yongdu_txt(City, Code):
    # 读取数据
    try:
        with open(f"{Code}-{City}-拥堵指数（政区）.txt", "r") as f:
            old_data = f.readlines()
            print(old_data[-1])
    except:
        print('文件不存在')

    # 写入数据
    with open(f"{Code}-{City}-拥堵指数（政区）.txt", "a") as f:
        url=f'https://jiaotong.baidu.com/trafficindex/city/districtrank?cityCode={Code}&roadtype=0' # 实时
        print(f'{City}:{url}')
        try:
            response = requests.get(url, timeout=20) # #发出请求并json化处理
        except :
            print('正在准备重新启动------------------------------------------\n')
            time.sleep(10)
            response = requests.get(url, timeout=20) # #发出请求并json化处理
            print('启动中----------------------------------------------------\n')
        # 获取时间信息
        today = datetime.datetime.now() # 今天
        yesterday = datetime.datetime.now() - datetime.timedelta(days=1) # 昨天
        today_str = today.strftime("%Y-%m-%d,%H:%M")
        yesterday_str = yesterday.strftime("%Y-%m-%d,%H:%M")
        # print(today)
        # print(yesterday)
        # print(today_str)
        # print(yesterday_str)
        day = yesterday_str # 预设日期为昨天
        if len(response.text) > 200:
            data_dict = json.loads(response.text) # 字典化
            data_list = data_dict['data']['list']
            for i in range(len(data_list)):
                data_list_dic = data_list[i]
                data_time = data_list_dic['timeHuman'] # 202110241425
                data_district_name = data_list_dic['district_name']
                data_district_code = data_list_dic['district_code']
                data_index = data_list_dic['index']
                data_index_level = data_list_dic['index_level']
                data_speed = data_list_dic['speed']
                data_length = data_list_dic['length']
                data_center = data_list_dic['center']
                f.write(f"City:{City} Code:{Code} Year:{data_time[:4]} Month:{data_time[4:6]} Day:{data_time[6:8]} Hour:{data_time[8:10]} Minute:{data_time[10:]} District:{data_district_name} DistrictCode:{data_district_code} Index:{data_index} IndexLevel:{data_index_level} Speed:{data_speed} Length:{data_length} Center:{data_center}\n") # 续写数据写入数据
    print('完成')

代码（实时拥堵道路排行）

import requests
import json
import time
import datetime


def daolu_yongdu_txt(City, Code):
    # 读取数据
    try:
        with open(f"{Code}-{City}-拥堵指数（道路）.txt", "r") as f:
            old_data = f.readlines()
            print(old_data[-1])
    except:
        print('文件不存在')

    # 写入数据
    with open(f"{Code}-{City}-拥堵指数（道路）.txt", "a") as f:
        url=f'https://jiaotong.baidu.com/trafficindex/city/roadrank?cityCode={Code}&roadtype=0' # 实时
        print(f'{City}:{url}')
        try:
            response = requests.get(url, timeout=20) # #发出请求并json化处理
        except :
            print('正在准备重新启动------------------------------------------\n')
            time.sleep(10)
            response = requests.get(url, timeout=20) # #发出请求并json化处理
            print('启动中----------------------------------------------------\n')
        # 获取时间信息
        today = datetime.datetime.now() # 今天
        yesterday = datetime.datetime.now() - datetime.timedelta(days=1) # 昨天
        today_str = today.strftime("%Y-%m-%d,%H:%M")
        yesterday_str = yesterday.strftime("%Y-%m-%d,%H:%M")
        # print(today)
        # print(yesterday)
        # print(today_str)
        # print(yesterday_str)
        day = yesterday_str # 预设日期为昨天
        if len(response.text) > 200:
            data_dict = json.loads(response.text) # 字典化
            data_list = data_dict['data']['list']
            for i in range(len(data_list)):
                data_list_dic = data_list[i]
                data_id = data_list_dic['id']
                data_time = data_list_dic['time'] # 202110241425
                data_district_type = data_list_dic['district_type']
                data_roadsegid = data_list_dic['roadsegid']
                data_speed = data_list_dic['speed']
                data_yongdu_length = data_list_dic['yongdu_length']
                data_road_type = data_list_dic['road_type']
                data_road_name = data_list_dic['roadname']
                data_index = data_list_dic['index']
                data_index_level = data_list_dic['index_level']
                data_length = data_list_dic['length']
                data_semantic = data_list_dic['semantic']
                data_links = data_list_dic['links']
                data_location = data_list_dic['location']

                infor_a = f"City:{City} Code:{Code} Year:{data_time[:4]} Month:{data_time[4:6]} Day:{data_time[6:8]} Hour:{data_time[8:10]} Minute:{data_time[10:]} "
                infor_b = f"DistrictType:{data_district_type} Roadsegid:{data_roadsegid} Speed:{data_speed} Yongdu_Length:{data_yongdu_length} RoadType:{data_road_type} RoadName:{data_road_name} "
                infor_c = f"Index:{data_index} IndexLevel:{data_index_level} Semantic:{data_semantic} Links:{data_links} Location:{data_location}\n"
                infor = infor_a + infor_b + infor_c
                f.write(infor) # 续写数据写入数据
    print('完成')

代码调用

可以发现，跳转到指定城市的url必须输入对应城市的code，因此调用上面任意一个函数的时候必须输入城市名和对应的code。
也就是代码里City和Code参数。
关于Code我已经很贴心地整理为字典了，如下：

YDCityCode = {
                "北京":131,"成都":75,"东莞":119,"上海":289,"深圳":340,"苏州":224,"天津":332,"武汉":218,"西安":233,"郑州":268,"重庆":132,
                "保定":307,"佛山":138,"广州":257,"杭州":179,"合肥":127,"济南":288,"金华":333,"昆明":104,"临沂":234,"南京":315,"宁波":180,
                "青岛":236,"沈阳":58,"石家庄":150,"唐山":265,"潍坊":287,"温州":178,"无锡":317,"长沙":158,"沧州":149,"常州":348,"大连":167,
                "德州":372,"福州":300,"赣州":365,"贵阳":146,"哈尔滨":48,"邯郸":151,"呼和浩特":321,"惠州":301,"济宁":286,"嘉兴":334,"廊坊":191,
                "洛阳":153,"南昌":163,"南宁":261,"南通":161,"南阳":309,"泉州":134,"厦门":194,"绍兴":293,"台州":244,"太原":176,"乌鲁木齐":92,
                "新乡":152,"邢台":266,"徐州":316,"烟台":326,"盐城":223,"长春":53,"中山":187,"淄博":354,"潮州":201,"大理":111,"大同":355,
                "桂林":142,"海口":125,"衡水":208,"衡阳":159,"湖州":294,"淮安":162,"江门":302,"拉萨":100,"兰州":36,"乐山":79,"连云港":347,
                "柳州":305,"茂名":139,"绵阳":240,"南充":291,"秦皇岛":148,"清远":197,"三亚":121,"汕头":303,"韶关":137,"泰安":325,"西宁":66,
                "咸阳":323,"扬州":346,"阳泉":357,"宜宾":186,"银川":360,"云浮":258,"湛江":198,"张家口":264,"漳州":255,"肇庆":338,"镇江":160,"珠海":140,
                }

所以，在调用的时候只要将City和Code首先赋值，然后直接用以下代码就可以获取数据。
比如这里我要获取南京的数据，那么：

City = '南京'
Code = YDCityCode[City]

yongdu_txt(City, Code) # 当日拥堵指数
his_yongdu_txt(City, Code) # 历史拥堵指数
zhengqu_yongdu_txt(City, Code) # 行政区拥堵排行
daolu_yongdu_txt(City, Code) # 道路拥堵排行