北京市城镇街道边界数据

北京市城镇街道边界数据

  • 目标网站:

https://beijing.tianditu.gov.cn/bjtdt-main/electronicindex.html
在这里插入图片描述

  • 最终数据:
{"code": "110101", "area_name": "东城区", "gbcode": "110101001", "down_name": "东华门街道", "address": "北京市东城区东安门大街55号", "geomPoint": "[39.9145862116844,116.40103833605]", "geomPolygon": "[[[[39.9229507331899,116.404653024318],[39.9229065800747,116.404821416975],[39.9228966605525,116.404987497475],[39.9228984142305,116.405401833128],[39.9229096692459,116.406454151438],[39.9229235224027,116.407651909908],[39.922933244074,116.408990577753],[39.9229371129575,116.410394363709],[39.9229459101257,116.411249547928],[39.9207894770528,116.411352462746],[39.9199761814556,116.411390483384],[39.9169247089083,116.41148187339],[39.9155844681487,116.411538733026],[39.9150429377814,116.411557840022],[39.9125433449994,116.411653770705],[39.9098860462041,116.411728632071],[39.90895740536,116.411695640442],[39.9068865337866,116.411787777784],[39.9063408665378,116.411843353188],[39.9059363478842,116.411881948493],[39.9047941828053,116.411973223386],[39.904389988807,116.411997282948],[39.9023956639288,116.412104379614],[39.8997216746156,116.412186074028],[39.8997098287456,116.411603361906],[39.8996940654288,116.410758101705],[39.8996821862839,116.410222743385],[39.8996674436976,116.409342232458],[39.899624171918,116.40876949022],[39.8996110373196,116.408294372987],[39.8995950752526,116.407799786331],[39.8995733710143,116.407027918903],[39.8995576625562,116.406418633613],[39.899552892552,116.406226325285],[39.899530719767,116.405508632118],[39.8995277529035,116.405412627691],[39.8995008110137,116.40454315864],[39.8994714733299,116.40358319351],[39.8994486926031,116.402886828664],[39.8994297043175,116.402313322002],[39.8994219512621,116.401981583683],[39.8994029773656,116.401361777224],[39.8993812038795,116.400652261593],[39.8993756127944,116.400464425593],[39.8993582235033,116.399911177959],[39.8993412326118,116.399369505498],[39.8993246329255,116.398852561696],[39.899309830984,116.398395600876],[39.899295829439,116.397959158089],[39.8992888650891,116.397648468402],[39.8992874576501,116.397619792619],[39.8992616210271,116.396879761192],[39.8992446067533,116.396360977377],[39.8992217900536,116.396162083314],[39.8991714747838,116.395814534913],[39.8990888324837,116.395229940509],[39.8990103009846,116.394747554056],[39.8989723127221,116.394474202423],[39.8989374594962,116.394217557494],[39.8989001754028,116.393956174038],[39.8988183397946,116.393355013223],[39.8988035090748,116.392914630506],[39.8987844677291,116.392351391404],[39.8987658229844,116.391801437986],[39.8987563351368,116.39141630242],[39.8987473527082,116.391037084394],[39.8987381580396,116.390677729693],[39.8987332756202,116.390447544619],[39.8987290775849,116.390280494649],[39.8987348224542,116.390077147143],[39.8987344240545,116.38980920673],[39.8987419864536,116.389654210374],[39.9024368474826,116.389421814765],[39.9067009919573,116.389201308194],[39.9066226547118,116.386396941466],[39.9070631048784,116.386373265014],[39.9071717204986,116.38650906714],[39.9092937715903,116.386415744491],[39.9092871157079,116.385549614623],[39.91048415472,116.385448735871],[39.9104932513625,116.385614966558],[39.9105883555682,116.385616309245],[39.9105888043299,116.38570106765],[39.9109982710534,116.385666720743],[39.9110028800789,116.385793090779],[39.9115227529727,116.385803848469],[39.9115531536552,116.386548522197],[39.9161805486934,116.386312527501],[39.9169714232926,116.386206752839],[39.9209185549474,116.386056056142],[39.9210458171108,116.390554532466],[39.92144707842,116.39053600913],[39.9219842191964,116.390511213022],[39.922089105328,116.393333290999],[39.9220103175222,116.393338229176],[39.9220309632584,116.394131849405],[39.9220640034511,116.395272682588],[39.9220915361955,116.396396747912],[39.9221925291619,116.396609428582],[39.9226073738313,116.397158197593],[39.9227491789315,116.397344269123],[39.922868228486,116.39750544562],[39.9228810303354,116.398091747836],[39.9228994214712,116.399302901708],[39.9229121342876,116.400274362872],[39.9229282798163,116.401496658444],[39.9229552181089,116.402913043902],[39.9229741443414,116.40447607641],[39.9229507331899,116.404653024318]]]]"}
......

1. 行政区街道

  • F12 检查网页结构,获取请求信息。
请求网址:https://beijing.tianditu.gov.cn/tianditu_pro/region/queryRegion
请求方法:POST
请求参数:无
解析响应数据
  • 网页查看响应数据是json 数据,其中需要的数据是 data 字段,字段结构 base64 加密,可以使用 Python 自带 base64 库进行解密。
import base64
def decrypt(info: str) -> obj:
    """解密"""
    json_str = base64.b64decode(info).decode("utf8")
    return json.loads(json_str)

2. 对应街道数据获取

  • 如:解析 东华门街道 数据

  • 请求信息:

请求网址:https://beijing.tianditu.gov.cn/tianditu_pro/region/queryRegionDetails
请求方法:POST
请求参数:{
	"eyJnYmNvZGUiOiIxMTAxMDEwMDEifQ==": ""
}
解析请求参数
  • 观察请求参数,base64 加密。反向解密,获取原始字符串。
加密字符串:eyJnYmNvZGUiOiIxMTAxMDEwMDEifQ==
解密后对应数据:'{"gbcode":"110101001"}'
解析请求响应数据
  • 加密方式与请求行政区数据一样,以同样的方式解密数据。

3. 代码

# -*- coding: utf-8 -*-
import os
import json
import time
import base64
import random
import requests

USER_AGENT = [
    "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.163 Safari/535.1",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36",
    "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
    "User-Agent,Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4094.1 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko",
    "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)",
    "User-Agent:Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
    "User-Agent, Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36 Edg/96.0.1054.34",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36"
]


class SpiderGPS(object):
    def __init__(self):
        self.file_name = "data.json"
        self.area_set = set()

    def prepare(self):
        pass

    def load_file(self):
        """文件加载"""
        if not os.path.exists(self.file_name):
            return
        for line in open(self.file_name, "rb"):
            if not line:
                continue
            line_obj = json.loads(line.decode("utf8"))
            gbcode = line_obj.get("gbcode")
            self.area_set.add(gbcode)

    def res_post(self, url, params=None):
        """数据请求"""
        while True:
            time.sleep(random.randint(1, 2))
            try:
                user_agent = random.choice(USER_AGENT)
                headers = {
                    "User-Agent": user_agent
                }
                response = requests.post(url, headers=headers, timeout=3, data=params)
                res_obj = response.json()
                code = res_obj.get("code")
                if code != 1:
                    return
                data = res_obj.get("data")
                data_obj = self.decrypt(data)
                return data_obj
            except Exception as e:
                time.sleep(5)
                print(f"请求错误。{e}")

    def decrypt(self, info: str) -> list:
        """解密"""
        json_str = base64.b64decode(info).decode("utf8")
        # print(json_str)
        return json.loads(json_str)

    def process(self):
        self.load_file()
        url = "https://beijing.tianditu.gov.cn/tianditu_pro/region/queryRegion"
        self.parse_locations(url)

    def parse_locations(self, url):
        """解析乡镇街道"""
        data_obj = self.res_post(url)
        for one_data in data_obj:
            code = one_data.get("gbcode")
            name = one_data.get("name")
            children_list = one_data.get("children")
            for children in children_list:
                gbcode = children.get("gbcode")
                children_name = children.get("name")
                # {"gbcode":"110101001"}
                if gbcode in self.area_set:
                    continue
                msg = self.parse_polygon(gbcode)
                old_dict = {
                    "code": code,
                    "area_name": name,
                    "gbcode": gbcode,
                    "down_name": children_name
                }
                old_dict.update(msg)
                self.write_file(old_dict)

    def parse_polygon(self, gbcode):
        """解析乡镇街道边界GPS"""
        url = "https://beijing.tianditu.gov.cn/tianditu_pro/region/queryRegionDetails"
        source_str = '{"gbcode":"' + gbcode + '"}'
        base64_str = base64.b64encode(source_str.encode("utf8"))
        params = {
            base64_str: ""
        }
        data_obj = self.res_post(url, params=params)
        res_dict = {
            "address": data_obj.get("address"),
            "geomPoint": data_obj.get("geomPoint"),
            "geomPolygon": data_obj.get("geomPolygon")
        }
        return res_dict

    def write_file(self, info_dict):
        """数据写入文件"""
        with open(self.file_name, "ab") as f:
            f.write(json.dumps(info_dict, ensure_ascii=False).encode("utf8"))
            f.write("\n".encode("utf8"))


def main():
    s = SpiderGPS()
    s.process()


if __name__ == '__main__':
    main()

4. 注意事项

  1. 因为各省份网页结构不一样,目前天地图上只有北京市数据可以这样解析;
  2. 网站中地图来源:使用 GIS for Leaflet 框架实现,如果需要将边界数据在其他地图上显示,需要进行坐标转换。

5. 数据

  • 网站原始数据:

链接:https://pan.baidu.com/s/1g0bngymx9Weqg9N_mC96Sw
提取码:3hhh

  • 高德地图坐标转换数据:

链接:https://pan.baidu.com/s/1eUEGHncXvK_WMh70repggg
提取码:66cs

  • 0
    点赞
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值