python爬虫城市数据

最新推荐文章于 2023-02-14 14:34:45 发布
最新推荐文章于 2023-02-14 14:34:45 发布
阅读量571 收藏 1
点赞数
分类专栏： 爬虫 文章标签： python 大数据
原文链接：https://blog.csdn.net/u012200261/article/details/56480023
版权

示例一

# -*- coding:utf-8 -*-
import csv
import json
import codecs
import os
import string
import sys
import time
import urllib.request
from urllib.parse import quote
from builtins import object, float, range, int, len, open, list, str

class BaiDuPOI(object):
def init(self, itemy, loc):
self.itemy = itemy
self.loc = loc

<span class="token keyword">def</span> <span class="token function">urls</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    api_key <span class="token operator">=</span> baidu_api
    urls <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token keyword">for</span> pages <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        url <span class="token operator">=</span> <span class="token string">'http://api.map.baidu.com/place/v2/search?query='</span> <span class="token operator">+</span> self<span class="token punctuation">.</span>itemy <span class="token operator">+</span> <span class="token string">'&amp;bounds='</span> <span class="token operator">+</span> self<span class="token punctuation">.</span>loc <span class="token operator">+</span> <span class="token string">'&amp;page_size=20&amp;page_num='</span> <span class="token operator">+</span> <span class="token builtin">str</span><span class="token punctuation">(</span>
            pages<span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token string">'&amp;output=json&amp;ak='</span> <span class="token operator">+</span> api_key
        urls<span class="token punctuation">.</span>append<span class="token punctuation">(</span>url<span class="token punctuation">)</span>
    <span class="token keyword">return</span> urls

<span class="token keyword">def</span> <span class="token function">baidu_search</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token triple-quoted-string string">'''json_sel = []
    for url in self.urls():
        s = quote(url, safe=string.printable)
        json_obj = urllib.request.urlopen(s).read().decode('utf-8')
        data = json.loads(json_obj)
        for item in data['results']:
            jname = item["name"]
            jlat = item["location"]["lat"]
            jlng = item["location"]["lng"]
            js_sel = jname + ',' + str(jlat) + ',' + str(jlng)
            json_sel.append(js_sel)
    return json_sel
    '''</span>
    data <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token keyword">for</span> url <span class="token keyword">in</span> self<span class="token punctuation">.</span>urls<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        s <span class="token operator">=</span> quote<span class="token punctuation">(</span>url<span class="token punctuation">,</span> safe<span class="token operator">=</span>string<span class="token punctuation">.</span>printable<span class="token punctuation">)</span>
        json_obj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
        data<span class="token punctuation">.</span>append<span class="token punctuation">(</span>json<span class="token punctuation">.</span>loads<span class="token punctuation">(</span>json_obj<span class="token punctuation">)</span><span class="token punctuation">)</span>
        time<span class="token punctuation">.</span>sleep<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span>  <span class="token comment"># 休眠1秒</span>
    <span class="token keyword">return</span> data

class LocaDiv(object):
def init(self, loc_all):
self.loc_all = loc_all

<span class="token keyword">def</span> <span class="token function">lat_all</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    lat_sw <span class="token operator">=</span> <span class="token builtin">float</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>loc_all<span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token string">','</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
    lat_ne <span class="token operator">=</span> <span class="token builtin">float</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>loc_all<span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token string">','</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
    lat_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">(</span><span class="token punctuation">(</span>lat_ne <span class="token operator">-</span> lat_sw <span class="token operator">+</span> <span class="token number">0.0001</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">0.2</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>  <span class="token comment"># 0.1为网格大小，可更改</span>
        lat_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span>lat_sw <span class="token operator">+</span> <span class="token number">0.2</span> <span class="token operator">*</span> i<span class="token punctuation">)</span>  <span class="token comment"># 0.05</span>
    lat_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span>lat_ne<span class="token punctuation">)</span>
    <span class="token keyword">return</span> lat_list

<span class="token keyword">def</span> <span class="token function">lng_all</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    lng_sw <span class="token operator">=</span> <span class="token builtin">float</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>loc_all<span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token string">','</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
    lng_ne <span class="token operator">=</span> <span class="token builtin">float</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>loc_all<span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token string">','</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
    lng_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">(</span><span class="token punctuation">(</span>lng_ne <span class="token operator">-</span> lng_sw <span class="token operator">+</span> <span class="token number">0.0001</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">0.3</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>  <span class="token comment"># 0.2为网格大小，可更改</span>
        lng_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span>lng_sw <span class="token operator">+</span> <span class="token number">0.3</span> <span class="token operator">*</span> i<span class="token punctuation">)</span>  <span class="token comment"># 0.2为网格大小，可更改</span>
    lng_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span>lng_ne<span class="token punctuation">)</span>
    <span class="token keyword">return</span> lng_list

<span class="token keyword">def</span> <span class="token function">ls_com</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    l1 <span class="token operator">=</span> self<span class="token punctuation">.</span>lat_all<span class="token punctuation">(</span><span class="token punctuation">)</span>
    l2 <span class="token operator">=</span> self<span class="token punctuation">.</span>lng_all<span class="token punctuation">(</span><span class="token punctuation">)</span>
    ab_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>l1<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        a <span class="token operator">=</span> <span class="token builtin">str</span><span class="token punctuation">(</span>l1<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">)</span>
        <span class="token keyword">for</span> i2 <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>l2<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
            b <span class="token operator">=</span> <span class="token builtin">str</span><span class="token punctuation">(</span>l2<span class="token punctuation">[</span>i2<span class="token punctuation">]</span><span class="token punctuation">)</span>
            ab <span class="token operator">=</span> a <span class="token operator">+</span> <span class="token string">','</span> <span class="token operator">+</span> b
            ab_list<span class="token punctuation">.</span>append<span class="token punctuation">(</span>ab<span class="token punctuation">)</span>
    <span class="token keyword">return</span> ab_list

<span class="token keyword">def</span> <span class="token function">ls_row</span><span class="token punctuation">(</span>self<span class="token punctuation">)</span><span class="token punctuation">:</span>
    l1 <span class="token operator">=</span> self<span class="token punctuation">.</span>lat_all<span class="token punctuation">(</span><span class="token punctuation">)</span>
    l2 <span class="token operator">=</span> self<span class="token punctuation">.</span>lng_all<span class="token punctuation">(</span><span class="token punctuation">)</span>
    ls_com_v <span class="token operator">=</span> self<span class="token punctuation">.</span>ls_com<span class="token punctuation">(</span><span class="token punctuation">)</span>
    ls <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
    <span class="token keyword">for</span> n <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>l1<span class="token punctuation">)</span> <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">0</span> <span class="token operator">+</span> <span class="token builtin">len</span><span class="token punctuation">(</span>l1<span class="token punctuation">)</span> <span class="token operator">*</span> n<span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>l2<span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>l2<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">*</span> n <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
            a <span class="token operator">=</span> ls_com_v<span class="token punctuation">[</span>i<span class="token punctuation">]</span>
            b <span class="token operator">=</span> ls_com_v<span class="token punctuation">[</span>i <span class="token operator">+</span> <span class="token builtin">len</span><span class="token punctuation">(</span>l2<span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">]</span>
            ab <span class="token operator">=</span> a <span class="token operator">+</span> <span class="token string">','</span> <span class="token operator">+</span> b
            ls<span class="token punctuation">.</span>append<span class="token punctuation">(</span>ab<span class="token punctuation">)</span>
    <span class="token keyword">return</span> ls

if name == ‘main’:
doc = open(‘NTPOI.csv’, ‘a+’)
writer = csv.writer(doc)

<span class="token comment"># ak</span>
baidu_api <span class="token operator">=</span> <span class="token string">"*********************"</span>  <span class="token comment"># 这里填入你的百度API的ak</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"开始爬取数据，请稍等..."</span><span class="token punctuation">)</span>
start_time <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
loc <span class="token operator">=</span> LocaDiv<span class="token punctuation">(</span><span class="token string">'31.69,120.54,32.65,121.95'</span><span class="token punctuation">)</span>
locs_to_use <span class="token operator">=</span> loc<span class="token punctuation">.</span>ls_row<span class="token punctuation">(</span><span class="token punctuation">)</span>

<span class="token keyword">for</span> loc_to_use <span class="token keyword">in</span> locs_to_use<span class="token punctuation">:</span>
    par <span class="token operator">=</span> BaiDuPOI<span class="token punctuation">(</span><span class="token string">'学校'</span><span class="token punctuation">,</span> loc_to_use<span class="token punctuation">)</span>  <span class="token comment"># 请修改爬取的类别</span>
    <span class="token triple-quoted-string string">'''
    a = par.baidu_search()
    for ax in a:
        writer.writerow(a)
   '''</span>

    listdata <span class="token operator">=</span> par<span class="token punctuation">.</span>baidu_search<span class="token punctuation">(</span><span class="token punctuation">)</span>
    <span class="token keyword">for</span> resultIndex <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>listdata<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>  <span class="token comment"># 提取返回的结果</span>
        resultlist<span class="token operator">=</span>listdata<span class="token punctuation">[</span>resultIndex<span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">"results"</span><span class="token punctuation">]</span>
        <span class="token keyword">for</span> pIndex <span class="token keyword">in</span>  <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>resultlist<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
             writer<span class="token punctuation">.</span>writerow<span class="token punctuation">(</span><span class="token builtin">list</span><span class="token punctuation">(</span>resultlist<span class="token punctuation">[</span>pIndex<span class="token punctuation">]</span><span class="token punctuation">.</span>values<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    <span class="token comment"># time.sleep(1)  # 休眠1秒</span>

doc<span class="token punctuation">.</span>close<span class="token punctuation">(</span><span class="token punctuation">)</span>
end_time <span class="token operator">=</span> time<span class="token punctuation">.</span>time<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"学校爬取完毕，用时%.2f秒"</span> <span class="token operator">%</span> <span class="token punctuation">(</span>end_time <span class="token operator">-</span> start_time<span class="token punctuation">)</span><span class="token punctuation">)</span>
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127

示例二

# 提取城市的POI点信息并将其保存至CSV
import csv
import string
import urllib
import json
from urllib.parse import quote

left_bottom = [120.89,31.83]; # 设置区域左下角坐标（百度坐标系）
right_top = [121.40,32.47]; # 设置区域右上角坐标（百度坐标系）
part_n = 8; # 设置区域网格（2*2）
url0 = ‘http://api.map.baidu.com/place/v2/search?’;
x_item = (right_top[0]-left_bottom[0])/part_n;
y_item = (right_top[1]-left_bottom[1])/part_n;
query = ‘学校’; #搜索关键词设置
ak = ‘OGTkT2pv********************mm9mek’; #百度地图api信令
n = 0; # 切片计数器

datacsv=open(“baidu.csv”, “a+”, encoding=“utf-8”);
csvwriter = csv.writer(datacsv, dialect=(“excel”))

for i in range(part_n):
for j in range(part_n):
left_bottom_part = [left_bottom[0]+ix_item,left_bottom[1]+jy_item]; # 切片的左下角坐标
right_top_part = [right_top[0]+ix_item,right_top[1]+jy_item]; # 切片的右上角坐标
for k in range(20):
url = url0 + ‘query=’ + query + ‘&page_size=20&page_num=’ + str(k) + ‘&scope=1&bounds=’ + str(left_bottom_part[1]) + ‘,’ + str(left_bottom_part[0]) + ‘,’+str(right_top_part[1]) + ‘,’ + str(right_top_part[0]) + ‘&output=json&ak=’ + ak;
s=quote(url, safe=string.printable)
data = urllib.request.urlopen(s);
hjson = json.loads(data.read().decode(‘utf-8’));
if hjson[‘message’] == ‘ok’:
results = hjson[‘results’];
for m in range(len(results)): # 提取返回的结果
csvwriter.writerow(list(results[m].values()))
n += 1;
print(‘第’,str(n),‘个切片入库成功’)

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35

如图：

在这里插入图片描述
在这里插入图片描述

                                </div><div><div></div></div>
            <link href="https://csdnimg.cn/release/phoenix/mdeditor/markdown_views-60ecaf1f42.css" rel="stylesheet">
                            </div>
        </article>
逼格怪兽
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
【Python爬虫案例学习10】基于Requests爬取拉勾网招聘信息，并保存至本地csv文件
人生苦短， 还不用Python？
08-05 1204
开发环境： MacBook Air (13-inch, Early 2015) macOS High Sierra 10.13.6 1.6GHZ Inter Core i5 Python：V 3.7.0 拉勾网Html页面分析 首先，使用自己账号登陆拉勾网，这个在分析header中会带有cookies等信息，在spider模拟访问请求的url时，被反爬虫的概率更小些（ps：拉勾网总不至于不让用...
python实现空气质量指数爬取，输出csv文件
weixin_42118531的博客
09-20 1644
""" 空气质量指数计算方法 定量描述空气质量状况的无量纲指数 版本7.0 智能爬取网页数据 获取所有城市的AQI bs = BeautifulSoup( url, html_parser, 指定解析器 enoding 指定编码格式（确保和网页编码一致） ) bs.find_all('a') 按照类型查找节点 按照属性查找节点 bs.find_all('...
python爬虫——保存数据为.csv文件
weixin_56067972的博客
11-15 2万+
爬虫存储数据为.csv文件
python爬虫练习案例：最受欢迎的影评数据保存csv文件
m0_67144365的博客
05-30 445
python爬虫练习案例：最受欢迎的影评保存csv文件
python爬取某省财政政策内容并保存为csv文件，最后用excel打开
weixin_44783823的博客
08-21 2229
目标网址 要求 保存形式 分析 代码及主要注释 csv文件用excel打开问题
Python爬虫-爬取目标城市酒店数据
11-12
通过python爬虫采集城市的酒店数据 内容概要：使用python采集酒店数据 适用人群：做酒店数据市场调研，数据分析报告的人群 使用场景及目标：需要依靠python3环境，执行爬虫脚本 其他说明：需要使用开发者工具捕捉...
119城市距离Python爬虫及源数据
05-23
119城市距离Python爬虫及源数据，使用python爬虫得到119个城市的相互距离。
Python爬虫实例_城市公交网络站点数据的爬取方法
09-20
下面小编就为大家分享一篇Python爬虫实例_城市公交网络站点数据的爬取方法，具有很好的参考价值，希望对大家有所帮助。一起跟随小编过来看看吧
python爬虫.zip
最新发布
03-01
爬虫通常由搜索引擎、数据挖掘工具、监测系统等应用于网络数据抓取的场景。 爬虫的工作流程包括以下几个关键步骤： URL收集： 爬虫从一个或多个初始URL开始，递归或迭代地发现新的URL，构建一个URL队列。这些URL...
Python爬虫数据可视化分析大作业.zip
05-29
Python爬虫数据可视化分析大作业，Python疫情大数据分析，涉及网络爬虫、可视化分析、GIS地图、情感分析、舆情分析、主题挖掘、威胁情报溯源、知识图谱、预测预警及AI和NLP应用等。
python 爬虫网页内容并写入CSV文件
niuyongliang110的博客
11-27 2463
python 爬虫 CSV
BeautifulSoup爬虫之保存CSV文件
ningyingqi的博客
12-09 4682
爬虫保存数据到CSV文件一.闲话一般我们写爬虫时都会保存为简单的text文件，但是当我们爬取的数据量很大我们想方便统计或者想存长时间保存 这个时候我们怎么办？我们可以保存信息为CSV格式 或者直接保存到数据库中。python提供了这样的包给我们！接下来我们以“中彩网往期双色球信息”为例给大家演示下如何保存信息CSV格式。二.干活 依然是爬虫三部曲：分析网页获取目标网址 ，爬取信息，保存信息。 1
python爬新闻并保存到csv_在python3.6中如何将爬取到的数据存入到csv文件中？
weixin_39582656的博客
11-21 411
我正在弄一个爬虫程序爬取豆瓣十部电影的相关数据，现在已经爬取出数据代码如下：from urllib import requestfrom bs4 import BeautifulSoupresponse = request.urlopen("https://movie.douban.com/chart")html = response.read()soup=BeautifulSoup(html,'...
用Python爬取51job保存到CSV
小胖墩
11-26 1565
#导入相关 import requests from lxml import etree import csv def spider(url_): response = requests.get(url_, headers=headers) response.encoding = ‘gbk’ return etree.HTML(response.text) #获取网页内容 def download...
Python爬取全国最新房价信息保存为CSV文件,进行简单的数据分析
分享既学习
07-09 4523
一、 使用Python爬取房价信息 1.话不多说先看代码 from bs4 import BeautifulSoup import requests class SpiderHours(object): def __init__(self): self.url = "http://www.creprice.cn/rank/cityforsale.html" ...
python 爬虫去获得省市区县的数据
rigou11的博客
02-14 2324
爬虫
python爬取中国-省-市-区县的geoJSON格式地图数据
Aimee的博客
03-04 2509
github地址：https://github.com/sky-Aimee/geoJson-cn
python爬虫爬取全国省市区
chenpang2969的博客
05-14 2168
#之前写的代码 from requests_html import HTMLSession import requests import time import re import datetime import json session = HTMLSession() import mys...
Python爬虫（二）--利用百度地图API批量获取城市所有的POI点
热门推荐
打酱油的葫芦娃
04-15 5万+
上篇关于爬虫的文章，我们讲解了如何运用Python的requests及BeautifuiSoup模块来完成静态网页的爬取，总结过程，网页爬虫本质就两步： 1、设置请求参数（url，headers，cookies，post或get验证等）访问目标站点的服务器； 2、解析服务器返回的文档，提取需要的信息。
python爬虫并数据可视化
08-30
Python爬虫和数据可视化是通过Python编程语言来实现的一种技术。Python爬虫是指使用Python编写程序，通过模拟浏览器的行为来获取互联网上的数据。数据可视化是指使用图表、图形等可视化方式将数据呈现出来，使之更加...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值