(爬虫)各大网站的查询接口和传参方式

腾讯天气

https://tianqi.qq.com/index.htm

ql = '海南'
qe = '三亚'
ser = 'https://wis.qq.com/weather/common?source=pc&weather_type=observe%7Cforecast_1h%7Cforecast_24h%7Cindex%7Calarm%7Climit%7Ctips%7Crise&province={}&city={}'.format(ql, qe)
print(ser)

百度地图

这个网站调的是百度地图接口

http://www.ditu6.com/

ql = '阜新'
qe = '北京'
ser = 'http://api.map.baidu.com/?qt=nav&c=1&sn=2%24%24%24%24%24%24{}%24%240%24%24%24%24&en=2%24%24%24%24%24%24{}%24%240%24%24%24%24&sy=0&ie=utf-8&oue=1&fromproduct=jsapi&res=api'.format(ql,qe)
print(ser)

唯品会主页搜索

https://www.vip.com/

偶然找到的所以没有截图 这个需要 把加载速度调慢寻找

ql = '耐克'
ser = 'https://category.vip.com/ajax/getSuggest.php?callback=searchSuggestions&keyword={}&_=1555241988506'.format(ql)
print(ser)

京东搜索 列表页商品

https://www.jd.com/

ql = '三叶草'
ser = 'https://search-x.jd.com/Search?callback=jQuery8928249&area=1&enc=utf-8&keyword={}&adType=7&page=1&ad_ids=291%3A24&xtest=new_search&_=1555292292157'.format(ql)
print(ser)

京东评论

某一个商品下的评论

https://item.jd.com/4222282.html#comment

点击页标去获取接口

去除json数据里面的html标签

from w3lib.html import remove_tags

pip3 install w3lib # 安装
b = remove_tags(title)  # 使用
print(b)

淘宝热卖

https://uland.taobao.com/sem/tbsearch?spm=a2e15.8261149.07626516003.1.41f529b4mtIYwm&refpid=mm_26632360_8858797_29866178&clk1=499c60f0a644c5c9aab9e4f635d41815&keyword=女装2019新款潮&page=2&_input_charset=utf-8

腾讯新闻

https://new.qq.com/ch/milite/

今日头条天气

https://www.toutiao.com/stream/widget/local_weather/data/?city=北京

from w3lib.html import remove_tags
import json
import requests
from lxml import etree
import re



qe = '北京'
ser = 'https://www.toutiao.com/stream/widget/local_weather/data/?city={}'.format(qe)
response = requests.get(ser, headers={
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
}, timeout=50)

we = json.loads(response.text)
content = we['data']
city = content['city'] # 城市
dayweather = content['weather']

for many_weather in dayweather['forecast_list']:
    wind_direction = many_weather['wind_direction'] # 风向
    high_temperature = many_weather['high_temperature'] # 最高气温
    low_temperature = many_weather['low_temperature'] # 最低气温
    condition = many_weather['condition'] # 天气状况
    date = many_weather['date'] # 天气日期
    wind_level = many_weather['wind_level'] #天气风速


    print(city,wind_direction,high_temperature,low_temperature,condition,date,wind_level)

优酷视频

不是VIP

是vip

搜索ip　查询地址

import json
import requests
from lxml import etree
import re
import socket

qe = '127.0.0.1'

ser = 'http://ip.taobao.com/service/getIpInfo.php?ip={}'.format(qe)

response = requests.get(ser, headers={
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
}, timeout=50)


we = json.loads(response.text)

country = we['data']['country'] # 国家
province = we['data']['city'] # 省
city = we['data']['region'] # 市
myip = we['data']['ip'] # ip
network = we['data']['isp'] # 网络渠道

print(country,province,city,myip,network)

hostname = 'cn.bing.com'
addr = socket.gethostbyname(hostname)
print ('The address of', hostname, 'is', addr)