很多数据，需要用搜索引擎帮忙搜下网页，再进行爬。找了半天就发现google和bing的可用--免费，但有一定限制。
google得接vpn什么的，你懂地。。。
使用bing搜索引擎服务，首先你需要一个ms帐号，然后去申请一个接入APP ID，免费版的有使用次数和频率限制。
bing搜索中文页： http://cn.bing.com/partners/developers#BingSearchApi
bing搜索APP ID页： https://www.microsoft.com/cognitive-services/en-us/subscriptions
bing搜索API: https://dev.cognitive.microsoft.com/docs/services/56b43eeccf5ff8098cef3807/operations/56b4447dcf5ff8098cef380d
bing搜索测试页：https://dev.cognitive.microsoft.com/docs/services/56b43eeccf5ff8098cef3807/operations/56b4447dcf5ff8098cef380d/console
如果返回http_code为200表示调用成功，结果是一个json格式的字符串，包含网页url、描述、日期等。
def t5():
import json
import requests
search_key = '黄金'
search_count = 10
search_offset = 10
search_language = 'zh-cn' #mkt
ms_app_id = 'a499*****************' #APP ID
url = 'https://api.cognitive.microsoft.com/bing/v5.0/search?q=%s&count=%d&offset=%d&mkt=%s&safesearch=Off' % (
search_key, search_count, search_offset, search_language)
headers = {'Ocp-Apim-Subscription-Key': ms_app_id} #必需，用于验证
ret = requests.get(url=url, headers=headers)
ret_code = ret.status_code
if not 200 == ret_code:
print ret_code
ret.close()
return
ret_content = ret.content
ret.close()
ret_content_dict = json.loads(ret_content)
ret_items = ret_content_dict.get('webPages').get('value')
for i in ret_items:
print i
响应数据：
{
"_type": "SearchResponse",
"webPages": {
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=hLmod1xUQ8sDTA4JbO1WZgkMJxe7ha9J5lEvKntxvxw&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591&p=DevEx,5437.1",
"totalEstimatedMatches": 14100000,
"value": [
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.0",
"name": "国际黄金价格走势图-黄金频道-和讯网",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=QoccehSy0QAhzg1f_JmllllfGvDb6dednFmprKry0Sw&v=1&r=http%3a%2f%2fgold.hexun.com%2fhjxh%2f&p=DevEx,5103.1",
"thumbnailUrl": "https://www.bing.com/th?id=OIP.M7e0fabd47cfc3d75f0581207207d9fb9o0&pid=Api",
"displayUrl": "gold.hexun.com/hjxh",
"snippet": "品种 中间价 买入价 卖出价 最高价 最低价 今开盘 昨收盘 涨跌量 更新时间 美元账户黄金 1275.05 1273.75 1276.35 1283.92 1267.72 1267.72 1275.10-1.35 2016-10-29 04:00:00",
"deepLinks": [
{
"name": "纸黄金",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=-bKusIVEhz02UqGzDdFOs1ibm2SDxq66RC1ahvdaivg&v=1&r=http%3a%2f%2fgold.hexun.com%2faccount%2f&p=DevEx,5096.1",
"snippet": "指黄金的纸上交易，投资者的买卖交易记录只在个人预先开立的“黄金存折账户”上体现，而不涉及实物金的提取。盈利模式即通过低买高卖，获取差价利润。"
},
{
"name": "实物金",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=csDxPOP2bxG85JED-8VY5ExkhGhP0CiLoPy7X416T3E&v=1&r=http%3a%2f%2fdata.gold.hexun.com%2fGoldHistoryPage.aspx&p=DevEx,5099.1",
"snippet": "时间 品牌 产品 价格 手工费 涨跌 2016-11-07 中国黄金 足金饰品 330.00----2016-11-07 中国黄金 高端工艺金条 355.00----2016-11-07 中国黄金 工艺金条 308.90----2016-11-07"
},
{
"name": "金市动态",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=Rad4DWLTda_sa8wjqBcKSB4CQjwcefoRVKDdEHDJEcA&v=1&r=http%3a%2f%2fgold.hexun.com%2fmarket%2f&p=DevEx,5094.1",
"snippet": "剑心解盘：9.18-9.19原油沥青一周行情回顾及下周布局操作建议 剑心解盘：8.26-27周评原油沥青行情分析及操作建议，原油宁贵沥青下周行情预测及操作建议"
},
{
"name": "实物黄金",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=QkRsF_ZBY43VUim_elwqfij-1iCbFxtepkKJA536Q6w&v=1&r=http%3a%2f%2fgold.hexun.com%2fbullion%2findex.html&p=DevEx,5095.1",
"snippet": "菜百简介 菜百主要经营足金、千足金饰品、金条、白玉、珠宝、金银币章、奥运特许商品及18K金、银饰品等大类商品。[详细]"
},
{
"name": "沙龙首页",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=pAoPOZbZHmTruR3EsGZl0dhlZ_AL6FoIiLfHEgCEVtY&v=1&r=http%3a%2f%2fsalon.hexun.com%2fgold%2findex.aspx&p=DevEx,5098.1",
"snippet": "2012年12月4日短期黄金下方支撑1705-1708 2012年12月3日今年年前白银高点在37一线 2012年11月29日白银12月35是较强阴力位 2012年11月28日白银近期回到33.5以下概率不 ..."
},
{
"name": "现货黄金",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=4LiFFvOy2k4jktXUHhSiYShyLQ3OcJx-0oGwEvGDPk0&v=1&r=http%3a%2f%2fgold.hexun.com%2fspot%2findex.html&p=DevEx,5100.1",
"snippet": "黄金现货市场上实物黄金的主要形式是金条（Gold Bullion）和金块，也有金币、金质奖章和首饰等。金条有低纯度的砂金和高纯度的条金，条金一般重400盎司。"
}
],
"dateLastCrawled": "2016-10-30T10:18:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.1",
"name": "黄金（贵金属）_百度百科",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=tFhHEizzpk-OEzaqAoHLPhW8Jb_6-PxoF5MZjJ3QsJQ&v=1&r=http%3a%2f%2fbaike.baidu.com%2fview%2f5937.htm&p=DevEx,5117.1",
"displayUrl": "baike.baidu.com/view/5937",
"snippet": "黄金（Gold）是化学元素金（化学元素符号Au）的单质形式，是一种软的，金黄色的，抗腐蚀的贵金属。金是最稀有、最珍贵和最被人看重的金属之一。国际上一般黄金都是 ...",
"dateLastCrawled": "2016-10-31T16:35:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.2",
"name": "黄金行情_东方财富网",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=UKVszzZuN1jaW-6lWooqOGjIvbr0FD8m_eswRJDDYrk&v=1&r=http%3a%2f%2fquote.eastmoney.com%2fcenter%2fgold.html&p=DevEx,5130.1",
"displayUrl": "quote.eastmoney.com/center/gold.html",
"snippet": "东方财富网黄金行情提供权威、全面、专业、及时的行情数据，将黄金行情、国际贵金属期货、国际贵金属现货、上海黄金期货、上海黄金现货、货币贵金属、纸 黄金 ...",
"dateLastCrawled": "2016-11-05T19:31:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.3",
"name": "黄金网_国内最具影响力的黄金投资服务平台",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=3rfiJbhnHx7qNre3kYIhMLvafJb_TCZiTNVuDaXdThg&v=1&r=http%3a%2f%2fgold.cnfol.com%2f&p=DevEx,5144.1",
"thumbnailUrl": "https://www.bing.com/th?id=OIP.M886fd1231b6be43eaec53890052a18d8o2&pid=Api",
"displayUrl": "gold.cnfol.com",
"snippet": "黄金网为您提供最及时、最全面全球24小时黄金、白银市场最新动态、国际经济数据和财经大事，满足贵金属投资市场参与者对基本面消息及时、全面的需求；提供业界最 ...",
"dateLastCrawled": "2016-11-06T01:19:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.4",
"name": "黄金价格|白银价格|黄金|白银_新浪财经-贵金属_新浪网",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=_pEXiOXNdR73LGeqyiUQPVhGMQcXt9LU5OtzS8HqrZQ&v=1&r=http%3a%2f%2ffinance.sina.com.cn%2fnmetal%2f&p=DevEx,5158.1",
"thumbnailUrl": "https://www.bing.com/th?id=OIP.M83d7877098e4a17b6497e3957777cb55o0&pid=Api",
"displayUrl": "finance.sina.com.cn/nmetal",
"snippet": "新浪贵金属提供全面专业的贵金属资讯及报价服务,覆盖黄金、白银等多个品种的现货和期货市场的投资理财服务.例如提供黄金价格,白银价格,纸黄金,黄金期货等贵金属信息。",
"dateLastCrawled": "2016-11-05T18:06:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.5",
"name": "中国黄金投资网(中国黄金网),黄金,金价,黄金价格,今日金价,纸 ...",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=CKVXV-pMW-evHTkLcQSW8LDY5601qWGGRKMsCnjgbZE&v=1&r=http%3a%2f%2fwww.cngold.org%2f&p=DevEx,5172.1",
"displayUrl": "www.cngold.org",
"snippet": "中国黄金网提供国际国内黄金价格,纸黄金价格,今日金价,铂金(白金)价格,黄金T+D开户,白银价格,纸黄金价格走势图,伦敦金,黄金投资资讯,金价实时行情,金价走势图 ...",
"dateLastCrawled": "2016-11-05T10:55:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.6",
"name": "COMEX黄金(GC)期货行情,新闻,报价_新浪财经_新浪网",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=YgrXMCJAerRufageBbWIQLPtALhl7HQn9DWik5yqdjY&v=1&r=http%3a%2f%2ffinance.sina.com.cn%2ffutures%2fquotes%2fGC.shtml&p=DevEx,5186.1",
"displayUrl": "finance.sina.com.cn/futures/quotes/GC.shtml",
"snippet": "新浪财经-期货频道为您提供COMEX黄金(GC)期货行情,期货资料,期货新闻,报价,机构报告,评论,现货价格,持仓分析等与COMEX黄金(COMEX黄金)期货相关的信息与 ...",
"dateLastCrawled": "2016-11-05T13:16:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.7",
"name": "黄金_理财频道_腾讯网",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=-lcwVd1Mt0qmAasOXzJTwzdbSSsmxryMumr6_1hPRdE&v=1&r=http%3a%2f%2ffinance.qq.com%2fgold%2f&p=DevEx,5200.1",
"displayUrl": "finance.qq.com/gold",
"snippet": "腾讯黄金频道为您提供专业丰富的黄金资讯，包括黄金，白银，实时行情，分析评论等内容。 ... 股市再遭杀戮 避险推升黄金大涨 金价目前交易于所有均线上方，5 ...",
"dateLastCrawled": "2016-10-22T03:26:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.8",
"name": "行业价值资讯领导者-贵金属频道-金融界",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=0o8GT89KHtGV5Vk3KBG1gwmEVJ5CtdPxc43XEVaI85k&v=1&r=http%3a%2f%2fgold.jrj.com.cn%2f&p=DevEx,5213.1",
"thumbnailUrl": "https://www.bing.com/th?id=OIP.Md28f6f64d420827d0c08eca680c94805o1&pid=Api",
"displayUrl": "gold.jrj.com.cn",
"snippet": "金融界黄金频道，国内最早的黄金频道，最权威的黄金投资指南，行情24小时实时更新，内容覆盖现货黄金、纸黄金、黄金T+D、黄金期货、现货白银、纸白银及铂金钯金等 ...",
"dateLastCrawled": "2016-11-06T03:09:00"
},
{
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.9",
"name": "和讯黄金-国内黄金门户专业权威网站",
"url": "http://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=oCHwBwAb3-CWzxFjXqLPH6PnrL8y1Cm9GwwwwNcCZhA&v=1&r=http%3a%2f%2fgold.hexun.com%2f&p=DevEx,5225.1",
"displayUrl": "gold.hexun.com",
"snippet": "和讯黄金为中国最大的黄金专业网站，国际财经资讯、贵金属市场最新动态、专家金评发表量第一，最专业最全的黄金白银市场数据，拥有中国最火的外汇论坛和博客互动 ...",
"dateLastCrawled": "2016-11-05T20:29:00"
}
]
},
"relatedSearches": {
"id": "https://api.cognitive.microsoft.com/api/v5/#RelatedSearches",
"value": [
{
"text": "黄金外汇交易平台",
"displayText": "黄金外汇交易平台",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=nxYqrIsLkTTqL72Bkk6Svzcd75ZPVQvHuKAbldl70DE&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e5%25a4%2596%25e6%25b1%2587%25e4%25ba%25a4%25e6%2598%2593%25e5%25b9%25b3%25e5%258f%25b0&p=DevEx,5438.1"
},
{
"text": "黄金白银价格",
"displayText": "黄金白银价格",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=yl9EcI_HpemXrC1gRk87sd8_IeDEn4tkLmk8499KaO8&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e7%2599%25bd%25e9%2593%25b6%25e4%25bb%25b7%25e6%25a0%25bc&p=DevEx,5439.1"
},
{
"text": "黄金投资公司",
"displayText": "黄金投资公司",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=u6JrcBTpg0HR3RPlP4HlyBjkkvqJjJVlsxpZQa0ft6E&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e6%258a%2595%25e8%25b5%2584%25e5%2585%25ac%25e5%258f%25b8&p=DevEx,5440.1"
},
{
"text": "国际黄金价格网",
"displayText": "国际黄金价格网",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=EZ4s0Zz5yG6SP9CkjCm3k2Pno4QxB0BeS3hb7qjVv5U&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e5%259b%25bd%25e9%2599%2585%25e9%25bb%2584%25e9%2587%2591%25e4%25bb%25b7%25e6%25a0%25bc%25e7%25bd%2591&p=DevEx,5441.1"
},
{
"text": "黄金收购",
"displayText": "黄金收购",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=fs94jtaF3YmvI7f4NoMatRX7RYnr5Hkliqr3JjAinFw&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e9%25bb%2584%25e9%2587%2591%25e6%2594%25b6%25e8%25b4%25ad&p=DevEx,5442.1"
},
{
"text": "国际黄金期货",
"displayText": "国际黄金期货",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=x8FskOKYJT5k2w20DN2xbRrU3UNSEauMv9INljPhmog&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e5%259b%25bd%25e9%2599%2585%25e9%25bb%2584%25e9%2587%2591%25e6%259c%259f%25e8%25b4%25a7&p=DevEx,5443.1"
},
{
"text": "搜狐黄金",
"displayText": "搜狐黄金",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=sNftjaCimcu4OKF6255d2yhu7HJf_CdNa3xQsDF3rQI&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e6%2590%259c%25e7%258b%2590%25e9%25bb%2584%25e9%2587%2591&p=DevEx,5444.1"
},
{
"text": "期货 黄金 股票",
"displayText": "期货 黄金 股票",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=-4pROTsvM90_hEltKIbVgxHiCDBlzsR77Fy90P15DHc&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e6%259c%259f%25e8%25b4%25a7%2b%25e9%25bb%2584%25e9%2587%2591%2b%25e8%2582%25a1%25e7%25a5%25a8&p=DevEx,5445.1"
},
{
"text": "中国黄金期货",
"displayText": "中国黄金期货",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=dvcMe80xmGUA8p_NAc4hHfyL_NXX90Ql3G10ZRqXzHA&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e4%25b8%25ad%25e5%259b%25bd%25e9%25bb%2584%25e9%2587%2591%25e6%259c%259f%25e8%25b4%25a7&p=DevEx,5446.1"
},
{
"text": "在线黄金",
"displayText": "在线黄金",
"webSearchUrl": "https://www.bing.com/cr?IG=C1AB388388994FA1B48C400EEEFE3E25&CID=35A25E6A5FDB67910FBB57A05EC9669A&rd=1&h=JKjSWXuNjinKVqTYxd-Yx-6TnyC3TZt9BzMWWMAWKkg&v=1&r=https%3a%2f%2fwww.bing.com%2fsearch%3fq%3d%25e5%259c%25a8%25e7%25ba%25bf%25e9%25bb%2584%25e9%2587%2591&p=DevEx,5447.1"
}
]
},
"rankingResponse": {
"mainline": {
"items": [
{
"answerType": "WebPages",
"resultIndex": 0,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.0"
}
},
{
"answerType": "WebPages",
"resultIndex": 1,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.1"
}
},
{
"answerType": "WebPages",
"resultIndex": 2,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.2"
}
},
{
"answerType": "WebPages",
"resultIndex": 3,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.3"
}
},
{
"answerType": "WebPages",
"resultIndex": 4,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.4"
}
},
{
"answerType": "WebPages",
"resultIndex": 5,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.5"
}
},
{
"answerType": "WebPages",
"resultIndex": 6,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.6"
}
},
{
"answerType": "WebPages",
"resultIndex": 7,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.7"
}
},
{
"answerType": "WebPages",
"resultIndex": 8,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.8"
}
},
{
"answerType": "WebPages",
"resultIndex": 9,
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#WebPages.9"
}
}
]
},
"sidebar": {
"items": [
{
"answerType": "RelatedSearches",
"value": {
"id": "https://api.cognitive.microsoft.com/api/v5/#RelatedSearches"
}
}
]
}
}
}
扫一扫
1111
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
