问题记录-python写mapper测试时出现urllib.error.HTTPError: HTTP Error 404: Not Found

最新推荐文章于 2024-08-21 14:38:09 发布

学战到底

最新推荐文章于 2024-08-21 14:38:09 发布

阅读量1.6w

点赞数

分类专栏：大数据 hadoop python 文章标签： hadoop mapreduce python

本文链接：https://blog.csdn.net/SHENNONGZHAIZHU/article/details/51387814

版权

在使用Python进行Hadoop mapreduce测试时遇到HTTPError 404错误，问题出在URL请求上。原本使用'www.bing.com/images/asvnc'作为搜索接口，但在mapper中改为'cn.bing.com/images/async'后问题得到解决。原因可能是直接访问'www.bing.com'时被重定向到了'cn.bing.com'。

摘要由CSDN通过智能技术生成

hadoop@ub1401:~/python/pythonfile$ cat keyword.txt
sheep	2
dog,3
firework 3
hadoop@ub1401:~/python/pythonfile$ cat keyword.txt | ./mappertest1-1.py
Traceback (most recent call last):
  File "./mappertest1-1.py", line 58, in <module>
    response = urllib.request.urlopen('https://www.bing.com/images/asvnc?q=' + urllib.parse.quote_plus(keyword) + '&async=content&first=' + str(current) + '&adlt=' + adlt)
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 469, in open
    response = meth(req, response)
  File "/usr/lib/python3.4/urllib/request.py", line 579, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.4/urllib/request.py", line 501, in error
    result = sel