爬虫第一天

爬虫步骤：

1. 导入requests

import requests

2. url和请求头

url = 'http://example.webscraping.com/places/default/user/login'	#要请求的页面地址
headers = {'User-Agent':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0'}

3. 暂存数据

req = requests.get(url=url).content.decode()
#下面这个可以不要的
with open('example.html','w',encoding='utf-8') as f:
	f.write(req)

4. 数据过滤

req_key = re.findall(r'name="_formkey" type="hidden" value="(.*?)"',req)[0]

5. 数据保存

with open('date.txt’,'w',encoding='utf-8') as f:
	f.write(req_key)

补充：

能够使用with的条件：具有__enter__和__exit__方法

class WithObject(object):
    def __enter__(self):
        pass
    def __exit__(self, exc_type, exc_val, exc_tb):
        pass

翻译小程序：

import requests
import sys
import json
word = ' '.join(sys.argv[1:])

req_url = 'http://fanyi.youdao.com/translate'
Form_Date = {}
Form_Date['i'] = word
Form_Date['doctype'] = 'json'

response = requests.post(req_url,data=Form_Date)
html = response.content.decode()
html = json.loads(html)
print('❤ '*len(word))
print(word)
print('❤ '*len(word))
print(html['translateResult'][0][0]['tgt'])
print('❤ '*len(word))

结果：