D5 使用requests的post请求，爬取GitHub登录界面

Github登录界面: https://github.com/login

登录时的Post请求信息

1. general
   a. Request URL: https://github.com/session
   b. Request Method: POST
2. Respond Headers
3. Request Headers
   a. Cookie: has_recent_activity=1; logged_in=no; _ga=GA1.2.1904824943.1561115204; _gat=1; tz=Asia%2FShanghai; _octo=GH1.1.1268982742.1561115206; _gh_sess=M3pZdDFWS1dZcjJJWUNxMzliVFRDZlZWY3d2ZnhIS2s3SlVMcDJ1Um9kYlZzMGw5WE9jbi9DVmovS1ZUZU5leDFOVnAwL21idWR2V3YzQjRNZVpsWk9jdjdGVWl6Zk1jV1BUbzJhd2wyRktlWEVua3J1U0VFMDhacGQ4SHNQamhTTlBBcjZ2aDlORUhQald2amxkRUd3aDV0QU4xWE42bTg3OW0rOWo2UHIzcUpySkM5UDBoYkpObHJldm51ZFQwVGxOdWJCOHpwRnF0ZXd0ZEpYYWt4Y1pONVhCZ2p5Z2dUUGV5b0o5S0ZubEkvYlpaSDhmTmtCSlFidTZNZUVKZUQ2bVdHZnQ5b0dvMXdoYUR5ODJHelE9PS0tVERnaW4xeVBuNlFNMDY2U21GWHpRZz09–9e9422017c6e227b2b9efd4d80955bc5d4520f9e
   b. User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
   c. Referer: https://github.com/login
4. Form data
   a. commit: Sign in
   b. utf8: ✓
   c. authenticity_token: f259J04r0E+5Whh4BN0CsRi7vqmWss227qLe7fHappEkSIFk/qyulp0t7bNJWZ7IMwxBI7hLW7udy2/gGW0DXg==
   d. login:
   e. password:
   f. webauthn-support: supported

分析：

1. 需要到refer界面找authenticity_token
2. 需要设置Cookie

实验代码

'''
URL：https://github.com/login
Method：GET

request headers:
'Cookie':'_ga=GA1.2.1851509965.1553350679; _octo=GH1.1.1609141208.1553350744; _device_id=896f41142a01902ae6fc03b373aaba6f; logged_in=no; has_recent_activity=1; _gat=1; tz=Asia%2FShanghai; _gh_sess=QXFSelVJNHlVTVdOM2w5SkUyempSMUlmVHJHcHJVaGJRVzViaW9BN3dQYzZneXpXRm1RSi9STEI5N2hEM3pQZ1ZJcVYxUW1La0xzclZzdWtraGJkdHZuUEdnZE8xbXVhYzlsMHBWYnZuUjljdkVKQmVtUlZOdjh5dUd1LzV0empMdjlTbm03L1c3ZzZ5enNzWlRRenNHRko5d1FQcGxKUGMyT0YzNWVJWHVrVGFRMytudklOTkVtdHdzR2FubEd0WWRQV3B4eGJtYjVwQnJYVkdTY0p6SFovV1Z0RGRUVWNQM20xdU9KSGQ0WjlmSHg2S0cybnBxV2U5QkJYcHdYb25ndkRrbHhlN1NCMWRLSlhQL1lBN2tVb0dBK2J2VDd4bS9kQWNtZDFrc2lEYlNYa1Y3Z080UndtR1VyTXlCZVFleDZESjFMNWVqS24ySVlVQmYzMVBGM2ZBRFpsZDdjOFR2dXFxdjZqK0pzPS0tUG1XalpoNzhSVklrMzQxVnNidFFIdz09--8d6ea59ffb357864b69489d61a43727bfa0ef81f'

'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36'
'''
import requests
import re
url = 'https://github.com/login'
headers = {
'Cookie':'_ga=GA1.2.1851509965.1553350679; _octo=GH1.1.1609141208.1553350744; _device_id=896f41142a01902ae6fc03b373aaba6f; logged_in=no; has_recent_activity=1; _gat=1; tz=Asia%2FShanghai; _gh_sess=QXFSelVJNHlVTVdOM2w5SkUyempSMUlmVHJHcHJVaGJRVzViaW9BN3dQYzZneXpXRm1RSi9STEI5N2hEM3pQZ1ZJcVYxUW1La0xzclZzdWtraGJkdHZuUEdnZE8xbXVhYzlsMHBWYnZuUjljdkVKQmVtUlZOdjh5dUd1LzV0empMdjlTbm03L1c3ZzZ5enNzWlRRenNHRko5d1FQcGxKUGMyT0YzNWVJWHVrVGFRMytudklOTkVtdHdzR2FubEd0WWRQV3B4eGJtYjVwQnJYVkdTY0p6SFovV1Z0RGRUVWNQM20xdU9KSGQ0WjlmSHg2S0cybnBxV2U5QkJYcHdYb25ndkRrbHhlN1NCMWRLSlhQL1lBN2tVb0dBK2J2VDd4bS9kQWNtZDFrc2lEYlNYa1Y3Z080UndtR1VyTXlCZVFleDZESjFMNWVqS24ySVlVQmYzMVBGM2ZBRFpsZDdjOFR2dXFxdjZqK0pzPS0tUG1XalpoNzhSVklrMzQxVnNidFFIdz09--8d6ea59ffb357864b69489d61a43727bfa0ef81f',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36'
}
res = requests.get(url,headers=headers)
token = re.findall('authenticity_token" value="(.*?)" ',res.text,re.S)[0]
cookies = res.cookies.get_dict()
print(res.headers)

'''
URL:https://github.com/session
method = post
Referer : https://github.com/login
User-Agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36   

FROM DATA
commit: Sign in
utf8: ✓
authenticity_token: dlRTo8UcMMhkDyspzN5QdJYGGJAxMUA9Muwc+xcObRJtgO6npkBoXENZZXZHcl15YqlcXqFZlqMkWDnTmekSLA==
login: 
password: 
webauthn-support: supported
'''

url2 = 'https://github.com/session'
headers2 ={
    'Referer' : 'https://github.com/login',
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36'
}
from_data = {
    'commit':'Sign in',
    'utf8': '✓',
    'authenticity_token':'dlRTo8UcMMhkDyspzN5QdJYGGJAxMUA9Muwc+xcObRJtgO6npkBoXENZZXZHcl15YqlcXqFZlqMkWDnTmekSLA==',
    'login':'845371222@qq.com',
    'password':'zhuqiong6566',
    'webauthn-support': 'supported'
}

res = requests.post(url2,headers=headers2,data=from_data,cookies=cookies)
print(res.status_code)

with open('github.html','w',encoding='utf-8') as f:
    f.write(res.text)

实验结果