python登录网页_使用python请求登录网站

I'm trying to login to https://www.voxbeam.com/login using requests to scrape data. I'm a python beginner and I have done mostly tutorials, and some web scraping on my own with BeautifulSoup.

Looking at the HTML:

I understand I should be using the method post, and sending userName and password

I'm trying this:

import requests

import webbrowser

url = "https://www.voxbeam.com/login"

login = {'userName': 'xxxxxxxxx',

'password': 'yyyyyyyyy'}

print("Original URL:", url)

r = requests.post(url, data=login)

print("\nNew URL", r.url)

print("Status Code:", r.status_code)

print("History:", r.history)

print("\nRedirection:")

for i in r.history:

print(i.status_code, i.url)

# Open r in the browser to check if I logged in

new = 2 # open in a new tab, if possible

webbrowser.open(r.url, new=new)

I’m expecting, after a successful login to get in r the URL to the dashboard, so I can begin scraping the data I need.

When I run the code with the authentication information in place of xxxxxx and yyyyyy, I get the following output:

Original URL: https://www.voxbeam.com/login

New URL https://www.voxbeam.com/login

Status Code: 200

History: []

Redirection:

Process finished with exit code 0

I get in the browser a new tab with www.voxbeam.com/login

Is there something wrong in the code?

Am I missing something in the HTML?

It’s ok to expect to get the dashboard URL in r, or to be redirected and trying to open the URL in a browser tab to check visually the response, or I should be doing things in a different way?

I been reading many similar questions here for a couple of days, but it seems every website authentication process is a little bit different, and I checked http://docs.python-requests.org/en/latest/user/authentication/ which describes other methods, but I haven’t found anything in the HTML that would suggest I should be using one of those instead of post

I tried too

r = requests.get(url, auth=('xxxxxxxx', 'yyyyyyyy'))

but it doesn’t seem to work either.

解决方案

As said above, you should send values of all fields of form. Those can be find in the Web inspector of browser. This form send 2 addition hidden values:

url = "https://www.voxbeam.com//login"

data = {'userName':'xxxxxxxxx','password':'yyyyyyyyy','challenge':'zzzzzzzzz','hash':''}

# note that in email have encoded '@' like uuuuuuu%40gmail.com

session = requests.Session()

r = session.post(url, headers=headers, data=data)

Also, many sites have protection from a bot like hidden form fields, js, send encoded values, etc. As variants you could:

1) Use a cookies from manual login:

url = "https://www.voxbeam.com"

headers = {'user-agent': "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36"}

cookies = {'PHPSESSID':'zzzzzzzzzzzzzzz', 'loggedIn':'yes'}

s = requests.Session()

r = s.post(url, headers=headers, cookies=cookies)

2) Use module Selenium:

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

url = "https://www.voxbeam.com//login"

driver = webdriver.Firefox()

driver.get(url)

u = driver.find_element_by_name('userName')

u.send_keys('xxxxxxxxx')

p = driver.find_element_by_name('password')

p.send_keys('yyyyyyyyy')

p.send_keys(Keys.RETURN)