I'm currently attempting to scrape data from a specific page on LinkedIn. I have a script that is able to log into LinkedIn, but I run into a snag when I try to access the page containing the data. When I call requests.get(data_url), I end up with the html for the LinkedIn loading screen that is displayed before LinkedIn loads the actual page content. Is there a way to make requests wait for LinkedIn to display the site data before actually scraping the html data? I essentially need to let the page fully render before I can 'get' the contents. My current script is below.
import requests
from bs4 import BeautifulSoup
client = requests.Session()
HOMEPAGE_URL = 'https://www.linkedin.com'
LOGIN_URL = 'https://www.linkedin.com/uas/login-submit'
html = client.get(HOMEPAGE_URL).content
soup = BeautifulSo