如何用python爬取数据_如何使用Python抓取实时流数据？

I was trying to scrape the number of flights for this webpage https://www.flightradar24.com/56.16,-49.51

The number is highlighted in the picture below:

The number is updated every 8 seconds.

This is what I tried with BeautifulSoup:

import requests

from bs4 import BeautifulSoup

import time

r=requests.get("https://www.flightradar24.com/56.16,-49.51")

c=r.content

soup=BeautifulSoup(c,"html.parser")

value=soup.find_all("span",{"class":"choiceValue"})

print(value)

But that always returns 0:

[0]

View source also shows 0, so I understand why BeautifulSoup returns 0 too.

Anyone know any other method to get the current value?

解决方案

The problem with your approach is that the page first loads a view, then performs regular requests to refresh the page. If you look at the network tab in the developer console in Chrome (for example), you'll see the requests to https://data-live.flightradar24.com/zones/fcgi/feed.js?bounds=59.09,52.64,-58.77,-47.71&faa=1&mlat=1&flarm=1&adsb=1&gnd=1&air=1&vehicles=1&estimated=1&maxage=7200&gliders=1&stats=1

The response is regular json:

{

"full_count": 11879,

"version": 4,

"afefdca": [

"A86AB5",

56.4288,

-56.0721,

233,

38000,

420,

"0000",

"T-F5M",

"B763",

"N641UA",

1473852497,

"LHR",

"ORD",

"UA929",

0,

0,

"UAL929",

0

],

...

"aff19d9": [

"A12F78",

56.3235,

-49.3597,

251,

36000,

436,

"0000",

"F-EST",

"B752",

"N176AA",

1473852497,

"DUB",

"JFK",

"AA291",

0,

0,

"AAL291",

0

],

"stats": {

"total": {

"ads-b": 8521,

"mlat": 2045,

"faa": 598,

"flarm": 152,

"estimated": 464

},

"visible": {

"ads-b": 0,

"mlat": 0,

"faa": 6,

"flarm": 0,

"estimated": 3

}

}

}

I'm not sure if this API is protected in any way, but it seems like I can access it without any issues using curl.

More info: