如何通过爬虫获取全球公交站点数据（Azure Maps API+Moovit)

最近做的一个实验需要全球公交站点的数据，查阅了一下官方文档，由于本人没有信用卡不能注册Azure，把思路写下，如果谁照着做成功的话，希望可以发份数据给我。

https://docs.microsoft.com/bs-latn-ba/rest/api/maps/mobility/getmetroareainfopreview

上面是Azure Maps API的官方教程链接，可以看到入参如下图：

官方提供参考的URL:GET https://atlas.microsoft.com/mobility/metroArea/info/json?api-version=1.0&query=121&detailType=agencies

query用的MetroId，首先这个MetroId不太好弄到列表，不过Moovit和Azure合作了，目测用的相同的编码。

下面是Moovit的官网，分地区，遍历一遍然后就能找到所有MetroId

以第一个Albania为例:

可以知道Tirana对应的超链接，Tirana的MetroId就是5771。

Mobility - Get Metro Area Info Preview

这个返回的JSON如下：

{ "metroName": "NYC-NJ", "agencies": [ { "agencyId": "121---10368", "agencyKey": "LIBUS", "agencyName": "NICE bus", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---10369", "agencyKey": "GOOGLESTAM_TRANSIT", "agencyName": "CT Transit- Stamford", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---10370", "agencyKey": "0", "agencyName": "Norwalk Transit District", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---516", "agencyKey": "NJB", "agencyName": "NJ Transit", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1175189", "agencyKey": "42700", "agencyName": "Statue Cruises", "transitType": { "transitType": "Ferry", "captionOverride": "Ferry" } }, { "agencyId": "121---1058901", "agencyKey": "SCT", "agencyName": "Suffolk County Transit", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1175188", "agencyKey": "42705", "agencyName": "Seastreak", "transitType": { "transitType": "Ferry", "captionOverride": "Ferry" } }, { "agencyId": "121---1217492", "agencyKey": "1", "agencyName": "NYC Ferry Shuttle bus", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---726988", "agencyKey": "LI", "agencyName": "LIRR", "transitType": { "transitType": "Rail", "captionOverride": "Rail" } }, { "agencyId": "121---1217493", "agencyKey": "1", "agencyName": "NYC Ferry", "transitType": { "transitType": "Ferry", "captionOverride": "Ferry" } }, { "agencyId": "121---855111", "agencyKey": "MTA NYCT", "agencyName": "MTA Subway", "transitType": { "transitType": "Subway", "captionOverride": "Subway" } }, { "agencyId": "121---855110", "agencyKey": "MTA NYCT", "agencyName": "Staten Island Railway", "transitType": { "transitType": "Rail", "captionOverride": "Rail" } }, { "agencyId": "121---5844", "agencyKey": "151", "agencyName": "PATH", "transitType": { "transitType": "Rail", "captionOverride": "Rail" } }, { "agencyId": "121---965398", "agencyKey": "631", "agencyName": "Port Authority of NY and NJ", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---865943", "agencyKey": "5942", "agencyName": "Suffolk County Transit", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1020565", "agencyKey": "24415", "agencyName": "Staten Island Ferry", "transitType": { "transitType": "Ferry", "captionOverride": "Ferry" } }, { "agencyId": "121---865942", "agencyKey": "14639", "agencyName": "Long Beach NY", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1358024", "agencyKey": "72", "agencyName": "Ulster County Area Transit", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1325321", "agencyKey": "58939", "agencyName": "Hampton Jitney", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1440628", "agencyKey": "63882", "agencyName": "Housatonic Area Regional Transit", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---775148", "agencyKey": "119", "agencyName": "AirTrain", "transitType": { "transitType": "Tram", "captionOverride": "Light Rail" } }, { "agencyId": "121---427", "agencyKey": "1", "agencyName": "Metro-North Railroad", "transitType": { "transitType": "Rail", "captionOverride": "Rail" } }, { "agencyId": "121---856508", "agencyKey": "5705", "agencyName": "Roosevelt Island Cable Car", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1460323", "agencyKey": "65856", "agencyName": "Downtown Alliance", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---856507", "agencyKey": "5705", "agencyName": "Roosevelt Island Cable Car", "transitType": { "transitType": "CableCar", "captionOverride": "Cable car" } }, { "agencyId": "121---1227748", "agencyKey": "SLE", "agencyName": "Shore Line East", "transitType": { "transitType": "Rail", "captionOverride": "Rail" } }, { "agencyId": "121---856506", "agencyKey": "5703", "agencyName": "NY Waterway", "transitType": { "transitType": "Ferry", "captionOverride": "Ferry" } }, { "agencyId": "121---1367715", "agencyKey": "67", "agencyName": "Dutchess County Public Transit", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---10169", "agencyKey": "NJT", "agencyName": "NJ Transit Rail", "transitType": { "transitType": "Rail", "captionOverride": "Rail" } }, { "agencyId": "121---10170", "agencyKey": "NJT", "agencyName": "NJ Transit Rail", "transitType": { "transitType": "Tram", "captionOverride": "Light Rail" } }, { "agencyId": "121---857463", "agencyKey": "MTA NYCT", "agencyName": "MTA Bus", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---11515", "agencyKey": "WCDOT", "agencyName": "Bee-Line Bus", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } }, { "agencyId": "121---1177901", "agencyKey": "0", "agencyName": "LANTA", "transitType": { "transitType": "Bus", "captionOverride": "Bus" } } ] }

以第一个为例：根据第一个的agencyId:可以构造https://moovitapp.com/index/en/public_transit-lines-NYCNJ-121-10368

我们可以看到每条路线对应一个链接，以第一个为例121-10368-587966-0：121是MetroId，10368是agencyId，587966是LineGroupId,我们需要的就是LineGroupId.

链接到 https://docs.microsoft.com/bs-latn-ba/rest/api/maps/mobility/gettransitlineinfopreview

示例：GET https://atlas.microsoft.com/mobility/transit/line/json?api-version=1.0&metroId=121&query=121---373227&detailType=[ "stops" ]

121是MetroId,373227是线路的LineGroupId，返回的是带左边的站点数据JSON

示例如下：

{ "lineGroup": { "lineGroupId": "121---373227", "agencyId": "121---857463", "agencyName": "MTA Bus", "lineNumber": "B25", "caption1": "Dwntn Bklyn & DUMBO - Broadway Junction", "caption2": "B25 Dwntn Bklyn & DUMBO - Broadway Junction", "color": "6CBE45", "transitType": "Bus" }, "lines": [ { "lineId": "121---1227940", "lineGroupId": "121---373227", "direction": "forward", "agencyId": "121---857463", "agencyName": "MTA Bus", "lineNumber": "B25", "lineDestination": "Broadway Jct Alabama Av Via Fulton", "mostFrequentPatternId": "121---1581548", "transitType": "Bus" }, { "lineId": "121---1227941", "lineGroupId": "121---373227", "direction": "backward", "agencyId": "121---857463", "agencyName": "MTA Bus", "lineNumber": "B25", "lineDestination": "Downtown Bklyn Front St Via Fulton", "mostFrequentPatternId": "121---1701359", "transitType": "Bus" } ], "stops": [ { "stopId": "121---14011984", "stopKey": "303017", "stopName": "Fulton St/Ashland Pl", "position": { "latitude": 40.687781, "longitude": -73.978221 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013637", "stopKey": "307259", "stopName": "Fulton St/Thomas S Boyland St", "position": { "latitude": 40.678336, "longitude": -73.913138 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013639", "stopKey": "307263", "stopName": "Fulton St/Van Sinderen Av", "position": { "latitude": 40.677816, "longitude": -73.903542 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013638", "stopKey": "307260", "stopName": "Fulton St/Rockaway Av", "position": { "latitude": 40.678189, "longitude": -73.910359 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013633", "stopKey": "307255", "stopName": "Fulton St/Ralph Av", "position": { "latitude": 40.678762, "longitude": -73.92125 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013632", "stopKey": "307254", "stopName": "Fulton St/Buffalo Av", "position": { "latitude": 40.67894, "longitude": -73.924108 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---31066255", "stopKey": "901211", "stopName": "East New York Av/Alabama Av", "position": { "latitude": 40.676719, "longitude": -73.900118 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013635", "stopKey": "307257", "stopName": "Fulton St/Saratoga Av", "position": { "latitude": 40.678489, "longitude": -73.915892 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013634", "stopKey": "307256", "stopName": "Fulton St/Howard Av", "position": { "latitude": 40.678615, "longitude": -73.918564 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011701", "stopKey": "302376", "stopName": "Fulton St/Grand Av", "position": { "latitude": 40.682362, "longitude": -73.961824 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011700", "stopKey": "302375", "stopName": "Fulton St/St James Pl", "position": { "latitude": 40.682662, "longitude": -73.9633 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14193658", "stopName": "Front St/York St", "position": { "latitude": 40.702464, "longitude": -73.991858 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011703", "stopKey": "302378", "stopName": "Fulton St/Franklin Av", "position": { "latitude": 40.680982, "longitude": -73.955289 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013751", "stopKey": "307631", "stopName": "Fulton St/Eastern Py", "position": { "latitude": 40.678066, "longitude": -73.908188 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011702", "stopKey": "302377", "stopName": "Fulton St/Classon Av", "position": { "latitude": 40.681579, "longitude": -73.958134 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011697", "stopKey": "302371", "stopName": "Fulton St/Clermont Av", "position": { "latitude": 40.683904, "longitude": -73.968617 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011696", "stopKey": "302370", "stopName": "Fulton St/Carlton Av", "position": { "latitude": 40.68478, "longitude": -73.970798 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011699", "stopKey": "302374", "stopName": "Fulton St/Washington Av", "position": { "latitude": 40.682882, "longitude": -73.964338 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011698", "stopKey": "302372", "stopName": "Fulton St/Clinton Av", "position": { "latitude": 40.683332, "longitude": -73.966593 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013629", "stopKey": "307251", "stopName": "Fulton St/Troy Av", "position": { "latitude": 40.679501, "longitude": -73.934459 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011964", "stopKey": "302948", "stopName": "Fulton St/Flatbush Av Ext", "position": { "latitude": 40.688976, "longitude": -73.981278 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011708", "stopKey": "302383", "stopName": "Fulton St/Kingston Av", "position": { "latitude": 40.67988, "longitude": -73.941381 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013631", "stopKey": "307253", "stopName": "Fulton St/Utica Av", "position": { "latitude": 40.67919, "longitude": -73.928636 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013630", "stopKey": "307252", "stopName": "Fulton St/Schenectady Av", "position": { "latitude": 40.679398, "longitude": -73.932497 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013689", "stopKey": "307491", "stopName": "Fulton St/Hoyt St", "position": { "latitude": 40.69069, "longitude": -73.985547 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011705", "stopKey": "302380", "stopName": "Fulton St/Nostrand Av", "position": { "latitude": 40.680301, "longitude": -73.949161 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013881", "stopKey": "307926", "stopName": "Fulton St/Albany Av", "position": { "latitude": 40.679664, "longitude": -73.937573 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011704", "stopKey": "302379", "stopName": "Fulton St/Bedford Av", "position": { "latitude": 40.6805, "longitude": -73.952805 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011963", "stopKey": "302947", "stopName": "Fulton St/Bond St", "position": { "latitude": 40.689709, "longitude": -73.983099 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011707", "stopKey": "302382", "stopName": "Fulton St/Brooklyn Av", "position": { "latitude": 40.680033, "longitude": -73.944167 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011962", "stopKey": "302945", "stopName": "Fulton St/Smith St", "position": { "latitude": 40.69148, "longitude": -73.987534 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011706", "stopKey": "302381", "stopName": "Fulton St/Marcy Av", "position": { "latitude": 40.680116, "longitude": -73.945667 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14406637", "stopName": "Tillary St/Cadman Pz E", "position": { "latitude": 40.696046, "longitude": -73.990383 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011693", "stopKey": "302359", "stopName": "Cadman Plz W/Cadman Tower", "position": { "latitude": 40.698348, "longitude": -73.991565 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011692", "stopKey": "302358", "stopName": "Cadman Pz W/Clark St", "position": { "latitude": 40.696696, "longitude": -73.991464 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013676", "stopKey": "307460", "stopName": "Adams St/Bklyn Supreme Crt", "position": { "latitude": 40.69256, "longitude": -73.989131 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011695", "stopKey": "302369", "stopName": "Fulton St/S Portland Av", "position": { "latitude": 40.686059, "longitude": -73.974034 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14013934", "stopKey": "308025", "stopName": "Water St/Main St", "position": { "latitude": 40.703318, "longitude": -73.991102 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011694", "stopKey": "302368", "stopName": "Fulton St/Lafayette Av", "position": { "latitude": 40.687146, "longitude": -73.97669 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011689", "stopKey": "302355", "stopName": "Old Fulton St/Elizabeth Pl", "position": { "latitude": 40.702666, "longitude": -73.993952 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011691", "stopKey": "302357", "stopName": "Cadman Pz W/Middagh St", "position": { "latitude": 40.699591, "longitude": -73.99129 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" }, { "stopId": "121---14011690", "stopKey": "302356", "stopName": "Old Fulton St/Henry St", "position": { "latitude": 40.701021, "longitude": -73.991449 }, "mainTransitType": "Bus", "mainAgencyId": "121---857463", "mainAgencyName": "MTA Bus" } ] }