这也敢爬，你离牢饭不远了，爬虫逆向实战案例

重要的事情放在最前面，请您一定看仔细：

本文以学习和交流爬虫技术，尤其是JS逆向技术为目的。大家重点学习其中的思路和过程，请不要模仿爬取文中举例的网站，如果要爬取，请爬取不要超过3页，否则后果自负。

Crawl It!

爬虫技术万年第一步——分析网页结构和数据加载方式：

结果一打开网页是这样的，看来网站维护人员也过节去了。那就假期回来再继续写吧。。

本文暂停于 2021.2.11 上午 10:25

春节假期一眨眼就过完了，我又回来“搞事情”了！

• 打开url，显示数据页面。

• 通过Chrome浏览器的Network选项卡可以得知，翻页是由ajax实现的。

响应数据如下：

"95780ba0943730051dccb5fe3918f9fe1b6f2130681f99d5620c5497aa480f133ef544506ed439f
4bddb1ca63d7c066964c776ea098085c5cd9112e7cbcf533c5823e8b4bb0d542e5e2591fb6a46134
7bbb39b1ed0de45bce0d2512d6a0834f5c1d80d738d51820fb78c1aa2481fd43c825924bc101ad71
917930074d902df9764fe7f29f5a6cb917ea3e617ca76a348186f29d547a8b2bd4044cc9d9036fe2
b22e953f4aeea13705d9d1583300371749d5a882f6c50176739ca1171a50db15aae8c6550591dd8a
e62ad68dc69935e3fcf57379cc625c94acabb2effc4d5505f214f94a2a0e50110d9647d20bb189f4
4acad6184304fa46dff0048b0f58dd1b5d8a945794bbe9e66898b8e363192a79b84cde408f90a02c
9c9b85df2b6d5de973fc00874b223acd2da7df8642ce1e1e64aa6ff550e7b96db782d348f8209d5f
d59b9e26a2aa7ee9c4621cfc5a6d757d91e3862469c39b3b5152054a50667e0f8dfad00fafe55917
e938398c09fc682becfd9aa54f68ce6fd7e023cc7db416727a1572389c798d85f1cd348ff918a4d0
9e67c41a719ac57b6c06a05e954b84a6f0303192b740d3d5ed9c9cc4fe8f0c4de71aa12edc33069d
55f311894109c8c686745fcd22ed2cb33f5deeb6ca9af98c0097e86b450e98a9b1011587011014a3
5b63a3d55c17d92849f56d5e20e76847238c37620d2098b96c153accc9df0df7722868ff86028415
1bf7d7d0aa78ffeb89c882d592994ff09a235180f6f1ff6092fd60ce7eae7c69fc30ba47239f513e
08b9eb3e530cf08647933ea18d2c34c4cb9515b82d4153172311afb9af5d8bdd770d9d9410e75381
cf0f2aa2ca5cb33ccf3276ed0c7dfeda58b7c953bd7225b4395b07e922b75c26c3b5ee302ac3f0ae
40dc31a55ef92afdc7909c7f2e900e1ae8bf92764571c5a5ff27671fbce00c27fc379d64ca61d84f
95558bd2af9b83dd58349922543b8c8e0dc43dead3a102025ce19f4db76f5346b5bbd8784cefa854
51ef22ccd3967f1f6b2eb6f3a9114cc7f26ce107b25985c35ac9a82f11b637fd4a52c2c4f6966f05
3ec48eea3d056268304e58540c3eac82dd70e668e2b24707daeb5d1026b6ad7e7efb638bd51838f7
18c690d0ca4f421246e09432169a0c84ee96adcd4082f5796f474c636ed6d19702e8e5c45421e98a
56d94324160196673363f729ec7cc207f478fbaa865af04ab243cd1a5d3cf54b22393a04aeff51fd
42d33a3d76d11814a540332db0d9f61172e99b3f08f75b081367da26edae0ee7ff7dcab16883ab28
82e48eb95177cdf3d337a1836c949182e7bd5b155aef25c87f399664a367fbe961f1e532295dbce5
0c5a1f7acd891849bae2e6169eb8b9bfbe62e85e740686bc4d3a1ff47cc224507f7be3578751d571
6f006eebdced3c82eabbd500e55b8ff610cef3d947a317f103bf2cbde3fdfc2a0bfdc7e26bdcfee7
9a6967fd66eb01fc84e5c07718747c603e52719d045fdd19ff01f4a00921db26c1761fb744eebf73
262688e1802da1c1a0320196dc721f1a604512589fd23cc01895760881ba4a7af28e5e66b921a14b
85b7f86cd317e691aadf81a8d2b463318058490b19293a35d2fe87ffdecd220cd24fbee8d4073c26
bc61a3a0ad31377db1e660bb5b15c397d15ff820832ffb46f4008ced73bb1133dc957e6e3ec3d829
309addb1aa1e3a431d06b1c8aef6c43ed040f5ca311910bc0d70c34d8aca94d5b7084cfe46fb793d
e87a027f7bfe4d60489d05d3a0e661e9405a768548cdb69bb8a8e52aa0c775684b3e30f18cc059c9
38b451b3376a1736a21df7809791ed8144a893cd831245c64321064f45c67721413018e6bfef3ebe
572cda88723a970ae924cdde1b0a205dbf80b2e5aa55e6baac55b4a30232ee3bad287a40d4916a42
414f6805955ad4cb32bc1c7a4946d77cc9aa4582add2e68976a9a85003acec58f21327c30750e7c1
0615b94a66912b93237e5cc5568a5a53b7214801d3f9903bff3f445e33cc9fda9d19137917abbee4
c08d178095e5956acbb22dcd91639943ace2c24510b7206d92d011f44c055973c7e25c3de3553a5d
ad8457f0b0023e5cfef2f66ed0288b320f37e8313d061b9b75add7c7848ad722e616cad7c7c12359
018c34f723e37afaf21d466654312495a08beb81cb77e2c2bae6d0151fdaa68df623a70618cd6d3c
b66c1475c8eb24829742f10b90bfb9b5794f5eeba7226f7530ba86a69804cf8fbb4c6985c843c0c4
0a3f5bf88bdf55437b7bfcb0bb52f3bae7ba8ee035923b0dcceb144a30f7d2559103d512b4c786c1
710ebba3cedde63e99eee1c5f6ff7991d4b013ba8f2b223bd791baa5e6962a687415b947efb5dea9
07e3dc7d3eb19d9adbc4e9498c5362460ceb945533e3c4c049e93ce9711ca776d54279a01a226998
c5160efd85151618f565c38e9beed091441d2bd1e7677e032d13a25817f051241ca97995188f45cb
be1602636e855f827f35122b6b8cc887f013402b032fc4d1e24939c7cf2c5e686fdb4c4565022ba6
11726a8b73b5c7f8e77502d73b56b58f14239eebe1fcf356b20367e41a7434ce793b738a6d2b7f7d
3a6753a950641bce3fe01428fd777ddadab33fdde61b9fc56c527d89f52ec4cce77fca7ff693ba1d
7e45225eff225fded5e6708e73c20f6fb72b8ce8b080f3f1f055d5b4b624d0b4020a599b9dbebd9e
88c567cd894a94d846022de3298b171556155bec89323dc12a49137f06acb0f3fbad45aa55f94ebc
31458ccb0289dc403540b98e3a36d9b48a975bbb055dccfc57d87a6213f1fc57db70919dab450c3f
21da6daec9d0df7a847fd632a3e9a882b0e14e15565d06c7f437e7e69e9648f1cb9b022d6118d08e
7a0407d15a820b3249bff0ce986c8dc05ca7eca74b6302080a24c898437e1bbc5d380eb93a783710
407fa1d5389c127ab201e2508cdbc63b51a9d5c546d6bcee9f5e0b3a4b92239509884edc05c27aa1
76f64a21f4a7d4bfe997b4a4fb1b95c9f7e1cf1a85c33df25848be1ae83fe4a948095c38d66dbe10
73d8fa6e1bda1fc8a2913d8731d78b1bd6ed8dda5597da4e63d9dadbb8eadb1612198fc88342afd6
dd2442170683a5d55d27dea2475bf57fec14663906dbb6beec6c91a012f909e01626bf9a1a2a452f
c696b52e4edabe499f349cfeff364b28d8bf82aa9a5cfaae80c4b92e22f03c58b3cca43363f1a64e
cfa7c652de4e676b3d18029e0fb606e8ca0e5f2960b16705eeb7d34f48ef3d6f1a6f2e9dd2ce32fb
44af7cfff512488d7bfe6f57f40b5681a18bf2b6a60827b8aa888c81afbcaff7063fc4aeed89b137
5d35f4f7af1ce3cf9c5b3c81c4175651a981f755f422c30713d3e6ca4f3763f95c0f0b1d1f51c32b
769157a7c0a25f050560bb851fbf581a7186d515bd88e100fc91a56b39b86da71e8d028485358e58
31b00d2e653485ee7f404cf8848a681f3fafceb06138e090fc2caf2b03a4978173e1c6e71821ada8
9ceba3302cc16cfaa4b9bf472fa79156772e5ffdab8b44606e258176fa651d13e7388c13ee17d965
4b3e5564c24ccf478015014da5b0dcc8deb213cb0558c185339b22c490a332b4d4f9cf700cf2662d
ec0f815c8418e2573621713717463e4606b5903f66d7bc0581369a1586e84ebd7ba27277aecbd67f
737e35ab87ac78c428588bd7a1a26e4bc2834779d3399e06790bf1cb1ea77f665a449b9e6b3f459d
bd14bbf1534b4f19292f35fbb5942ec505fcf654235d8e1b556843c87b6b868b4e049733e85f551f
78e3587dd6223183f038be2482259638a0650cf6eff6f1bf5383d4774e5b1e5bd20dc839df78ae23
cca22d23d30f2fa1a49307220474a371666908619339f83cc7ae3fcd37a37db640a06970b39db055
a2a8a0d067fe72516b9b9eccc51a69dcd19e961bfe3b131e8130ec2e648d20aafafaceb7f05f03e5
a4e75cd3dadece5d15cdb26c7e6c39a4fcf400490a668e4221f4c0129cfeb46078775392218c65db
37935c172fc5c10f2b0079a195698acd00f736071593604a1e088db94b29319b75f759cc0e6fb3a4
e226a2305dfb7ad0f0f3b9b02b229953e030ac50fe75560f7a98ccfff5a3dd06617349b4fd1ee5ac
db33789b5724e5c3e4441d13a11974b11406bca931d7077b0b2e1e9c4c7a6eb7245fc5779c1cf922
0f64e3bbd4c1fc1cd5000c879700dcb35e5fd77508fa96063de2c7531ae658bf048156b7678b4b41
bf66f00eb6198503671668115bcfcea46ab1a07a7c077d339bb24f65757fe8cfc944cbc1ab7aae39
b19c40619f28d86e58b1f39e52202ff220092ded48323531264e8271129c644ca7e6c4ee03ccbae9
02b990257b1a55daec1b919f33fff441198653032d1026ac1b404de3562c6f851664fd534b04e544
c43dd82ac7ebeaccc7a9a479678c2371665e99815e6a0c60b8b66a21fb0275dd5ee2c26cd69ddff0
b173b40b90e6a602a30c5f0b33e8b933de6f13520af43adf5b87e22ee5af4c307a27b2f788feaefc
668d87af90bf442df137fd26c8e29a6d40c5cb5fbd551752ef0660477013614528661901067c6d64
a95917b9b419180c4d3d3f0a1edb8521d90700ad70a2f4d0717d5bdc24766802ece3be32dc45bfa0
79cd7ff464608926728f9b395f609f53e42c7ba4136abd566d53d0b1819fbef45358df803eadd071
a250c3ca1abd7c2426975dbdfc5dc8369c79e785a4f6c0ed5a7152ec4912df7bf97914abca3f0b4f
616905ed128809a5ead9d9fe57425a16a32fb0ff3e81e409718814eca8226676d3cc8df61b8b0827
c5c54696a5c5c25f697fb842ca316a36a09dfb48b6fc9f5fdc3717affb3477d98259126821e2fedb
292daba634bff50e1e6136dfc83a4441bc8f2ed5350792b6c4668e76f5db7d64b4f40e94bb749471
19881f0077536a67422278bf0cc97614c2f7800ad08fc3e04e5ea3b010a6c16af0a8e16dfb15b780
e6eb284f197ce9d3d370536bb10c1f48762e7842a651875c0d47c530fc9fd5705fad0b0f3a533500
153e810ad751ff15af761011399a162029a77b4fe937edd5b8987ac01e2c89f1642716fe83627124
7f26c983bea9c09ae415a7d367f5a3bfd1d5cf3e3eafd235d0a222869a6d2f3403937a2f6bb0b7eb
6568c743a39ee90bb7b8efd6d1b5e863da2cae9642eedb758d44f2dd749