0.参考
https://developer.mozilla.org/zh-CN/docs/Web/CSS/Pseudo-elements
https://developer.mozilla.org/zh-CN/docs/Web/CSS/::before
https://developer.mozilla.org/en-US/docs/Web/API/Window/getComputedStyle
0.1 Use with pseudo-elements
getComputedStyle can pull style info from pseudo-elements (for example, ::after
, ::before
, ::marker
, ::line-marker
—see spec here).
<style> h3::after { content: ' rocks!'; } </style> <h3>generated content</h3> <script> var h3 = document.querySelector('h3'); var result = getComputedStyle(h3, ':after').content; console.log('the generated content is: ', result); // returns ' rocks!' </script>
0.2 手动设置伪元素信息
0.3 理论上应该是可以直接通过 js 读取伪元素 content 的。。。然而这里读取不到 ::before 的 content
1.初步抓取
1.1不加载js VS 加载js
1.2 requests仅见一张table
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
In [1]: import requests ...: from scrapy import Selector ...: ...: ...: url = 'https://car.autohome.com.cn/config/series/3170.html' ...: s = requests.Session() ...: s.verify = False ...: s.headers = { ...: 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36' ...: } ...: ...: r = s.get(url) ...: sel = Selector(text=r.text) ...: ...: sel.css('table') ...: Out[1]: [<Selector xpath='descendant-or-self::table' data='<table class="t1 txtCen" width="100%">\n '>]
1.3 selenium可见所有table
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
In [3]: from scrapy import Selector ...: from selenium import webdriver ...: ...: ...: url = 'https://car.autohome.com.cn/config/series/3170.html' ...: dr = webdriver.Chrome() ...: dr.get(url) ...: sel = Selector(text=dr.page_source) ...: sel.css('table') ...: DevTools listening on ws://127.0.0.1:12968/devtools/browser/2e9adb31-7510-421f-ac13-835350af144e Out[3]: [<Selector xpath='descendant-or-self::table' data='<table class="t1 txtCen" width="100%">\n '>, <Selector xpath='descendant-or-self::table' data='<table cellspacing="0" cellpadding="0" c'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_side" cellspacing="0" cel'>, <Selector xpath='descendant-or-self::table' data='<table cellspacing="0" cellpadding="0" c'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_0" cellspacing="0" cellpa'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_1" cellspacing="0" cellpa'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_2" cellspacing="0" cellpa'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_3" cellspacing="0" cellpa'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_4" cellspacing="0" cellpa'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_5" cellspacing="0" cellpa'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_100" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_101" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_102" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_103" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_104" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_105" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_106" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_107" cellspacing="0" cell'>, <Selector xpath='descendant-or-self::table' data='<table id="tab_108" cellspacing="0" cell'>]
1.4 提取table信息
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
In [4]: for t in sel.css('table#tab_0'): ...: print('@'*10) ...: for row in t.xpath('.//tr'): ...: print('-'*100) ...: for col in row.xpath('.//th|.//td'): ...: print(''.join(col.xpath('.//node()/text()').extract()), end='\t') ...: print() ...: @@@@@@@@@@ ---------------------------------------------------------------------------------------------------- 基本参数 ---------------------------------------------------------------------------------------------------- 厂 - - - - - - - - - - - - ---------------------------------------------------------------------------------------------------- 级别 ---------------------------------------------------------------------------------------------------- 能源类型 汽油 汽油 汽油 汽油 汽油 汽油 汽油 汽油 汽油 汽油 汽油 汽油 ---------------------------------------------------------------------------------------------------- 上市 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 ---------------------------------------------------------------------------------------------------- (kW) 110 110 110 110 140 140 110 110 110 110 140 140 ---------------------------------------------------------------------------------------------------- (N·m) 250 250 250 250 320 320 250 250 250 250 320 320 ---------------------------------------------------------------------------------------------------- 发动机 1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 2.0T 190马力 L4 2.0T 190马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 2.0T 190马力 L4 2.0T 190马力 L4 ---------------------------------------------------------------------------------------------------- 变速箱 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 7挡双离合 ---------------------------------------------------------------------------------------------------- 长*宽*高(mm) 4312*1785*1426 4321*1785*1426 4312*1785*1426 4321*1785*1426 4312*1785*1426 4321*1785*1426 4457*1796*1417 4462*1796*1417 4457*1796*1417 4462*1796*1417 4457*1796*1417 4462*1796*1417 ---------------------------------------------------------------------------------------------------- 车身结构 5门5座两厢车 5门5座两厢车 5门5座两厢车 5门5座两厢车 5门5座两厢车 5门5座两厢车 4门5座三厢车 4门5座三厢车 4门5座三厢车 4门5座三厢车 4门5座三厢车 4门5座三厢车 ---------------------------------------------------------------------------------------------------- 最高车速(km/h) 215 215 215 215 230 230 215 215 215 215 230 230 ---------------------------------------------------------------------------------------------------- 官方0-100km/h加速(s) 8.4 8.4 8.4 8.4 7.4 7.4 8.4 8.4 8.4 8.4 7.4 7.4 ---------------------------------------------------------------------------------------------------- 0-100km/h加速(s) - - - - - - - - - - - - ---------------------------------------------------------------------------------------------------- 100-0km/h制动(m) - - - - - - - - - - - - ---------------------------------------------------------------------------------------------------- 工信部(L/100km) 5.7 5.7 5.7 5.7 6.2 6.2 5.6 5.6 5.6 5.6 6.1 6.1 ---------------------------------------------------------------------------------------------------- (L/100km) - - - - - - - - - - - - ---------------------------------------------------------------------------------------------------- 整车 三10公里 三10公里 三10公里 三10公里 三10公里 三10公里 三10公里 三10公里 三10公里 三10公里 三10公里 三10公里
1.5 对比可见提取table信息不全
2.问题分析
2.1 开发者工具显示 '商' 为CSS伪元素的content属性
2.2 使用正常浏览器硬性刷新未见CSS文件存在有效对应信息
2.3 检查发现class name存在规律:hs_kw数字_configyk
.hs_kw22_configyk::before {
content: "商";
}
.hs_kw44_configyk::before {
content: "一汽";
}
2.4 selector提取存在多个class name后缀,而且无法提取到其中文字
In [19]: sel = Selector(text=dr.page_source) In [20]: sel.css('span[class*="hs_kw"]').extract() Out[20]: ['<span class="hs_kw34_configyk"></span>', '<span class="hs_kw66_configyk"></span>', ... '<span class="hs_kw42_optionLA"></span>', ]
2.5 使用正常浏览器全局搜索 hs_kw 和 hs_kw22_configyk 和 configyk ,仅发现关联文件 3170.html
2.6 Sources面板对 3170.html 进行美化排版,复制到 notepad++ 搜索关键字定位到 js 代码段
继续搜索
2.7 复制其中一个代码片段到 Sources 面板作为 snippet,移除头尾 <script> </script>,添加断点
2.8 运行代码片段,发现可疑变量
2.9 根据关键字定位到相关函数(运气好的话),添加log语句,重新运行输出中间变量
3 完整代码
http://www.cnblogs.com/my8100/p/9028359.html
4 运行结果
5.彩蛋(前端都这么直白?!)