实习笔记

尚未解决的问题

selenium偶然性错误

selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette

有时运行脚本的时候会出现上面的信息，可能是因为系统中有大量的firefox相关的进程导致的，把这些进程删除之后就可以了。至于为什么突然多出来这么多进程我就不知道了。
杀死相关进程的指令

kill -9 $(ps -ef | grep firefox | awk '{print $ 3}')
kill -9 $(ps -ef | grep firefox | awk '{print $ 2}')

代理

mitmproxy: Error starting proxy server: Address already in use

运行mitmproxy是出现上述错误，原因是端口被占用了。使用netstat -Wplant这个命令可以查看端口的详细信息

同时发现系统中运行了好多浏览器，怀疑是这些浏览器导致8080端口总是被占用。
删除geckodriver进程（命令kill -9 $(ps -ef | grep geckodriver | awk '{print $2}')）

但是mitmproxy还是不能在默认端口打开，必须要使用

mitmproxy -p 3128

------------------------------------------------------------------------------

已解决的问题

Xvfb

Xvfb -ac :7 -screen 0 1280x1024x8
export  DISPLAY=:7

firefox

"selenium.common.exceptions.NoSuchWindowException: Message: Browsing context has been discarded"

一般是网站的问题（网上的答案说是网站的问题）或者是使用的浏览器崩了（比如在win7上的火狐浏览器自动化运行的时候，你的鼠标瞎在浏览器上面动，很有可能影响selenium的鼠标操作，造成上述问题，或者ubuntu中的浏览器崩了，如果是ubuntu中的浏览器崩了，需要杀死firefox的绝大多数线程，只留下下面截图中的线程即可，然后重新运行脚本）

cookie实现免登陆

通过脚本将火狐浏览器运行并成功通过账号登录到爱奇艺平台之后使用如下语句获取cookie

f1 = open('aiqiyi_cookie.txt', 'w')
f1.write(json.dumps(driver.get_cookies()))
f1.clos

调用cookie

driver.get(url)#先加载url

print '1'
driver.refresh()
with open('./cookie.txt', 'r') as f:
    cookies = f.read()
    cookies = json.loads(cookies)

for mem in cookies:
    driver.add_cookie(mem)
driver.refresh()

这样就可以免登陆了，但是还没研究一个cookie文件如何存放多个平台的cookie。目前的效果是一个cookie文件只有一个站点的cookie。

获取元素属性

<span>asdf</span>

想要获取上面的元素中的"asdf"，运行下面的语句

driver.find_element_by_xpath('/html/body/div[4]/div[1]/div[1]/div[1]/div[2]/div/div/div/div/div[1]/div[2]/div[2]/div[1]/div[3]/span[3]').get_attribute('textContent')

mitmproxy使用

字符集问题

Error: mitmproxy requires a UTF console environment.
Set your LANG environment variable to something like en_US.UTF-8

遇到上述问题重新设置一下LANG这个全局变量就好了

LANG="en_US.UTF-8"

视频合并

/usr/local/services/videoParse-1.0/sbin/ffmpeg -f concat -safe 0 -i list.txt -c copy -y ./out.mp4

其中list.txt存放的是分片信息(本次操作的视频分片存放在工作目录的上级目录中，所以下面的分片名会有"…/")

file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00001.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00002.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00003.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00004.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00005.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00006.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00007.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00008.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00009.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00010.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00011.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00012.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00013.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00014.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00015.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00016.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00017.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00018.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00019.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00020.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00021.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00022.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00023.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00024.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00025.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00026.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00027.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00028.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00029.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00030.ts
file ../03000600005C8266991A21A2FCEFF5811F81C4-062C-48B5-9C74-67F24B4FD18F-00031.ts

为Xvfb安装中文字体（ubuntu，命令行）

不过下面的方法仅仅适用于firefox

apt-get install xvfb xfonts-base xfonts-75dpi xfonts-100dpi firefox
apt-get install xfonts-wqy

如果是chrome，则需要先下载simsun.ttc文件，并将其放置到/usr/share/fonts/simsun.ttc，然后依次执行以下命令

mkfontdir
mkfontscale
fc-cache -fv

之后再运行chrome，就可以看到汉字了。

关于Chrome的参考文章
https://blog.csdn.net/u011550708/article/details/84587439
https://www.aityp.com/centos7%E5%AE%89%E8%A3%85%E4%B8%AD%E6%96%87%E5%AE%8B%E4%BD%93-phantomjs/