python爬虫——百度文本审核技术

hello大家好我还是那个萌新，今天给大家带来用post爬取百度文本审核技术的文章！

百度文本审核技术（如图 ）：

 

目标：在终端输入文字，获取网页响应结果。

网址：https://ai.baidu.com/tech/textcensoring

F12打开测试工具，选择Natwork，然后在网页中输入任意文字，点击分析，可以发现响应列表中多出了一个叫aidemo的请求：

 显而易见，这个请求就是api接口~

点开查看：

 hhhh简单了吧~直接调用接口即可~

代码如下：

import requests

url = 'https://ai.baidu.com/aidemo'

text = input('输入需要审核的文字（最多200字）：   ')

data = {'content': text,
'type': 'textcensor',
'apiType': 'censor',
'requestTime': '1628780167667',
'token': '2fe1cba870'}

headers = {'accept': '*/*',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'zh-CN,zh;q=0.9',
'content-length': '103',
'content-type': 'application/x-www-form-urlencoded',
'cookie': 'BAIDUID_BFESS=EAB301E34BAB5B3829977A4FDE00ED44:FG=1; BAIDUID=EAB301E34BAB5B385D72A84F2B5F0276:FG=1; BIDUPSID=EAB301E34BAB5B385D72A84F2B5F0276; PSTM=1628757735; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; __yjs_duid=1_cc834c6c33b9da38de2c8e4c95cf28491628775855270; Hm_lvt_8b973192450250dd85b9011320b455ba=1628775852; Hm_lpvt_8b973192450250dd85b9011320b455ba=1628780471; ab_sr=1.0.1_YjI5OTFlOWQ5OWI1ZjIwM2NmN2U5M2QyMTFiYjY2NWUxY2QxMWJhOTliOGYyN2Q1YWYwM2JhY2ZmZDU0NmQ4NTU1ZTJiNDRiN2RjMWRjOTNjYjdiNTE0YmFiOGQ3ZDE4NjljNmQ0MDk3NTg2N2QyZDJiZDA0OGI4MDdmYzRlYzkzMDI3M2Q1NWJhMzgwMzdhZDE1MGUyNGQ2MWM5MmMwMw==; __yjs_st=2_MjYxNDkyMTMwMWNiNzM2ZGU4NDM0NmQ4YTcyMTM2MTgxMTU0OWY5NzU4NzY2NjI0OTRiNTkyN2VjZjg0MDIwMjBiMWY1Zjg1YzFjNjNiY2QxNzliY2JlZWU0N2YwNGJlYjA5YzhiYzk0NGRhNTMxNGJjMjYxMjE1M2Y5YjJhNWQ1ZDE4NzFlMTNlM2VmODc2MTNlY2NlN2NlMDRkMDE1NDA2NjI1MzY5YmU0YTY4Mzc1YWYzYjRiNzUxZDRiYmJkMzRlODIyMTFiYWE3YjhjMjlhM2RmZDZkOTQ5OGRlMmQ5MjRkNWM2YWJlNDNkM2JjNmYzNzlmMTk4YmYzMmQxYl83X2U4YzNlODYy',
'origin': 'https://ai.baidu.com',
'referer': 'https://ai.baidu.com/tech/textcensoring',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'}
res = requests.post(url,data=data,headers=headers)

但是，有时候会出现如下情况：

这时候正常人会重新点击分析按钮，所以我们要重新发送请求，直到请求成功，于是代码可以写成这样：

def posts():
    global js,res

    res = requests.post(url,data=data,headers=headers)

    js = res.json()

try:
    posts()
except:
    posts()

这样就可以一直请求直到成功

接下来可能比较难

点击测试工具的Preview查看：

 

展开data，data下有一个result，同样展开，下面有pass和reject（如果审核通过就没有reject），pass和reject下有数字（范围0~5）：

 全部展开，发现每个数字下面都有一个score，明显就是我们要的数据

          ↑↑↑↑↑↑↑↑↑↑↑↑↑pass数字下的score对应的是通过项的分数 ↑↑↑↑↑↑↑↑↑↑↑↑↑

                     ↑↑↑↑↑↑↑↑↑↑↑↑↑reject下对应的就是不通过的分数啦 ↑↑↑↑↑↑↑↑↑↑↑↑↑

rights = js['data']['result']['pass']
wrongs = js['data']['result']['reject']
for worng in wrongs:
    y = round(float(worng['score']),3)
    print('{}   分数：{}'.format(y))
    print('\n')
print('--------------------------------------------')

for right in rights:
    y = round(float(right['score']),3)
    print('{}   分数：{}'.format(y))
    print('\n')

提取各项的分数，然后用round四舍五入到小数点第三位

只输出分数当然不行，谁知道这些分数代表什么啊？

我们要把分数和名称一一对应，我的方法是建立一个字典：

dic = {'1':'暴恐违禁','2':'文本色情','3':'政治敏感','4':'恶意推广','5':'低俗辱骂','6':'低质灌水'}

这些序号可以在data→result→pass/reject→label找到

下面是代码：

dic = {'1':'暴恐违禁','2':'文本色情','3':'政治敏感','4':'恶意推广','5':'低俗辱骂','6':'低质灌水'}

rights = js['data']['result']['pass']
wrongs = js['data']['result']['reject']
for worng in wrongs:
    x = dic[str(worng['label'])]
    y = round(float(worng['score']),3)
    print('{}   分数：{}'.format(x,y))
    print('\n')
print('--------------------------------------------')

for right in rights:
    x = dic[str(right['label'])]
    y = round(float(right['score']),3)
    print('{}   分数：{}'.format(x,y))
    print('\n')

我们还需要测试的综合结果：

全部代码：

import requests

url = 'https://ai.baidu.com/aidemo'

text = input('输入需要审核的文字（最多200字）：   ')

data = {'content': text,
'type': 'textcensor',
'apiType': 'censor',
'requestTime': '1628780167667',
'token': '2fe1cba870'}

headers = {'accept': '*/*',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'zh-CN,zh;q=0.9',
'content-length': '103',
'content-type': 'application/x-www-form-urlencoded',
'cookie': 'BAIDUID_BFESS=EAB301E34BAB5B3829977A4FDE00ED44:FG=1; BAIDUID=EAB301E34BAB5B385D72A84F2B5F0276:FG=1; BIDUPSID=EAB301E34BAB5B385D72A84F2B5F0276; PSTM=1628757735; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; __yjs_duid=1_cc834c6c33b9da38de2c8e4c95cf28491628775855270; Hm_lvt_8b973192450250dd85b9011320b455ba=1628775852; Hm_lpvt_8b973192450250dd85b9011320b455ba=1628780471; ab_sr=1.0.1_YjI5OTFlOWQ5OWI1ZjIwM2NmN2U5M2QyMTFiYjY2NWUxY2QxMWJhOTliOGYyN2Q1YWYwM2JhY2ZmZDU0NmQ4NTU1ZTJiNDRiN2RjMWRjOTNjYjdiNTE0YmFiOGQ3ZDE4NjljNmQ0MDk3NTg2N2QyZDJiZDA0OGI4MDdmYzRlYzkzMDI3M2Q1NWJhMzgwMzdhZDE1MGUyNGQ2MWM5MmMwMw==; __yjs_st=2_MjYxNDkyMTMwMWNiNzM2ZGU4NDM0NmQ4YTcyMTM2MTgxMTU0OWY5NzU4NzY2NjI0OTRiNTkyN2VjZjg0MDIwMjBiMWY1Zjg1YzFjNjNiY2QxNzliY2JlZWU0N2YwNGJlYjA5YzhiYzk0NGRhNTMxNGJjMjYxMjE1M2Y5YjJhNWQ1ZDE4NzFlMTNlM2VmODc2MTNlY2NlN2NlMDRkMDE1NDA2NjI1MzY5YmU0YTY4Mzc1YWYzYjRiNzUxZDRiYmJkMzRlODIyMTFiYWE3YjhjMjlhM2RmZDZkOTQ5OGRlMmQ5MjRkNWM2YWJlNDNkM2JjNmYzNzlmMTk4YmYzMmQxYl83X2U4YzNlODYy',
'origin': 'https://ai.baidu.com',
'referer': 'https://ai.baidu.com/tech/textcensoring',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'}

def posts():
    global js,res

    res = requests.post(url,data=data,headers=headers)

    js = res.json()

try:
    posts()
except:
    posts()

print(res,'\n')

dic = {'1':'暴恐违禁','2':'文本色情','3':'政治敏感','4':'恶意推广','5':'低俗辱骂','6':'低质灌水'}

rights = js['data']['result']['pass']
wrongs = js['data']['result']['reject']
yesornot = False

for worng in wrongs:
    yesornot = True
    x = dic[str(worng['label'])]
    y = round(float(worng['score']),3)
    print('{}   分数：{}'.format(x,y))
    print('\n')
print('--------------------------------------------')

for right in rights:
    x = dic[str(right['label'])]
    y = round(float(right['score']),3)
    print('{}   分数：{}'.format(x,y))
    print('\n')

if yesornot == True:
    print('综合评价：审核不通过')
else:
    print('综合评价：审核通过')

 输出结果：

画质不行，大家有好的工具推荐一下~

以上就是全部内容，谢谢阅读，点个赞再走吧~