与风景对话_交互式旅游推荐系统_数据预处理与分析（七)基于规则的离群数据识别

我们发现不仅会出现英文报错，也会出现中文报错，这些报错多源于浏览器插件错误或者爬虫无法正确的识别文章中的视频文件导致的报错，因为我们也需要对其进行处理，但是由于是中文类型的报错，导致我们无法直接采用识别主题语言的方式来对其进行识别，因此我们这次采取基于规则的离群数据识别，即基于关键词和正则表达式的识别

一.问题分析 

首先我们还是对问题进行分析，我们在出现中文类型的报错时主要是浏览器报错

您的浏览器版本暂不支持播放视频，请更新至最新版本！Video Player is loading.Play  Time0:00/Duration1:00Loaded:26.01%0:00Stream TypeLIVESeek to live, currently playing  Time-1:00Playback Rate1xChaptersChaptersDescriptionsdescriptions off,  settings, opens subtitles settings  off, selectedAudio  is a modal window.Beginning of dialog window. Escape will cancel and close the window.--- Size50%75%100%125%150%175%200%300%400%Text Edge   Sans-SerifMonospace Sans- SerifMonospace   all settings to the default  Modal DialogEnd of dialog window.

大家好，我叫泰山，是一名来自成都华阳的gai（街）娃宠物小精灵训练师，即便我只有3岁8个月，但我有一个比身体大百倍的梦想——成为宇宙最强“宠物小精灵训练家”，追求那无穷尽的星辰大海。本次我背上行囊，带着我最爱的两个“宠物小精灵”——dad和mum，为此梦赴一场前路不明的旅途…主人公：小泰给小泰讲书，哄他睡觉，管他温饱，包办他帅气的小精灵——mum可以陪疯闹，裸泳，打怪兽，下苦力的小精灵——dad科普：想成为最优秀的宠物小精灵训练师，就必须参加宠物小精灵联盟比赛，得到认可。而获得宠物小精灵联盟参赛权的唯一办法是：在各地道馆与小精灵比赛中获得胜利，并且集齐至少八枚徽章才可以参赛。道馆——以靠宠物小精灵对决进行比赛为目的的场所，胜出的挑战者能够获得打赢该道馆的凭证——道馆徽章。每个人都要经历被生活一拳一脚的“教育”，才能摸索到自己梦想的边界，

像这个报错，主要是因为文章中出现了视频导致我们无法正确的展示，因此我们可以采用基于正则表达式的方式来对文章进行删减。我们通过对报错信息的分析与检测发现，大部分的报错信息都会出现以下英文字符，而且存在出现以下中文字符的情况，这样可以最大程度的确保不会出现遗漏，尽可能的消除文件中的错误文本。

error", "404", "not found", "exception", "failed", "warning", "null",
    "undefined", "unable to", "invalid", "server error", "timeout",
    "unauthorized", "forbidden", "service unavailable", "bad request",
    "conflict", "internal server error", "gateway timeout", "network error",
    "bad gateway"

   "错误", "404", "找不到", "异常", "失败", "警告", "空值", "未定义",
    "无法", "无效", "服务器错误", "超时", "未授权", "禁止", "服务不可用",
    "错误请求", "冲突", "内部服务器错误", "网关超时", "网络错误", "错误网关"

 

二.代码实现

1.导入库

import re
import json

2.对已有的错误进行正则表达式识别

def remove_video_description(text):
    # 定义视频描述的正则表达式模式
    pattern = r"您的浏览器版本暂不支持播放视频.*?End of dialog window\."
    # 使用正则表达式删除匹配到的文本
    cleaned_text = re.sub(pattern, "", text, flags=re.DOTALL)
    return cleaned_text

 3.使用关键词删除匹配到的文本行

def remove_outliers_by_keywords(text, keywords):
    # 使用关键词删除匹配到的文本行
    cleaned_lines = []
    for line in text.split('\n'):
        if not any(keyword.lower() in line.lower() for keyword in keywords):
            cleaned_lines.append(line)
    cleaned_text = '\n'.join(cleaned_lines)
    return cleaned_text

4.从储存中读取文章进行处理后保存

# 读取本地文件
file_path = 'C:/Users/48594/Desktop/深度学习/res/去哪儿上海.json'  
# 处理每篇文章
for i, entry in enumerate(data):
    if 'text' in entry:
        text = entry['text']
        # 删除视频描述部分
        text = remove_video_description(text)
        # 删除包含特定关键词的离群数据
        text = remove_outliers_by_keywords(text, keywords)
        # 更新处理后的文本
        data[i]['text'] = text

# 保存处理后的文件
cleaned_file_path = 'C:/Users/48594/Desktop/深度学习/res/cleaned_file.json'  # 
with open(cleaned_file_path, 'w', encoding='utf-8') as file:
    json.dump(data, file, ensure_ascii=False, indent=4)

5.定义关键词

# 定义扩展的关键词列表
english_keywords = [
    "error", "404", "not found", "exception", "failed", "warning", "null",
    "undefined", "unable to", "invalid", "server error", "timeout",
    "unauthorized", "forbidden", "service unavailable", "bad request",
    "conflict", "internal server error", "gateway timeout", "network error",
    "bad gateway"
]

chinese_keywords = [
    "错误", "404", "找不到", "异常", "失败", "警告", "空值", "未定义",
    "无法", "无效", "服务器错误", "超时", "未授权", "禁止", "服务不可用",
    "错误请求", "冲突", "内部服务器错误", "网关超时", "网络错误", "错误网关"
]

6.运行程序

if __name__ == "__main__":
    data_file_path = 'C:/Users/48594/Desktop/深度学习/res/outliers.json'

    # 加载数据
    data = load_data(data_file_path)
    # 调用离群数据识别方法
    outlier_indices = detect_outliers(data)

总体代码

import re
import json

def remove_video_description(text):
    # 定义视频描述的正则表达式模式
    pattern = r"您的浏览器版本暂不支持播放视频.*?End of dialog window\."
    # 使用正则表达式删除匹配到的文本
    cleaned_text = re.sub(pattern, "", text, flags=re.DOTALL)
    return cleaned_text

def remove_outliers_by_keywords(text, keywords):
    # 使用关键词删除匹配到的文本行
    cleaned_lines = []
    for line in text.split('\n'):
        if not any(keyword.lower() in line.lower() for keyword in keywords):
            cleaned_lines.append(line)
    cleaned_text = '\n'.join(cleaned_lines)
    return cleaned_text

# 读取本地文件
file_path = 'C:/Users/48594/Desktop/深度学习/res/去哪儿上海.json'  # 替换为您的文件路径

with open(file_path, 'r', encoding='utf-8') as file:
    data = json.load(file)

# 定义扩展的关键词列表
english_keywords = [
    "error", "404", "not found", "exception", "failed", "warning", "null",
    "undefined", "unable to", "invalid", "server error", "timeout",
    "unauthorized", "forbidden", "service unavailable", "bad request",
    "conflict", "internal server error", "gateway timeout", "network error",
    "bad gateway"
]

chinese_keywords = [
    "错误", "404", "找不到", "异常", "失败", "警告", "空值", "未定义",
    "无法", "无效", "服务器错误", "超时", "未授权", "禁止", "服务不可用",
    "错误请求", "冲突", "内部服务器错误", "网关超时", "网络错误", "错误网关"
]

keywords = english_keywords + chinese_keywords

# 处理每篇文章
for i, entry in enumerate(data):
    if 'text' in entry:
        text = entry['text']
        # 删除视频描述部分
        text = remove_video_description(text)
        # 删除包含特定关键词的离群数据
        text = remove_outliers_by_keywords(text, keywords)
        # 更新处理后的文本
        data[i]['text'] = text

# 保存处理后的文件
cleaned_file_path = 'C:/Users/48594/Desktop/深度学习/res/cleaned_file.json'  # 替换为您的保存路径
with open(cleaned_file_path, 'w', encoding='utf-8') as file:
    json.dump(data, file, ensure_ascii=False, indent=4)

print("文件处理完成并已保存。")
if __name__ == "__main__":
    data_file_path = 'C:/Users/48594/Desktop/深度学习/res/outliers.json'

    # 加载数据
    data = load_data(data_file_path)
    # 调用离群数据识别方法
    outlier_indices = detect_outliers(data)

    # 打印离群数据索引