Python正则表达式做文本预处理，去掉特殊符号

在进行文本训练和处理之前难免要进行下预处理，过滤掉没有用的符号等，简单用python 的正则表达式过滤一下。

#!/usr/bin/python
# encoding: UTF-8
import re

# make English text clean
def clean_en_text(text):
    # keep English, digital and space
    comp = re.compile('[^A-Z^a-z^0-9^ ]')
    return comp.sub('', text)

# make Chinese text clean
def clean_zh_text(text):
    # keep English, digital and Chinese
    comp = re.compile('[^A-Z^a-z^0-9^\u4e00-\u9fa5]')
    return comp.sub('', text)

if __name__ == '__main__':
    text_en = '$How old are you? Could you give me your pen?'
    text_zh = '$你好！我是个程序猿，标注码农￥'
    print(clean_en_text(text_en))
    print(clean_zh_text(text_zh))

测试

(base) ➜  src python test.py
How old are you Could you give me your pen
你好我是个程序猿标注码农