java nltk_如何使用NLTK tokenizer摆脱标点符号？

删除标点(它将删除 . 以及使用下面的代码处理标点符号的一部分)

tbl = dict.fromkeys(i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith('P'))

text_string = text_string.translate(tbl) #text_string don't have punctuation

w = word_tokenize(text_string) #now tokenize the string

输入/输出样本：

direct flat in oberoi esquire. 3 bhk 2195 saleable 1330 carpet. rate of 14500 final plus 1% floor rise. tax approx 9% only. flat cost with parking 3.89 cr plus taxes plus possession charger. middle floor. north door. arey and oberoi woods facing. 53% paymemt due. 1% transfer charge with buyer. total cost around 4.20 cr approx plus possession charges. rahul soni

['direct', 'flat', 'oberoi', 'esquire', '3', 'bhk', '2195', 'saleable', '1330', 'carpet', 'rate', '14500', 'final', 'plus',