自然语言处理作业A1

作业地址
参考资料

1. 任务1：把HTML格式转为JSON数据，再用python的JSON包，把JSON数据转为python能使用的数据结构(dicts, lists…)（chaos2json.py）

Your implementation should have at least one regular expression (to extract the textual content of each line), and use NLTK’s word_tokenize function as the tokenizer. You may also use built-in string methods/operations and write your own helper functions.
The word_tokenize function does not separate hyphens, but this text uses hyphens in place of dashes, so your code should separate them.

Hint 1: The HTML contains (nonstandard) tags like at the beginning of each line. The number is the line within the stanza (between 1 and 4). Ignore ellipsis lines indicating removed stanzas.
Hint 2: When converting to JSON, use the indent argument to make it more human-readable.
(This script should not take extremely long to implement, but it will probably take you longer than you expect.)

from urllib import request
from bs4 import BeautifulSoup
from nltk import word_tokenize
import re
import json

url = 'file:///E:/学习文档/数据集/a1/chaos.html'

# 打开URL，返回HTML信息
def open_url(url):
    # 根据当前URL创建请求包
    req = request.Request(url)
    # 添加头信息，伪装成浏览器访问
    req.add_header('User-Agent',
                   'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36')
    # 发起请求
    response = request.urlopen(req)
    # 返回请求到的HTML信息
    return response.read()

# 用正则定位
def find_tag(url, regex = '<xxx.>(.*?)(<br>|</p>)'):
    # (.*?)
    # .是除了\n的任意字符
    # *是取之前字符的0个或者n个
    # ？是去之前字符的0个或者1个；也可以解释为非贪婪模式
    # （）圆括号，举例说明，eg： a(b)c