基于Trie树进行拆分字符串变成拼音音节(一):构建拼音音节模型

一.前言

在智能客服领域,需要具备识别用户输入拼音并将其转为中文,进而来搜索与之相关问题,但用户输入的拼音内容是一个长字符串,此时需要把字符串进行中文拼音音节拆分,用逗号’,'将每个音节分开,因此本篇文章主要介绍构建基于Trie的拼音音节模型,在下篇文章介绍拆分音节,再下下篇文章介绍拼音识别成中文

二.构建模型

2.1 Trie树简要介绍

Trie树形结构属于字典树,整个树的根节点是空的, 根节点的子节点可以有多个,每个子节点智能存放单个元素,用于我们拼音模型就是根节点的子结点有26个,分别由a-z组成,每个子节点又可以当作其下的根节点,Trie字典树节点的结构为:

class TrieNode:
    def __init__(self):
        self.value = None
        self.children = {}

Trie树结构例图

2.2 构建拼音模型

首先我们需要中文拼音词库pinyin.txt,里面包含各种音节,将这些音节加入到trie树,变成拼音字典树,例如:

a
ai
aing
an
ang
ao
ba
bai
ban
bang
bao
bei
.
.
.

初始化字典树:

class Trie:
    def __init__(self):
        self.root = TrieNode()  # 初始化节点
        self.trie_path = Pinyin_Trie_Path  # 生成模型的目录文件 
        self.pinyin_path = Pinyin_Path # 中文拼音库目录文件

插入trie节点:

 def insert(self, key):
        node = self.root
        for char in key:
            if char not in node.children:
                child = TrieNode()
                node.children[char] = child
                node = child
            else:
                node = node.children[char]
        node.value = key

遍历音节字符串获取单个字母i,如果trie树已存在该单个字母,则继续在该字母节点的子节点中添加,例如:目前有音节zhao,而trie树已有zh,则遍历zhao,添加a作为h的子节点,再添加o作为a的子节点.如果字母i不属于子节点中,则需要新建一个TrieNode节点,存放新的字母.

    def build_trie(self):
        trie = Trie()
        for line in open(self.pinyin_path):
            word = line.strip().lower()
            trie.insert(word)
        with open(self.trie_path, 'wb') as f:
            pickle.dump(trie, f)

此段代码是构建拼音模型的代码,它逐行读取中文拼音词库pinyin.txt的拼音并转换成小写,并调用insert()方法插入Trie树中,最后导出拼音模型文件pinyin_Trie.model

    def search(self, key):
        node = self.root
        matches = []
        for char in key:
            if char not in node.children:
                break
            node = node.children[char]
            if node.value:
                matches.append(node.value)
        return matches

该代码主要功能是根据新输入的拼音去查找比对trie树的内容,比如输入zhuanye,比对trie后,会把[zhu,zhua,zhuan],[ye]返回

三.全部代码

Proj_path = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
Pinyin_Trie_Path = os.path.join(Proj_path, "main/pinyin_Trie.model")
Pinyin_Path = os.path.join(Proj_path, "main/assets/dataset/pinyin.txt")
# 定义trie字典树节点
class TrieNode:
    def __init__(self):
        self.value = None
        self.children = {}

class Trie:
    def __init__(self):
        self.root = TrieNode()
        self.trie_path = Pinyin_Trie_Path
        self.pinyin_path = Pinyin_Path

    # 添加树节点Pinyin_Path
    def insert(self, key):
        node = self.root
        for char in key:
            if char not in node.children:
                child = TrieNode()
                node.children[char] = child
                node = child
            else:
                node = node.children[char]
        node.value = key

    # 查找节点
    def search(self, key):
        node = self.root
        matches = []
        for char in key:
            if char not in node.children:
                break
            node = node.children[char]
            if node.value:
                matches.append(node.value)
        return matches

    def build_trie(self):
        trie = Trie()
        for line in open(self.pinyin_path):
            word = line.strip().lower()
            trie.insert(word)
        with open(self.trie_path, 'wb') as f:
            pickle.dump(trie, f)

有问题的,欢迎评论一起讨论!