bert第三篇：tokenizer

最新推荐文章于 2024-08-16 16:16:06 发布

iterate7

最新推荐文章于 2024-08-16 16:16:06 发布

阅读量3.3w

点赞数 54

分类专栏：机器学习深度学习文章标签： bert tokenizer wordpiece berttokenizer bpe

本文链接：https://blog.csdn.net/iterate7/article/details/108959082

版权

本文深入探讨BERT中的tokenizer，包括BasicTokenizer、WordpieceTokenizer和FullTokenizer的作用，强调WordPiece分词概念及其优点。文章还介绍了如何训练自定义的tokenizer，并讨论了BPE算法在解决词汇表外（OOV）问题上的作用。作者建议结合实践理解tokenizer的本质，以实现更高效的信息表示。

摘要由CSDN通过智能技术生成

文章目录

tokenizer基本含义

tokenizer就是分词器；只不过在bert里和我们理解的中文分词不太一样，主要不是分词方法的问题，bert里基本都是最大匹配方法。

最大的不同在于“词”的理解和定义。比如：中文基本是字为单位。
英文则是subword的概念，例如将"unwanted"分解成[“un”, “##want”, “##ed”] 请仔细理解这个做法的优点。
这是tokenizer的一个要义。

bert里涉及的tokenizer

BasicTokenzer

主要的类是BasicTokenizer，做一些基础的大小写、unicode转换、标点符号分割、小写转换、中文字符分割、去除重音符号等操作，最后返回的是关于词的数组（中文是字的数组）

 def tokenize(self, text):
    """Tokenizes a piece of text."""
    text = convert_to_unicode(text)
    text = self._clean_text(text)

    # This was added on November 1st, 2018 for the multilingual and Chinese
    # models. This is also applied to the English models now, but it doesn't
    # matter since the English models were not trained on any Chinese data
    # and generally don't have any Chinese data in them (there are Chinese
    # characters in the vocabulary because Wikipedia does have some Chinese
    # words in the English Wikipedia.).
    text = self._tokenize_chinese_chars(text)

    orig_tokens = whitespace_tokenize(text)
    split_tokens = []
    for token in orig_tokens:
      if self.do_lower_case:
        token = token.lower()
        token = self._run_strip_accents(token)
      split_tokens.extend(self._run_split_on_punc(token))

    output_tokens = whitespace_tokenize(" ".join(split_tokens))
    return output_tokens

BasicTokenzer是预处理。

wordpiecetokenizer

另外一个则是关键wordpiecetokenizer，就是基于vocab切词。

  def tokenize(self, text):
    """Tokenizes a piece of text into its word pieces.

    This uses a greedy longest-match-first algorithm to perform tokenization
    using the given vocabulary.

    For example:
      input = "unaffable"
      output = ["un", "##aff", "##able"]

    Args:
      text: A single token or whitespace separated tokens. This should have
        already been passed through `BasicTokenizer.

    Returns:
      A list of wordpiece tokens.
    """

    text = convert_to_unicode(text)

    output_tokens = []
    for token in whitespace_tokenize(text):
      chars = list(token)
      if len(chars) > self.max_input_chars_per_word:
        output_tokens.append(self.unk_token)
        continue

      is_bad = False
      start = 0
      sub_tokens = []
      while start < len(chars):
        end = len(chars)
        cur_substr = None

        #找个单词，找不到end向前滑动；还是看代码实在！！！
        while start < end:
          substr = "".join(chars[start:end])
          if start > 0:
            substr = "##" + substr
          if substr in self.vocab:
            cur_substr = substr
            break
          end -= 1
        if cur_substr is None:
          is_bad = True
          break
        sub_tokens.append(cur_substr)
        start = end

      if is_bad:
        output_tokens.append(self.unk_token)
      else:
        output_tokens.extend(sub_tokens)
    return output_tokens