深度学习中的token和tokenization

Tokenization is a way of separating a piece pf text into smaller units called tokens.And tokens are the building blocks of Natural Language,which can be either words, characters, or subwords.
举个例子，“Never give up”，Tokenization 成words tokens：Never-give-up。
再举个例子，“smarter”，characters tokens：s-m-a-r-t-e-r ；subword tokens：smart-er。
————————————————
版权声明：本文为CSDN博主「weixin_44716423」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/weixin_44716423/article/details/119003736