GPT Prompt中英文token对比

点击↑上方↑蓝色“编了个程”关注我~

这是Yasin的第 93 篇原创文章

官方的token解释

->

Our models understand and process text by breaking it down into tokens. Tokens can be words or just chunks of characters. For example, the word “hamburger” gets broken up into the tokens “ham”, “bur” and “ger”, while a short and common word like “pear” is a single token. Many tokens start with a whitespace, for example “ hello” and “ bye”. The number of tokens processed in a given API request depends on the length of both your inputs and outputs. As a rough