【学习总结】python transformers 预处理 YelpReviewFull 数据集，并展示

1. 数据简介

Yelp是一家总部位于美国旧金山的跨国公司，它开发Yelp.com网站和Yelp移动应用程序。Yelp是一个用户对餐馆等场所进行评价的网站。（维基百科）

Yelp 评论数据集包含来自 Yelp 的评论。 它是从 Yelp 数据集挑战 2015 数据中提取的。
Yelp 评论数据集

支持的任务：
text-classification、sentiment-classification
该数据集主要用于文本分类：给定文本，预测情感。

数据结构：
一个数据包括文本和相应的标签：

1. ‘text’: 评论文本使用双引号（“）转义，任何内部双引号都通过2个双引号（”"）转义。换行符使用反斜杠后跟一个 “n” 字符转义，即 “\n”。
2. ‘label’: 对应于评论的分数（0-4）0 表示1星，4表示5星

{
   'label': 0,
 'text': "This place is absolute garbage...  Half of the tees are not available, including all the grass tees.  It is cash only, and they sell the last bucket at 8, despite having lights.  And if you finish even a minute after 8, don't plan on getting a drink.  The vending machines are sold out (of course) and they sell drinks inside, but close the drawers at 8 on the dot.  There are weeds grown all over the place.  I noticed some sort of batting cage, but it looks like those are out of order as well.  Someone should buy this place and turn it into what it should be."}

2. 数据显示

load_dataset.py :

from datasets import load_dataset

dataset = load_dataset("yelp_review_full")

print(dataset)
print(type(dataset))
print(dataset["train"][11])

输出结果：

DatasetDict({
   
    train: Dataset({
   
        features: ['label', 'text'],
        num_rows: 650000
    })
    test: Dataset({
   
        features: ['label', 'text'],
        num_rows: 50000