hugginface相关数据集整理

swaption2009/20k-en-zh-translation-pinyin-hsk
翻译
Source: https://mnemosyne-proj.org/cards/20000-chinese-sentences-translations-and-pinyin
Contributed by: Brian Vaughan http://brianvaughan.net/

RUCAIBox/Translation
翻译
WMT14 English-French (wmt14-fr-en)
WMT16 Romanian-English (wmt16-ro-en)
WMT16 German-English (wmt16-de-en)
WMT19 Czech-English (wmt19-cs-en)
WMT13 Spanish-English (wmt13-es-en)
WMT19 Chinese-English (wmt19-zh-en)
WMT19 Russian-English (wmt19-ru-en).

dbarbedillo/SMS_Spam_Multilingual_Collection_Dataset
The text has been further translated into Spanish, Chinese, Arabic, Bengali, Russian, Portuguese, Indonesian, Urdu, Japanese, Punjabi, Javanese, Turkish, Korean, Marathi, Ukrainian, Swedish, and Norwegian using M2M100_418M a multilingual encoder-decoder (seq-to-seq) model trained for Many-to-Many multilingual translation created by Facebook AI.
The original English text was taken from- https://www.kaggle.com/uciml/sms-spam-collection-dataset Hindi, German and French taken from - https://www.kaggle.com/datasets/rajnathpatel/multilingual-spam-data

projecte-aina/ca_zh_wikipedia
中文-加拿大语之间的翻译

wanng/wukong100m
简介 Brief Introduction
取自Noah-Wukong多语言多模态数据集中的中文部分，一共100M个图文对。
A subset from Noah-Wukong (a multimodal dataset), around 100M image-text pairs (only Chinese).

MMChat
基于图片的聊天对
is a large-scale dialogue dataset that contains image-grounded dialogues in Chinese. Each dialogue in MMChat is associated with one or more images (maximum 9 images per dialogue). We design various strategies to ensure the quality of the dialogues in MMChat.

Jiangjie/ekar_chinese
Explainable Knowledge-intensive Analogical Reasoning benchmark (E-KAR).

Hello-SimpleAI/HC3-Chinese
人工和-ChatGPT对比数据集

kuroneko5943/weibo16
微博情感

wangrui6/Zhihu-KOL
知乎问题答案，一个问题，多个答案，根据赞同数量可以排序

silver/personal_dialog
中文个人对话，多轮回

medical_dialog
医学上的病人和医生的对话

mteb/amazon_massive_intent
亚马逊意图识别

qanastek/MASSIVE
意图识别，ner，

GEM/RiSAWOZ
对话多轮

sunzeyeah/chinese_chatgpt_corpus
train_data_external_v1.jsonl
prompt: prompt, string
answers: list of answers
answer: answer, string
score: score of answer, int
prefix: prefix to the answer, string

BelleGroup/generated_train_0.5M_CN
BELLE: Bloom-Enhanced Large Language model Engine
prompt_cn.txt: 生成所使用的提示语
0.5M生成的数据 ： 为了方便模型训练，huggingface开源数据将原始生成文件中的"instruction"、"input"字段合并成"input"字段，"output"字段修改为"target"字段。