https://huggingface.co/datasets/yuyijiong/LongData-Corpus
1、hf的数据
在开发机上要设置ssh key,然后cat复制之后在设置在hf上
2、中文小说数据在云盘上
下载:
#!/bin/bash
# Base URL
base_url="https://cloud.tsinghua.edu.cn/d/0670fcb14d294c97b5cf/files/?p=%2F%E4%B8%AD%E6%96%87%E5%B0%8F%E8%AF%B4_"
# Loop to download files from 1 to 85
for i in {1..85}
do
# Format the number with leading zeros (e.g., 001, 002, ..., 085)
num=$(printf "%d" $i)
# Construct the URL
url="${base_url}${num}.jsonl.zst&dl=1"
# Download the file
wget -O "中文小说_${num}.j