简介
微调过程不再细说, 参考link进行即可. 主要是数据集.
1.3b模型微调训练占用资源信息
top信息
评估
根据DeepSeek-coder的Evaluation试进行对微调后的模型进行评估. 其中的评估库主要是evol-teacher和human-eval.
进入DeepSeek-coder下的Evaluation中的Human-eval文件夹, 新建一个eval_ins.sh
文件, 填入以下内容
LANG="python"
OUPUT_DIR="output"
MODEL="deepseek-coder-1.3b-instruct"
CUDA_VISIBLE_DEVICES=0,1 python eval_instruct.py \
--model "deepseek-ai/$MODEL" \ # 修改路径为微调保存的模型路径
--output_path "$OUPUT_DIR/${LANG}.$MODEL.jsonl" \
--language $LANG \
--temp_dir $OUPUT_DIR
随后启动bash eval_ins.sh
model /home/LLM/DeepSeek-Coder/finetune/output/checkpoint-14500
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
load tokenizer <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'> from /home/stlinpeiyang/lpy22/LLM/DeepSeek-Coder/finetune/output/checkpoint-14500 over.
Read 164 examples for evaluation over.
Generating: 1%|▌ | 2/164 [07:10<10:00:15, 222.32s/it]Failed to extract code block with error `list index out of range`:
>>> Task: Python/2
>>> Output:
def truncate_number(number: float) -> float:
""" Given a positive floating point number, it can be decomposed into
and integer part (largest integer smaller than given number) and decimals
(leftover part always smaller than 1).
Return the decimal part of the number.
>>> truncate_number(3.5)
0.5
"""
integer_part = int(number)
decimal_part = number - integer_part
return decimal_part
Generating: 6%|██▋ | 10/164 [28:04<7:23:26, 172.77s/it]
耗时很久.
deepseek-coder-6.7b-instruct验证数据
(py_LLM) orange:~/lpy22/LLM/DeepSeek-Coder/Evaluation/HumanEval$ bash eval_6.7b.sh
model /home/LLM/text-generation-webui/models/deepseek-coder-6.7b-instruct
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
load tokenizer <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'> from /home/LLM/text-generation-webui/models/deepseek-coder-6.7b-instruct over.
Loading checkpoint shards: 100%|████████████████████████████████████████████| 2/2 [00:02<00:00, 1.12s/it]
Read 164 examples for evaluation over.
Generating: 100%|███████████████████████████████████████████████████| 164/164 [11:58:55<00:00, 263.02s/it]
Generate all over!!!
Save 164 processed examples into ./output/python.deepseek-coder-6.7b-instruct.jsonl over!
Reading samples...
100%|████████████████████████████████████████████████████████████████| 164/164 [00:00<00:00, 11619.16it/s]
Running test suites...
100%|███████████████████████████████████████████████████████████████████| 164/164 [00:14<00:00, 11.32it/s]
{'pass@1': 0.7865853658536586}
python {'pass@1': 0.7865853658536586} /home/LLM/text-generation-webui/models/deepseek-coder-6.7b-instruct
deepseek-coder-1.3b-instruct-finetune验证数据
Generating: 100%|███████████████████████████████████████████████████| 164/164 [10:58:16<00:00, 240.83s/it]
Generate all over!!!
Save 164 processed examples into ./output/python.deepseek-coder-1.3b-instruct-finetune.jsonl over!
Reading samples...
100%|███████████████████████████████████████████████████████████████████| 164/164 [00:00<00:00, 13670.37it
Running test suites...
100%|██████████████████████████████████████████████████████████████████████| 164/164 [00:13<00:00, 12.25it
{'pass@1': 0.6158536585365854}
python {'pass@1': 0.6158536585365854} /home/orange/LLM/DeepSeek-Coder/finetune/output/checkpoi14500
20140511