DeepSeek-coder 微调训练记录

简介

微调过程不再细说, 参考link进行即可. 主要是数据集.

1.3b模型微调训练占用资源信息

top信息

评估

根据DeepSeek-coder的Evaluation试进行对微调后的模型进行评估. 其中的评估库主要是evol-teacher和human-eval.

进入DeepSeek-coder下的Evaluation中的Human-eval文件夹, 新建一个eval_ins.sh文件, 填入以下内容

LANG="python"
OUPUT_DIR="output"
MODEL="deepseek-coder-1.3b-instruct"

CUDA_VISIBLE_DEVICES=0,1 python eval_instruct.py \
    --model "deepseek-ai/$MODEL" \  # 修改路径为微调保存的模型路径
    --output_path "$OUPUT_DIR/${LANG}.$MODEL.jsonl" \
    --language $LANG \
    --temp_dir $OUPUT_DIR

随后启动bash eval_ins.sh

model /home/LLM/DeepSeek-Coder/finetune/output/checkpoint-14500
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
load tokenizer <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'> from /home/stlinpeiyang/lpy22/LLM/DeepSeek-Coder/finetune/output/checkpoint-14500 over.
Read 164 examples for evaluation over.
Generating:   1%|▌                                            | 2/164 [07:10<10:00:15, 222.32s/it]Failed to extract code block with error `list index out of range`:
>>> Task: Python/2
>>> Output:
def truncate_number(number: float) -> float:
    """ Given a positive floating point number, it can be decomposed into
    and integer part (largest integer smaller than given number) and decimals
    (leftover part always smaller than 1).

    Return the decimal part of the number.
    >>> truncate_number(3.5)
    0.5
    """
    integer_part = int(number)
    decimal_part = number - integer_part
    return decimal_part

Generating:   6%|██▋                                          | 10/164 [28:04<7:23:26, 172.77s/it]

耗时很久.

---

deepseek-coder-6.7b-instruct验证数据

(py_LLM) orange:~/lpy22/LLM/DeepSeek-Coder/Evaluation/HumanEval$ bash eval_6.7b.sh
model /home/LLM/text-generation-webui/models/deepseek-coder-6.7b-instruct
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
load tokenizer <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'> from /home/LLM/text-generation-webui/models/deepseek-coder-6.7b-instruct over.
Loading checkpoint shards: 100%|████████████████████████████████████████████| 2/2 [00:02<00:00,  1.12s/it]
Read 164 examples for evaluation over.
Generating: 100%|███████████████████████████████████████████████████| 164/164 [11:58:55<00:00, 263.02s/it]
Generate all over!!!
Save 164 processed examples into ./output/python.deepseek-coder-6.7b-instruct.jsonl over!
Reading samples...
100%|████████████████████████████████████████████████████████████████| 164/164 [00:00<00:00, 11619.16it/s]
Running test suites...
100%|███████████████████████████████████████████████████████████████████| 164/164 [00:14<00:00, 11.32it/s]
{'pass@1': 0.7865853658536586}
python {'pass@1': 0.7865853658536586} /home/LLM/text-generation-webui/models/deepseek-coder-6.7b-instruct

---

deepseek-coder-1.3b-instruct-finetune验证数据

Generating: 100%|███████████████████████████████████████████████████| 164/164 [10:58:16<00:00, 240.83s/it]
Generate all over!!!
Save 164 processed examples into ./output/python.deepseek-coder-1.3b-instruct-finetune.jsonl over!
Reading samples...
100%|███████████████████████████████████████████████████████████████████| 164/164 [00:00<00:00, 13670.37it
Running test suites...
100%|██████████████████████████████████████████████████████████████████████| 164/164 [00:13<00:00, 12.25it
{'pass@1': 0.6158536585365854}
python {'pass@1': 0.6158536585365854} /home/orange/LLM/DeepSeek-Coder/finetune/output/checkpoi14500

---

20140511