ModelScope系列之开发环境安装

ModelScope系列之开发环境安装

本篇文章介绍ModelScope使用所需的环境安装指南。

ModelScope Library目前支持tensorflow，pytorch深度学习框架进行模型训练、推理， 在Python 3.8+, Pytorch 1.11+, Tensorflow上测试可运行。

本机环境如下：

本机操作系统：Win10 22H2
CPU：Intel i7-12700
MEM：64G
Disk： 1TB SSD
GPU：RTX 3060 12G

参考文档：

环境安装

Python环境配置

参考文档下载windows安装包，安装配置Anaconda环境。

安装完成后，搜索打开Anaconda Prompt，执行如下命令为modelscope library创建对应的python环境。

conda create -n modelscope python=3.10
conda activate modelscope

安装深度学习框架

# 安装pytorch
pip3 install torch torchvision torchaudio -i https://pypi.tuna.tsinghua.edu.cn/simple

# 安装Tensorflow
pip3 install tensorflow -i https://pypi.tuna.tsinghua.edu.cn/simple

ModelScope Library 安装

ModelScope Libarary由核心框架，以及不同领域模型的对接组件组成。

1. 通过pip安装

如果只需要ModelScope模型和数据集访问等基础能力，可以只安装ModelScope的核心框架。

# 安装
pip install modelscope
# 升级
pip install modelscope --upgrade

但如果需要进一步具体使用ModelScope平台上承载的，包括多模态，NLP，CV，语音等不同领域的模型，来进行模型推理以及模型训练、微调等能力，则需要安装各个领域上不同的依赖。

例如需体验多模态领域的模型，可执行如下命令安装领域依赖：

pip install "modelscope[multi-modal]" 

如仅需体验NLP领域模型，可执行如下命令安装领域依赖:

pip install "modelscope[nlp]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

如仅需体验CV领域模型，可执行如下命令安装领域依赖:

pip install "modelscope[cv]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

如仅需体验语音领域模型，可执行如下命令安装领域依赖:

pip install "modelscope[audio]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

如仅需体验科学计算领域模型，请执行如下命令：

pip install "modelscope[science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

如果您所有领域的模型功能都想体验，可执行如下命令

pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

2. 通过源码安装

git clone git@github.com:modelscope/modelscope.git
cd modelscope
git fetch origin master
git checkout master

安装依赖，以NLP领域模型，可执行如下命令安装依赖，对比pip方案，将命令中的modelscope改为.即可。

pip install ".[nlp]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

3. 安装验证

安装成功后，即可使用对应领域模型进行推理，训练等操作。这里我们以NLP领域为例。安装后，可执行如下命令，运行中文分词任务，来验证安装是否正确：

python -c "from modelscope.pipelines import pipeline;print(pipeline('word-segmentation')('今天天气不错，适合 出去游玩'))"

(modelscope) C:\Users\xsdwl>python -c "from modelscope.pipelines import pipeline;print(pipeline('word-segmentation')('今天天气不错，适合 出去游玩'))"
2024-05-09 23:38:16,541 - modelscope - INFO - PyTorch version 2.3.0 Found.
2024-05-09 23:38:16,543 - modelscope - INFO - TensorFlow version 2.16.1 Found.
2024-05-09 23:38:16,543 - modelscope - INFO - Loading ast index from C:\Users\xsdwl\.cache\modelscope\ast_indexer
2024-05-09 23:38:16,543 - modelscope - INFO - No valid ast index found from C:\Users\xsdwl\.cache\modelscope\ast_indexer, generating ast index from prebuilt!
2024-05-09 23:38:16,619 - modelscope - INFO - Loading done! Current index file version is 1.14.0, with md5 1a8e54824de97d329fb743182d69f765 and a total number of 976 components indexed
2024-05-09 23:38:19,435 - modelscope - WARNING - Model revision not specified, use revision: v1.0.3
Downloading: 100%|██████████████████████████████████████████████████████████████████| 243k/243k [00:00<00:00, 5.78MB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████| 976/976 [00:00<00:00, 978kB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████| 161/161 [00:00<00:00, 161kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████| 46.7k/46.7k [00:00<00:00, 2.66MB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████| 388M/388M [00:03<00:00, 115MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████| 5.96k/5.96k [00:00<00:00, 6.12MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████| 30.0/30.0 [00:00<?, ?B/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████| 107k/107k [00:00<00:00, 5.22MB/s]
2024-05-09 23:38:27,301 - modelscope - INFO - initiate model from C:\Users\xsdwl\.cache\modelscope\hub\damo\nlp_structbert_word-segmentation_chinese-base
2024-05-09 23:38:27,301 - modelscope - INFO - initiate model from location C:\Users\xsdwl\.cache\modelscope\hub\damo\nlp_structbert_word-segmentation_chinese-base.
2024-05-09 23:38:27,304 - modelscope - INFO - initialize model from C:\Users\xsdwl\.cache\modelscope\hub\damo\nlp_structbert_word-segmentation_chinese-base
You are using a model of type bert to instantiate a model of type structbert. This is not supported for all configurations of models and can yield errors.
2024-05-09 23:38:28,140 - modelscope - WARNING - No preprocessor field found in cfg.
2024-05-09 23:38:28,141 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-05-09 23:38:28,142 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': 'C:\\Users\\xsdwl\\.cache\\modelscope\\hub\\damo\\nlp_structbert_word-segmentation_chinese-base'}. trying to build by task and model information.
2024-05-09 23:38:28,152 - modelscope - INFO - cuda is not available, using cpu instead.
2024-05-09 23:38:28,155 - modelscope - WARNING - No preprocessor field found in cfg.
2024-05-09 23:38:28,155 - modelscope - WARNING - No val key and type key found in preprocessor domain of configuration.json file.
2024-05-09 23:38:28,155 - modelscope - WARNING - Cannot find available config to build preprocessor at mode inference, current config: {'model_dir': 'C:\\Users\\xsdwl\\.cache\\modelscope\\hub\\damo\\nlp_structbert_word-segmentation_chinese-base', 'sequence_length': 512}. trying to build by task and model information.
C:\Users\xsdwl\.conda\envs\modelscope\lib\site-packages\transformers\modeling_utils.py:1051: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
{'output': ['今天', '天气', '不错', '，', '适合', '出去', '游玩']}

SWIFT安装

SWIFT是ModelScope官方提供的大模型（LLM&SD模型）训练推理框架。如果你对这类模型的训练过程有需求，推荐使用SWIFT。

1. Wheel包安装

可以使用pip进行安装：

pip install ms-swift -U

2. 源代码安装

git clone https://github.com/modelscope/swift.git
cd swift
pip install -e .