Server - 服务器配置 Conda 和 Jupiter Lab 的环境

SpikeKing

已于 2024-02-18 15:12:02 修改

阅读量744

点赞数 2

分类专栏： Server&Program 文章标签： conda python 开发语言

于 2022-06-26 16:26:32 首次发布

本文链接：https://blog.csdn.net/caroline_wendy/article/details/125471196

版权

Server&Program 专栏收录该内容

53 篇文章 1 订阅

订阅专栏

欢迎关注我的CSDN：https://spike.blog.csdn.net/
本文地址：https://blog.csdn.net/caroline_wendy/article/details/125471196

Conda和Jupiter是服务器研发算法必备，两者连用提升开发效率。

Conda和Jupiter

安装Conda

miniconda可以首次安装，也可以复用环境。

首次安装Conda

安装miniconda，其它版本miniconda或者anaconda均可，命令如下：

bash files/Miniconda3-py38_4.10.3-Linux-x86_64.sh

实际下载地址：https://repo.anaconda.com/miniconda/，约98.8M

修改路径，默认即可：

[/home/user/miniconda3] >>>

注意：配置在共享区域，速度较慢，不推荐。

复用Conda环境

复制.condarc和.bashrc至当前环境，TORCH_HOME是torch模型的存储目录，如下：

cp /nfs/user/files/.condarc /home/user/.conda/.

# 添加配置命令
# my configs
export TORCH_HOME=/nfs/user/workspace/torch_home/

其中，.condarc是conda的配置：

channels:
  - defaults
show_channel_urls: true
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
channel_priority: disabled
allow_conda_downgrades: true

激活conda：

source /home/user/.bashrc

创建conda环境，--clone可选，建议使用tmux，建议备份环境：

# Tmux可选
tmux new -s conda-init

# 复制创建
conda create -n torch-def --clone /nfs/user/conda/envs/torch-def/  # 创建
# 直接创建
conda create -n torch-def python=3.8

conda activate # 激活
conda deactivate  # 不激活

# 删除环境
# conda remove -n torch-def --all

# 备份环境
cp -r /home/user/miniconda3 /nfs/user/conda_env/miniconda3-jupyter-20220626

安装 pip 环境：

# python 2.7
curl "https://bootstrap.pypa.io/pip/2.7/get-pip.py" > get-pip-2.7.py
python get-pip-2.7.py

# python 3
curl 'https://bootstrap.pypa.io/get-pip.py' > get-pip.py
python get-pip.py

配置pip源：

mkdir ~/.pip
vim ~/.pip/pip.conf

[global]
# trusted-host = pypi.tuna.tsinghua.edu.cn
# index-url = https://pypi.tuna.tsinghua.edu.cn/simple/
index-url = http://mirrors.aliyun.com/pypi/simple/
trusted-host = mirrors.aliyun.com

安装pytorch，使用pip安装较慢，建议使用conda：

时间比较长，使用tmux持续窗口。
必须安装pytorch cuda 11版本，否则无法正常运行，具体cuda版本，参考nvidia-smi。

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

测试GPU是否可用，数据是否可用，参考，两个都需要测试。

# 测试环境
ngpu= 1
# Decide which device we want to run on
device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")
print("驱动为：",device)
print("GPU型号： ",torch.cuda.get_device_name(0))


# 测试张量
import 	torch
import  time
print(torch.__version__)
print(torch.cuda.is_available())

a = torch.randn(100, 10)
b = torch.randn(10, 200)
device = torch.device('cuda')
a = a.to(device)
b = b.to(device)
c = torch.matmul(a, b)
print(a.device, c.device)

卸载Conda环境

删除conda文件夹，删除.bashrc的启动命令，和删除配置文件，即可：

参考：恢复anaconda环境, 卸载anaconda, 重装anaconda

vim .bashrc
rm -rf ~/.condarc ~/.conda ~/.continuum

配置Jupiter

配置Jupiter Notebook

Jupyter安装

pip install ipython
pip install jupyter

配置Jupyter环境，参考After installing with pip, “jupyter: command not found”

vim ~/.bashrc

# shift+g跳转到文件尾部，gg跳转到文件头部
# jupyter
export PATH=$PATH:~/.local/bin

设置密码，可以设置为123或其他密码。

ipython
# from notebook.auth import passwd # 旧版
from jupyter_server.auth import passwd
passwd()

# 每次都不同
'argon2:$argon2id$......'

exit

配置密码：

mkdir ~/.jupyter
vim ~/.jupyter/jupyter_notebook_config.py

# 说明
c.ServerApp.ip='*' #设置访问notebook的ip，*表示所有IP，这里设置ip为都可访问  
c.ServerApp.password=u'argon2:$argon2id$v=19...' #填写刚刚生成的密文  
c.ServerApp.open_browser=False # 禁止notebook启动时自动打开浏览器(在linux服务器一般都是ssh命令行访问，没有图形界面的。所以，启动也没啥用)  
c.ServerApp.port=8901 #指定访问的端口，默认是8888。  

# 实际，不能有空格或汉字
c.ServerApp.ip='*'
c.ServerApp.password=u'argon2:$argon2id$v=19...'
c.ServerApp.open_browser=False
c.ServerApp.port=8901
c.ServerApp.notebook_dir="/"

进入工作环境，先尝试启动jupyter，再切换nohup模式启动：

cd /nfs/user

jupyter notebook --allow-root
nohup jupyter notebook --allow-root > nohup.jupyter-53.out &

ps aux | grep notebook  # 查看是否启动

访问地址，需要替换为真实IP地址，使用ifconfig查询eth0，例如：http://172.xx.0.xx:8889/，密码123

配置Jupiter Lab

额外安装包：

python -m pip install --upgrade pip
pip install jupyterlab

启动Jupiter Lab

nohup jupyter lab --port=8901 --no-browser --allow-root > nohup.jupyter-lab.out &

当使用SSH登录服务器时，无法直接获取IP地址。需要配置SSH隧道转发：

ssh -N -f -L localhost:8899:localhost:8899 [yourname]@xxx.xxx.xxx.xxx

手动关闭隧道转发：

参考：ssh -L (error: bind: Address already in use)

lsof -ti:8899 | xargs kill -9

配置Conda环境到Jupiter Notebook

配置conda环境到jupyter，参考jupyter notebook 如何配置conda环境：

python -m ipykernel install --user --name torch-new --display-name "torch-new"

其他jupyter的kernel环境命令：

jupyter kernelspec list  # 显示所用kernel
jupyter kernelspec remove torch-new  # 删除kernel

配置 Bash

参考，注意替换[user]为自己的目录，执行 source ~/.bashrc 配置成功。

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar

# make less more friendly for non-text input files, see lesspipe(1)
[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
    xterm-color|*-256color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
    if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
	# We have color support; assume it's compliant with Ecma-48
	# (ISO/IEC-6429). (Lack of such support is extremely rare, and such
	# a case would tend to support setf rather than setaf.)
	color_prompt=yes
    else
	color_prompt=
    fi
fi

if [ "$color_prompt" = yes ]; then
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

# colored GCC warnings and errors
#export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01'

# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'

# Add an "alert" alias for long running commands.  Use like so:
#   sleep 10; alert
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi

# my configs
export PATH=$PATH:~/.local/bin
export TORCH_HOME=/nfs_baoding/[user]/workspace/torch_home/
export LD_LIBRARY_PATH=/home/[user]/miniconda3/lib:$LD_LIBRARY_PATH 

alias bos='bcecmd --conf-path /etc/bceconf/ bos'

# Added by cryoSPARC:
export PATH="/nfs_baoding/[user]/workspace/cryosparc/cryosparc_master/bin":$PATH
export PATH="/home/[user]/.aspera/connect/bin":$PATH

SpikeKing

关注

2
点赞
踩
1

收藏

觉得还不错? 一键收藏
打赏
0
评论
Server - 服务器配置 Conda 和 Jupiter Lab 的环境

Conda和Jupiter是服务器研发算法必备。可以首次安装，也可以复用环境。安装miniconda，其它版本miniconda或者anaconda均可，命令如下：实际下载地址：https://repo.anaconda.com/miniconda/，约98.8M修改路径，默认即可：注意：配置在共享区域，速度较慢，不推荐。复
复制链接

扫一扫