ubuntu server（包括树莓派）初始化配置（20200801更新）

>>> docker pull ubuntu
>>> docker run -v /home/pi/share:/root/share -p 8888:8888 -p 3100:3100 -itd ubuntu:latest
>>> docker exec -it xxxx /bin/bash

• 安装zsh、oh-my-zsh

# 安装zsh
>>> apt install zsh
# 修改默认sh
>>> chsh -s /bin/zsh
# 安装git
>>> apt install git
# 安装curl
>>> apt install curl
# 安装oh-my-zsh
>>> sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
# 重启
>>> reboot now

• 安装vim修改配置

>>> apt install vim
>>> vim .vimrc
set nocompatible "去掉vi的一致性"
set number "显示行号"
set guioptions-=r " 隐藏滚动条"    
set guioptions-=L
set guioptions-=b
set showtabline=0 "隐藏顶部标签栏"
set guifont=Monaco:h13 "设置字体"        
syntax on   "开启语法高亮"
set background=dark     "设置背景色"
set nowrap  "设置不折行"
set fileformat=unix "设置以unix的格式保存文件"
set cindent     "设置C样式的缩进格式"
set tabstop=4   "设置table长度"
set shiftwidth=4        "同上"
set showmatch   "显示匹配的括号"
set scrolloff=5     "距离顶部和底部5行"
set laststatus=2    "命令行为两行"
set fenc=utf-8      "文件编码"
set backspace=2
set mouse=a     "启用鼠标"
set selection=exclusive
set selectmode=mouse,key
set matchtime=5
set ignorecase      "忽略大小写"
set incsearch
set hlsearch        "高亮搜索项"
set noexpandtab     "不允许扩展table"
set autoread
set cursorline      "突出显示当前行"
set cursorcolumn        "突出显示当前列"
set ts=4
set expandtab
set autoindent

• 安装中文

>>> apt install language-pack-zh-hans
>>> vim /var/lib/locales/supported.d/local
en_US.UTF-8 UTF-8
zh_CN.UTF-8 UTF-8
zh_CN.GBK GBK
zh_CN GB2312
>>> locale-gen
>>> apt-get install fonts-droid-fallback ttf-wqy-zenhei ttf-wqy-microhei fonts-arphic-ukai fonts-arphic-uming
>>> vim .zshrc
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
>>> source .zshrc

• 安装ss、proxyxxxxxx

略

• 安装Python与包

>>> apt install python3, python3-dev
>>> python3 -m pip install --upgrade pip
# >>> apt install python3-pip
# 安装lxml时需要下面两个包,否则会报错
>>> apt install libxml2-dev libxslt1-dev
# 安装matplotlib时需要下面的包，否则会报错
>>> apt install libfreetype6-dev
>>> pip3 install pandas numpy matplotlib lxml xlrd xlsxwriter requests bs4 impyla sqlalchemy apscheduler pymysql py3amf

注:
sqlalchemy依赖impyla,否则会报错:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "crawl_dw_so_customerinfo.py", line 64, in main
    engine = data.create_engine()
  File "crawl_dw_so_customerinfo.py", line 15, in create_engine
    engine = sqlalchemy.create_engine('impala://{}:{}/{}'.format(self.ip, self.port, self.database))
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/__init__.py", line 435, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/langhelpers.py", line 240, in load
    "Can't load plugin: %s:%s" % (self.group, name)
sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:impala

如果无法安装impyla,错误中有:

xxxxxxxxxx: fatal error: Python.h: No such file or directory

则先安装python3-dev
如果要使用pyhanlp,则需要安装OpenJDK

apt install openjdk-8-jre
pip3 install pyhanlp

• 设置时区

Ubuntu server默认没有设置时区,会导致无法使用apscheduler进行定时运行程序,安装tzdata设置时区:

>>> apt install tzdata
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  tzdata
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 190 kB of archives.
After this operation, 3,106 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main armhf tzdata all 2019b-0ubuntu0.18.04 [190 kB]
Fetched 190 kB in 2s (102 kB/s)                    
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package tzdata.
(Reading database ... 21572 files and directories currently installed.)
Preparing to unpack .../tzdata_2019b-0ubuntu0.18.04_all.deb ...
Unpacking tzdata (2019b-0ubuntu0.18.04) ...
Setting up tzdata (2019b-0ubuntu0.18.04) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
Configuring tzdata
------------------

Please select the geographic area in which you live. Subsequent configuration questions will narrow this down by presenting a list of cities, representing the time zones in which they are located.

  1. Africa  2. America  3. Antarctica  4. Australia  5. Arctic Ocean  6. Asia  7. Atlantic Ocean  8. Europe  9. Indian Ocean  10. Pacific Ocean  11. System V timezones  12. US  13. None of the above
Geographic area: 6

Please select the city or region corresponding to your time zone.

  1. Aden      9. Baghdad   17. Chita       25. Dushanbe          33. Irkutsk    41. Kashgar       49. Macau         57. Omsk        65. Rangoon        73. Taipei    81. Ujung Pandang  89. Yekaterinburg
  2. Almaty    10. Bahrain  18. Choibalsan  26. Famagusta         34. Istanbul   42. Katmandu      50. Magadan       58. Oral        66. Riyadh         74. Tashkent  82. Ulaanbaatar    90. Yerevan
  3. Amman     11. Baku     19. Chongqing   27. Gaza              35. Jakarta    43. Khandyga      51. Makassar      59. Phnom Penh  67. Sakhalin       75. Tbilisi   83. Urumqi
  4. Anadyr    12. Bangkok  20. Colombo     28. Harbin            36. Jayapura   44. Kolkata       52. Manila        60. Pontianak   68. Samarkand      76. Tehran    84. Ust-Nera
  5. Aqtau     13. Barnaul  21. Damascus    29. Hebron            37. Jerusalem  45. Krasnoyarsk   53. Muscat        61. Pyongyang   69. Seoul          77. Tel Aviv  85. Vientiane
  6. Aqtobe    14. Beirut   22. Dhaka       30. Ho Chi Minh City  38. Kabul      46. Kuala Lumpur  54. Nicosia       62. Qatar       70. Shanghai       78. Thimphu   86. Vladivostok
  7. Ashgabat  15. Bishkek  23. Dili        31. Hong Kong         39. Kamchatka  47. Kuching       55. Novokuznetsk  63. Qostanay    71. Singapore      79. Tokyo     87. Yakutsk
  8. Atyrau    16. Brunei   24. Dubai       32. Hovd              40. Karachi    48. Kuwait        56. Novosibirsk   64. Qyzylorda   72. Srednekolymsk  80. Tomsk     88. Yangon
Time zone: 70


Current default time zone: 'Asia/Shanghai'
Local time is now:      Tue Sep 24 17:06:40 CST 2019.
Universal Time is now:  Tue Sep 24 09:06:40 UTC 2019.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

---

2019-09-28 更新

• 设置双网卡并开机自动执行

>>> ip route add 10.0.0.0/8 via 10.x.x.x
>>> route delete default gw 10.x.x.x enp3s0

机器重新启动后无法生效，通过将上面的命令添加至rc.local开机自动运行

>>> sudo cp /lib/systemd/system/rc-local.service /etc/systemd/system/rc-local.service
>>> sudo vim /etc/rc.local
#!/bin/sh -e

ip route add 10.0.0.0/8 via 10.x.x.x
route delete default gw 10.x.x.x enp3s0

exit 0
>>> sudo chmod a+x /etc/rc.local
>>> sudo systemctl enable rc-local
>>> sudo reboot now
# 启动
>>> sudo systemctl start rc-local.service
# 查看启动日志
>>> sudo systemctl status rc-local.service

参考

如果在’sudo systemctl enable rc-local’后出现:

The unit files have no installation config (WantedBy, RequiredBy, Also, Alias
settings in the [Install] section, and DefaultInstance for template units).
This means they are not meant to be enabled using systemctl.
Possible reasons for having this kind of units are:
1) A unit may be statically enabled by being symlinked from another unit's
   .wants/ or .requires/ directory.
2) A unit's purpose may be to act as a helper for some other unit which has
   a requirement dependency on it.
3) A unit may be started when needed via activation (socket, path, timer,
   D-Bus, udev, scripted systemctl call, ...).
4) In case of template units, the unit is meant to be enabled with some
   instance name specified.

则

>>> sudo vim /etc/systemd/system/rc-local.service
...
[Install]
WantedBy=multi-user.target

参考

如果每次重启后报错:

● rc-local.service - /etc/rc.local Compatibility
   Loaded: loaded (/etc/systemd/system/rc-local.service; enabled; vendor preset: enabled)
  Drop-In: /usr/lib/systemd/system/rc-local.service.d
           └─debian.conf
   Active: failed (Result: exit-code) since Sat 2019-12-21 03:25:33 UTC; 3min 3s ago
     Docs: man:systemd-rc-local-generator(8)
  Process: 1249 ExecStart=/etc/rc.local start (code=exited, status=2)

Dec 21 03:25:33 ubuntu systemd[1]: Starting /etc/rc.local Compatibility...
Dec 21 03:25:33 ubuntu rc.local[1249]: Error: Nexthop has invalid gateway.
Dec 21 03:25:33 ubuntu systemd[1]: rc-local.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Dec 21 03:25:33 ubuntu systemd[1]: rc-local.service: Failed with result 'exit-code'.
Dec 21 03:25:33 ubuntu systemd[1]: Failed to start /etc/rc.local Compatibility.

则可能是由于网络还没启动完全就执行那两条命令,因而用sleep延迟一会儿再执行.

>>> sudo vim /etc/rc.local
#!/bin/sh -e

sleep 1
ip route add 10.0.0.0/8 via 10.8.15.1
route delete default gw 10.8.15.1 eth1

exit 0

---

2019-10-02更新

• 使用中科大源

>>> vim /etc/apt/sources.list
# 将 http://ports.ubuntu.com/ 全部替换为 http://mirrors.ustc.edu.cn/

---

2019-11-18更新

• 配置双网卡

# 显示所有网卡
>>> ifconfig -a
# 启用xxxx网卡
>>> sudo ifconfig xxxx up
# 网卡配置
>>> sudo vim /etc/netplan/50-cloud-init.yaml
network:
    ethernets:
        eth0:
            addresses:
            - 192.168.1.202/24
            gateway4: 192.168.1.1
            nameservers:
                addresses: [8.8.8.8, 8.8.4.4]
            optional: true
        eth1:
            addresses:
            - 10.8.15.52/24
            optional: true
    version: 2
# 使其生效
>>> sudo netplan apply

参考：https://www.hi-linux.com/posts/49513.html

• 设置ssh root用户登录

>>> sudo passwd root
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully
>>> sudo sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
>>> sudo service ssh restart

参考

• 修改username、hostname

# 使用root账号登录
>>> vim /etc/hostname
>>> usermod -l <newname> -d /home/<newname> -m <oldname>
>>> sudo reboot now

参考

• 安装docker (以下方法会安装edge版，而非stable版，不建议使用)

安装

>>> curl -sSL https://get.docker.com | sh

配置权限

>>> cat /etc/group | grep docker # 查找 docker 组，确认其是否存在
>>> groups # 列出自己的用户组，确认自己在不在 docker 组中
# 如果 docker 组不存在，则添加之：
>>> sudo groupadd docker
# 将当前用户添加到 docker 组
>>> sudo gpasswd -a ${USER} docker
# 重启机器
>>> sudo reboot now

---

2019-12-29更新

Kerberos认证依赖库

impala使用Kerberos认证，则需要安装库

>>> apt install libsasl2-dev libsasl2-2 libsasl2-modules-gssapi-mit
>>> pip3 install cython
>>> pip3 install thrift_sasl==0.2.1

如果不安装cython则在安装thrift_sasl时会报错

    ...
    thriftpy/transport/cybase.c:3191:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
     3191 |     tstate->exc_value = value;
          |             ^~~~~~~~~
          |             curexc_value
    thriftpy/transport/cybase.c:3192:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
     3192 |     tstate->exc_traceback = tb;
          |             ^~~~~~~~~~~~~
          |             curexc_traceback
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

若thift_sasl不为0.2.1版，则有可能报错：

AttributeError: 'TSocket' object has no attribute 'isOpen'

参考：
https://github.com/cloudera/impyla/issues/268#issuecomment-522853969
https://github.com/cloudera/impyla/issues/149#issuecomment-398270409
https://github.com/cloudera/impyla/issues/352#issuecomment-540530826

2023-07-25更新

经过验证，目前连接Kerberos认证的impala已不需要安装如下依赖

libsasl2-dev libsasl2-2 libsasl2-modules-gssapi-mit
cython
thrift
thrift_sasl

只需要

>>> pip3 install impyla
>>> pip3 install thriftpy2

---

2020-01-06更新

安装jupyterhub

>>> apt install nodejs
>>> apt install npm
>>> apt install libffi-dev
>>> npm install -g configurable-http-proxy
>>> pip3 install notebook
>>> python3 -m pip install jupyterhub
>>> pip3 install --upgrade cython
>>> pip3 install --upgrade jupyterthemes
>>> jt -t oceans16 -f fira -fs 13 -cellw 90% -ofs 11 -dfs 11 -T
>>> cd /etc
>>> mkdir jupyterhub
>>> cd jupyterhub
>>> jupyterhub --generate-config
>>> adduser gaoxiang
>>> chgrp -R gaoxiang notebook
>>> chown -R gaoxiang notebook

参考：https://blog.cequn.wang/2018/03/10/%E8%85%BE%E8%AE%AF%E4%BA%91-jupyter-notebook%E5%92%8Cjupyterhub/

---

2020-01-13更新

pip切换源、更新pip包

使用apt install python3-pip安装的pip3无法使用

pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

因而需要重新安装
因而需要升级至最新版本

# >>> python3 -m pip install --upgrade pip
# >>> apt-get remove python3-pip
# >>> apt autoremove
>>> pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple -U pip

更新全部pip包

>>> pip3 freeze --local | grep -v '^-e' | cut -d = -f 1  | xargs -n1 pip3 install -U

如果升级后thrift_sasl自动变为0.3.0，则需要降为0.2.1版

>>> pip3 install thrift_sasl==0.2.1

---

2020-04-26更新

thrift-sasl 0.4.0+版已经修复问题，不再需要指定安装0.2.1版本！

>>> pip3 install thrift_sasl

参考：https://github.com/cloudera/impyla/issues/268#issuecomment-578007708

---

2020-05-07更新

• 安装docker
  https://blog.csdn.net/MacwinWin/article/details/103643877

---

2020-06-11更新

• 使用中科大源

方式1：

>>> vim /etc/apt/sources.list
# 将 http://ports.ubuntu.com/ 全部替换为 http://mirrors.ustc.edu.cn/

方式2（树莓派）

>>> sudo sed -i 's/ports.ubuntu.com/mirrors.ustc.edu.cn/g' /etc/apt/sources.list

• 使用清华源

sed -i 's/archive.ubuntu.com/mirrors.tuna.tsinghua.edu.cn/g' /etc/apt/sources.list

---

2020-06-16更新

• zsh下docker命令自动补全

>>> vim ~/.zshrc
plugins=(git docker docker-compose)
>>> source $ZSH/oh-my-zsh.sh

---

2020-07-01更新

om my zsh没有正常启动

如果出现oh my zsh没用启动则：

>>> sudo vim ~/.zshrc
# 结尾添加
source $ZSH/oh-my-zsh.sh
>>> source ~/.zshrc

---

2020-08-01更新

安装docker-compose

树莓派安装

https://blog.csdn.net/MacwinWin/article/details/107726017

Linux服务器安装

官方给的地址下载docker-compose在下载速度非常慢，尤其是阿里云ECS上

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   638  100   638    0     0    492      0  0:00:01  0:00:01 --:--:--   492
  0 11.6M    0  118k    0     0   1290      0  2:38:19  0:01:34  2:36:45     0

所以要换成国内链接，注意版本号要改成最新的

>>> curl -L https://get.daocloud.io/docker/compose/releases/download/1.26.2/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   423  100   423    0     0     72      0  0:00:05  0:00:05 --:--:--   117
100 11.6M  100 11.6M    0     0  1341k      0  0:00:08  0:00:08 --:--:-- 4709k
>>> sudo chmod +x /usr/local/bin/docker-compose
>>> sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

2021-08-03

thrift-sasl 0.4.3版似乎已经修复问题，不再需要指定安装0.2.1版本!

>>> pip3 install thrift_sasl