天池大赛-flink极客挑战赛之ubuntu18.04环境配置

一、安装软件环境

ubuntun 镜像：http://mirrors.aliyun.com/ubuntu-releases/18.04/

mac执行直接安装自带的Ubuntu18.04环境，我用的是PD虚拟机

 

1、安装ssh工具

apt-get install openssh-server

测试 

2、安装net-tools

apt install net-tools

3、sudo apt install lrzsz

借助CRT工具上传文件用

4、apt install vim

 

四、工程环境

1、安装JDK

export JAVA_HOME=/usr/local/jdk1.8
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=.:${JAVA_HOME}/bin:$PATH

2、安装python3.7

wget https://www.python.org/ftp/python/3.7.6/Python-3.7.6.tgz

首先安装依赖包

sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
sudo apt install build-essential -y
sudo apt install libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev -y
sudo apt-get install zlib1g-dev

sudo apt-get install build-essential python-dev python-setuptools python-pip python-smbus -y
sudo apt-get install build-essential libncursesw5-dev libgdbm-dev libc6-dev -y
sudo apt-get install zlib1g-dev libsqlite3-dev tk-dev -y
sudo apt-get install libssl-dev openssl -y
sudo apt-get install libffi-dev -y

报错

Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.0g-2ubuntu4.1
  404  Not Found [IP: 91.189.91.38 80]
Ign:2 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-doc all 1.1.0g-2ubuntu4.1
Err:2 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-doc all 1.1.0g-2ubuntu4.1
  404  Not Found [IP: 91.189.91.38 80]

解决方案：sudo apt-get update  即可

继续安装参考上一篇文章：https://blog.csdn.net/u010848845/article/details/108126882

创建软链接

ln -s /usr/local/python37/bin/python3.7 /usr/bin/python3

ln -s /usr/local/python37/bin/pip3.7 /usr/bin/pip3

删除软连接 rm -rf /usr/bin/python3

可能报错 ‘CommandNotFound’

 cp /usr/lib/python3/dist-packages/lsb_release.py /usr/local/python37/lib/python3.7/

 

3、安装flink/kakfa/ai_flow/analytics_zoo等

pip3 install kafka-python   -i https://pypi.tuna.tsinghua.edu.cn/simple/
python3 -m pip install apache-flink==1.11.0  -i https://pypi.tuna.tsinghua.edu.cn/simple/
sudo pip3 install --user  pyproxima2-2.2.0_lite-cp37-cp37m-linux_x86_64.whl -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com
sudo pip3 install --user  ai_flow-0.1-py3-none-any.whl  -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com
 
sudo pip3 install --user  analytics_zoo_serving-0.9.0.dev0-py3-none-any.whl  -i http://mirrors.aliyun.com/pypi/simple --trusted-host mirrors.aliyun.com

4、Ubuntu 18.04 LTS 更换国内源——解决终端下载速度慢的问题

可以参考这篇文章：https://zhuanlan.zhihu.com/p/61228593

5、添加环境变量

vim /etc/profile  

export ENV_HOME=/home/parallels/apps/resource/tianchi_ai_flow 
export TASK_ID=10
export PYTHONPATH=/home/parallels/apps/resource/tianchi_ai_flow/package/python_codes
export PATH=/usr/local/python37/bin:$PATH

生效命令：source /etc/profile  

测试

root@parallels-Parallels-Virtual-Platform:~# python3
Python 3.7.6 (default, Sep 20 2020, 08:17:05) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os             
>>> import sys
/home/parallels/apps/resource/tianchi_ai_flow
>>> 
>>> 
>>> 
>>> print(os.environ['TASK_ID']) 
10
>>> data_set = '{}/data_set/'.format(os.environ['ENV_HOME'])
>>> print(data_set)
/home/parallels/apps/resource/tianchi_ai_flow/data_set/
>>> 

 

ModuleNotFoundError: No module named '_sqlite3'

sudo apt-get install libsqlite3-dev

进入python3所在目录重新编译,就是有configure那个文件夹
sudo ./configure
sudo make
sudo makeinstall

python3 kafka-source.py 

ModuleNotFoundError: No module named '_bz2'

mv  /usr/local/python37/lib/python3.7/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so /usr/local/python37/lib/python3.7/lib-dynload/_bz2.cpython-37m-x86_64-linux-gnu.so  

在查阅资料后发现是虚拟机中安装的Python3.7没有_bz2.cpython-37m-x86_64-linux-gnu.so这个文件，但是可以在3.6版本中找到该文件。

随后笔者在Python3.6的安装目录中找到了_bz2.cpython-36m-x86_64-linux-gnu.so，然后通过指令将这个文件copy到了3.7.4的对应目录中，然后将_bz2.cpython-36m-x86_64-linux-gnu.s重命名为_bz2.cpython-37m-x86_64-linux-gnu.so后，再次运行代码就不提示错误了。

参考：https://www.cnblogs.com/ikww/p/11544020.html

 

 

又报错：ModuleNotFoundError: No module named 'yaml'

 pip3 install pyyaml

File "/root/.local/lib/python3.7/site-packages/kafka/producer/simple.py", line 54
    return '<SimpleProducer batch=%s>' % self.async
                                                  ^

因为py3.7里面async已经变成了关键字。所以导致了不兼容。
 
解决办法：
使用最新的kafka版本，但是pyPI上的kafka还没有被替换成最新的，可以使用下面的方法升级kafka python

pip3 uninstall kafka

pip3 install kafka-python

ModuleNotFoundError: No module named 'zoo'
root@parallels-Parallels-Virtual-Platform:/home/parallels/apps/resource/tianchi_ai_flow/package/python_codes# pip3 install zoo

 

ModuleNotFoundError: No module named 'zoo'

 find / -name 'zoo' 找到zoo的路径

/tmp/pip-install-4pwmbiki/analytics-zoo-serving/zoo

要用analytics-zoo-serving的zoo包

cp -r zoo/ /usr/local/python37/lib/python3.7/site-packages/

ModuleNotFoundError: No module named 'redis'

pip3 install redis

ModuleNotFoundError: No module named 'cv2'

pip3 install opencv-python -i https://pypi.tuna.tsinghua.edu.cn/simple/ (这种安装会卡死)

pip3  install opencv-python==4.2.0.34 -i https://pypi.tuna.tsinghua.edu.cn/simple/

ModuleNotFoundError: No module named 'httpx'
 pip3  install httpx -i https://pypi.tuna.tsinghua.edu.cn/simple/ 

 

启动 /root/.local/bin/cluster-serving-start

 

在unbuntu 安装了一个idea

最后发现 ai_flow_master.py报下面的问题

原来是系统时区问题

执行dpkg-reconfigure tzdata命令，将linux时区修改为Asia/Shanghai

root@parallels-Parallels-Virtual-Platform:/home/parallels/apps/resource/tianchi_ai_flow_guanfan/package/python_codes/index#  python3 -i
Python 3.7.6 (default, Sep 23 2020, 20:03:36) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tzlocal import get_localzone
>>> get_localzone()
<DstTzInfo 'Asia/Shanghai' LMT+8:06:00 STD>
>>> 

待续。。。