使用Docker简化离线Python部署

In cases when a production server does not have access to the Internet or to the internal network, you will need to bundle up the Python dependencies (as wheel files) and interpreter along with the source code.

如果生产服务器无法访问Internet或内部网络，则需要将Python依赖项（如wheel文件）和解释器与源代码捆绑在一起。

This post looks at how to package up a Python project for distribution internally on a machine cut off from the Internet using Docker.

这篇文章探讨了如何打包一个Python项目，以便在内部使用Docker与Internet隔离的机器上内部分发。

目标 (Objectives)

By the end of this post, you will be able to…

在这篇文章的结尾，您将能够...

1. Describe the difference between a Python wheel and egg
2. Explain why you may want to build Python wheel files within a Docker container
3. Spin up a custom environment for building Python wheels using Docker
4. Bundle and deploy a Python project to an environment without access to the Internet
5. Explain how this deployment setup can be considered immutable

1. 描述Python轮子和鸡蛋之间的区别
2. 解释为什么您可能要在Docker容器中构建Python wheel文件
3. 启动自定义环境以使用Docker构建Python轮子
4. 将Python项目捆绑并部署到无需访问Internet的环境
5. 解释如何将该部署设置视为不变的

情境 (Scenario)

The genesis for this post came from a scenario where I had to distribute a legacy Python 2.7 Flask app to a Centos 5 box that did not have access to the Internet due to security reasons.

这篇文章的起源来自一个场景，我不得不将一个旧版Python 2.7 Flask应用程序分发到由于安全原因而无法访问Internet的Centos 5盒子。

Python wheels (rather than eggs) are the way to go here.

Python轮子（而不是鸡蛋）是到达这里的方法。

Python wheel files are similar to eggs in that they are both just zip archives used for distributing code. Wheels differ in that they are installable but not executable. They are also pre-compiled, which saves the user from having to build the packages themselves; and, thus, speeds up the installation process. Think of them as lighter, pre-compiled versions of Python eggs. They’re particularly great for packages that need to be compiled, like lxml or NumPy.

Python wheel文件与egg类似，它们都是用于分发代码的zip存档。 轮子的区别在于它们是可安装的，但不能执行。 它们也经过预编译，从而使用户不必自己构建软件包。 从而加快了安装过程。 可以将它们视为更轻巧的Python鸡蛋的预编译版本。 它们对于需要编译的软件包特别有用 ，例如lxml或NumPy 。

For more on Python wheels, check out Python on Wheels and The Story of Wheel.

有关Python轮子的更多信息，请查看Python on Wheels和The Story of Wheel 。

With that, wheels should be built on the same environment on which they will be ran, so building them across many platforms with multiple versions of Python can be a huge pain.

这样一来，应该在运行轮子的环境中构建轮子，因此在使用多个版本的Python的多个平台上构建轮子可能会非常麻烦。

This is where Docker comes into play.

这就是Docker发挥作用的地方。

束 (Bundle)

Before beginning, it’s important to note that we will be using Docker simply to spin up an environment for building the wheels. In other words, we’ll be using Docker as a build tool rather than as a deploy environment.

在开始之前，请务必注意，我们将仅使用Docker来提升构建轮子的环境。 换句话说，我们将使用Docker作为构建工具，而不是部署环境。

Also, keep in mind that this process is not just for legacy apps – it can be used for any Python application.

另外，请记住，此过程不仅适用于旧版应用程序，还可以用于任何Python应用程序。

Stack:

堆栈：

• OS: Centos 5.11
• Python version: 2.7
• App: Flask
• WSGI: gunicorn
• Web server: Nginx

• 作业系统：Centos 5.11
• Python版本：2.7
• 应用：烧瓶
• WSGI：Gunicorn
• Web服务器：Nginx

Want a challenge? Replace one of the pieces from the above stack. Use Python 3.6 or perhaps a different version of Centos, for example.

想要挑战吗？ 替换上面堆栈中的一块。 例如，使用Python 3.6或其他版本的Centos。

If you’d like to follow along, clone down the base repo:

如果您想继续，请克隆基本存储库：

 $ git clone git@github.com:testdrivenio/python-docker-wheel.git
$ git clone git@github.com:testdrivenio/python-docker-wheel.git
$ $ cd python-docker-wheel
cd python-docker-wheel

Again, we need to bundle the application code along with the Python interpreter and dependency wheel files. cd into the “deploy” directory and then run:

同样，我们需要将应用程序代码与Python解释器和依赖项滚轮文件捆绑在一起。 cd进入“ deploy”目录，然后运行：

Review the deploy/build_tarball.sh script, taking note of the code comments:

回顾一下deploy / build_tarball.sh脚本，并注意以下代码注释：

 #!/bin/bash

#!/bin/bash

USAGE_STRINGUSAGE_STRING == "USAGE: build_tarball.sh {VERSION_TAG}"

"USAGE: build_tarball.sh {VERSION_TAG}"

VERSIONVERSION == $1
$1
if if [ -z [ -z "" ${${ VERSIONVERSION }} " " ]] ; ; then
    then
    echo echo "ERROR: Need a version number!" >"ERROR: Need a version number!" > && 2
    2
    echo echo "" ${${ USAGE_STRINGUSAGE_STRING }} " >" > && 2
    2
    exit exit 1
1
fi

fi

# Variables
# Variables
WORK_DIRECTORYWORK_DIRECTORY =app-v= app-v "" ${${ VERSIONVERSION }} "
"
TARBALL_FILETARBALL_FILE == "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} ".tar.gz

" .tar.gz

# Create working directory
# Create working directory
if if [ -d [ -d "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} " " ]] ; ; then
    rm -rf then
    rm -rf "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "/
" /
fi
mkdir fi
mkdir "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "

"

# Cleanup tarball file
# Cleanup tarball file
if if [ -f [ -f "wheels/wheels" "wheels/wheels" ]] ; ; then
    rm then
    rm "" ${${ TARBALL_FILETARBALL_FILE }} "
"
fi

fi

# Cleanup wheels
# Cleanup wheels
if if [ -f [ -f "" ${${ TARBALL_FILETARBALL_FILE }} " " ]] ; ; then
    rm -rf then
    rm -rf "wheels/wheels"
"wheels/wheels"
fi
mkdir fi
mkdir "wheels/wheels"

"wheels/wheels"

# Copy app files to the working directory
cp -a ../project/app.py ../project/requirements.txt ../project/run.sh ../project/test.py # Copy app files to the working directory
cp -a ../project/app.py ../project/requirements.txt ../project/run.sh ../project/test.py "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "/

" /

# remove .DS_Store and .pyc files
find # remove .DS_Store and .pyc files
find "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} " -type f -name " -type f -name '*.pyc' -delete
find '*.pyc' -delete
find "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} " -type f -name " -type f -name '*.DS_Store' -delete

'*.DS_Store' -delete

# Add wheel files
cp ./# Add wheel files
cp ./ "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "/requirements.txt ./wheels/requirements.txt
" /requirements.txt ./wheels/requirements.txt
cd wheels
docker build -t docker-python-wheel .
docker run --rm -v cd wheels
docker build -t docker-python-wheel .
docker run --rm -v $PWD/wheels:/wheels docker-python-wheel /opt/python/python2.7/bin/python -m pip wheel --wheel-dir$PWD /wheels:/wheels docker-python-wheel /opt/python/python2.7/bin/python -m pip wheel --wheel-dir =/wheels -r requirements.txt
mkdir ../= /wheels -r requirements.txt
mkdir ../ "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "/wheels
cp -a ./wheels/. ../" /wheels
cp -a ./wheels/. ../ "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "/wheels/
" /wheels/
cd ..

cd ..

# Add python interpreter
cp ./Python-2.7.14.tar.xz ./# Add python interpreter
cp ./Python-2.7.14.tar.xz ./ ${${ WORK_DIRECTORYWORK_DIRECTORY }/
cp ./get-pip.py ./} /
cp ./get-pip.py ./ ${${ WORK_DIRECTORYWORK_DIRECTORY }/

} /

# Make tarball
tar -cvzf # Make tarball
tar -cvzf "" ${${ TARBALL_FILETARBALL_FILE }} " " "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "/

" /

# Cleanup working directory
rm -rf # Cleanup working directory
rm -rf "" ${${ WORK_DIRECTORYWORK_DIRECTORY }} "/
" /

Here, we:

在这里，我们：

1. Created a temporary working directory
2. Copied over the application files to that directory, removing any .pyc and .DS_Store files
3. Built (using Docker) and copied over the wheel files
4. Added the Python interpreter
5. Created a tarball, ready for deployment

1. 创建一个临时工作目录
2. 将应用程序文件复制到该目录，删除所有.pyc和.DS_Store文件
3. 内置（使用Docker）并复制到wheel文件中
4. 添加了Python解释器
5. 创建一个tarball，准备部署

Then, take note of the Dockerfile within the “wheels” directory:

然后，记下“ wheels”目录中的Dockerfile：

After extending from the base Centos 5.11 image, we configured a Python 2.7.14 environment, and then generated the wheel files based on the list of dependencies found in the requirements file.

从基本的Centos 5.11映像扩展后，我们配置了Python 2.7.14环境，然后根据在需求文件中找到的依赖项列表生成了wheel文件。

Here’s a quick video in case you missed any of that:

如果您错过了以下任何视频，请看以下快速视频：

With that, let’s configure a server for deployment.

这样，让我们​​为部署配置服务器。

环境设定 (Environment Setup)

We will be downloading and installing dependencies through the network in this section. Assume that you normally will not need to set up the server itself; it should already be pre-configured.

在本节中，我们将通过网络下载并安装依赖项。 假设您通常不需要设置服务器本身； 它应该已经预先配置。

Since the wheels were built on a Centos 5.11 environment, they should work on nearly any Linux environment. So, again, if you’d like to follow along, spin up a Digital Ocean droplet with the latest version of Centos.

由于这些轮子是在Centos 5.11环境下构建的，因此它们几乎可以在任何Linux环境下工作。 因此，再次，如果您想继续，请使用最新版本的Centos来启动Digital Ocean小滴。

Review PEP 513 for more information on building broadly compatible Linux wheels (manylinux1).

有关构建广泛兼容的Linux轮子（ manylinux1 ）的更多信息，请查阅PEP 513 。

SSH into the box, as a root user, and add the dependencies necessary for installing Python before continuing with this tutorial:

以root用户身份使用SSH进入框中，并在继续本教程之前添加安装Python所需的依赖项：

Next, install and then run Nginx:

接下来，安装并运行Nginx：

 $ yum -y install $ yum -y install 
    epel-release 
    epel-release 
    nginx

    nginx
$ sudo /etc/init.d/nginx start
$ sudo /etc/init.d/nginx start

Navigate to the server’s IP address in your browser. You should see the default Nginx test page.

在浏览器中导航到服务器的IP地址。 您应该看到默认的Nginx测试页面。

Next, update the Nginx config in /etc/nginx/conf.d/default.conf to redirect traffic:

接下来，更新/etc/nginx/conf.d/default.conf中的Nginx配置以重定向流量：

Restart Nginx:

重新启动Nginx：

 $ service nginx restart
$ service nginx restart

You should now see a 502 error in the browser.

现在，您应该在浏览器中看到502错误。

Create a regular user on the box:

在方框上创建一个普通用户：

Exit the environment when done.

完成后退出环境。

部署 (Deploy)

To deploy, first manually secure copy over the tarball along with with the setup script, setup.sh, to the remote box:

要进行部署，请首先将压缩包中的副本以及设置脚本setup.sh手动安全地保存到远程框中：

 $ scp app-v20180119.tar.gz <username>@<host-address>:/home/<username>
$ scp app-v20180119.tar.gz <username>@<host-address>:/home/<username>
$ scp setup.sh <username>@<host-address>:/home/<username>
$ scp setup.sh <username>@<host-address>:/home/<username>

Take a quick look at the setup script:

快速查看设置脚本：

This should be fairly straightforward: This script simply sets up a new Python environment and installs the dependencies within a new virtual environment.

这应该非常简单：此脚本仅设置一个新的Python环境并将依赖项安装在一个新的虚拟环境中。

SSH into the box and run the setup script:

SSH输入框并运行设置脚本：

 $ ssh <username>@<host-address>
$ ssh <username>@<host-address>
$ sh setup.sh $ sh setup.sh 20180119 <username>
20180119 <username>

This will take a few minutes. Once done, cd into the app directory and activate the virtual environment:

这将需要几分钟。 完成后， cd进入应用程序目录并激活虚拟环境：

Run the tests:

运行测试：

 $ python test.py
$ python test.py

Once complete, fire up gunicorn as a daemon:

完成后，启动gunicorn作为守护程序：

Feel free to use a process manager, like Supervisor, to manage gunicorn.

随意使用诸如Supervisor之类的流程管理器来管理gunicorn。

Again, check out the video to see the script in action!

同样，请观看视频以查看脚本的运行！

结论 (Conclusion)

In this article we looked at how to package up a Python project with Docker and Python wheels for deployment on a machine cut off from the Internet.

在本文中，我们研究了如何使用Docker和Python轮子打包Python项目，以将其部署在与Internet断开连接的计算机上。

With this setup, since we’re packaging the code, dependencies, and interpreter up, our deployments are considered immutable. For each new deploy, we’ll spin up a new environment and test to ensure it’s working before bringing down the old environment. This will eliminate any errors or issues that could arise from continuing to deploy on top of legacy code. Plus, if you uncover issues with the new deploy you can easily rollback.

通过此设置，由于我们将代码，依赖项和解释器打包在一起，因此我们的部署被认为是不可变的。 对于每个新的部署，我们将启动一个新环境并进行测试以确保它能够正常运行，然后再关闭旧环境。 这将消除继续在旧代码之上部署可能引起的任何错误或问题。 另外，如果您发现新部署的问题，则可以轻松回滚。

Looking for some challenges?

寻找一些挑战？

1. At this point, the Dockerfile and each of the scripts are tied to a Python 2.7.14 environment on Centos 5.11. What if you also had to deploy a Python 3.6.1 version to a different version of Centos? Think about how you could automate this process given a configuration file.

   For example:

   Alternatively, check out the cibuildwheel project, for managing the building of wheel files.

2. You probably only need to bundle the Python interpreter for the first deploy. Update the build_tarball.sh script so that it asks the user whether Python is needed before bundling it.

3. How about logs? Logging could be handled either locally or at the system-level. If locally, how would you handle log rotation? Configure this on your own.

1. 此时，Dockerfile和每个脚本都已与Centos 5.11上的Python 2.7.14环境绑定在一起。 如果还必须将Python 3.6.1版本部署到其他版本的Centos怎么办？ 考虑一下在给定配置文件的情况下如何自动执行此过程。

   例如：

    [
     {
       "os" : "centos" ,
       "version" : "5.11" ,
       "bit" : "64" ,
       "python" : [ "2.7.14" ]
     },
     {
       "os" : "centos" ,
       "version" : "7.40" ,
       "bit" : "64" ,
       "python" : [ "2.7.14" , "3.6.1" ]
     },
   ]
   

   或者，签出cibuildwheel项目，以管理wheel文件的构建。

2. 您可能只需要为第一次部署捆绑Python解释器。 更新build_tarball.sh脚本，以便在捆绑之前询问用户是否需要Python。

3. 日志怎么样？ 日志可以在本地或系统级别处理。 如果在本地，您将如何处理日志轮换？ 自行配置。

翻译自: https://www.pybloggers.com/2018/01/simplifying-offline-python-deployments-with-docker/