Python Virtual Environments

Python virtual environments - what?

Python virtual environments (venv) are a nifty little tool that allows you to create an isolated environment for your python code - and avoid potential conflicts between python applications and their dependencies.

Python virtual environments - why?

Imagine a situation when some of the python applications that you use (lets say Ansible and QuArK) both are using the same library. That is not an issue, unless both can use the same version of the library. Often however it's not the case - the older QuArK will most likely not be compatible with the newer library version that is required by Ansible. This creates a dependency conflict.

And that's when the python venv comes to the rescue!

You can use venv to separate both applications along with all the libraries they depend on - in two separate ecosystems.

It also comes very handy in situations like:

• when you need to manage some old Unix boxes with older version of python installed.
• when you need to develop applications that depend on different python versions.
• when a playbook/role you are writing uses a particular version of ansible module.

It is a good habit to create a separate venv for every single project that your are working on. Trust us , you will thank yourself later for that !

Python virtual environments - how?

Install and create virtual environment

The easiest way to install virtualenv module on python 2.7 is as usual trough pip - python package manager: pip install virtualenv. In python 3 venv tool is already present - the rest of this guide will assume that you're using python 3.

You can create new virtual environment (along with it's new directory) with:

python3 -m venv env_name

Virtual environment files will be created (missing parent directories in the path aswell). The result will look similar as below, with pip already set up to work in your virtual environment (unless venv was invoked with --without-pip option) so you don't need to explicitly command pip to install packages in the virtual environment:

root@25shmeckles:/env_name# tree -d
.
|-- bin
|-- include
|-- lib
|   `-- python3.6
|       `-- site-packages
|           |-- __pycache__
|           |-- pip
|           |   |-- __pycache__
|           |   |-- _vendor
|           |   |   `-- __pycache__
|           |   |-- commands
|           |   |   `-- __pycache__
|           |   |-- compat
|           |   |   `-- __pycache__
|           |   |-- models
|           |   |   `-- __pycache__
|           |   |-- operations
|           |   |   `-- __pycache__
|           |   |-- req
|           |   |   `-- __pycache__
|           |   |-- utils
|           |   |   `-- __pycache__
|           |   `-- vcs
|           |       `-- __pycache__
|           |-- pip-9.0.1.dist-info
|           |-- pkg_resources
|           |   |-- __pycache__
|           |   |-- _vendor
|           |   |   |-- __pycache__
|           |   |   `-- packaging
|           |   |       `-- __pycache__
|           |   `-- extern
|           |       `-- __pycache__
|           |-- pkg_resources-0.0.0.dist-info
|           |-- setuptools
|           |   |-- __pycache__
|           |   |-- _vendor
|           |   |   |-- __pycache__
|           |   |   `-- packaging
|           |   |       `-- __pycache__
|           |   |-- command
|           |   |   `-- __pycache__
|           |   `-- extern
|           |       `-- __pycache__
|           `-- setuptools-39.0.1.dist-info
|-- lib64 -> lib
`-- share
    `-- python-wheels

48 directories

As you can see the virtual environment has its own lib directory (lib64 being a link to it) which contains site-packages where the installed dependencies for this ecosystem will be placed.

Usage HowTo

At this moment you should have the virtual environment created and ready to go. Now you can activate it - which modifies your PATH env variable so the python will use the venv by default:

source env_name/bin/activate

It can be deactivated anytime (by running deactivate) and you can see the name of the venv was added to your prompt. Tip of the day: You can see all dependencies you installed in the specific virtual environment by running pip list in the activated venv.

(venv_example) ❰grzegorz.koper@pl.ibm.com❙~/gitprojects❱✔≻ pip list
Package    Version
---------- -------
pip        19.2.3
setuptools 40.8.0

Notice that the venv is "clean" only whats required for running pip is there by default. You can now install any package using pip inside your virtual environment. We will install ansible and molecule:

(venv_example) ❰grzegorz.koper@pl.ibm.com❙~/gitprojects❱✔≻ pip install ansible molecule
Collecting ansible
  Using cached https://files.pythonhosted.org/packages/04/25/48fee5f8048360d9375e01846fcf395dda58242ed1f25a2106b6794452eb/ansible-2.8.5.tar.gz
Collecting molecule
  Using cached https://files.pythonhosted.org/packages/af/be/3084cbedc051e179062cf887fd5933c1c5f1031200f62f718048f36dc604/molecule-2.22-py2.py3-none-any.whl
...

...
Running setup.py install for ansible ... done
Successfully installed MarkupSafe-1.1.1 PyYAML-5.1.2 ansible-2.8.5 ansible-lint-4.1.0 anyconfig-0.9.7 arrow-0.15.2 asn1crypto-0.24.0 aspy.yaml-1.3.0 atomicwrites-1.3.0 attrs-19.1.0 bcrypt-3.1.7 binaryornot-0.4.4 cerberus-1.3.1 certifi-2019.9.11 cffi-1.12.3 cfgv-2.0.1 chardet-3.0.4 click-7.0 click-completion-0.5.1 colorama-0.4.1 cookiecutter-1.6.0 cryptography-2.7 entrypoints-0.3 fasteners-0.15 flake8-3.7.8 future-0.17.1 git-url-parse-1.2.2 identify-1.4.7 idna-2.8 importlib-metadata-0.23 jinja2-2.10.1 jinja2-time-0.2.0 mccabe-0.6.1 molecule-2.22 monotonic-1.5 more-itertools-7.2.0 nodeenv-1.3.3 packaging-19.2 paramiko-2.6.0 pathspec-0.5.9 pbr-5.4.3 pexpect-4.7.0 pluggy-0.13.0 poyo-0.5.0 pre-commit-1.18.3 psutil-5.6.3 ptyprocess-0.6.0 py-1.8.0 pycodestyle-2.5.0 pycparser-2.19 pyflakes-2.1.1 pynacl-1.3.0 pyparsing-2.4.2 pytest-5.1.3 python-dateutil-2.8.0 python-gilt-1.2.1 requests-2.22.0 ruamel.yaml-0.16.5 ruamel.yaml.clib-0.1.2 sh-1.12.14 shellingham-1.3.1 six-1.12.0 tabulate-0.8.3 testinfra-3.2.0 toml-0.10.0 tree-format-0.1.2 urllib3-1.25.5 virtualenv-16.7.5 wcwidth-0.1.7 whichcraft-0.6.1 yamllint-1.17.0 zipp-0.6.0

With all that installed we run molecule:

(venv_example) ❰grzegorz.koper@pl.ibm.com❙~/gitprojects❱✔≻ molecule
Usage: molecule [OPTIONS] COMMAND [ARGS]...

   _____     _             _
  |     |___| |___ ___ _ _| |___
  | | | | . | | -_|  _| | | | -_|
  |_|_|_|___|_|___|___|___|_|___|

  Molecule aids in the development and testing of Ansible roles.

  Enable autocomplete issue:

    eval "$(_MOLECULE_COMPLETE=source molecule)"

Options:
  --debug / --no-debug    Enable or disable debug mode. Default is disabled.
...

After you deactivate the environment, you can see that no molecule is installed on your system, everything we did was done inside python venv!

(venv_example) ❰grzegorz.koper@pl.ibm.com❙~/gitprojects❱✔≻ deactivate
❰grzegorz.koper@pl.ibm.com❙~/gitprojects/venv_example❱✔≻ molecule
fish: Unknown command 'molecule'

Of course at any time you can activate your environment again! (There are different activation binaries prepared for different shells that you might have on your system. I myself are using fish shell in this example).

❰grzegorz.koper@pl.ibm.com❙~/gitprojects❱✔≻ source venv_example/bin/activate.fish
(venv_example) ❰grzegorz.koper@pl.ibm.com❙~/gitprojects❱✔≻ molecule
Usage: molecule [OPTIONS] COMMAND [ARGS]...

   _____     _             _
  |     |___| |___ ___ _ _| |___
  | | | | . | | -_|  _| | | | -_|
  |_|_|_|___|_|___|___|___|_|___|

  Molecule aids in the development and testing of Ansible roles.

  Enable autocomplete issue:

    eval "$(_MOLECULE_COMPLETE=source molecule)"

Options:
  --debug / --no-debug    Enable or disable debug mode. Default is disabled.
  -c, --base-config TEXT  Path to a base config.