virtualenv and virtualenvwrapper

Ian Bicking’s virtualenv tool has become the de-facto standard mechanism for isolating Python environments. Its purpose is simple: if you have a number of Python projects on a single machine, each with different dependencies (perhaps with dependencies on different versions of the same package), managing the dependencies in a single Python installation is nigh impossible.

virtualenv creates “virtual” Python installations, each with their own, segregated, site-packages. distribute and pip are also installed in such a way that pip install correctly installs packages to the virtualenv rather than the system Python installation. Switching back and forth between your virtualenv is a one-command process.

A separate tool, Doug Hellmann’s virtualenvwrapper, makes creating and managing multiple virtualenvs easier. Let’s go ahead and install both now:

Installation

$ sudo pip install `virtualenvwrapper`
...
Successfully installed `virtualenvwrapper` `virtualenv` `virtualenv`-clone stevedore
Cleaning up...

As you can see, the latter has a dependency on the former, so simply installing virtualenvwrapper is sufficient. Note that if you’re using Python 3, PEP-405, which gives Python native support for virtual environments through the venv package and pyvenv command, was implemented in Python 3.3. You should use that instead of the tools mentioned above.

Once you’ve installed virtualenvwrapper, you’ll need to add a line to your .zhsrc file (or .bashrc file for bash users):

$ echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.zshrc

This adds a number of useful commands to your shell (remember to source your .zshrc to actually make them available for the first time).

virtualenvwrapper commands

virtualenvwrapper_cd
virtualenvwrapper_expandpath
virtualenvwrapper_absolutepath
virtualenvwrapper_derive_workon_home
virtualenvwrapper_verify_workon_home
virtualenvwrapper_mktemp
virtualenvwrapper_tempfile
virtualenvwrapper_run_hook
virtualenvwrapper_setup_tab_completion
virtualenvwrapper_initialize
virtualenvwrapper_verify_resource
virtualenvwrapper_verify_virtualenv
virtualenvwrapper_verify_virtualenv_clone
virtualenvwrapper_verify_workon_environment
virtualenvwrapper_verify_active_environment
virtualenvwrapper_mkvirtualenv_help
mkvirtualenv
rmvirtualenv
virtualenvwrapper_show_workon_options
_lsvirtualenv_usage
lsvirtualenv
showvirtualenv
virtualenvwrapper_workon_help
workon
virtualenvwrapper_get_python_version
virtualenvwrapper_get_site_packages_dir
add2virtualenv
cdsitepackages
cdvirtualenv
lssitepackages
toggleglobalsitepackages
cpvirtualenv
virtualenvwrapper_verify_project_home
setvirtualenvproject
virtualenvwrapper_mkproject_help
mkproject
cdproject
mktmpenv
wipeenv
allvirtualenv
virtualenvwrapper

creating a project

While you can create a virtualenv directly with the mkvirtualenv command, creating a “project” using mkproject [OPTIONS] DEST_DIR is usually more useful. Since we have an existing project, however, we’ll simply create a new virtualenv for our project. We can do this with a simple command:

/tmp mkvirtualenv vulnpwn
New python executable in /Users/codeABC/.virtualenvs/vulnpwn/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/codeABC/.virtualenvs/vulnpwn/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/codeABC/.virtualenvs/vulnpwn/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/codeABC/.virtualenvs/vulnpwn/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/codeABC/.virtualenvs/vulnpwn/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/codeABC/.virtualenvs/vulnpwn/bin/get_env_details
(vulnpwn) /tmp

You’ll notice your shell prompt is now prepended by the name of your virtualenv (which I called “vulnpwn”, but obviously you can use whatever name you’d like). Now anything installed via pip install is installed to the site-packages of your virtualenv.

list projects

$ lsvirtualenv

stop working on the project

To stop working on your project and switch back to the system installation, use the deactivate command. You should see the virtualenv name that was prepended to your shell prompt disappear.

resume working on the project

To resume work on your project, run $ workon <project name> and you’ll be back in your virtualenv.

Aside from simply creating the virtualenv for your project, you’ll use it to do one more thing: generate your requirements.txt file. pip is capable of installing all of project’s dependencies by using a requirements file and the -r flag. To create this file, run the following command within your virtualenv (once your code is working with the virtualenv, that is):

(vulnpwn)$ pip freeze > requirements.txt

You’ll get a nice list of all of the requirements for your project, which can later be used by the setup.py file to list your dependencies. One note here: I often change the ‘==’ to ‘>=’ in requirements.txt to say “any version of this package after the one I’m working on.” Whether or not you should/need to do this is project specific, but I just thought I’d point it out.

Commit requirements.txt to your git repo. In addition, you can now add the packages listed there as the value for the install_requirements argument to distutils.setup in setup.py. Doing that now will ensure that, when we later upload the package to PyPI. It can be pip installed with automatically resolved dependencies.

References

1. http://virtualenv.readthedocs.io/en/stable/
2. http://virtualenvwrapper.readthedocs.io/en/latest/
3. http://jeffknupp.com/blog/2013/08/16/open-sourcing-a-python-project-the-right-way/