Presentation: pip and virtualenv
I put together a walkthrough of pip and virtualenv for the django-district July meeting. It was a slide-less presentation, focusing mostly on talking points and command-line walkthroughs. This post provides a recap of what I covered.
virtualenv is a tool to create isolated Python environments. It was created by Ian Bicking to provide a solution to a consistent problem.
To illustrate the problem let’s pretend you have a Web server with two Web sites on it: mysite.com and anothersite.com. Each of your Web sites are sharing the global site-packages directory in /usr/lib/python2.x/site-packages.
The global site-packages directory has Django 1.0.2, Pinax 0.5 and PIL 1.1.6 among other things. The sites are purring along, living in complete harmony.
One day, the client for anothersite.com decides they’d like to take advantage of some of the new features in Django 1.1. Now we have a problem. How do we upgrade anothersite.com’s dependencies without affecting mysite.com?
anothersite.com and mysite.com need the ability to have their own set of dependencies. The way to achieve this is to create isolated Python environments using virtualenv.
virtualenv creates an environment that has its own installation directories and isolates itself from other virtual environments. The environment contains a site-packages directory, installs setuptools and a Python interpreter that is aware of its environment.
This allows us to install a different set of dependencies for each of our sites. Now anothersite.com can upgrade to Django 1.1 and Pinax trunk without affecting mysite.com at all.
virtualenv is available on PyPI. Run the following command to install:
Working with virtualenv
While I could go into detail about the commands available with virtualenv, I honestly never use the virtualenv commands directly. I use suite of conveniences for virtualenv called virtualenvwrapper.
Doug Hellmann was kind enough to create virtualenvwrapper, which is best described on PyPI:
virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.
I put together a quick screencast to go over the wrappers and hooks that virtualenvwrapper provides as stated below. You can also view the source code on Bitbucket. Doug’s code is well commented and I found out about several of these wrappers only after taking a look at the source.
The wrappers provided by virtualenvwrapper (that I know of) are:
- mkvirtualenv (create a new virtualenv)
- rmvirtualenv (remove an existing virtualenv)
- workon (change the current virtualenv)
- add2virtualenv (add external packages in a .pth file to current virtualenv)
- cdsitepackages (cd into the site-packages directory of current virtualenv)
- cdvirtualenv (cd into the root of the current virtualenv)
- deactivate (deactivate virtualenv, which calls several hooks)
One of the coolest things about virtualenvwrapper is the ability to provide hooks when an event occurs. Hook files can be placed in ENV/bin/ and are simply plain-text files with shell commands. virtualenvwrapper provides the following hooks:
virtualenvwrapper is available on PyPI. Run the following command to install:
Add two lines to your .bashrc to set the location where the virtual environments should live and the location of the script installed with this package:
export WORKON_HOME=$HOME/.virtualenvs source /usr/local/bin/virtualenvwrapper_bashrc
You should also take a look at Doug’s blog post on virtualenvwrapper, which provides a great overview of why he created virtualenvwrapper and provides detailed setup instructions.
Another tool created by Ian Bicking, pip stands for pip installs Python packages. It is a replacement for easy_install that provides some great improvements including requirements files and support for version control systems.
Requirements files are plain text files that contain a list of packages to be installed. These text files allow you to create repeatable installations. As illustrated in this example file, there are several ways to specify a required package.
Package name with version requirements:
Direct URL to a tarball containing a setup.py script:
Editable checkouts from VCS repositories:
-e svn+http://svn.myproject.org/svn/MyProject/trunk#egg=MyProject -e git+http://git.myproject.org/MyProject/#egg=MyProject -e hg+ssh://firstname.lastname@example.org/MyProject/#egg=MyProject -e bzr+https://bzr.myproject.org/MyProject/trunk#egg=MyProject
Editable checkouts from VCS repositories with revision information:
-e svn+http://svn.myproject.org/svn/MyProject/trunk@2019#egg=MyProject -e git://git.myproject.org/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709#egg=MyProject -e hg+http://hg.myproject.org/MyProject/@2019#egg=MyProject -e bzr+https://bzr.myproject.org/MyProject/trunk/@2019#egg=MyProject
View my sample requirements file for more examples of checking out with revision information including branches, tags and dates.
Once you have a requirements file, Installing a package using pip is as simple as the following command:
pip install -r /path/to/requirements.txt
I should note that you don’t have to create a requirements file. You can simply issue any of the commands listed in the requirements file directly from the command line like so:
pip install Django>=1.1 pip install http://effbot.org/downloads/Imaging-1.1.6.tar.gz pip install -e svn+http://myrepo/svn/MyApp#egg=MyApp
pip doesn’t just install packages. It also provides a few other commands, including freeze:
pip freeze > /path/to/requirements.txt
This will inspect the current environment and generate a requirements files that contains explicit version number for each of the installed packages. I generally use this as a starting point for creating a stable requirements file.
There are other commands as well. I’d encourage you to take a look at the documentation for pip to learn more about the various options.
pip is available on PyPI. Run the following command to install:
You can also check out a copy from bitbucket. Run one of the following commands:
hg clone http://bitbucket.org/ianb/pip/
Using virtualenv, virtualenvwrapper and pip together can help you create isolated, repeatable environments with ease. We use these tools on a daily basis at Discovery Creative and it has saved us countless hours in addition to providing documentation for requirements for our applications and sites.