Configuring Emacs as a Python IDE

http://pedrokroger.net/configuring-emacs-python-ide/

Emacs is a huge beast. It can read email, play tetris, act as a file manager,display google maps, and even edit videos. It has support for many, many programming languages and has many features to programming. Unfortunately, emacs doesn’t have a full programming environment for python out-of-the-box.

In this post I’ll show how to configure emacs to write Python programs. We want to have not only basics things like syntax highlighting, but also code completion, easy access to Python’s documentation, ability to check for common mistakes, run unit tests, debugging, and a good interactive programming environment.

This setup is based on ipython and python-mode, but it’s also possible to userope, ropemacs, and the auto complete mode as we can see here. I didn’t have much luck with ropemacs in the past but I’ll try it again in the future.

Tools installation and configuration

First we should install the tools we need.

• python-mode. There are two python modes; python.el and python-mode.el. Although python.el comes with GNU Emacs 22, I recommend you use python-mode.el since it has support for ipython. (There’s some talkabout merging the two modes, but I don’t know it’s current status). Download and install python-mode and put the following in your .emacs:[code](require ‘python-mode)
  (add-to-list ‘auto-mode-alist ‘("\.py\’" . python-mode))
  [/code]
• ipython. Ipython is a powerful python shell with advanced features and nice code completion. Check [ipython’s website]for documentation and screencasts. If you’re using debian or ubuntu, install it with:[code]sudo apt-get install ipython
  [/code]

To configure ipython, edit ~/.ipython/ipy_user_conf.py and add your options. I like to use ipy_greedycompleter so it will complete things like “foo”.TAB.

• ipython.el. It allows you to use python-mode with ipython. Download and install ipython.el and put the following in your .emacs:[code](require ‘ipython)
  [/code]
• lambda-mode. This is only for aesthetics. When you install lambda-mode.el it will display a lambda character (λ) when you type lambda:

The full name “lambda” will still be there, emacs is only displaying it differently. I like lambda-mode because it makes the code look shorter, neater, and cooler. Download and install lambda-mode.el and put the following in your .emacs:

[code](require ‘lambda-mode)
(add-hook ‘python-mode-hook #’lambda-mode 1)
[/code]

I had to set the variable lambda-symbol to the following, but YMMV:

[code](setq lambda-symbol (string (make-char ‘greek-iso8859-7 107)))
[/code]

There are other ways to have pretty lambdas and pretty symbols in emacs, check the Emacs Wiki page to see which one you prefer.

• anything. We will use anything for code completion. Download and install anything.el, anything-ipython.el, and anything-show-completion.el, and add the following to your .emacs:[code](require ‘anything) (require ‘anything-ipython)
  (when (require ‘anything-show-completion nil t)
  (use-anything-show-completion ‘anything-ipython-complete
  ‘(length initial-pattern)))
  [/code]
• Change comint keys. Comint is a minor-mode to deal with interpreter commands in an emacs buffer. It’s used by many modes, including ipython. Comint uses M-p and M-nto cycle backwards and forward through input history. I prefer to use the up and down arrow keys:[code](require ‘comint)
  (define-key comint-mode-map (kbd "M-") ‘comint-next-input)
  (define-key comint-mode-map (kbd "M-") ‘comint-previous-input)
  (define-key comint-mode-map [down] ‘comint-next-matching-input-from-input)
  (define-key comint-mode-map [up] ‘comint-previous-matching-input-from-input)
  [/code]
• pylookup. We will use pylookup to search python’s documentation within emacs. First install the python documentation locally. If you use debian or ubuntu you can install it with one command:[code]apt-get install python2.6-doc
  [/code]

If the python documentation is not available as a package for your system, download it from the python website and unzip it somewhere in your computer. Then download pylookup and follow the instructions in the README file. Run pylookup.py to index the database by running:

[code]./pylookup.py -u file:///usr/share/doc/python2.6-doc/html
[/code]

This command will generate a database file (pylookup.db). Naturally, you need to put the correct path for the python documentation. Next you need to add the following to your .emacs. Again, replace the values to match your system.

[code](autoload ‘pylookup-lookup "pylookup")
(autoload ‘pylookup-update "pylookup")
(setq pylookup-program "~/.emacs.d/pylookup/pylookup.py")
(setq pylookup-db-file "~/.emacs.d/pylookup/pylookup.db")
(global-set-key "C-ch" ‘pylookup-lookup)
[/code]

• autopair. A common feature in modern editors is to insert pairs of matching elements such as parenthesis, quotes, and braces. There are many ways to do this in emacs and installing autopair.el is probably the easiest one. You can configure it to work in all modes and exclude the modes you don’t to have it activated. For instance, I have it configured to work globally, except in the lisp-mode, where I use paredit:[code](autoload ‘autopair-global-mode "autopair" nil t)
  (autopair-global-mode)
  (add-hook ‘lisp-mode-hook #'(lambda () (setq autopair-dont-activate t)))
  [/code]

I also have the following to in my .emacs to make autopair work with python single and triple quotes:

[code](add-hook ‘python-mode-hook
#'(lambda () (push ‘(?’ . ?’)
(getf autopair-extra-pairs :code))
(setq autopair-handle-action-fns
(list #’autopair-default-handle-action
#’autopair-python-triple-quote-action))))
[/code]

• pep8 and pylint. These scripts are good to check your program for style errors and common mistakes. Install pylint and pep8:apt-get install pylint pep8

and download and install python-pep8.el and python-pylint.el to integrate them with emacs and add the following to your .emacs:

[code](require ‘python-pep8)
(require ‘python-pylint)
[/code]

• delete trailing space. When you type the return key in python-mode it’ll invoke py-newline-and-indent. As the manual says, it “deletes the whitespace before point, inserts a newline, and takes an educated guess as to how you want the new line indented.” This is great and I love this feature, but one unintended consequence is that you may end up with blank lines with trailing spaces. Python doesn’t care, but the pep8 tool will complain. Put the following in your .emacs to delete the trailing spaces when saving a file:[code](add-hook ‘before-save-hook ‘delete-trailing-whitespace)
  [/code]
• ipdb package. The ipdb package makes it easy to set breakpoints in the ipdb debugger. It’s easy to install it using easy_install (duh :-P):[code]easy_install ipdb
  [/code]
• reimport package. reimport is a replacement for Python’s reloadfunction that reloads all modules and sub-modules consistently. It’s useful for long-running programs and interactive development. Install it witheasy_install:[code]sudo easy_install reimport
  [/code]
• Templates. Yasnippet is a textmate-like templates for emacs. Download Yasnippet and pet the following in your .emacs:[code](require ‘yasnippet-bundle)
  (yas/initialize)
  (yas/load-directory "~/.emacs.d/my-snippets/")
  [/code]

You can also install flymake to check the syntax on the fly. I find it distracting, so I don’t use it.

All right, these are the tools we need! Now we can go on and use these features.

Using python-mode

So now we’re ready to go. Open a python file, say test.py, type C-c ! (py-shell) to start the python interpreter. Go back to test.py (with C-x o) and type C-c C-c (pt-execute-buffer) to send the whole buffer to the interpreter. In the following picture we can see how it works; it’s neat to be able to write code in one file and test it in the interpreter without the whole edit-compile-run-test cycle:

Code completion

Code completion is very useful and I find a must to be able to complete methods for both built-in and user-defined classes. For instance, if you type “os.path.” and M-tab emacs will show all methods available in os.path:

This completion method uses anything and you can use C-n and C-p to navigate through the completion items (when you’re done type return). This works for used-defined methods and functions too:

Access to documentation

Python has excellent documentation and it’s a huge bust in productivity to be able to access the documentation without leaving the editor. By using pylookup we can type C-c h to search any term in the python documentation. For instance, if we type “os.path.basename” and then C-c h to invoke the documentation, pylookup will open the documentation page foros.path.basename using the default browser configured in emacs (usually firefox):

However, many people prefer to use w3m inside emacs to browse technical documentation. With w3m we can read the documentation without leaving emacs, making it easy to copy code snippets and not needing to switch to an external program (to much switching can kill focus). Here’s the same documentation in the above picture but using w3m this time:

If you search for something with more than one possible answer, pylookup will ask you to choose among some possibilities, for instance, if I search for “print” I’ll get many options to choose from:

Code quality

The pep8 script checks if your code follow the PEP8 style guide. You can run it inside emacs by typing M-x pep8. The key M-x ` will jump to the place in the source code where the next error occurs:

You can run pylint inside emacs with M-x pylint, it behaves in asimilar way to pep8:

Unit tests

It’s possible to run unit tests inside emacs if we use the compilation mode. Type M-x compile and replace what will appear (most likely “make -k”) with your favorite python program to run unit tests. As with pep8 and pylint (they both use the compilation mode), you can type M-x to navigate to the next error. In the following screenshot we can see thattest_square_of_100_is_1000is not correct:

Debugging

There are many ways to debug python code. I like to use ipdb, ipython’s debugger, because it has advanced features like code completion and syntax highlighting. With the -pdb flag Ipython will go directly to the debugger when an exception is raised, instead of just printing a stack trace. (you may want to edit ~/.ipython/ipythonrc to make the change permanent).

To set breakpoints in the regular python debugger, you need to add the following lines to your code:

[code]import pdb; pdb.set_trace()
[/code]

With the ipdb package we can set breakpoints in a similar way if using ipdb:

[code]import ipdb; ipdb.set_trace()
[/code]

I like to make emacs highlight the lines that set breakpoints, so I can easily visualize where the breakpoints are and remember to remove them from the code after I’m done debugging. I use the following code to accomplish that:

[code](defun annotate-pdb ()
(interactive)
(highlight-lines-matching-regexp "import pdb")
(highlight-lines-matching-regexp "pdb.set_trace()"))

(add-hook ‘python-mode-hook ‘annotate-pdb)
[/code]

Notice in the following screenshot how the line that sets the breakpoint is highlighted and the program execution stopped at the breakpoint. I used the command n (next) to advance to the next statement:

I have a function to add breakpoints mapped to C-c C-t so I can set breakpoints easily:

[code](defun python-add-breakpoint ()
(interactive)
(py-newline-and-indent)
(insert "import ipdb; ipdb.set_trace()")
(highlight-lines-matching-regexp "^[ ]*import ipdb; ipdb.set_trace()"))

(define-key py-mode-map (kbd "C-c C-t") ‘python-add-breakpoint)
[/code]

Templates

For textmate-like templates I use Yasnippet (be sure to watch its demo). With it you can define templates easily and quickly to fill things for you, like a new-style class with documentation and an __init__ method. Python doesn’t have much boilerplate, but using yasnippet can help you to write code even faster.

Finding your way in the source code

Emacs allows us to navigate through source code by using tags. When you see a function call you can jump to its definition with M-. (find-tag) and jump back with M-* (pop-tag-mark). Another useful command is tags-query-replace, to rename functions, methods, etc. To use it we need to generate a TAGS file. Emacs comes with the etags command but I recommend exuberant tags:

[code]sudo apt-get install exuberant-ctags
[/code]

Usually I put the following code in a Makefile to generate a TAGS file for a project:

[code]ctags-exuberant -e -R –languages=python –exclude="__init__.py"
[/code]

Dealing with multiple files

As we have seen, we can program in an interactive style in python-mode by sending the buffer to the python interpreter with C-c C-c. This works well for single file scripts and libraries, but no so well for more complex modules with sub-modules. For instance, if you open file4.py in the mainmodule bellow and execute it with C-c C-c it’ll fail if it depends on other submodules.

[code]mainmodule
|— __init__.py
|— submodule1
|— __init__.py
|— file1.py
|— file2.py
|— submodule2
|— __init__.py
|— file3.py
|— file4.py
[/code]

We can define a master file using the pt-master-file variable; python-mode will execute the file set in the variable instead of the current buffer’s file. This variable can be set as a file variable, but I prefer not to pollute every single freaking .py file in a module with a file variable, so I use a directory variableinstead. Another advantage of using directory variables in this case is that each member of a team can set the pt-master-file to reflect their file system layout. Create a file called .dir-locals.el in mainmodule‘s root with the following content:

[code]((python-mode . ((py-master-file . "/path/to/interactivetest.py")
(tags-file-name . "/path/to/TAGS"))))
[/code]

(Note that I also define the tags’ filename, so emacs’ll automatically load it when I read a python file located in this directory.)

To make this work I use a master file called interactivetest.py in the module’s root to re-import the module with the reimport package. This file is not really part of the module and I don’t even check it under version control. Now, every time we hit C-c C-c, regardless of what file we are changing, python-mode will execute interactivetest.py again, and update the whole module. For instance, this is what I have in the interactivetest.py for my aristoxenus library:

[code]import reimport
import aristoxenus
reimport.reimport(aristoxenus)
[/code]

But I may add a few things in the interactivetest.py file to make testing things interactively easier and to be able to save it between coding sessions:

[code]from aristoxenus.parse import humdrum
foo = humdrum.parse_string("**kernn4C")
[/code]

Of course this is not a substitute for unit tests, but it’s nice to have fooavailable in the REPL so I can inspect it and quickly see what’s going on.

Conclusion

As you can see, we can have a nice programming environment for python in emacs. It may look like a lot of work, but in reality we only need to download and configure already existing packages. However, it would be nice to have a full-fledged development environment for python like slime.

I didn’t mention many things like the emacs code browser, the git interface, and the window session manager. They are not particularly specific to python, but they contribute to turn emacs into a powerful tool for development. Maybe I’ll mention then in another post.

I intend to make a screencast in the future showing these features, so stay tuned. Meanwhile, what do you use to develop python code?