Python for Bash scripters: A well-kept secret

http://magazine.redhat.com/2008/02/07/python-for-bash-scripters-a-well-kept-secret/

Python for Bash scripters: A well-kept secret

by Noah Gift

Hey you, ya you! Do you write Bash scripts?

Come here, I have a secret to tell you.

Python is easy to learn, and more powerful than Bash. I wasn’t supposed to tell you this–it’s supposed to be a secret. Anything more than a few lines of Bash could be done better in Python. Python is often just as portable as Bash too. Off the top of my head, I can’t think of any *NIX operating systems, that don’t include Python. Even IRIX has Python installed.

If you can write a function in Bash, or even piece together a few commands into a script and make it executable, then you can learn Python. What usually throws Bash scripters off is they see something object-oriented like this:

class FancyObjectOriented(object):
    def __init__(self, stuff = "RegularStuff"):
        self.stuff = stuff
    def printStuff(self):
        print "This method prints the %s object" % self.stuff

Object-oriented programming can be a real challenge to get the hang of, but fortunately in Python it is 100% optional. You don’t need to have a Computer Science degree to program in Python–you can get started immediately if you know a few shortcuts. My goal here is to show Average Joe Bash scripter how to write in Python some of the things they would normally write in Bash. Even though it seems unbelievable, you can be a beginning Python programmer, by the end of this article.

Baby steps

The very first thing to understand about Python, is that whitespace is significant. This can be a bit of a stumbling block for newcomers, but it will be old hat very quickly. Also, the shebang line is different than it should be in Bash:

Python Shebang Line:

#!/usr/bin/env python

Bash Shebang Line:

#!/usr/bin/env bash

Knowing these two things, we can easily create the usual ‘Hello World’ program in Python, although whitespace won’t come into play just yet. Open up your favorite text editor and call the python script, hello.py, and the bash script hello.sh.

Python Hello World script:

#!/usr/bin/env python
print "Hello World"

Bash Hello World script:

#!/usr/bin/env bash
echo Hello World

Make sure that you make each file executable by using chmod +x hello.py, and chmod +x hello.sh. Now if you run either script–./hello.py or ./hello.sh–you will get the obligatory “Hello World.”

Toddler: System calls in Python

Now that we got ‘Hello World’ out of the way, lets move on to more useful code. Typically most small Bash scripts are just a bunch of commands either chained together, or run in sequence. Because Python is also a procedural language, we can easily do the same thing. Lets take a look at a simple example.

In order to take our toddler steps it is important to remember two things:

1. Whitespace is significant. Keep this in mind–I promise we will get to it. It is so important that I want to keep reminding you!

2. A module called subprocess needs to be imported to make system calls.

It is very easy to import modules in Python. You just need to put this statement at the top of the script to import the module:

import subprocess

Lets take a look at something really easy with the subprocess module. Lets execute an ls -l of the current directory.

Python ls -l command:

#!/usr/bin/env python
import subprocess
subprocess.call("ls -l", shell=True)

If you run this script it will do the exact same thing as running ls -l in Bash. Obviously writing 2 lines of Python to do one line of Bash isn’t that efficient. But let’s run a few commands in sequence, just like we would do in Bash so you can get comfortable with how a few commands run in sequence might look. In order to do that I will need to introduce two new concepts: one for Python variables and the other for lists (known as ‘arrays’ in Bash). Lets write a very simple script that gets the status of a few important items on your system. Since we can freely mix large blocks of Bash code, we don’t have to completely convert to Python just yet. We can do it in stages. We can do this by assigning Bash commands to a variable.

Note:

If you are cutting and pasting this text, you MUST preserve the whitespace. If you are using vim you can do that by using paste mode  :set paste

PYTHON
Python runs a sequence of system commands.

#!/usr/bin/env python
import subprocess

#Note that Python is much more flexible with equal signs.  There can be spaces around equal signs.
MESSAGES = "tail /var/log/messages"
SPACE = "df -h"

#Places variables into a list/array
cmds = [MESSAGES, SPACE]

#Iterates over list, running statements for each item in the list
#Note, that whitespace is absolutely critical and that a consistent indent must be maintained for the code to work properly
count=0
for cmd in cmds:
    count+=1
    print "Running Command Number %s" % count
    subprocess.call(cmd, shell=True)

BASH
Bash runs a sequence of system commands.

#!/usr/bin/env bash

#Create Commands
SPACE=`df -h`
MESSAGES=`tail /var/log/messages`

#Assign to an array(list in Python)
cmds=("$MESSAGES" "$SPACE")

#iteration loop
count=0
for cmd in "${cmds[@]}"; do
    count=$((count + 1))
    printf "Running Command Number %s \n" $count
    echo "$cmd"
done

Python is much more forgiving about the way you quote and use variables, and lets you create a much less cluttered piece of code.

Childhood: Reusing code by writing functions

We have seen how Python can implement system calls to run commands in sequence, just like a regular Bash script. Let’s go a little further and organize blocks of code into functions. As I mentioned earlier, Python does not require the use of classes and object-oriented programming techniques, so most of the full power of the language is still at our fingertips—even if we’re only using plain functions.

Let’s write a simple function in Python and Bash and call them both in a script.

Note:

These two scripts will deliver identical output in Bash and Python, although Python handles default keyword parameters automatically in functions. With Bash, setting default parameters is much more work.

PYTHON:

#!/usr/bin/env python
import subprocess

#Create variables out of shell commands
MESSAGES = "tail /var/log/messages"
SPACE = "df -h"

#Places variables into a list/array
cmds = [MESSAGES, SPACE]

#Create a function, that takes a list parameter
#Function uses default keyword parameter of cmds
def runCommands(commands=cmds):
    #Iterates over list, running statements for each item in the list
    count=0
    for cmd in cmds:
        count+=1
        print "Running Command Number %s" % count
        subprocess.call(cmd, shell=True)

#Function is called
runCommands()

BASH:

#!/usr/bin/env bash

#Create variables out of shell commands
SPACE=`df -h`
MESSAGES=`tail /var/log/messages`
LS=`ls -l`
#Assign to an array(list in Python)
cmds=("$MESSAGES" "$SPACE")

function runCommands ()
{
    count=0
    for cmd in "${cmds[@]}"; do
        count=$((count + 1))
        printf "Running Command Number %s \n" $count
        echo "$cmd"
    done
}

#Run function
runCommands

Teenager: Making reusable command-line tools

Now that you have the ability to translate simple Bash scripts and functions into Python, let’s get away from the nonsensical scripts and actually write something useful. Python has a massive standard library that can be used by simple importing modules. For this example we are going to create a robust command-line tool with the standard library of Python, by importing the subprocess and optparse modules.

You can later use this example as a template to build your own tools that combine snippits of Bash inside of the more powerful Python. This is a great way to use your current knowledge to slowly migrate to Python.

Embedding Bash to make Python command-line tools^[1]:

#!/usr/bin/env python
import subprocess
import optparse
import re

#Create variables out of shell commands
#Note triple quotes can embed Bash

#You could add another bash command here
#HOLDING_SPOT="""fake_command"""

#Determines Home Directory Usage in Gigs
HOMEDIR_USAGE = """
du -sh $HOME | cut -f1
"""

#Determines IP Address
IPADDR = """
/sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | head -1
"""

#This function takes Bash commands and returns them
def runBash(cmd):
    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    out = p.stdout.read().strip()
    return out  #This is the stdout from the shell command

VERBOSE=False
def report(output,cmdtype="UNIX COMMAND:"):
   #Notice the global statement allows input from outside of function
   if VERBOSE:
       print "%s: %s" % (cmdtype, output)
   else:
       print output

#Function to control option parsing in Python
def controller():
    global VERBOSE
    #Create instance of OptionParser Module, included in Standard Library
    p = optparse.OptionParser(description='A unix toolbox',
                                            prog='py4sa',
                                            version='py4sa 0.1',
                                            usage= '%prog [option]')
    p.add_option('--ip','-i', action="store_true", help='gets current IP Address')
    p.add_option('--usage', '-u', action="store_true", help='gets disk usage of homedir')
    p.add_option('--verbose', '-v',
                action = 'store_true',
                help='prints verbosely',
                default=False)

    #Option Handling passes correct parameter to runBash
    options, arguments = p.parse_args()
    if options.verbose:
        VERBOSE=True
    if options.ip:
        value = runBash(IPADDR)
        report(value,"IPADDR")
    elif options.usage:
        value = runBash(HOMEDIR_USAGE)
        report(value, "HOMEDIR_USAGE")
    else:
        p.print_help()

#Runs all the functions
def main():
    controller()

#This idiom means the below code only runs when executed from command line
if __name__ == '__main__':
    main()

Python’s secret sysadmin weapon: IPython

The skeptics in the Bash crowd are just about to say, “Python is pretty cool, but it isn’t interactive like Bash.” Actually, this is not true. One of the best kept secrets of the Python world is IPython. I asked the creator of IPython, Fernando Perez, how IPython stacks up to classic Unix interactive shells. Rather than trying to replicate what he said, I’ll simply quote directly:

IPython is a replacement for the Python interactive environment that tries to incorporate the most common shell-like usage patterns in a natural way, while keeping 100% syntactic compatibility with the Python language itself. In IPython, commands like ‘cd’ or ‘ls’ do what you’d expect of them, while still allowing you to type normal Python code. And since IPython is highly customizable, it ships with a special mode that activates even more defaults for shell-like behavior. IPython custom modes are called profiles, and the shell profile can be requested via:

ipython -p sh

This will enable all the shell-like features by default. The links below show some basic information about the shell-like usage of IPython, though we still lack a comprehensive guide for all of the features that actually exist under the hood.

http://ipython.scipy.org/moin/Cookbook/IpythonShell
http://ipython.scipy.org/moin/Cookbook/JobControl

IPython also contains a set of extensions for interactively connecting and manipulating tabular data, called ‘ipipe,’ that enables a lot of sophisticated exploration of filesystem objects and environment variables. More information about ipipe can be found here:

http://ipython.scipy.org/moin/UsingIPipe

It is quite possible to use IPython as the only interactive shell for simple systems administration tasks. I recently wrote an article for IBM Developerworks, in which I demonstrated using IPython to perform interactive SNMP queries using Net-SNMP with Python bindings:

Summary

Even if you can barely string together a few statements in Bash, with a little work you can learn Python and be productive very quickly. Your existing Bash skills can be slowly converted to Python skills. And before you know it, you will be a full-fledged Python programmer.

I find Python easier to program in than Bash; you don’t have to deal with hordes of escaping scenarios, for one. Bash has its place–usually when you don’t have the ability to run Python–as Python beats the pants off Bash as a scripting language.

I have included a link to all of the examples, and will have a souped-up version of the Python command-line tool with a few extra tricks sometime soon.

Let me close with saying that if you are interested in replacing Bash with Python, try to start out on the best possible foot and write tests that validate what you think you wrote actually works. This is a huge leap in thinking, but it can propel your code and productivity to the next level. The easiest way to get started with testing in Python is to use doctests, and I have enclosed a link at the bottom of this article. Good luck!

References

• Subversion Repository For Examples
• Checklist Based Testing For SysAdmins
• Doctests
• Online Bash Scripting Guide
• Python Tutorial
• IPython
• Jeff Rush Show Me Do Tutorial
• PEP8
• Net-SNMP and IPython

^[1] This code example has been corrected. Feb 08, 2008, 11AM EST

About the author

Noah Gift is currently co-authoring a book for O’Reilly, “Python For *Nix Systems Administration,” (working title) due sometime in 2008. He works as a software engineer for Racemi, dealing with Bash, Python, SNMP and a slew of *nix operating systems, including AIX, HP-UX, Solaris, Irix, Red Hat, Ubuntu, Free BSD, OS X, and anything else that has a shell. He is giving a talk at PyCon 2008–the annual Python Programming convention being held in Chicago–on writing *nix command line tools in Python. When not sitting in front of a terminal, you might find him on a 20 mile run on a Sunday afternoon.

This entry was posted by Noah Gift on Thursday, February 7th, 2008 at 6:17 pm and is filed under documentation,Fedora, Red Hat Enterprise Linux, technical. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.