Sh - the Bourne Shell
Last update Fri Jul 21 08:45:02 EDT 2006
Table of Contents
Copyright 2001, 2005 Bruce Barnett and General Electric Company
All rights reserved
You are allowed to print copies of this tutorial for your personal use, and link to this page, but you are not allowed to make electronic copies, or redistribute this tutorial in any form without permission.
How to build your own complex commands from the simple commands in the UNIX toolbox. This tutorial discusses of Bourne shell programming, describing features of the original Bourne Shell. The newer POSIX shells have more features. I first wrote this tutorial before the POSIX shells were standardized. So the information describe here should work in POSIX shells as it is a subset of the POSIX specifications.
You're not getting the most out of UNIX if you can't write shell programs!
This sections covers the Bourne shell. The manual pages are only 10 pages long, so it shouldn't be difficult to learn, right? Well, apparently I'm wrong, because most of the people I know learned one shell to customize their environment, and stayed with the C shell ever since. I understand the situation. It's hard enough to learn one shell language, and after struggling with one shell for a while, they are hesitant to learn another shell. After a few scripts, the new user decides the C shell is "good enough for now" and it ends right there. They never take the next step, and learn the Bourne shell. Well, perhaps this chapter will help.
The Bourne shell is considered the primary shell in scripts. All UNIX systems have it, first of all. Second, the shell is small and fast. It doesn't have the interactive features of the C shell, but so what? Interactive features aren't much use in scripts. There are also some features the Bourne shell has that the C shell doesn't have. All in all, many consider the Bourne shell to be the best shell for writing portable UNIX scripts.
What is a shell, anyway? It's simple, really. The UNIX operating system is a complex collection of files and programs. UNIX does not require any single method or interface. Many different techniques can be used. The oldest interface, which sits between the user and the software, is the shell. Twenty five years ago many users didn't even have a video terminal. Some only had a noisy, large, slow hard-copy terminal. The shell was the interface to the operating system. Shell, layer, interface, these words all describe the same concept. By convention, a shell is a user program that is ASCII based, that allows the user to specify operations in a certain sequence.
There are four important concepts in a UNIX shell:
- ·
- The user interacts with the system using a shell.
- ·
- A sequence of operations can be scripted, or automatic, by placing the operations in a script file.
- ·
- A shell is a full featured programming language, with variables, conditional statements, and the ability to execute other programs. It can be, and is, used to prototype new programs.
- ·
- A shell allows you to easily create a new program that is not a "second-class citizen," but instead is a program with all of the privileges of any other UNIX program.
The last two points are important. DOS does not have a shell that has as many features as the Bourne shell. Also, it is impossible to write a DOS script that emulates or replaces existing commands.
Let me give an example. Certain DOS programs understand the meta-character "*." That is, the "RENAME" command can be told
- RENAME *.OLD *.NEW
Files "A.OLD" and "B.OLD" will be renamed "A.NEW" and "B.NEW." The DOS commands "COPY" and "MOVE" also understand the character "*." The "TYPE" and "MORE" commands, however, do not. If you wanted to create a new command, it too, would not understand that an asterisk is used to match filenames. You see, each DOS program is given the burden of understanding filename expansion. Consequently, many programs do not.
UNIX is a different story. The shell is given the burden of expanding filenames. The program that sees the filenames is unaware of the original pattern. This means all programs act consistently, as filename expansion can be used with any command. It also means a shell script can easily replace a C program, as far as the user is concerned. If you don't like the name of a UNIX utility, it is easy to create a new utility to replace the current program. If you wanted to create a program called "DIR" you could simply create a file containing
- #!/bin/sh
ls $*
The shell does the hard part.
The other difference between the DOS batch file and the UNIX shell is the richness of the shell language. It is possible to do software development using the shell as the top level of the program. Not only is it possible, but it is encouraged. The UNIX philosophy of program development is to start with a shell script. Get the functionality you want. If the end results has all of the functionality, and is fast enough, then you are done. If it isn't fast enough, consider replacing part (or all) of the script with a program written in a different language (e.g. C, Perl). Just because a UNIX program is a shell script does not mean it isn't a "real" program.
Because there are many shells available, it is important to learn to distinguish between the different shells. Typing commands for one shell when you are using another is bound to cause confusion. I know from personal experience. This was aggravated by the fact that many books I used to learn UNIX never mentioned that other shells existed. Therefore, the first step is to make sure you are using the proper shell.
You can execute the following command to determine your default shell (The command you type is in boldface):
- % echo $SHELL
/bin/csh
While this identifies your default shell, it does not accurately identify the shell you are currently using. I will give you a better way to find out later. Because this column discusses the Bourne shell, any commands discussed here will only work right if you are using the Bourne shell. You have two choices: place these commands in a Bourne shell script. That is, create a file, make the first line read
- #!/bin/sh
Make the second line, and those following, contain the commands you want to test. Then make it executable by typing
- chmod +x filename
Once you do this, you can test the script by typing
- ./filename
The second method is to create a new window (if you desire). Then type
- sh
You may see a change in the characters the shell gives you as a prompt, like the example below:
- % /bin/sh
$
The Bourne shell will execute each line you type, until an end of file is found. To put it another way, when you type Control-D, the Bourne shell will terminate, and return you to the shell you were previously using. This is the same action the shell takes when a script file is executed, and the end of the script file is reached.
The basic actions of the shell are simple. It reads a line. This is either from a file, a script, or from a user. First, meta-characters are "handled." Second, the name of the executable is found. Third, the arguments are passed to the program. Fourth, the file redirection is setup. Lastly, the program is executed.
As the shell reads each line, it "handles" any special characters. This includes variable evaluation (variables start with a "$)," and filename expansion. Expansion of filenames occurs when the characters "*," "?," or "[" occur in a word. A question mark matches a single character. An asterisk matches any number of characters, including none. Square brackets are used to specify a range or particular combination of characters. Inside square brackets, a hyphen is used to specify a range or characters. Also, if the first character inside the square brackets is an exclamation point, the complement of the range is used. Let me give some examples:
+--------------------------------------------------------------------+
| Table 1 |
| Examples of filename expansion|
+--------------------------------------------------------------------+
Pattern Matches
* Every file in the current directory
? Files consisting of one character
?? Files consisting of two characters
??* Files consisting of two or more characters
[abcdefg] Files consisting of a single letter from a to g.
[gfedcba] Same as above
[a-g] Same as above
[a-cd-g] Same as above
[a-zA-Z0-9] Files that consist of a single letter or number
[!a-zA-Z0-9] Files that consist of a single character not a letter or number
[a-zA-Z]* Files that start with a letter
?[a-zA-Z]* Files whose second character matches a letter.
*[0-9] Files that end with a number
?[0-9] Two character filename that end with a number
*.[0-9] Files that end with a dot and a number
+--------------------------------------------------------------------+
As you can see, the dot is not a special character. Filenames may or may not have a dot. UNIX Programers use the dot to standardize on the type of source code of each file, but that is just a convention. There is another convention, which concerns the shell:
- Files whose name starts with a dot are not normally listed.
Again, it is a convention, but
ls,
find and the various shells follow this convention. This allows some files to be "secret," or perhaps invisible, by default. You must explicitly ask for these files, by including the dot as part of the filename. The pattern ".*" matches all hidden files. Remember that two hidden files are always in every directory, ".," which indicate the present directory, and "..," which indicates the directory above the current directory. If you want to match all hidden files except these two directories, there is no easy way to specify a pattern that will always match all files except the two directories. I use
- .??
or
- .[a-zA-Z]*
As I said, this does not match all combinations, but works most of the time. Hackers (or crackers, if you prefer) break into computers and often use strange filenames, like ". " or ".. " to hide their traces. You may not have noticed, but there was a space in these filenames. Refering to files with spaces in the names require quoting, which I will cover later. Personally, all of my hidden files are matched by ".[a-z]*" and all of my hidden directories are matched by ".[A-Z]*." This works because I made up my own convention, and always follow it.
The slash is also special, as it is used to indicate a directory path. Filename expansion does not expand to match a slash, because a slash can never be part of the filename. Also, the same rules for filename expansion of hidden files applies if the pattern follows a slash. If you want to match hidden files in a subdirectory, you must specify the explicit pattern. Table 2 lists some examples.
+---------------------------------------------------------------------------+
|Table 2 |
| Filename Expansion with directories|
|Pattern Matches |
+---------------------------------------------------------------------------+
* All non-invisible files
abc/* All non-invisible files in directory abc
abc/.*All invisible files in directory abc
*/*All non-invisible files in all subdirectories below
*/.* All invisible files in all subdirectories below
+---------------------------------------------------------------------------+
Filename expansions are based on the current directory, unless the filename starts with a slash.
The Bourne shell differs from the C shell if the meta-characters do not match any file. If this happens, the pattern is passed to the program unchanged. (The C shell will either do this, or generate an error, depending on a variable).
If you are not sure how something will expand, use the echo command to check. It generates output more compact than ls, and it will not list contents of directories like ls will. You will also notice the output is sorted alphabetically. The shell not only expans filenames, but sorts them, for all applications.
Once the shell expands the command line, it breaks up the line into words, and takes the first word as the command to be executed. (The special Bourne variable "IFS" contains the characters used to "break up" the line. Normally, this variable contains a space and a tab.) Afterwards, the first word is used as the name of the program. If the command is specified without an explicit directory path, the shell then searches the different directories specified by the "PATH" variable, until it finds the program specified.
If you have been following the points I made, it should not surprise you that a valid UNIX command might be
- *
The contents of the directory determines the results, but if you created a file called "0," which contains
#!/bin/sh
echo Hey! You forgot to specify the command!
Click here to get file:
0and if it is the first file (alphabetically) in your directory, then executing "*" would give you an error message. (Provided the current directory was in your search path).
So you see, filename expansion can be anywhere on a command line. You can execute programs with long names without typing the entire name. However, filename expansion only works if the file is in the directory you specify. If you wanted to execute the program "my_very_own_program" without typing the complete filename, you could type
- my_* arguments
as long as "my_*" expanded to a unique program name. If this was in another directory, then you you have to specify the directory path:
- /usr/local/bin/my_* arguments
Understanding the relationship between the shell and the programs allows you to add features. Some people create a file called "-i" in a directory. If someone then types
- rm *
while in this directory, the first argument will probably be "-i." This filename is passed to the
rm program, which assumes the hyphen indicates an argument. Therefore
rm runs with the interactive option, protecting programs from accidental deletion.
One last point. Many DOS users complain that they can't execute
- RENAME *.OLD *.NEW
I admit that this is a little awkward to do in UNIX. I'd like to say two things in defense. First, the above usage of filename expansion is an "unnatural act," as far as the UNIX philosophy is concerned. There are many advantages to the shell handling the filename expansion, and perhaps one disadvantage in one case. I believe the advantages of the UNIX philosophy far outweight the disadvanges.
Second, I don't believe it is a disadvantage anyway. Renaming files like that is wrong. The only reason to do so is because DOS does it that way, and you have to to this because you are limited to 11 characters. If you must rename them, append a string to the end instead of changing the original filename. This is UNIX. You can have filenames 256 characters long, so this approach isn't a problem. So if you must rename them, use
for i in *.OLD
do
mv $i $i.NEW
done
This alows you to undo what you did, and retains the original filename. Even better, move the files into another directory, letting them keep their original name. I would suggest you type
- mkdir Old
mv *.OLD Old
This makes undoing your action very easy, and works for files of any name, and not just "*.OLD."
The first problem shell programmers experience is quotation marks. The standard keyboard has three quotation marks. Each one has a different purpose, and only two are used in quoting strings. Why quote at all, and what do I mean by quoting? Well, the shell understands many special characters, called meta-characters. These each have a purpose, and there are so many, beginners often suffer from meta-itis. Example: The dollar sign is a meta-character, and tells the shell the next word is a variable. If you wanted to use the dollar sign as a regular character, how can you tell the shell the dollar sign does not indicate a variable? Answer: the dollar sign must be quoted. Why? Quoted characters do not have a special meaning. Let me repeat this with emphasis.
Quoted characters do not have a special meaning
A surprising number of characters have special meanings. The lowly space, often forgotten in many books, is an extremely important meta-character. Consider the following:
- rm -i file1 file2
The shell breaks this line up into four words. The first word is the command, or program to execute. The next three words are passed to the program as three arguments. The word "-i" is an argument, just like "file1." The shell treats arguments and options the same, and does not know the difference between them. In other words, the program treats arguments starting with a hyphen as special. The shell doesn't much care, except that it follows the convention. In this case,
rm looks at the first argument, realizes it is an option because it starts with a hyphen, and treats the next two arguments as filenames. The program then starts to delete the files specifies, but firsts asks the user for permission because of the "-i" option. The use of the hyphen to indicate an option is a convention. There is no reason you can't write a program to use another character. You could use a forward slash, like DOS does, to indicate a hyphen, but then your program would not be able to distinguish between an option and a path to filename whose first characters is a slash.
Can a file have a space in the name? Absolutely. This is UNIX. There are few limitations in filenames. As far as the operating system is concerned, You can't have a filename contain a slash or a null. The shell is a different story, and one I don't plan to discuss.
Normally, a space delineates arguments. To include a space in a filename, you must quote it. Another verb used in the UNIX documentations is "escape;" this typically refers to a single character. You "quote" a string, but "escape" a meta-character. In both cases, all special characters are treated as regular characters.
Assume, for a moment, you had a file named "file1 file2," This is one file, with a space between the "1" and the "f." If this file is to be deleted, one way to quote the space is
- rm 'file1 file2'
There are other ways to do the same. Most people consider the quotation mark as something you place at the beginning and end of the string. A more accurate description of the quoting process is a switch, or toggle. The following variations are all equivalent:
- rm 'file1 file2'
rm file1' 'file2
rm f'ile1 file'2
In other words, when reading a shell script, you must remember the current "quoting state." The quote toggles the state. Therefore if you see a quotation mark in the middle of a line, it may be turning the toggle on or off. You must start from the beginning, and group the quotation marks in pairs.
There are two other forms or quoting. The second uses a backslash "," which only acts to "escape" the next character. The double quotation mark is similar to the single quotes used above, but weaker. I'll explain strong and weak quotation later on. Here is the earlier example, this time using the other forms of quoting:
- rm "file1 file2"
rm file1 file2
rm file1" "file2
A very confusing problem is placing quotation marks within quotation marks. It can be done, but it is not always consistent or logical. Quoting a double quote is perhaps the simplist, and does what you expect:
- echo '"'
echo """
echo "
The backslash is different. Look at the three variations:
- echo ''
echo ""
echo
As you can see, single quotes and double quotes behave differently. A double quote is weaker, and does not quote a backslash. Single quotes are different again. You can escape them with a backslash, or quote them with double quotes:
- echo '
echo "'"
The following does not work:
- echo '''
It is identical to
- echo '
Both examples start a quoting operation, but do not end the action. In other words, the quoting function will stay toggled, and will continue until another single quote is found. If none is found, the shell will read the rest of the script, until an end of file is found.
Earlier I described single quotes as strong quoting, and double quotes as weak quoting. What is the difference? Strong quoting prevents characters from having special meanings, so if you put a character inside single quotes, what you see is what you get. Therefore, if you are not sure if a character is a special character or not, use strong quotation marks.
Weak quotation marks treat most characters as plain characters, but allow certain characters (or rather meta-characters) to have a special meaning. As the earlier example illustrates, the backslash within double quotation marks is a special meta-character. It indicates the next character is not, so it can be used before a backslash and before a double quotation mark, escaping the special meaning. There are two other meta-characters that are allowed inside double quotation marks: the dollar sign, and the back quote.
Dollar signs indicate a variable. One important variable is "HOME" which specifies your home, or starting directory. The following examples illustrates the difference:
- $ echo '$HOME'
$HOME
$ echo '$HOME'
$HOME
$ echo "$HOME"
/home/barnett
$ echo "$HOME"
$HOME
The back quote does command substitution. The string between backquotes is executed, and the results replaces the backquoted string:
- $ echo 'The current directory is `pwd`'
The current directory is `pwd`
$ echo 'The current directory is `pwd`'
The current directory is `pwd`
$ echo "The current directory is `pwd`"
The current directory is `/home/barnett`
$ echo "The current directory is `pwd`"
The current directory is `pwd`
There is a large difference between the C shell and the Bourne shell when a quote is larger than a line. The C shell is best suited for interactive sessions. Because of this, it assumes a quote ends with the end of a line, if a second quoute character is not found. The Bourne shell makes no assumptions, and only stops quoting when you specify a second quotation mark. If you are using this shell interactively, and type a quotation mark, the normal prompt changes, indicating you are inside a quote. This confused me the first time it happened. The following Bourne shell example illustrates this:
- $ echo 'Don't do this'
> ls
> pwd
> '
Dont do this
ls
pwd
$
This is a minor inconvenience if you use the shell interactively, but a large benefit when writing shell scripts that contain multiple lines of quoted text. I used the C shell for my first scripts, but I soon realized how awkward the C shell was when I included a multi-line awk script insice the C shell script. The Bourne shell's handling of awk scripts was much easier:
#!/bin/sh
# Print a warning if any disk is more
# than 95% full.
/usr/ucb/df | tr -d '%' | awk '
# only look at lines where the first field contains a "/"
$1 ~ /// { if ($5 > 95) {
printf("Warning, disk %s is %4.2f%% full ",$6,$5);
}
}'
Click here to get file:
diskwarn.sh
Having two types of quotation marks simplifies many problems, as long as you remember how meta-characters behave. You will find that the easiest way to escape a quotation mark is to use the other form of quotation marks.
- echo "Don't forget!"
echo 'Warning! Missing keyword: "end"'
Earlier I showed how to include a quote within quotes of the same kind. As you recall, you cannot place a single quote within a string terminated by single quotes. The easiest solution is to use the other type of quotation marks. But there are times when this is not possible. There is a way to do this, but it is not obvious to many people, especially those with a lot of experience in computer languages. Most languages, you see, use special characters at the beginning and end of the string, and has an escape to insert special characters in the middle of the string. The quotation marks in the Bourne shell are
not used to define a string. There are used to
disable or
enable interpretation of meta-characters. You should understand the following are equivalent:
- echo abcd
echo 'abcd'
echo ab'c'd
echo a"b"cd
echo 'a'"b"'c'"d"
The last example protects each of the four letters from special interpretation, and switches between strong and weak quotation marks for each letter. Letters do not need to be quoted, but I wanted a simple example. If I wanted to include a single quote in the middle of a string delineated by a single quote marks, I'd switch to the different form of quotes when that particular character is encountered. That is, I'd use the form
- 'string1'"string2"'string3'
where string2 is a single quote character. Here is the real example:
- $ echo 'Strong quotes use '"'"' and weak quotes use "'
Strong quotes use ' and weak quotes use "
It is confusing, but if you start at the beginning, and following through, you will see how it works.
Change the quoting mid-stream is also very useful when you are inserting a variable in the middle of a string. You could use weak quotes:
- echo "My home directory is $HOME, and my account is $USER"
You will find that this form is also useful:
- echo 'My home directory is '$HOME', and my account is '$USER
When you write your first multi-line
awk or
sed script, and discover you want to pass the value of a variable to the middle of the script, the second form solves this problem easily.
The Bourne shell has a very simple syntax for variables:
- variable=value
The characters used for variable names is limited to letters, numbers and the underscore character. It cannot start with a number. Unlike the C shell, spaces are important when defining Bourne shell variables. Whitespace (spaces, tabs or newlines) terminate the value. If you want whitespace in a variable, it must be quoted:
- question='What is the filename? '
Multiple assignments can be placed on one line:
- A=1 B=2 C=3 D=4
Do not put a space after the equals sign. This terminates the value. The command
- a=date
sets the variable "a" to be equal to "date," but the command
- a= date
sets "a" to be the empty string, and
executes the date command. The "date" command? Yes. Which introduces...
Notice how two commands are executed on one line: the variable is changed, and the "date" program is executed. It is not obvious that this is valid. The manual page doesn't mention this. Even stranger is some commands can be executed, while others cannot. The "date" command is an external program. That is, the command is not built into the shell, but is an external executable. Other commands are internal command, built into the shell. "Echo" and "export" are shell built-in commands, and can follow the variable assignment. You might see an environment variable defined like this:
- VAR=/usr/lib; export VAR
But the following works just as well:
- VAR=/usr/lib export VAR
Some of the built--in commands cannot be on the same line, like "for" or "if." The "echo" command does, but it may not do what you think. Don't believe me? I'll give you an example, and you have to guess what the results will be. I suspect that that 99.9999% of you would guess wrong. Put on your thinking caps. You'll need it.
Ready?
What does the following Bourne shell commands do?
- a=one; echo $a
a=two echo $a
a=three echo $a >$a
I have to be honest. I failed the test myself. Well, I got partial credit. But I wrote the quiz. The first line is simple: "one" is output to the screen. The second line behaves differently. The value of variable "a" is set to "two," but the echo command outputs "one." Remember, the shell reads the lines, expands metacharacters, and then passes it to the programs. The shell treats built-in commands like external commands, and expands the meta-characters before executing the built-in commands. Therefore the second line is effectively
- a=two echo one
and
then the command is executed, which changes the value of the variable
after it is used.
Ready for a curve ball? What does the last line do? It creates a file. The file contains the word "two." For $64,000 and a trip to Silicon Valley, what is the name of the file that is created? For those to thought the answer is "two," I'm terrible sorry, you didn't win the grand price. We do have a nice home version of this game, and a year's supply of toothpaste. The correct answer is "three." In other words
- echo $a >$a
is interpreted as
- echo second >third
I am not fooling you. The variable "$a" has two different values on the same line! The C shell doesn't do this, by the way. The Bourne shell evaluates metacharacters twice: one for the commands and arguments, and a second time for file redirection. Perhaps Mr. Bourne designed the shell to behave this way because he felt
- a=output >$a
ought to use the
new value of the variable, and not the value
before the line was executed, which might be undefined, and would certainly have undesirable resultsa. Although the real reason is that the above command treats the "a" variable like an environment variable, and sets the variable, marks it for export, and then executes the command. Because the variable is "passed" to the command by the environment, the shell simply sets the standard output to the appropriate file, and then processes the line for variables, and lastly executing the command on the line. More on this later.
If you with to examine the values of all of your current variables, use the command "set:"
- $ set
DISPLAY=:0.0
HOME=/home/barnett
IFS=
LD_LIBRARY_PATH=/usr/openwin/lib:/usr/openwin/lib/server
LOGNAME=barnett
MAILCHECK=600
OPENWINHOME=/usr/openwin
PATH=/home/barnett/bin:/usr/ucb:/usr/bin
PS1=$
PS2=>
PWD=/home/barnett
SHELL=/bin/csh
TERM=vt100
USER=barnett
$
Notice the alphabetical order of the variables, and the equals character between the variable and the value. The "set" command is one way to determine which shell you are currently using. (The C shell puts spaces between the variable and the value.) Also note the assortment of variables already defined. These are environment variables.
UNIX provides a mechanism to pass information to all processes created by a parent process by using environment variables. When you log onto a system, you are given a small number of variables, predefined. You can add to this list in your shell start-up files. Every program you execute will inherit these variables. But the information flow is one-way. New UNIX users find this confusing, and cannot understand why a shell script can't change their current directory. Picture it this way: suppose you executed hundreds of programs, and they all wanted to change their environment to a different value. It should be obvious that they can't all control the same variable. Imagine hundreds of programs trying to change the directory you are currently using! Perhaps these variables ought to be called hereditary, and not environmental. Children processes inherit these values form the parents, but can never change how the parents were created. That would require time-travel, a feature not currentable available in commercial UNIX systems.
As I mentioned before, the shell command "export" is used to update environment variables. The command
- export a b c
marks the variables "a" "b" and "c," and all child processes will inherit the current value of the variable. With no arguments, it lists those variables so marked. The command "export" is necessary. Changing the value of an environment variable does not mean this change will be inherited. Example:
- HOME=/
myprogram
When "myprogram" executes, the value of the "HOME" variable is not changed. However, in this example:
- HOME=/
export HOME
myprogram
the program
does get the modified value. Another way to test this is to start a new copy of the shell, and execute the "export" command. No variables are reported.
The command marks the variable. It does not copy the current value into a special location in memory. A variable can be marked for export before it is changed. That is,
- export HOME
HOME=/
myprogram
works fine.
There are several special variables the shell uses, and there are special variables the system defined for each user. SunOS and Solaris systems use different environment variables. If in doubt, check the manual pages. I'll describe some the important Solaris variables.
The "PATH" environment variable lists directories that contain commands. When you type an arbitrary command, the directories listed are searched in the order specified. The colon is used to separate directory names. An empty string corresponds to the current directory. Therefore the searchpath
- :/usr/bin:/usr/ucb
contains three directories, with the current directory being searched first. This is dangerous, as someone can create a program called "ls" and if you change your current directory to the one that contains this program, you will execute this trojan horse. If you must include the current directory, place it last in the searchpath.
- /usr/bin:/usr/ucb:
The "HOME" variable defines where the "cd" goes when it is executed without any arguments. The HOME evvironment variable is set by the login process.
When you execute the "cd" command, and specify a directory, the shell searches for that directory inside the current working directory. You can add additional directories to this list. If the shell can't find the directory in the current directory, it will look in the list of directories inside this variable. Adding the home directory, and the directory above the current directory is useful:
- CDPATH=$HOME:..
export CDPATH
The "IFS" variable lists the characters used to terminate a word. I discussed this briefly earlier. Normally, whitespace separates words, and this variable contains a space, a tab and a new line. Hackers find this variable interesting, because it can be used to break into computer systems. A poorly written program may carelessly execute "/bin/ps." A hacker may redefine the PATH variable, and define IFS to be "/." When the program executes "/bin/ps," the shell will treat this as "bin ps." In other words, the program "bin" is executed with "ps" as an argument. If the hacker has placed a program called "bin" in the searchpath, then the hacker gains privileged access.
The "PS1" variable specifies the prompt printed before each command. It is normally "$ ." The current directory cannot be placed inside this prompt. Well, some people make a joke, and tell a new user to place a period inside this variable. A "."
does signifies the current directory, however, most users prefer the actual name.
The "PS2" environment variable defines the secondary prompt, This is the prompt you see when you execute a multi-line command, such as "for" or "if." You also see it when you forget to terminate a quote. The default value is "> ."
The "MAIL" variable specifies where your mailbox is located. It is set by the login process.
The "MAILCHECK" variable specifies how often to check for mail, in seconds. The default value is 600 seconds (10 minutes). If you set it to zero, every time the shell types a prompt, it will check for mail.
This variable defines the accounting file, used by the
acctcom and
acctcms commands.
The "MAILPATH" variable lists colon-separated filenames. You can add a "%" after the filename, and specify a special prompt for each mailbox.
In addition, several environment variables are specified by the login process. "TERM" defines the terminal type, and "USER" or "LOGNAME" defines your user ID. "SHELL" defines your default shell, and "TZ" specifies your time zone. Check the manual pages, and test your own environment to find out for sure. The external program "env" prints all current environment variables.
Earlier, I discussed simple variables in the Bourne shell. Now is the time to go into more detail. Suppose you wanted to append a string to a variable. That is, suppose you had a variable "X" with the value of "Accounts," but you wanted to add a string like ".old," or "_new" making "Accounts.old" or "Accounts_new," perhaps in an attempt to rename a file. The first one is easy. The second requires a special action. In the first case, just add the string
- mv $X $X.old
The second example, however, does not work:
- mv $X $X_new # WRONG!
The reason? Well, the underscore character is a valid character in a variable name. Therefore the second example evaluates two variables, "X" and "X_new." If the second one is undefined, the variable will have a value of nothing, and the shell will convert it to
- mv Accounts
The mv command will take the offered arguments, and complain, as it always wants two or more variables. A similar problem will occur if you wish to add a letter or number to the value of a variable.
There are several solutions. The first is to use shell quoting. Remember, quoting starts and stops the shell from treating the enclosed string from interpretation. All that is needed is to have a quote condition start or stop between the two strings passed to the shell. Place the variable in one string, and the constant in the other. If the variable "x" has the value "home," and you want to add "run" to the end, all of the following combinations are equal to "homerun:"
- $x"run"
$x'run'
$xrun
$x''run
$x""run
"$x"run
There is another solution, using curly braces:
- ${x}run
This is a common convention in UNIX programs. The C shell also uses the same feature. The UNIX
make utility uses this in makefiles, and requires braces for all variable references longer than a single letter. (
Make uses either curly braces or parenthesis).
This form for variables is very useful. You could standardize on it as a convention. But the real use comes from four variations of this basic form, briefly described below:
+-------------------------------------------------------------+
|Form Meaning |
+-------------------------------------------------------------+
|${variable?word} Complain if undefined |
|${variable-word} Use new value if undefined |
|${variable+word} Opposite of the above |
|${variable=word} Use new value if undefined, and redefine. |
+-------------------------------------------------------------+
Why are these forms useful? If you write shell scripts, it is good practice to gracefully handle unusual conditions. What happens if the variable "d" is not defined - and you use the command below?
d=`expr $d + 1`
You get "expr: syntax error"
The way to fix this is to have it give an error if "d" is not defined.
d=`expr "${d?'not defined'}" + 1`
The "?" generates an error: "sh: d: not defined"
If instead, you wanted it to silently use zero, use
d=`expr "${d-0}" + 1`
This uses "0" if "d" is undefined.
If you wish to set the value if it's undefined, use "="
echo $z
echo ${z=23}
echo $z
The first echo outputs a blank line. The next 2 "echo" commands output "23."
Note that you can't use
new=`expr "${old=0}" + 1`
to change the value of "old" because the expr command is run as a subshell script, and changing the value of "old" in that shell doesn;t change the value in the parent shell.
I've seen many scripts fail with strange messages if certain variables aren't defined. Preventing this is very easy, once you master these four methods of referring a Bourne shell variable. Let me describe these in more detail.
The first variation is used when something unusual happens. I think of it as the "Huh???" option, and the question mark acts as the mnemonic for this action. As an example, assume the following script is executed:
- #!/bin/sh
cat ${HOME}/Welcome
But suppose the environment variable "HOME" is not set. Without the question mark, you might get a strange error. In this case, the program cat would complain, saying file "/Welcome" does not exist. Change the script to be
- #!/bin/sh
cat ${HOME?}/Welcome
and execute it, and you will get the following message instead:
- script: HOME: parameter null or not set
As you can see, changing all variables of the form "$variable" to "${variable?}" provides a simple method to improve the error reporting. Better still is a message that tells the user how to fix the problem. This is done by specifying a word after the question mark. Word? Yes, the manual pages says a word. In a typical UNIX-like way, that word is very important. You can place a single word after the question mark. But only one word. Perfect for one-word insults to those who forget to set variables:
- cat ${HOME?Dummy}/Welcome
This is a perfect error message if you wish to develop a reputation. Some programmers, however, prefer to keep their jobs and friends. If you fall into that category, you may prefer to give an error message that tells the user how to fix the problem. How can you do that with a single word?
Remember my discussion earlier on quoting? And how the shell will consider a whitespace to be the end of the word unless quoted? The solution should be obvious. Just quote the string, which makes the results one word:
- cat ${HOME?"Please define HOME, and try again"}/Welcome
Simple, yet this makes a shell script more user-friendly. It's not a real shell error, as it doesn't cause the shell to exit, and is not a syntax error. If you want a real error to occur, then you must do this yourself with the "exit" command.
The next variation doesn't generate an error. It simply provides a variable that didn't have a value. Here is an example, with user commands in boldface:
- $ echo Y is $Y
Y is
$ echo Y is ${Y-default}
Y is default
$ Y=new
$ echo Y is ${Y-default}
Y is new
$
Think of the hyphen as a mnemonic for an optional value, as the hyphen is used to specify an option on a UNIX command line. Like the other example, the word can be more than a single word. Here are some examples:
- ${b-string}
${b-$variable}
${b-"a phrase with spaces"}
${b-"A complex phrase with variables like $HOME or `date`"}
${b-`command`}
${b-`wc -l </etc/passwd`}
${b-`ypcat passwd | wc -l`}
Any command in this phrase is only executed if necessary. The last two examples counts the number of lines in the password file, which might indicate the maximum number of users. Remember - you can use these forms of variables in place of the simple variable reference. So instead of the command
- echo Maximum number of users are $MAXUSERS
change it to
- echo Maximum number of users are ${MAXUSERS-`wc -l </etc/passwd`}
If the variable is set, then the password file is never checked.
The third variation uses a plus sign, instead of a minus. The mnemonic is "plus is the opposite of the minus." This is appropriate, as the command does act the opposite as the previous one. In other words, if the variable is set, then ignore the current value, and use the new value. This can be used as a debug aid in Bourne shell scripts. Suppose you wanted to know when a variable was set, and what the current value is. A simple way to do this is to use the
echo command, and echo nothing when the variable has no value by using:
- echo ${A+"Current value of A is $A"}
This command does print a blank line if A does not have a value. To eliminate this, use either the Berkeley version of echo, or the System V version of echo:
- /usr/bin/echo ${A+"A = $A"}"c"
/usr/ucb/echo -n ${A+"A = $A"}
Don't forget that these variations are used when you reference a variable, and do not change the value of the variable. Well, the fourth variation is different, in that it
does change the value of the variable, if the variable is undefined. It acts like the hyphen, but if used, redefines the variable. The mnemonic for this action? The equals sign. This should be easy to remember, because the equals sign is used to assign values to variables:
- $ echo Y is $Y
Y is
$ echo Y is ${Y=default}
Y is default
$ echo Y is $Y
Y is default
$
As you use these features, you may wish to test the behavior. But how do you undefine a variable that is defined? If you try to set it to an empty string:
- A=
you will discover that the above tests do not help. As far as they are concerned, the variable is defined. It just has the value of nothing, or null as the manual calls it. To undefine a variable, use the unset command:
- unset A
or if you wish to unset several variables
- unset A B C D E F G
As you can see, there is a different between a variable that has a null value, and a variable that is undefined. While it might seem that all one cares about is defined or undefined, life is rarely so simple. Consider the following:
- A=$B
If B is undefined, is A also undefined? No. Remember, the shell evaluates the variables, and then operates on the results. So the above is the same as
- A=
which defines the variable, but gives it an empty, or null value. I think most scripts don't care to know the difference between undefined and null variables. They just care if the variables have a real value or not. This makes so much sense, that later versions of the Bourne shell made it easy to test for both cases by creating a slight variation of the four forms previously described: a colon is added after the variable name:
+-------------------------------------------------------------------------------------------+
|Form Meaning |
+-------------------------------------------------------------------------------------------+
|${variable:?word} Complain if undefined or null |
|${variable:-word} Use new value if undefined or null |
|${variable:+word} Opposite of the above |
|${variable:=word} Use new value if undefined or null, and redefine. |
+-------------------------------------------------------------------------------------------+
Notice the difference between "${b-2}" and "${b:-2}" in the following example:
- $ # a is undefined
$ b=""
$ c="Z"
$ echo a=${a-1}, b=${b-2}, c=${c-3}
a=1, b=, c=Z
$ echo a=${a:-1}, b=${b:-2}, c=${c:-3}
a=1, b=2, c=Z
One last point - the special word in one of these formats is only evaluated if necessary. Therefore the
cd and
pwd commands in the following: is only executed if the word is executed:
- echo ${x-`cd $HOME;pwd`}
Also - the evaluation occurs in the current shell, and not a sub-shell. The command above will change the current directory, but the one below will not, as it executes the commands in a new shell, which then exits.
- echo `cd $HOME;pwd`
Earlier, I discussed Bourne shell variables, and various ways to use them. So far I have only given you the foundation of shell programming. It's time for discussing special Bourne shell variables, which will allow you to write useful scripts. These special variables are identified by the dollar sign, and another character. If the character is a number, it's a positional parameter. If it's not a letter or number, it's a special purpose variable.
The most important concept in shell scripts is passing arguments to a script. A script with no options is more limited. The Bourne shell syntax for this is simple, and similar to other shells, and
awk. As always, the dollar sign indicates a variable. The number after the dollar sign indicates the position on the command line. That is, "$1" indicates the first parameter, and "$2" indicates the second. Suppose you wanted to create a script called
rename that takes two arguments. Just create a file with that name, that contains the following:
#!/bin/sh
# rename: - rename a file
# Usage: rename oldname newname
mv $1 $2
Click here to get file:
rename0.sh
Then execute "chmod +x rename" and you have a new UNIX program. If you want to add some simple syntax checking to this script, using the techniques I discussed earlier, change the last line to read:
- mv ${1?"missing: original filename"} ${2?"missing new filename"}
This isn't very user friendly. If you do not specify the first argument, the script will report:
- rename: 1: missing: original filename
As you can see, the missing variable, in this case "1," is reported, which is a little confusing. A second way to handle this is to assign the positional variables to new names:
#!/bin/sh
# rename: - rename a file
# Usage: rename oldname newname
oldname=$1
newname=$2
mv ${oldname:?"missing"} ${newname:?"missing"}
Click here to get file: rename.sh
This will report the error as follows:
- rename: oldname: missing
Notice that I had to add the colons before the question mark. Earlier I mentioned how the question mark tests for undefined parameters, while the colon before the question mark complains about empty parameters as well as undefined parameters. Otherwise, the mv command might have complained that it had insufficient arguments.
The Bourne shell can have any number of parameters. However, the positional parameters variables are limited to numbers 1 through 9. You might expect that $10 refers to the tenth argument, but it is the equivalent of the value of the first argument with a zero appended to the end of the value. The other variable format, ${10}, ought to work, but doesn't. The Korn shell does support the ${10} syntax, but the Bourne shell requires work-arounds. One of these is the shift command. When this command is executed, the first argument is moved off the list, and lost. Therefore one way to handle three arguments follows:
- #!/bin/sh
arg1=$1;shift;
arg2=$1;shift;
arg3=$1;shift;
echo first three arguments are $arg1 $arg2 and $arg3
The
shift command can shift more than one argument; The above example could be:
- #!/bin/sh
arg1=$1
arg2=$2
arg3=$3;shift 3
echo first three arguments are $arg1 $arg2 and $arg3
This technique does make it easier to add arguments, but the error message is unfriendly. All you get is "cannot shift" as an error. The proper way to handle syntax errors requires a better understanding of testing and branching, so I will postpone this problem until later.
There is a special positional parameter, at location zero, that contains the name of the script. It is useful in error reporting:
- echo $0: error
will report "rename: error" when the rename script executes it. This variable is not affected by the shift command.
Another work-around for the inability for specifying parameters 10 and above is the "$*" variable. The "*" is similar to the filename meta-character, in that it matches all of the arguments. Suppose you wanted to write a script that would move any number of files into a directory. If the first argument is the directory, the following script would work:
#!/bin/sh
# scriptname: moveto
# usage:
# moveto directory files.....
directory=${1:?"Missing"};shift
mv $* $directory
Click here to get file:
moveto.sh
If this script was called "moveto" then the command
- moveto /tmp *
could easily move hundreds of files into the specified directory. However, if any of the files contain a space in the name, the script would not work. There is a solution, however, using the $@ variable.
The "$@" variable is very similar to the the "$*" variable. Yet, there is a subtle, but important distinction. In both cases, all of the positional parameters, starting with $1, are listed, separated by spaces. If there are spaces inside the variables, then "$@" retains the spaces, while "$*" does not. An example will help. Here is a script, called
EchoArgs, that echoes its arguments:
#!/bin/sh
# Scriptname: EchoArgs
# It echoes arguments
#First - make sure we are using the Berkeley style echoes
PATH=/usr/ucb:$path;export PATH
E="echo -n"
# echo the name of the script
${E} $0:
# now echo each argument, but put a space
# before the argument, and place single quotes
# around each argument
${E} " '${1-"?"}'"
${E} " '${2-"?"}'"
${E} " '${3-"?"}'"
${E} " '${4-"?"}'"
${E} " '${5-"?"}'"
${E} " '${6-"?"}'"
${E} " '${7-"?"}'"
echo
Click here to get file: EchoArgs.sh
Second, here is a script that tests the difference:
#!/bin/sh
EchoArgs $*
EchoArgs $@
EchoArgs "$*"
EchoArgs "$@"
Click here to get file: TestEchoArgs.sh
Now, let's execute the script with arguments that contain spaces:
- ./TestEcho "a b c" 'd e' f g
The script outputs the following:
- ./EchoArgs: 'a' 'b' 'c' 'd' 'e' 'f' 'g'
./EchoArgs: 'a' 'b' 'c' 'd' 'e' 'f' 'g'
./EchoArgs: 'a b c d e f g' '?' '?' '?' '?' '?' '?'
./EchoArgs: 'a b c' 'd e' 'f' 'g' '?' '?' '?'
As you can see, $* and $@ act the same when they are not contained in double quotes. But within double quotes, the $* variable treats spaces within variables, and spaces between variables the same. The variable $@ retains the spaces. Most of the time $* is fine. However, if your arguments will ever have spaces in them, then the $@ is required.
The "$#" variable is equal to the number of arguments passed to the script. If
newscript returned $# as a results, then both
- newscript a b c d
and
- newscript "a b c" 'd e' f g
would report 4. The command
- shift $#
"erases" all parameters because it shifts them away, so it is lost forever.
The variable "$$" corresponds to the process ID of the current shell running the script. Since no two processes have the same identification number, this is useful in picking a unique temporary filename. The following script selects a unique filename, uses it, then deletes it:
#!/bin/sh
filename=/tmp/$0.$$
cat "$@" | wc -l >$filename
echo `cat $filename` lines were found
/bin/rm $filename
Click here to get file:
CountLines0.sh
Another use of this variable is to allow one process to stop a second process. Suppose the first process executed
- echo $$ >/tmp/job.pid
A second script can kill the first one, assuming it has permissions, using
- kill -HUP `cat /tmp/job.pid`
The
kill command sends the signal specified to the indicated process. In the above case, the signal is the hang-up, or HUP signal. If you logged into a system from home, and your modem lost the connection, your shell would receive the HUP signal.
I hope you don't mind a brief discourse into signals, but these concepts are closely related, so it is worth while to cover them together. Any professional-quality script should terminate gracefully. That is, if you kill the script, there should be no extra files left over, and all of the processes should quit at the same time. Most people just put all temporary files in the /tmp directory, and hope that eventually these files will be deleted. They will be, but sometimes the temporary files are big, and can fill up the /tmp disk. Also some people don't mind if it takes a while for a script to finish, but if it causes the system to slow down, or it is sending a lot of error messages to the terminal, then you should stop all child processes of your script when your script is interrupted. This is done with a trap command, which takes one string, and any number of signals as an argument. Therefore a script that kills a second script could be written using:
- #!/bin/sh
# execute a script that creates /tmp/job.pid
newscript &
trap 'kill -HUP `cat /tmp/job.pid`' 0 HUP INT TERM
# continue on, waiting for the other to finish
<rest of the script deleted>
Signals are a very crude form of inter-process communication. You can only send signals to processes running under your user name. HUP corresponds to a hang-up, INT is an interrupt, like a control-C, and TERM is the terminate command. You can use the numbers associated with these signals if you wish, which are 1, 2, and 15. Signal number zero is special. It indicates the script is finished. Therefore setting a trap at signal zero is a way to make sure some commands are done at the end of the script. Signal 1, or the HUP signal, is generally considered to be the mildest signal. Many programs use this to indicate the program should restart itself. Other signals typically mean stop soon (15), while the strongest signal (9) cannot be trapped because it means the process should stop immediately. Therefore if you kill a shell script with signal 9, it cannot clean up any temporary files, even if it wanted to.
The previous example with $$ requires the proce