Objectives
1) Upon completion of this unit, you should be able to:
- Use tools for extracting, analyzing and manipulating text data
Tools for Extracting Text
1) File contents: less and cat
2) File Experts: head and tail
3) Extract by Column: cut
4) Extract by Keyworld: grep
Viewing File Contents - less and cat
1) Cat: dump one or more files to STDOUT
- Multiple files are concatenated together
2) less: view file or STDIN on page at a time
- Useful commands while viewing:
- /text searches for text
- n/N jumps to the next/previous match
- v opens the file in a text editor
- less is the pager used by man
Viewing File Excerpts - head and tail
1) head: Display the first 10 lines of a file
- Use –n to change number of lines displayed
2) tail: Display the last 10 lines of a file
- Use –n to change the number of lines displayed
- Use –f to follow the subsequent additions to the file
- Very useful for monitoring the log files!
Extracting Text by keyword - grep
1) Print lines of files to STDIN where a pattern is matched
- grep ‘john’ /etc/passwd
- date –help | grep year
2) Use –i to search case-insensitively
3) Use –n to print line number of matches
4) Use –v to print lines not containing pattern
5) Use –Ax to include the x lines after each match
6) Use –Bx to include the x lines before each match
7) Use –l to return the name of the files that containing the pattern
Extracting Text by Column - cut
1) Display a specified columns of file or STDIN data
- cut –d: –f1 /etc/passwd
- grep root /etc/passwd | cut –d: –f7
2) Use –d to specify the column delimiter (default is TAB)
3) Use –f to specify the column to print
4) Use –c to cut by characters
- cut –c2 –5 /etc/share/dict/words
Tools for Analyzing Text
1) Text Stats: wc
2) Sorting text: sort
3) Comparing files: diff and match
4) Spell check: aspell
Gathering Text Statistics - wc (word count)
1) Counts words, lines, bytes and characters
2) Can act upon a file or STDIN
3) Use –l for only line count
4) Use –w for only word count
5) Use –c for only byte count
6) Use –m for character count (not displayed)
Sorting Text - sort
1) Sort text to STDOUT – original file unchanged
- sort [options] file(s)
2) Common options
- -r performs a rerverse (descending) sort
- -n performs a numeric sort
- -f ignores (folds) case of characters in strings
- -u (unique) removes duplicate lines in output
- -t c uses c as a field separator
- -k x sorts by c-delimited field x, can be used mutiple times
sort –t : –k 3 –n /etc/passwd
Eliminating Duplicate Lines – sort and uniq
1) sort –u: removes duplicated lines from input
2) uniq: removes duplicate adjacent lines from input
- Use –c to count number of occurences
- Use with sort for best effect:
sort userlist.txt | uniq –c
Comparing Files – diff
1) Compares two files for differences
2) Use gvimdiff for graphical diff
- Provided by vim-X11 package
Duplicating File Changes – patch
1) diff output stored in a file is called a “patchfile”
- Use –u for “unified” diff, best in patchfiles
2) patch duplicates changes in other files (use with care!)
- Use –b to automatically backup changed file
diff –u foo.conf-broken foo.conf-works > foo.patch
patch –b foo.conf-broken foo.patch
Spell Checking with aspell
1) Interactively spell-check files:
- aspell check letter.txt
2) Non-interactively list mis-spelled words in STDIN
- aspell list < letter.txt
- aspell list < letter.txt | wc –l
Tools for Manipulating Text – tr and sed
1) Alter (translate) Character: tr
- Converts characters in one set to corresponding characters in another set
- Only reads data from STDIN
$ tr ‘a-z’ ‘A-Z’ < lowercase.txt
2) Alter Strings: sed
- stream editor
- Performs search/replace operations on a stream of text
- Normally does not alter source file
- Use –i.bak to backup and alter source file
- -i : case-insensitive
- -g: global
sed ‘s/cat/dog/’ pets
sed ‘s/cat/dog/gi’ pets
Sed Examples
1) Quote search and replace instructions!
2) sed addresses
- sed ‘s/dog/cat/g’ pets
- sed ‘1,50s/dog/cat/g’ pets ###the replacement will only be performed on lines 1 to 50
- sed ‘digby/,/duncan/s/dog/cat/g’ pets ###the replacement will only start on the line that contains the string ‘digby’ and continuing through the line that contains ‘duncan’
3) Multiple sed instructions
- sed –e ‘s/dog/cat/’ –e ‘s/hi/lo/’ pets
- sed –f myedits pets
Special Characters for Complex Searches Regular Expressions
1) ^ represents beginning of line
2) $ represents end of line
3) Character classes as in bash:
- [abc], [^abc]
- [[:upper:]], [^[:upper:]]
4) Used by:
- grep, sed, less, others
End of Unit8
1) Questions and Answers
2) Summary
- Extracting Text: cat, less, head, tail, grep, cut
- Analyzing Text: wc, sort, uniq, diff, patch
- Manipulating Text: tr, sed
- Special Search Characters: ^, $, [abc], [[:alpha:]], [^[:alpha:]], etc
[:digit:]
Only the digits 0 to 9
Only the digits 0 to 9
[:alnum:]
Any alphanumeric character 0 to 9 OR A to Z or a to z.
Any alphanumeric character 0 to 9 OR A to Z or a to z.
[:alpha:]
Any alpha character A to Z or a to z.
Any alpha character A to Z or a to z.
[:blank:]
Space and TAB characters only.
Space and TAB characters only.
[:xdigit:]
Hexadecimal notation 0-9, A-F, a-f.
Hexadecimal notation 0-9, A-F, a-f.
[:punct:]
Punctuation symbols . , " ' ? ! ; : # $ % & ( ) * + - / < > = @ [ ] / ^ _ { } | ~
Punctuation symbols . , " ' ? ! ; : # $ % & ( ) * + - / < > = @ [ ] / ^ _ { } | ~
[:print:]
Any printable character.
Any printable character.
[:space:]
Any whitespace characters (space, tab, NL, FF, VT, CR). Many system abbreviate as /s.
Any whitespace characters (space, tab, NL, FF, VT, CR). Many system abbreviate as /s.
[:graph:]
Exclude whitespace (SPACE, TAB). Many system abbreviate as /W.
Exclude whitespace (SPACE, TAB). Many system abbreviate as /W.
[:upper:]
Any alpha character A to Z.
Any alpha character A to Z.
[:lower:]
Any alpha character a to z.
Any alpha character a to z.
[:cntrl:]
Control Characters NL CR LF TAB VT FF NUL SOH STX EXT EOT ENQ ACK SO SI DLE DC1 DC2 DC3 DC4 NAK SYN ETB CAN EM SUB ESC IS1 IS2 IS3 IS4 DEL.
Control Characters NL CR LF TAB VT FF NUL SOH STX EXT EOT ENQ ACK SO SI DLE DC1 DC2 DC3 DC4 NAK SYN ETB CAN EM SUB ESC IS1 IS2 IS3 IS4 DEL.