MIT-Missing-Semester2: Shell Tools and Scripting

A. Lecture Notes: Shell Tools and Scripting

Shell Scripting

1、shell里的变量

注意单引号和双引号的区别。“”可以把foo的值给打印出来，一般常用“”。

2、shell里的函数

vim mcd.sh，加入下面的内容：

然后执行source mcd.sh

接着执行mcd test

带$的参数：

$0 - Name of the script

$1 to $9 - Arguments to the script. $1 is the first argument and so on.

$@ - All the arguments

$# - Number of arguments

$? - Return code of the previous command

$$ - Process identification number (PID) for the current script

!! - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-execute the command with sudo by doing sudo !!

$_ - Last argument from the last command. If you are in an interactive shell, you can also quickly get this value by typing Esc followed by .

3、||和&&

和一般编程一样：

4、STDOUT STDEER

command return标准输出到STDOUT，标准错误到STDERR，和一个exit code，如果是0则为正常，否则代表有异常。

5、拓展搜索

ls *.sh
打印所有以.sh为后缀的文件

ls project?
打印所有以project+一个字符的文件

6、{}和()。

convert image.{png,jpg}
# Will expand to
convert image.png image.jpg

cp /path/to/project/{foo,bar,baz}.sh /newpath
# Will expand to
cp /path/to/project/foo.sh /path/to/project/bar.sh /path/to/project/baz.sh /newpath

# Globbing techniques can also be combined
mv *{.py,.sh} folder
# Will move all *.py and *.sh files


mkdir foo bar
# This creates files foo/a, foo/b, ... foo/h, bar/a, bar/b, ... bar/h
touch {foo,bar}/{a..h}
touch foo/x bar/y
# Show differences between files in foo and bar
diff <(ls foo) <(ls bar)
# Outputs
# < x
# ---
# > y

7、以python脚本形式运行

#!/usr/local/bin/python
import sys
for arg in reversed(sys.argv[1:]):
    print(arg)

8、finding files

# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '*/test/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'

# Delete all files with .tmp extension
find . -name '*.tmp' -exec rm {} \;
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {}.jpg \;

9、Finding code

rg: ripgrep。

# Find all python files where I used the requests library
rg -t py 'import requests'
# Find all files (including hidden files) without a shebang line
rg -u --files-without-match "^#\!"
# Find all matches of foo and print the following 5 lines
rg foo -A 5
# Print statistics of matches (# of matched lines and files )
rg --stats PATTERN

B. Exercises

1、Read man ls and write an ls command that lists files in the following manner

• Includes all files, including hidden files
• Sizes are listed in human-readable format (e.g. 454M instead of 454279954)
• Files are ordered by recency
• Output is colorized

a：all、t：sort by recency、l：Output is colorized、h：human-readable。

ls -laht

2、Write bash functions marco and polo that do the following. Whenever you execute marco the current working directory should be saved in some manner, then when you execute polo, no matter what directory you are in, polo should cd you back to the directory where you executed macro. For ease of debugging you can write the code in a file marco.sh and (re)load the definitions to your shell by executing source marco.sh.

创建macro.sh：

#!/bin/sh
recent=$(pwd)

然后执行：

source macro.sh

创建polo.sh

polo () {
    cd $recent
}

然后执行：

source polo.sh

再执行

polo

就会进入macro.sh所在的路径了。

3、Say you have a command that fails rarely. In order to debug it, you need to capture its output but it can be time-consuming to get a failure run. Write a bash script that runs the following script until it fails and captures its standard output and error streams to files and prints everything at the end. Bonus points if you can also report how many runs it took for the script to fail.

#!/usr/bin/env bash
count=0

while true; do
	bash ./run.sh &> output.txt # 将./run.sh运行的输出和错误重定向至output.txt
	if [[ $? -ne 0 ]]; then  # 若上一个输出为错误，则停止循环
		break	
	fi
	count=$[count+1]
done
echo "total run times: $count"
cat output.txt

4、As we covered in the lecture find’s -exec can be very powerful for performing operations over the files we are searching for. However, what if we want to do something with all the files, like creating a zip file? As you have seen so far commands will take input from both arguments and STDIN. When piping commands, we are connecting STDOUT to STDIN, but some commands like tar take inputs from arguments. To bridge this disconnect there’s the xargs command which will execute a command using STDIN as arguments. For example ls | xargs rm will delete the files in the current directory.

find . -name "*.html" -print0 | xargs -0 zip a.zip

find . -name “*.html”：寻找html后缀的文件

-print0：每个文件打印一个空格

-0：指定间隔为空格

5、抓取当前目录下最新修改的单个文件。

ls -t | awk '{print $1; exit}'

-t：按最新修改的顺序打印当前目录下的文件

awk ‘{print $1; exit}’：抓取第一个字符

参考：
1、MIT-Missing-Semester。
2、xiaofanwudi。