【MIT Missing Semester 2】Shell Tools

这节主要是bash作为脚本语言的一些基本用法，以及一些常见的功能强大的shell tools

Shell Scripting

可以用变量、循环、条件等更为复杂的方式，主要还是为了完成shell-related任务

变量

foo=bar  # 将字符串bar赋值给变量foo
echo fool #这个会把fool理解为字符串

注意单引号和双引号是不一样的，单引号里面是字面量，不会进行变量替换；而双引号会替换

注意foo=bar里是不能有空格的，如果有空格foo = bar会理解为执行一个叫foo的程序，然后有参数=和bar，一般空格在shell里是来分割参数的
还可以将输出转化为变量 --> $(CMD)

foo=$(pwd)
echo $foo
echo "we are in $(pwd)"

另外<(CMD)可以进行process substitution，执行命令，然后将结果放到一个临时文件里，再把这个<()替换为临时文件
比如diff <(ls blogs) <(ls daily)

函数

vim mcd.sh

mcd () {
    mkdir -p "$1"
    cd "$1"
}

source mcd
mcd test
结果会新建一个test文件夹并进入
这里$1指的是第一个参数

• $0 - Name of the script

• $1 to $9 - Arguments to the script. $1 is the first argument and so on.

• $@ - All the arguments

• $# - Number of arguments

• $? - Return code of the previous command，0表示没问题；true总是返回0，false总是返回1；grep查找没找到的时候也会返回1

  • 
    

• $$ - Process identification number (PID) for the current script

• !! - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-execute the command with sudo by doing sudo !!

  • 比如mkdir xxxx发现没有权限，下一行可以直接sudo !!来替换这个路径

• $_ - Last argument from the last command. If you are in an interactive shell, you can also quickly get this value by typing Esc followed by . or Alt+.

逻辑

false总是返回1，true总是返回错误码0.可以用|| &&来连接命令；另外;总是会分割命令

false || echo "Oops, fail"
# Oops, fail

true || echo "Will not be printed"
#

true && echo "Things went well"
# Things went well

false && echo "Will not be printed"
#

true ; echo "This will always run"
# This will always run

false ; echo "This will always run"
# This will always run

分支

#!/bin/bash

echo "Starting program at $(date)" # Date will be substituted

echo "Running program $0 with $# arguments with pid $$"

for file in "$@"; do
    grep foobar "$file" > /dev/null 2> /dev/null
    # When pattern is not found, grep has exit status 1
    # We redirect STDOUT and STDERR to a null register since we do not care about them
    if [[ $? -ne 0 ]]; then
        echo "File $file does not have any foobar, adding one"
        echo "# foobar" >> "$file"
    fi
done

注意是双括号[[ ]]来进行比较
执行时可以./example.sh file1 file2 file3 之类

另外这里第一行的#!指出运行的程序在哪，shell可以区分这是不是个Bash程序
比如Python程序，直接python xxx.py是可以的，不过想要直接./xxx.py就需要在开头标明#!/usr/bin/python3之类的,可以查看PATH路径，写这个是个好习惯，可以让程序更可移植

shell globbing通配符

让一些重复的参数简化的方法

• 通配符 ? *用法类似正则表达式 比如rm foo? rm foo*
• 大括号{} 可以把共同的子串放在大括号前后，bash会自动补充，比如touch {foo,bar}.{a..c} rm {foo,bar}.?

语法检查

可以用shellcheck工具
可以直接复制在线检查 https://www.shellcheck.net

shell tools --查找

tldr 查找如何使用命令

不知道跟什么参数好，可以直接-h 或者–help来查，更详细版本的是man
简化版本的tldr可能会更好用 (too long don’t read)

find 查文件

# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '*/test/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'

还可以在找出来的文件上直接执行操作

# Delete all files with .tmp extension
find . -name '*.tmp' -exec rm {} \;
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {}.jpg \;

• 也可以用fd命令
• locate查找文件索引 只用名字，不过find可以查的更详细

查code

grep 查看文件中是否包含了什么内容
-c 获得上下文 -v 避开一些匹配 -r递归的查文件
其他包括ack rg ag等 rg可以给出color coding，更快

# Find all python files where I used the requests library
rg -t py 'import requests'
# Find all files (including hidden files) without a shebang line
rg -u --files-without-match "^#!"
# Find all matches of foo and print the following 5 lines
rg foo -A 5
# Print statistics of matches (# of matched lines and files )
rg --stats PATTERN

查shell 命令

history可以结合管道和grep

• Ctrl+R 向后寻找，按了之后可以输入substring来从history里查，可以ctrl+R结合fzf来查
• zsh的auto-suggestions插件
• 可以配置.bashrc或者.zshrc来改变history的一些属性

Directory Navigation

• fasd or autojump
• More complex tools exist to quickly get an overview of a directory structure: tree(给出文件结构), broot or even full fledged file managers like nnn or ranger.

练习

1.Read man ls and write an ls command that lists files in the following manner

• Includes all files, including hidden files
• Sizes are listed in human readable format (e.g. 454M instead of 454279954)
• Files are ordered by recency
• Output is colorized
  A sample output would look like this

-rw-r–r-- 1 user group 1.1M Jan 14 09:53 baz
drwxr-xr-x 5 user group 160 Jan 14 09:53 .
-rw-r–r-- 1 user group 514 Jan 14 06:42 bar
-rw-r–r-- 1 user group 106M Jan 13 12:12 foo
drwx------+ 47 user group 1.5K Jan 12 18:08 …

尝试回答：
ls -lh -1 -a -t
其中lh可以显示人类可读的单位大小， -1分行展示，-a显示隐藏文件

2.Write bash functions marco and polo that do the following. Whenever you execute marco the current working directory should be saved in some manner, then when you execute polo, no matter what directory you are in, polo should cd you back to the directory where you executed marco. For ease of debugging you can write the code in a file marco.sh and (re)load the definitions to your shell by executing source marco.sh.
尝试给一种方案：
在marco.sh中

echo $(pwd) > pwdfile
echo $(pwd)
echo "in pwdfile:"
cat pwdfile

在polo.sh里cd $1
执行的时候source polo.sh $(cat ./blogs/pwdfile )
3.Say you have a command that fails rarely. In order to debug it you need to capture its output but it can be time consuming to get a failure run. Write a bash script that runs the following script until it fails and captures its standard output and error streams to files and prints everything at the end. Bonus points if you can also report how many runs it took for the script to fail.

 #!/usr/bin/env bash

 n=$(( RANDOM % 100 ))

 if [[ n -eq 42 ]]; then
    echo "Something went wrong"
    >&2 echo "The error was using magic numbers"
    exit 1
 fi

 echo "Everything went according to plan"

一个尝试的回答，但是大概有问题：

cnt=1
flag=true
while [[ $flag == true ]];
do
       source prog1.sh
       code=$?
       cnt=cnt+1
       if [[ $code -ne 0 ]]; then
               echo "catch wrong"
               echo "wrong when cnt=$cnt"
               echo code>codefile
               flag=false
       fi
done

4.As we covered in the lecture find’s -exec can be very powerful for performing operations over the files we are searching for. However, what if we want to do something with all the files, like creating a zip file? As you have seen so far commands will take input from both arguments and STDIN. When piping commands, we are connecting STDOUT to STDIN, but some commands like tar take inputs from arguments. To bridge this disconnect there’s the xargs command which will execute a command using STDIN as arguments. For example ls | xargs rm will delete the files in the current directory.

Your task is to write a command that recursively finds all HTML files in the folder and makes a zip with them. Note that your command should work even if the files have spaces (hint: check -d flag for xargs).

If you’re on macOS, note that the default BSD find is different from the one included in GNU coreutils. You can use -print0 on find and the -0 flag on xargs. As a macOS user, you should be aware that command-line utilities shipped with macOS may differ from the GNU counterparts; you can install the GNU versions if you like by using brew.
尝试回答：
find . -name '*.html' | xargs tar cjf all_html.tar.gz
5.(Advanced) Write a command or script to recursively find the most recently modified file in a directory. More generally, can you list all files by recency?