Shell - MIT Missing Semester【1】

Topic 1: The Shell

• On Linux and Mac OS these paths are separated by forward slashes. The path / is the “root” of the file system , under which all directions and files lies. In the root directory, so the slash at the very beginning. It indicates that this is starting from the top of the file system, then look inside the next directory, then look for the file called echo. Everything lives under the root name spac, so all paths start with a slash or all absolute paths .
• On Windows, paths are usually separated by back slashes instead. There is one root for every partition, so you might have seen things like C colon backslash. So Windows has separate sort of file system path hierarchies for each Drive.
• A path that starts with forward slash is called an absolute path. Any other path is a relative path.

command/program

pwd print work directory
The cd expands to change directory.
ls list directory contents

• -l use a long listing format (hint: look at the permission bits of the file).

missing % ls -l
total 8
-rw-r--r--@ 1 owner  owning_group  61 Oct  9 21:44 semester

First, the d at the beginning of the line tells us that the content in the path is a directory, and the - means file.
Then follow three groups of three characters(rwx) indicate what permissions the owner of the file, the owning group, and everyone else respectively have on the relevant item.
A - indicates that the given principal does not have on the given permission.
To list its contents, a user must have read (r) permissions on that directory/file.
A w indicates that the given principle is allowed to modify the directory/file(i.e., add/remove files in directory).
To enter a directory or to execute a file, a user must have “search” (represented by “execute”: x) permissions on that directory or file (and its parents).

mv to rename/move a file
cp to copy a file
rm remove a file or directory 删除文件/文件夹
rmdir remove a directory if it is empty 删除空文件夹
mkdir to make a new directory 新建文件夹

-p Create intermediate directories as required. 确保中间目录存在

cat prints the contents of a file. It also supports this kind of rewiring. cat which by default just duplicates its input to its output.
tail prints the last n lines of its input.
curl client URL 请求Web服务器

curl [options / URLs]

curl normally displays a progress meter during operating, indicating the amount of transferred data, transfer speeds and estimates time left, etc. You can also disable the progress meter completely with the -s, --silent option.
-I, --head (HTTP FTP FILE) Fetch the headers only.

sudo do something as super user. sudo su is saying run the following command as root and su is a complicates command that effectively gets you a shell as the super user. then type password and exit
touch change file access and modification times. If any file does not exist, it is created. 新建文件
sh is a POSIX(Portable Operating System Interface of UNIX，可移植操作系统接口)-compliant command interpreter (shell). （Interpreter is used to directly execute program instructions, which is done line by line.)
chmod: change file modes or Access Control Lists.

cmho u+x semester
cmho 777 semester

The grep utility searches any given input files, selecting lines that match one or more patterns.
echo: write arguments to the standard output.

character/prompt

yourmachinename: ~$

Current working directory:
The tilde(~) is short for “home”. 即环境变量$HOME，代表用户主目录 /Users/Owner。cd is the shortcut of cd~
The $ tells you that you are not the root user. % in zsh.
The pound symbol(#) indicates run this as root
The dash(-): the directory you were previously in.
In a path, . refers to the current directory, and .. to its parent directory. 当前目录和上级目录

• 键入脚本文件的路径后回车，可执行文件./filename
• 返回上一级目录 cd ..

The pipe is a vertical bar(|). It means take the output of the program to the left, and make it the input of the program to the right

Quoting 引用

Quoting is used to remove the special meaning of certain characters or words to the shell.

• Escape character(\) preserves the literal value of next character that follows.
• Enclosing characters in single quotes(') preserves the literal of each character within the quotes. 单引号保留字面值。A single quote may not occur between single quotes, even when preceded by a blaskslah.
  -双引号保留除含特殊含义的元字符以外字符的字面值。 Enclosing characters in double quotes(”) preserves the literal value of all characters within the quotes, with the exception of ‘$’, ‘`’, ‘’, and, when the history expansion is enable, ‘!’.

flags and optionos

most commands accept flags and options (flags with values) that start with - to modify their behavior.
For example, running a program with the -h or --help flag will print some help text that tells you what flags and options are available.
How can I get a help message for zsh builtins?

% run-help <builtin>

man takes as an argument of a program, and shows you its manual page. Press q to exit.

Connecting programs

< file and > file let you rewire重定向 the input and output streams of a program to a file respectively.

• the left angle bracket(<) indicates rewire the input for this program to be the contents of this file, and the end angle bracket(>) means rewire the output preceding program into this file. 创建或清空写入
• A double end bracket(>>) which is append instead of overwrite to a file. 追加

echo sth. >> filename

shortcut

Ctrl+L: let’s you clear your terminal and go back to the top.

Others

#! Shebang(“hash-bang” redirects here.) When a text file with a shebang is used as if is an executable in a Unix-like operating system, the program loader mechanism parses the rest of the file’s initial as an interpreter directive. 指定解释器，后跟解释器位置。
Bash脚本的SheBang

Use | and > to write the “last modified” date output by semester into a file called last-modified.txt in your home directory.

Exercises

cd /tmp
tmp % mkdir missing
tmp % man touch
tmp % touch semester
tmp % echo '#!/bin/sh' > semester
tmp % echo curl --head --silent https://missing.csail.mit.edu >> semester
tmp % ./semester
zsh: command not found: semester
tmp % ls -l
tmp % chmod u+x semester
tmp % ./semester
tmp % ./semester | grep last-modified > ~/last-modified.txt

Topic 2 Shell Tools and Scripting

Shell Scripting

To assign variables in bash, use the syntax foo=bar and access the value of the variable with $foo.访问变量所存储数值的语法

% foo=bar
% echo $foo
bar
% foo = bar # In shell scripts the space character will perform arguments splitting.
zsh: command not found: foo
% echo "Value is %foo"
Value is bar
% echo 'Value is %foo' # Strings delimited with `'` are literal strings and will not substitute variable values wheras `"` delimited strings will.
Value is %foo 

vim is a programmer’s text editor.

命令模式：刚启动vi/vim即进入命令模式。常见的命令：i切换到输入模式；x删除当前光标所在处的字符；:切换到底线命令模式。
输入模式：esc退出输入模式，切换到命令模式。
底线命令模式：可以输入耽搁或多个字符的命令。wq保存文件并退出程序。esc推出底线命令模式。

source Shell builtin commands are commands that can be excused within the running shell’s process.
Unlike other scripting languages, bash uses a variety of special variables to refer to arguments, error codes, and other relevant variables.
$0 - Name of the script
$1 to $9 - Arguments to the script
$@ - All the arguments
$# - Number of arguments
$_ - Last argument from the last command.
$? - Return code of the previous command.
$$ - Process identification number (PID) for the current script
!! - Entire last command, including arguments. A common pattern is to execute a command only for it to fail due to missing permissions; you can quickly re-excute the command with sudo bu doing sudo !!

~/missing/tools % vim mcd.sh
i
mcd () {
	mkdir -p "$1"
	cd "$1" # `$1`to`$9`-Arguments to the script.
}
esc
:wq
~/m/t % source mcd.sh # excute this script in our shell and load it.
~/m/t % mcd test
~/m/t/test % cd ..
~/m/t % rmdir test
~/m/t % mkdir test
~/m/t % cd $_
~/m/t/test % cd ..
~/m/t % mkdir /mnt/new
mkdir: /mnt/new: Permission denied
~/m/t % sudo !!
sudo mkdir /mnt/new
Password:

A value of 0 usually means everything went OK; anything different from 0 means an error occurred.
Exit codes can be used to conditionally execute commands using && (and operator) and || (or operator), both of which are short-circuiting operators.
Commands can also be separated within the same line using a semicolon ;.

% echo "Hello"
Hello
% echo $?
0
% grep foobar mcd.sh
% echo $?
% 1
% false || echo "Oops fail"
Oops fail
% true || echo "Will be not be printed" # Since true will have a zero error code, then the second one will be short-circuited短路 and it won't be printed. 
% true && echo "Things went well"
Things went well
% false 

Another common pattern is wanting to get the output of a command as a variable. This can be done with command substitution 命令替换. Whenever you place $( CMD ) it will execute CMD, get the output of the command and substitute it in place.
A lesser known similar feature is process substitution 进程替换, <( CMD ) will execute CMD and place the output in a temporary file and substitute the <() with that file’s name.

~/m/t % cat <(ls) <(ls ..) # It will concatenate both files in current and parent dirs.
mch.sh
test
semester
testsemester
tools

test - check file type and compare values

-ne Interger1 is not equal to integer2

# example.sh
#!/bin/bash

echo "Starting program at $(date)" 

echo "Running program $0 with $# arguments with pid $$"

for file in "$@"; do
	grep foobar "$file" > /dev/null 2> /dev/null # ?
	# When pattern is not found, grep has exit status 1
	# We redirect STDOUT and STDERR to a null register since we do not care about them
	if [[ $? -ne 0 ]]; then
		echo "File $file does not have any foobar, adding one"
		echo "# foobar" >> "$file"
	fi # ?
done

% ./example.sh mch.sh example.sh
Starting program at Wed Oct 19 23:14:08 CST 2022
Running program ./example.sh with 2 arguments with pid 72439"
File mcd.sh does not have any foobar, adding one

Shell globbing 通配: Expanding expressions by carrying out filename expansion, you will provide arguments that are similar when launching scripts.

• Wildcards 通配符 - you can use ? and * to match one or any amount of characters respectively.
• Curly braces {} - Whenever you have a common substring in series of commands, you can use curly braces for bash to expand this automatically. 在一系命令中有公共字符串，开使用{}自动展开。This comes in very handy when moving or converting files.

convert image.{png,jpg}
# Will expand to
convert image.png image.jpg

mkdir foo bar
# This creates files foo/a, foo/b, ... foo/h, bar/a, ... bar/h
touch {foo, bar}/{a..h}
touch

Shell Tools

Finding how to use commands
the first-order approach is to call said command with the -h or --help flags.A more detailed approach is to use the man command, short for manual.
TLDR pages are a nifty complementary solution that focuses on giving example use cases of a command so you can quickly figure out which options to use.

Finding files
find will recursively search for files matching some criteria.
SYNOPSIS

find path -option [-print] [-exec -ok command] {} \; #在命令列上第一个-之前的部分为path，之后是expression。path是空字传则使用目前路径，expression是空字传则使用-print

listing files.

# Find all directories named src
find . -name src -type d
# Find all python files that have a folder named test in their path
find . -path '*/test/*.py' -type f
# Find all files modified in the last day
find . -mtime -1
# Find all zip files with size in range 500k to 10M
find . -size +500k -size -10M -name '*.tar.gz'

Perform actions over files that match your query. The property can be incredibly helpful to simplify what could be fairly monotonous tasks.

# Delete all files with .tmp extension
find . -name '*.tmp' -exec rm {} \;
# Find all PNG files and convert them to JPG
find . -name '*.png' -exec convert {} {}.jpg \;

================
Finding code grep

-C for getting Content around the matching line. grep -C 5 will print 5 lines before and after the match.
-v for inverting the match, i.e. print all lines that do not match the pattern.
grep xxxx filename | head/tail -x 匹配前/后5个结果

Fingding shell commands history
Directory Navigation

Exercises

ls

• -a Include directory entries whose names begin with a dot, including hidden files.
• -h When use with the -l option, use unit suffixes. Sizes are listed in human readable format (e.g. 454M instead of 454279954)
• -t Sort by descending time modified (most recently modified first). Files are ordered by recency. 按照最近使用时间降序排列
• -G Enable colorized output. This option is equivalent to defining CLICOLOR or COLORTERN in the environment and setting --color=auto.

~/m/t % ls -a -l -h -t -G --color=auto
total 40
-rw-r--r--@ 1 owner  owning_group   6.0K Oct 23 19:42 .DS_Store
drwxr-xr-x  7 owner  owning_group   224B Oct 20 22:28 .
-rw-r--r--  1 owner  owning_group    35B Oct 20 22:28 mch.sh
-rw-r--r--  1 owner  owning_group    44B Oct 19 23:14 mcd.sh
-rwxr--r--  1 owner  owning_group   439B Oct 19 23:09 example.sh
drwxr-xr-x  2 owner  owning_group    64B Oct 17 18:51 test
drwxr-xr-x  6 owner  owning_group   192B Oct 17 17:46 ..

Others

如何退出python3

~ % python3
>>> exit()
~ %

unzip lab00.zip # expand the zip archive
mv <source path> <destination path> # moving files

You can type open . on MacOS to open the current directory in your Finder.
source Shell builtin-in command can be executed within the running shell’s process.
export 设置或显示环境变量，效力仅限于该次登录