Bash Shell in Linux

原文http://blog.sina.com.cn/s/blog_8bb0a3bd010171cp.html

工作中经常在shell脚本中看到set的这两个用法，但就像生活中的很多事情，习惯导致忽视，直到出现问题才引起关注。

1. set -e

set命令的-e参数，linux自带的说明如下：
"Exit immediately if a simple command exits with a non-zero status."
也就是说，在"set-e"之后出现的代码，一旦出现了返回值非零，整个脚本就会立即退出。有的人喜欢使用这个参数，是出于保证代码安全性的考虑。但有的时候，这种美好的初衷，也会导致严重的问题。

真实案例：
脚本a.sh开头使用了"set -e"，且能正常运行。在几个月或更久以后，因需求升级，在脚本中增加了3行hadoop操作：
#!/bin/bash
set -e
...
/home/work/.../hadoopdfs -rmr /app/.../dir
/home/work/.../hadoopdfs -mkdir /app/.../dir
/home/work/.../hadoopdfs -put file_1 /app/.../dir/
...
这几行hadoop命令逻辑很简单：在hdfs上清除并新建一个目录，并将一份本地文件推送至这个目录，供后续使用。将这几行单拎出来，在命令行下执行，除了提示待删除的目录不存在，并没有什么问题，文件还是会被推送到指定的地方。

但第一次执行这个脚本的时候，却失败退出了，且导致调用该脚本的程序整体退出，造成了严重的后果。原因是hdfs上还没有这个目录，rmr这一行会返回255，这个值被脚本前方的"set-e"捕捉到，直接导致了脚本退出。

新增的代码本身并没有问题，先删除再新建目录，反而是保证数据安全的比较规范的操作，删除命令本身的容错性，可以保证后续命令正常执行。事实是这个脚本有好几百行，且逻辑比较复杂，在增加这几行代码的时候，开发人员已经不记得这个脚本里还有个"set-e"埋伏着了。

可见设置"set-e"，在脚本开发过程中可能很有帮助，而在开发完成后，特别是对于后期可能有升级的脚本，则可能是埋下了安全隐患。

2. set -o pipefail

对于set命令-o参数的pipefail选项，linux是这样解释的：
"If set, the return value of apipeline is the value of the last (rightmost) command to exit witha  non-zero  status,or zero ifall commands in the pipeline exit successfully. This option is disabled by default."

设置了这个选项以后，包含管道命令的语句的返回值，会变成最后一个返回非零的管道命令的返回值。听起来比较绕，其实也很简单：

# test.sh
set -opipefail
ls./a.txt |echo "hi" >/dev/null
echo$?

运行test.sh，因为当前目录并不存在a.txt文件，输出：
ls: cannot access ./a.txt: No such file or directory
2

# 设置了set -o pipefail，返回从右往左第一个非零返回值，即ls的返回值2

注释掉 set-o pipefail 这一行，再次运行，输出：
ls: cannot access ./a.txt: No such file or directory
0

#没有set -opipefail，默认返回最后一个管道命令的返回值

3. Shell Exit Code

Shell script itself doesn't return any exit codes. Programs (or commands) do. Usually programs and commands, if they do exit() by themselves, put some meaning in the exit code -- you can look it up in the manual. But usually these codes are well below 128 (usually, again). If the command was terminated by a signal, then most shells return <some_value>+<signal_number> as the exit code. For bash, some_value is 128. Your case looks like the last command or program of the script has been terminated by a signal number (141-128)=13, which is SIGPIPE on Linux.

(Also note that the above is correct if you source your script in thesame shell as you are in. If you run it in a new shell, then the exit  code is the exit code of the shell process.)

4. 在sed中引入shell变量

   1） eval sed ’s/$a/$b/’ filename
   2）sed "s/$a/$b/" filename
   3）sed ’s/’$a’/’$b’/’ filename 
   4）sed s/$a/$b/ filename

5. Is it possible to build a code for sending `pseudo-filesystem` to other machine?

Question:

I am on an effort to sending pseudo-filesystem to other machine.

In this case, I am using /proc/cpuinfo and /proc/meminfo as the pseudo-filesystem example.

I have two computer that can communicate each other in ssh.

Now, I am trying to build an *.sh code to secure copy some system file from other computer. The code was:

#INPUT
export MASTER=user@11.1.111.11
export HELPER=user@12.1.122.12


#Obtaining CPU and Memory Data
scp $MASTER:/proc/cpuinfo /master/.
scp $MASTER:/proc/meminfo /master/.
scp $HELPER:/proc/cpuinfo /helper/.
scp $HELPER:/proc/meminfo /helper/.

The idea is that I can run this script in any computer (either Master or Helper computer).

However, later on the copied files are blank and has no information inside.I try tosudo chmod andsudo chown the/proc/ folder but the system said permission denied. FYI, I do not activateroot user and just usingsudo all the time.

Can anybody guide me to some solutions please?

Additional info and improvisation:The full code is in jsfiddle.I am just trying to use collaboration option byjsfiddle though this code is not a JavaScript.http://jsfiddle.net/santosasandy/sWYLL/#&togetherjs=hCO3VuPwO4

Answer:

It appears that scp first uses stat to determine the size of the file being transferred, and then transfers up to that number of bytes. Because/proc is a pseudo-filesystem, and/proc/cpuinfo is a pseduo-file,stat reports its size as zero bytes. Therefore,scp transfers nothing.

In contrast, cp appears to just read blocks from the file until it can read no more, so the zero size reported bystat is irrelevant.

To copy the file to another system it seems you'll have to first use cp to make a local copy, and then usescp to transfer over the network.

or

You can use cat to read from the pseudo files, instead of trying to copy them. For example:

ssh $MASTER 'cat /proc/cpuinfo' > /master/cpuinfo

5. Bash字符串处理

http://bbs.51cto.com/thread-1158753-1.html

http://www.cnblogs.com/frydsh/p/3261012.html

字符串切片：${var:offset:length}

示例：
[root@localhost ~]#mypath="/etc/sysconfig/network-scripts/"    #定义一个变量，等会切这个变量
[root@localhost ~]# echo ${mypath:5}       #偏移5个字符显示
sysconfig/network-scripts/
[root@localhost ~]# echo ${mypath:10}     #偏移10个字符显示
nfig/network-scripts/
[root@localhost ~]# echo ${mypath:5:5}    #偏移5个字符，取5个字符
sysco

取出字符串的最后几个字符：${var: -length}
注意：-length之前有空白字符；
[root@localhost ~]# echo ${mypath: -10}
k-scripts/

基于模式取子串：

${var#*word}：自左而右，查找var变量中存储的字符串中第一次出现的由word所指明的字符，删除此字符及其左侧的所有内容；
示例：
[root@localhost ~]#mypath="/etc/sysconfig/network-scripts"
[root@localhost ~]# echo ${mypath#*/}
etc/sysconfig/network-scripts

${var##*word}：自左而右，查找var变量中存储的字符串中最后一次出现的由word所指明的字符，删除此字符及其左侧的所有内容；
[root@localhost ~]#mypath="/etc/sysconfig/network-scripts"
[root@localhost ~]# echo ${mypath##*/}
network-scripts

再举两个示例：
[root@localhost ~]#mypath="/etc/sysconfig/network-scripts"
[root@localhost ~]# echo ${mypath##*c}
ripts
[root@localhost ~]# echo ${mypath#*c}
/sysconfig/network-scripts

${var%word*}：自右而左，查找var变量中存储的字符串中第一次出现的由word所指明的字符，删除此字符及其右侧的所有内容；
${var%%word*}：自右而左，查找var变量中存储的字符串中最后一次出现的由word所指明的字符，删除此字符及其右侧的所有内容；
示例：
[root@localhost ~]#mypath="/etc/sysconfig/network-scripts"
[root@localhost ~]# echo ${mypath%c*}
/etc/sysconfig/network-s
[root@localhost ~]# echo ${mypath%%c*}
/et

示例：取一个URL的协议和端口
[root@localhost ~]#url="http://www.baidu.com:80"
[root@localhost ~]# echo ${url##*:}
80
[root@localhost ~]# echo ${url%%:*}
http

查找替换：   

${var/pattern/replacement}：查找var变量存储的字符中第一次由pattern匹配到的内容，并替换为replacement；
[root@localhost ~]#url="http://www.baidu.com:80"
[root@localhost ~]# echo ${url/www/WWW}
http://WWW.baidu.com:80
[root@localhost ~]# echo ${url/w/W}
http://Www.baidu.com:80

${var//pattern/replacement}：查找var变量存储的字符中所有能够由pattern匹配到的内容，并替换为replacement；
[root@localhost ~]# echo ${url//w/W}
http://WWW.baidu.com:80

${var/#pattern/replacement}：查找var变量存储的字符中最开始处能够由pattern匹配到的内容，并替换为replacement；
[root@localhost ~]# userinfo="root:x:0:0:rootuser:/root:/bin/bash"
[root@localhost ~]# echo ${userinfo/#root/ROOT}
ROOT:x:0:0:root user:/root:/bin/bash

${var/%pattern/replacement}：查找var变量存储的字符中最后位置能够由pattern匹配到的内容，并替换为replacement；
[root@localhost ~]# userinfo="root:x:0:0:rootuser:/root:/bin/root"
[root@localhost ~]# echo ${userinfo/%root/ROOT}
root:x:0:0:root user:/root:/bin/ROOT

查找删除：   

${var/pattern}：查找var变量存储的字符中第一次由pattern匹配到的内容，并删除；
${var//pattern}：查找var变量存储的字符中所有能够由pattern匹配到的内容，并删除；
${var/#pattern}：查找var变量存储的字符中最开始处能够由pattern匹配到的内容，并删除；
${var/%pattern}：查找var变量存储的字符中最后位置能够由pattern匹配到的内容，并删除；
示例：
[root@localhost ~]# userinfo="root:x:0:0:rootuser:/root:/bin/root"
[root@localhost ~]# echo ${userinfo/root}
:x:0:0:root user:/root:/bin/root
[root@localhost ~]# echo ${userinfo//root}
:x:0:0: user:/:/bin/
[root@localhost ~]# echo ${userinfo/#root}
:x:0:0:root user:/root:/bin/root
[root@localhost ~]# echo ${userinfo/%root}
root:x:0:0:root user:/root:/bin/

字符串大小写转换：

${var^^}：把var变量中的所有小写字母，统统替换为大写；
${var,,}：把var变量中的所有大写字母，统统替换为小写；
示例：
[root@localhost ~]# echo $userinfo
root:x:0:0:root user:/root:/bin/root
[root@localhost ~]# myinfo=${userinfo^^}
[root@localhost ~]# echo $myinfo
ROOT:X:0:0:ROOT USER:/ROOT:/BIN/ROOT
[root@localhost ~]# echo ${myinfo,,}
root:x:0:0:root user:/root:/bin/root

变量赋值：  

${var:-word}：如果变量var为空或未声明，则返回word所表示的字符串；否则，则返回var变量的值；
[root@localhost ~]# echo $name
                      #这行的值为空
[root@localhost ~]# echo ${name:-tom}
tom
[root@localhost ~]# name=hello
[root@localhost ~]# echo ${name:-tom}
hello

${var:=word}：如果变量var为空或未声明，则返回word所表示的字符串，并且把word赋值为var变量；否则，则返回var变量的值；
[root@localhost ~]# echo $name
                     #这行的值为空
[root@localhost ~]# name=${name:-tom}
[root@localhost ~]# echo $name
tom
[root@localhost ~]# name=${name:-jerry}
[root@localhost ~]# echo $name
tom

${var:?error}：如果变量var为空或未声明，则返回error为错误信息；否则，则返回var变量的值；
[root@localhost ~]# echo "User's name is${name:?wrong}"
-bash: name: wrong
[root@localhost ~]# name=tom
[root@localhost ~]# echo "User's name is${name:?wrong}"
User's name is tom

${var:+word}：如果变量var为空或未声明，忽略；否则，则返回word；
[root@localhost ~]# unset name
[root@localhost ~]# echo "User's name is${name:+wrong}"
User's name is
[root@localhost ~]# name=tom
[root@localhost ~]# echo "User's name is${name:+wrong}"
User's name is wrong