shell用法

单引号、双引号、反引号

（1）单引号’ '内剥夺所有字符的特殊含义，所有字符都是单纯的字符串而没有特殊功能。
如，$取参数等命令是无效的。

（2）双引号" "中除了字符串，特殊字符是没有被转义的。
如，$等特殊字符一样可以使用其功能。

（3）反引号``是命令替换，即把命令输出结果传给入变量。

[filter@idxdb1 ~/yy]$ cat test_yh.sh 
#!/bin/bash

a="hello://123"
echo "$a"
echo '$a'
echo `ls -l`

[filter@idxdb1 ~/yy]$ ls
analysis_diary  test  test_yh.sh

[filter@idxdb1 ~/yy]$ sh -x test_yh.sh 
hello://123
$a
total 12 drwxrwxr-x 2 filter filter 4096 May 10 14:26 analysis_diary drwxrwxr-x 4 filter filter 4096 May 29 16:41 test -rw-rw-r-- 1 filter filter 66 Jun 3 14:59 test_yh.sh

grep全局查找

Global search Regular Expression and Print out the line

#文件准备
[engine@client1v ~/yy/grep_test]$ ls
ceng1  file0.txt
[engine@client1v ~/yy/grep_test]$ pwd
/home/engine/yy/grep_test
[engine@client1v ~/yy/grep_test]$ cat file0.txt 
hello world
vim is a idea using on servers
shell is a script language used by thousands of people 
I think people like the two beautiful tools most of the time
余姚
[engine@client1v ~/yy/grep_test]$ ls ceng1/
file1.txt
[engine@client1v ~/yy/grep_test]$ cat ceng1/file1.txt 
hello test grep
today is fine
a beautiful bird fly in the sky

#中文查找
#-r   查找当前目录和递归查找当前目录的子目录
[engine@client1v ~/yy/grep_test]$ grep '余姚' -r /home/engine/yy/grep_test/*
/home/engine/yy/grep_test/file0.txt:余姚
[engine@client1v ~/yy/grep_test]$ grep '余姚' -r /home/engine/yy/grep_test/file0.txt 
余姚
[engine@client1v ~/yy/grep_test]$ grep '余姚' /home/engine/yy/grep_test/*   
/home/engine/yy/grep_test/file0.txt:余姚
[engine@client1v ~/yy/grep_test]$ grep '余姚' /home/engine/yy/grep_test/file0.txt 
余姚
[engine@client1v ~/yy/grep_test]$ grep '余姚' /home/engine/yy/grep_test/ceng1/*   
[engine@client1v ~/yy/grep_test]$ grep '余姚' /home/engine/yy/grep_test/ceng1/file1.txt

#英文查找
[engine@client1v ~/yy/grep_test]$ grep beautiful /home/engine/yy/grep_test/*                  
/home/engine/yy/grep_test/file0.txt:I think people like the two beautiful tools most of the time
[engine@client1v ~/yy/grep_test]$ grep beautiful -r /home/engine/yy/grep_test/* 
/home/engine/yy/grep_test/ceng1/file1.txt:a beautiful bird fly in the sky
/home/engine/yy/grep_test/file0.txt:I think people like the two beautiful tools most of the time
[engine@client1v ~/yy/grep_test]$ pwd                                             
/home/engine/yy/grep_test
[engine@client1v ~/yy/grep_test]$ grep beautiful ./*                                
./file0.txt:I think people like the two beautiful tools most of the time
------------------------------------------------------------------------------------[以下是错误查找方式]
[engine@client1v ~/yy/grep_test]$ grep beautiful /home/engine/yy/grep_test
[engine@client1v ~/yy/grep_test]$ grep beautiful /home/engine/yy/grep_test/
[engine@client1v ~/yy/grep_test]$ grep beautiful .
[engine@client1v ~/yy/grep_test]$ grep beautiful ./
[engine@client1v ~/yy/grep_test]$ 

#-c   只输出匹配行的计数
[engine@client1v ~/yy/grep_test]$ grep beautiful -c /home/engine/yy/grep_test/*
/home/engine/yy/grep_test/ceng1:0
/home/engine/yy/grep_test/file0.txt:1

#-l   只列出匹配的文件名
[engine@client1v ~/yy/grep_test]$ grep beautiful -r -l /home/engine/yy/grep_test/*
/home/engine/yy/grep_test/ceng1/file1.txt
/home/engine/yy/grep_test/file0.txt
----------------------------------------------------
#-L   只列出不匹配的文件名
[engine@client1v ~/yy/grep_test]$ grep '余姚' -r -L /home/engine/yy/grep_test/*          
/home/engine/yy/grep_test/ceng1/file1.txt
[engine@client1v ~/yy/grep_test]$ grep '余姚' -r -l /home/engine/yy/grep_test/*
/home/engine/yy/grep_test/file0.txt

#-w   只匹配整个单词，而不是字符串的一部分
[engine@client1v ~/yy1/grep_test]$ grep a -w /home/engine/yy/grep_test/ceng1/file1.txt
a beautiful bird fly in the sky
##today is fine这一行没有匹配

#grep pattern1 files | grep pattern2  显示既匹配 pattern1 又匹配pattern2 的行 [交集]
[engine@client1v ~/yy1/grep_test]$ grep a -w /home/engine/yy/grep_test/ceng1/file1.txt
a beautiful bird fly in the sky
[engine@client1v ~/yy1/grep_test]$ grep is -w /home/engine/yy/grep_test/ceng1/file1.txt 
today is fine
[engine@client1v ~/yy1/grep_test]$ grep a -w /home/engine/yy/grep_test/ceng1/file1.txt | grep is -w
[engine@client1v ~/yy1/grep_test]$

seq生成数组

[engine@client1v ~test]$ seq 1 3 10
1
4
7
10

for i in {1..5}
do
echo $i;
done

for i in $(seq 1 5)
do
echo $i;
done

在某个目录及其子目录下查找某个文件名

#目录递归查找精确文件名
find . -name "hadoop.py"

#目录递归查找匹配文件名
find . -name "*util*"

#指定查找深度为1
find . -maxdepth 1 -name "*util*"

输出文件指定行

#准备文件
[engine@client1v ~/yy1/grep_test]$ cat test_line.txt 
hello
this is a file
show you something
the thing may be related to shell
a useful tool called sed tail head
you could use them to find content in pointed lines
wait a minute
we will start as soon as possible

#输出文件第一行
[engine@client1v ~/yy1/grep_test]$ sed -n 1p test_line.txt 
hello
#输出文件第一至第三行
[engine@client1v ~/yy1/grep_test]$ sed -n '1,3p' test_line.txt 
hello
this is a file
show you something
#输出文件第一，第三，第四行
[engine@client1v ~/yy1/grep_test]$ sed -n '1p;3,4p' test_line.txt  
hello
show you something
the thing may be related to shell

#输出文件开头两行[默认10行]
[engine@client1v ~/yy1/grep_test]$ head -2 test_line.txt 
hello
this is a file

#输出文件末尾一行[默认10行]
[engine@client1v ~/yy1/grep_test]$ tail -1 test_line.txt  
we will start as soon as possible

#实时监控文件内容增加，默认10行
tail -f test_line.txt

后台运行进程

nohup允许用户在退出登录以后仍可以继续执行命令
&指定程序在后台执行，即在执行的命令后加上&；作用和bg命令相同
fg把后台任务放前台执行

[engine@client221v ~/yy1/get_page_info]$ nohup ./post_crawler.sh &
[1] 7658
[engine@client221v ~/yy1/get_page_info]$ nohup: ignoring input and appending output to `nohup.out'

[engine@client221v ~/yy1/get_page_info]$ jobs -l
[1]+  7658 Running                 nohup ./post_crawler.sh &

[engine@client221v ~/yy1/get_page_info]$ ps -aux | grep post_crawler.sh
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
engine      7658  0.0  0.0 107608  1500 pts/1    S    19:15   0:00 /bin/sh ./post_crawler.sh
engine      24721  0.0  0.0 103244   940 pts/1    S+   19:35   0:00 grep post_crawler.sh

[engine@client221v ~/yy1/get_page_info]$ kill -9 7658
[1]+  Killed                  nohup ./post_crawler.sh

[engine@client221v ~/yy1/get_page_info]$ jobs -l

命令行处理中文编码

[sch@crawl_service]$ export LC_ALL='zh_CN.utf8'

查看磁盘剩余空间及当前目录所在磁盘&文件大小

sch@ds201 [14:56:09]:~/yy1/train_data
$ df -h
文件系统                         容量  已用  可用 已用% 挂载点
/dev/mapper/VolGroup00-LogVol03  100G   90G   11G   90% /
devtmpfs                         126G     0  126G    0% /dev
tmpfs                            126G     0  126G    0% /dev/shm
tmpfs                            126G  4.0G  122G    4% /run
tmpfs                            126G     0  126G    0% /sys/fs/cgroup
/dev/sda1                       1014M  156M  859M   16% /boot
/dev/sdd2                        1.2T  1.2T   22G   99% /da2/sch/bigdata
/dev/mapper/VolGroup00-LogVol01  2.0G   33M  2.0G    2% /tmp
/dev/mapper/VolGroup00-LogVol02   10G  586M  9.5G    6% /var
/dev/md127                       1.5T  1.4T   69G   96% /da2
tmpfs                             26G   32K   26G    1% /run/user/1017

想查看当前目录train_data所在盘，找到当前目录或其上级目录有软连的目录，然后ll
sch@ds201 [14:58:39]:~/yy1/train_data
$ ll
lrwxrwxrwx   1 root            root              11 8月  18 2017 sch -> /da2/sch

sch@tt4 [19:49:14]:/home/sch
$ df -h /home/sch
文件系统        容量  已用  可用 已用% 挂载点
/dev/sdb         37T  177G   37T    1% /da1

sch@ds201 [14:59:09]:~/yy1/train_data
$du -ha pre_vectors.bin
3.5G    pre_vectors.bin