目录
背景
linux 不玩不知道,一玩吓一跳,如果会一些小命令,shell什么的,那你工作过程中会节省很多时间哦。
上次碰到了需要拆分文件的需求,本来打算写java或者python脚本分一下呢,还得用IO流,还得写代码,后来发现了一个好方法,,那就是直接通过linux里面的命令来进行操作,真尼玛爽,用着... 分享一下文件拆分和合并的操作...
大文件如何进行数据拆分合并?
split命令
可以指定按行数分割和按字节大小分割两种模式。
(1) 按行数分割
# -l 按行数分隔,300 表示行数, 所需拆分的文件 ,拆分后的文件前缀名
split -l 300 large_file.txt new_file_prefix
加上-d,使用数字后缀;加上--verbose,显示分割进度:(必须是linux,Mac不支持哦)
split -l 50000 -d large_file.txt part_ --verbose
(2) 按字节大小分割
split -b 10m large_file.log new_file_prefix
合并-cat命令
cat part_* > merge_file.txt
split和cat命令格式
$ split --h
Usage: split [OPTION]... [FILE [PREFIX]]
Output pieces of FILE to PREFIXaa, PREFIXab, ...;
default size is 1000 lines, and default PREFIX is 'x'.
With no FILE, or when FILE is -, read standard input.
Mandatory arguments to long options are mandatory for short options too.
-a, --suffix-length=N generate suffixes of length N (default 2) 后缀名称的长度 (默认为2)
--additional-suffix=SUFFIX append an additional SUFFIX to file names
-b, --bytes=SIZE put SIZE bytes per output file 每个输出文件的字节大小
-C, --line-bytes=SIZE put at most SIZE bytes of records per output file 每个输出文件每行的最大字节大小
-d use numeric suffixes starting at 0, not alphabetic 使用数字后缀代替字母后缀
--numeric-suffixes[=FROM] same as -d, but allow setting the start value
-e, --elide-empty-files do not generate empty output files with '-n' 不产生空的输出文件
--filter=COMMAND write to shell COMMAND; file name is $FILE 写入到shell命令行
-l, --lines=NUMBER put NUMBER lines/records per output file 设定每个输出文件的行数,默认行数是1000行
-n, --number=CHUNKS generate CHUNKS output files; see explanation below 产生chunks文件
-t, --separator=SEP use SEP instead of newline as the record separator; 使用新字符分割
'\0' (zero) specifies the NUL character
-u, --unbuffered immediately copy input to output with '-n r/...' 无需缓存
--verbose print a diagnostic just before each 显示分割进度
output file is opened
--help display this help and exit 显示帮助信息
--version output version information and exit 显示版本信息
The SIZE argument is an integer and optional unit (example: 10K is 10*1024).
Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000).
CHUNKS may be:
N split into N files based on size of input
K/N output Kth of N to stdout
l/N split into N files without splitting lines/records
l/K/N output Kth of N to stdout without splitting lines/records
r/N like 'l' but use round robin distribution
r/K/N likewise but only output Kth of N to stdout
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/split>
or available locally via: info '(coreutils) split invocation'
$ cat --h
Usage: cat [OPTION]... [FILE]...
Concatenate FILE(s) to standard output.
With no FILE, or when FILE is -, read standard input.
-A, --show-all equivalent to -vET
-b, --number-nonblank number nonempty output lines, overrides -n
-e equivalent to -vE
-E, --show-ends display $ at end of each line
-n, --number number all output lines
-s, --squeeze-blank suppress repeated empty output lines
-t equivalent to -vT
-T, --show-tabs display TAB characters as ^I
-u (ignored)
-v, --show-nonprinting use ^ and M- notation, except for LFD and TAB
--help display this help and exit
--version output version information and exit
Examples:
cat f - g Output f's contents, then standard input, then g's contents.
cat Copy standard input to standard output.
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/cat>
or available locally via: info '(coreutils) cat invocation'
小结
使用文件拆分为了干什么呢?我的场景是为了提高数据传输速率,需要用到文件的拆分,感觉linux命令好强哦,一行代码搞定了别的语言需要好多行代码,好玩不? 在大数据领域内,文件拆分合并还是比较常见的哦~
你学会了吗?