Linux 如何快速复制文件

最新推荐文章于 2024-07-25 15:27:56 发布

weixin_33721427

最新推荐文章于 2024-07-25 15:27:56 发布

阅读量1.4w

点赞数

文章标签：操作系统运维 python

原文链接：https://my.oschina.net/u/179750/blog/3041591

版权

2019独角兽企业重金招聘Python工程师标准>>>

http://www.cnblogs.com/paul8339/p/7802450.html

1，在需要对大量小文件进行移动或复制时，用cp、mv都会显得很没有效率，可以用tar先压缩再解压缩的方式。
　　

2，在网络环境中传输时，可以再结合nc命令，通过管道和tcp端口进行传输。
　nc和tar可以用来快速的在两台机器之间传输文件和目录，比ftp和scp要来得简单的多。
　由于nc是一个超轻量的命令，所以一般busybox都会集成它。当一个linux终端，比如linux pda,
　通过usblan的方式连接到另一台linux主机的时候，这样的嵌入式终端上一般不会集成ftp server, ssh server
　这样比较笨重的服务，这个时候， nc可能成为唯一的上传手段。
　　

比如将机器A上的mytest目录上传到到机器 B（192.168.0.11）上，只需要：
　　

在机器B上，用nc来监听一个端口，随便就好，只要不被占用；并且将收到的数据用tar展开。-l代表监听模式。

nc -l 6666 |tar -C /target_dir -zxf -

然后，在A上通过nc和 tar发送test目录。使用一致的6666的端口。

tar -zcvf - mytest |nc 192.168.0.11 6666

Sometimes a simple cp -a command is a very painful and slow process. It's true that -v (verbose) option can give you some information on the details of the copy process, but not normally the progress of it. In fact, cp -a is a quite slow process that sometimes is faster (and safer) implemented by tar, for example:

$ tar cf - . | (cd /dst; tar xvf -)

Usually faster, and more verbose. Another commands such as pv can help you too, to monitor the progress of a copy between two directories, for example:

$ tar cf - . | pv | (cd /dst; tar xf -)
2,06GB 0:00:09 [ 194MB/s] [  <=>                     ]

But copying several gigabytes/terabytes of data and many files between quite old NFS disks is painful via cp. Let's see two alternatives for:

Monitoring the progress of the copy and the copied files.
Skipping to next file before an error (gcp)
Syncing directories (rsync)
Copying files via network (rsync)

One of the better commands for doing copies is rsync, that allows you to synchronize two directories, and in this sense src/ can have live data, that incrementally is synced to dst/ in several executions of the command

$ rsync --info=progress2 -auvz ~/Music/ /data/music/

giving a result like this:

Jake Bugg - Jake Bugg Album 2012/
Jake Bugg - Jake Bugg Album 2012/01 - Lighting Bolt.mp3
  1,913,897,967  15%   22.79MB/s    0:01:20 (xfr#277, ir-chk=1019/1825)
Jake Bugg - Jake Bugg Album 2012/05 - Simple As This.mp3
  1,936,698,070  15%   22.80MB/s    0:01:21 (xfr#281, ir-chk=1015/1825)

You can also use it with -n option to perform a dry run (this is more used than the skype test call), that checks and lists the differences between the two given directories. You can use it too with "-e ssh" user@host:dst/ or without --info option in older versions of rsync. It is slower for copying but it does a lot of useful things such syncing, checkings md5sums.... You will remember rsync if something goes bad.

Another fantastic command for copy is gcp. Besides of progress estimation, gcp does not copy when the file exists, skips to the next file if occurs an error, and all the fails are written to a journal file.

$ gcp -rv ~/Music/* /data/music/
Copying 13.53 GiB   2% |#                                  | 165.50 MB/s ETA:  0:01:25

Please check journal: /home/cesar/.gcp/journal

$ cat /home/cesar/.gcp/journal

/home/cesar/Music/Alabama Shakes-Boys & Girls (2014)/01 - Alabama Shakes - Hold On.mp3
FAILED: already exists
/home/cesar/Music/Alabama Shakes-Boys & Girls (2014)/03 - Alabama Shakes - Hang Loose.mp3
FAILED: already exists

In an Alfresco context, many simple migrations (or restoring processes) are tracked via CIFS or Webdav drives. In these cases the above commands are useful. Even they can be useful, if you are doing a local copy in an Alfresco instance, for performing a later Filesystem Bulk process in Alfresco. From a system administrator point of view, when restoring huge contentstores or Lucene / SOLR indices, or moving backups, these commands can save you so much time.

Another day we took some time in alternatives for scp copies between two machines.

Some useful links for reading and just patience for copying:

NOTE: ~/Music and /data/music are simple tests on a local SSD disk.

转载于:https://my.oschina.net/u/179750/blog/3041591