fast copy

翻译 2011年10月12日 16:21:16

[edit] The normal way

To copy files recursively from src1/, src2/, to dest/ you do

cp -Rv src1/ src2/ dest/

[edit] The piped way

cp uses character by character copy. Using the kernel pipes support, we could copy files block by block. tar converts the directories recursively into a single stream.At one end we create a single stream out of source directories and at other end we extract this stream and put it in the destination directory. The transfer between the two ends is by means of pipes.

tar -Sc src1/ src2/ | tar -C dest/ -xv

This is not necessarily faster, but is much more flexible. Note that ext2, ext3 and most modern Linux file systems have a capability called 'sparse files'. This is used to store large files with lots of zeroed content in an efficient manner. Most of the time you should not care about this, but certain programs make extensive use of this feature (e.g. net-p2p/mldonkey). You should be careful when copying sparse files using this method, as your disk usage can explode if you forget to use the -S switch on the left side. Whereas the cp utility does handle sparse files automatically, tar (without -S) does not. Sparse files in the source directory will be stored in full representation on the destination directory.

[edit] Network with SCP

If you want to transfer a file, the following may be used

# Remote to local
scp -C remotebox:path/to/sourcefile .
# Local to remote
scp -C localfile remotebox:path/to/destination/

If multiple files or directories are to be transferred, the following may be used

# Remote to local
scp -rC remotebox:path/to/sourcedir .
# Local to remote
scp -Cr localdir remotebox:path/to/destination/

[edit] Network with SSH

A distinction must be whether from a local system to a remote or vice versa is to be copied:

# to the remote system
tar czv dir/files | ssh "tar xz -C /dir/"
# to the remote system with faster encryption 
tar czv ListOfFiles | ssh -c blowfish tar xz -C /home/user/PathToCopy
# from remote to local
ssh tar cz -C BeginDirCopyFiles |tar xz -C DirToCopy

Transfer of individual files can also be made with tar, but also suitable for this cat:

# local to remote
cat dir/file | ssh -C "cat > /dir/file"
# combined with gzip or bz2
cat dir/file | gzip | ssh -C "gunzip > /dir/file"
cat dir/file | bzip2 | ssh -C "bunzip2 > /dir/file"
# remote to local
ssh "cat /dir/file" > dir/file
# combined with gzip or bz2
ssh "cat /dir/file | gzip" | gunzip > dir/file
ssh "cat /dir/file | bzip2" | bunzip2 > dir/file

[edit] Network with netcat

Transfers over netcat can have miniscule CPU needs, unlike transfers over ssh. However the data is transmitted without encryption and authentication.

Destination box: nc -l -p 2342 | tar -C /target/dir -xz -
Source box: tar -cz /source/dir | nc Target_Box 2342

For further CPU use reduction, lzop can be used in place of the tar z option for much faster but less effective compression.

Destination box: nc -l -p 2342 | lzop -d | tar -C /target/dir -x -
Source box: tar -c /source/dir | lzop | nc Target_Box 2342

Network with socat Same idea as Netcat.

Destination box: socat -u - tcp4-listen:2342 | tar x -C /target/dir
Source box: tar c /source/dir | socat -u - tcp4:Target_Box:2342

We can use a variety of compression methods in a general way:

Destination box: socat -u - tcp4-listen:2342 | ${UNZIP} | tar x -C /target/dir
Source box: tar c /source/dir | ${ZIP} | socat -u - tcp4:Target_Box:2342

We then define ZIP as one of the following:

   *  cat
   *  lzop
   *  gzip
   *  bzip2

and UNZIP as:

   *  cat
   *  lzop -d
   *  gunzip
   *  bunzip2

What's the difference?

time -p tar /usr/src/linux-2.6.3 | ${ZIP} | cat > /dev/null
   *  cat: 182MB, 1.1sec
   *  lzop: 64MB, 4sec
   *  gzip --fast: 51MB, 9sec
   *  gzip: 41MB, 18sec
   *  gzip --best: 41MB, 49sec
   *  bzip2: 32MB, 134sec

(edit: umm... this is kind of irrelevant. It all depends on how fast your network connection is, and how fast each computer is. If your processors/disks are really slow and your network is really fast, strait cat would work best. if your processors/disks are really fast but your network is, say, a dialup, or just a slow wireless connection or something, you're way better off with the smallest transfer (like, bzip2 or gzip). Reality is, you're gonna be somewhere in the middle, but there's a reason that they put the kernel in .bz2 on

XenServer中Fast Copy与Full Copy的区别

原文地址: 详细跟踪了一下LVM-Based VDI与File-Based VDI的复制与链接过程,我们可以发现,Fast Copy与Ful...
  • kepa520
  • kepa520
  • 2016年05月30日 22:12
  • 322

fast memory copy code

void static fmemcpy(void *dest, void *src, int n) {#if 1 __asm {    mov  edi,dest    mov  esi,src   ...
  • firetoucher
  • firetoucher
  • 2004年12月15日 10:31
  • 1180


前言: 网络安装服务器套件 Cobbler(补鞋匠)从前,我们一直在做装机民工这份很有前途的职业。自打若干年前 Red Hat 推出了 Kickstart,此后我们顿觉身价倍增。不再需要刻了光盘...
  • zjq001X
  • zjq001X
  • 2017年05月16日 16:33
  • 384

Hadoop实现Clustering by fast search and find of density peaks

Hadoop实现Clustering by fast search and find of density peaks本篇博客参考:Clustering by fast search and find...
  • fansy1990
  • fansy1990
  • 2015年06月04日 18:07
  • 3127

“Clustering by Fast Search and Find of Density Peaks”《SCIENCE》上超赞的聚类算法这个博客写得深入,值得推荐。
  • zhenxin066
  • zhenxin066
  • 2014年11月05日 20:42
  • 1655

夜谈 Java中 Fail-fast 和 Fail-safe 的原理与使用

在我们详细讨论这两种机制的区别之前,首先得先了解并发修改。 1.什么是同步修改? 当一个或多个线程正在遍历一个集合Collection,此时另一个线程修改了这个集合的内容(添加,删除或者修改)。这...
  • bigtree_3721
  • bigtree_3721
  • 2017年03月27日 23:40
  • 693

HDFS FastCopy

一、背景 HDFS原始实现两个集群间数据复制使用的是DistCp的方式,主要通过提交Job的并行读取文件数据,写入到目标文件,这种方式需要使所有block数据经由源集群DN-》client节点-...
  • macyang
  • macyang
  • 2013年07月23日 16:35
  • 1900


  • fyhuihack
  • fyhuihack
  • 2013年02月20日 10:20
  • 1368


fast算法是由 ed Rosten,tom drummond 等提出的。算法的主页: 论文下载地址: http...
  • jia_zhengshen
  • jia_zhengshen
  • 2014年01月02日 18:55
  • 1720


在实时的视频流处理中,需要对每一帧特征提取,对算法处理速度上有很高的要求,传统的SIFT,Harris等特征点提取很难满足。由此提出Fast(Features from Accelerated Se...
  • luoshixian099
  • luoshixian099
  • 2015年09月08日 18:34
  • 5699
您举报文章:fast copy