linux怎样复制文件快,linux – 如何尽快复制文件？

我在machineA上运行我的

shell脚本,它将文件从machineB和machineC复制到machineA.

如果文件不在机器B中,那么它应该在machineC中是肯定的.所以我会尝试从机器B首先复制,如果没有在机器B然后我将去machineC复制相同的文件.

在machineB和machineC中,这个文件夹里面会有一个这样的YYYYMMDD文件夹 –

/data/pe_t1_snapshot

所以无论日期是上述文件夹中YYYYMMDD格式的最新日期,我将选择该文件夹作为我需要开始复制文件的完整路径 –

所以假设如果这是最新的日期文件夹20140317里面的/ data / pe_t1_snapshot那么这将是我的完整路径 –

/data/pe_t1_snapshot/20140317

从我需要开始在machineB和machineC中复制文件.我需要从machineB和machineC中复制machineA中的大约400个文件,每个文件大小为1.5 GB.

目前我有我的下面的shell脚本,工作正常,因为我使用的是scp,但不知何故需要〜2个小时来复制400个文件在machineA这是太长了我,我猜. 🙁

以下是我的shell脚本 –

#!/bin/bash

readonly PRIMARY=/export/home/david/dist/primary

readonly SECONDARY=/export/home/david/dist/secondary

readonly FILERS_LOCATION=(machineB machineC)

readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot

PRIMARY_PARTITION=(0 3 5 7 9) # this will have more file numbers around 200

SECONDARY_PARTITION=(1 2 4 6 8) # this will have more file numbers around 200

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1

echo $dir2

if [ "$dir1" = "$dir2" ]

then

# delete all the files first

find "$PRIMARY" -mindepth 1 -delete

for el in "${PRIMARY_PARTITION[@]}"

do

scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.

done

# delete all the files first

find "$SECONDARY" -mindepth 1 -delete

for sl in "${SECONDARY_PARTITION[@]}"

do

scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.

done

fi

我正在PRIMARY文件夹和SECONDARY_PARTITION文件中复制PRIMARY_PARTITION文件在machineA中SECONDARY文件夹中.

有没有办法在machineA中更快地移动文件.一次可以一次复制10个文件或并行复制5个文件,以加快此过程或任何其他方法？

注意：machineA在SSD上运行

更新： –

我尝试的并行Shell脚本,shell脚本的顶部与上述相同.

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]

then

find "$PRIMARY" -mindepth 1 -delete

for el in "${PRIMARY_PARTITION[@]}"

do

(scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.) &

WAITPID="$WAITPID $!"

done

find "$SECONDARY" -mindepth 1 -delete

for sl in "${SECONDARY_PARTITION[@]}"

do

(scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.) &

WAITPID="$WAITPID $!"

done

wait $WAITPID

echo "All files done copying."

fi

我遇到的并行shell脚本的错误 –

channel 24: open failed: administratively prohibited: open failed

channel 25: open failed: administratively prohibited: open failed

channel 26: open failed: administratively prohibited: open failed

channel 28: open failed: administratively prohibited: open failed

channel 30: open failed: administratively prohibited: open failed

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

channel 32: open failed: administratively prohibited: open failed

channel 36: open failed: administratively prohibited: open failed

channel 37: open failed: administratively prohibited: open failed

channel 38: open failed: administratively prohibited: open failed

channel 40: open failed: administratively prohibited: open failed

channel 46: open failed: administratively prohibited: open failed

channel 47: open failed: administratively prohibited: open failed

channel 49: open failed: administratively prohibited: open failed

channel 52: open failed: administratively prohibited: open failed

channel 54: open failed: administratively prohibited: open failed

channel 55: open failed: administratively prohibited: open failed

channel 56: open failed: administratively prohibited: open failed

channel 57: open failed: administratively prohibited: open failed

channel 59: open failed: administratively prohibited: open failed

mux_client_request_session: session request failed: Session open refused by peer

channel 61: open failed: administratively prohibited: open failed

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

mux_client_request_session: session request failed: Session open refused by peer

channel 64: open failed: administratively prohibited: open failed

mux_client_request_session: session request failed: Session open refused by peer

channel 68: open failed: administratively prohibited: open failed

channel 72: open failed: administratively prohibited: open failed

channel 74: open failed: administratively prohibited: open failed

channel 76: open failed: administratively prohibited: open failed

channel 78: open failed: administratively prohibited: open failed