多线程和多进程文件复制

自由，自由，自由；民主，民主，民主；法治，法治，法治

一、cmd 命令数据拷贝

# 命令
xcopy /s /e /i "C:\SourceFolder" "C:\DestinationFolder"

其中，/s 表示复制文件夹及其所有子文件夹和文件，/e 表示包括空的子文件夹，/i 表示如果目标路径不存在则视为目标路径为文件夹。

二、python多线程和多进程数据拷贝

import os
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed

def parallel_copy(source_dir, dest_dir, buffer_size=1024 * 1024, num_threads=4, num_processes=4):
    file_paths = []
    for root, dirs, files in os.walk(source_dir):
        for name in files:
            file_paths.append(os.path.join(root, name))

    with ThreadPoolExecutor(max_workers=num_threads) as thread_executor, \
            ProcessPoolExecutor(max_workers=num_processes) as process_executor:
        futures = []
        for file_path in file_paths:
            dest_file_path = os.path.join(dest_dir, os.path.relpath(file_path, source_dir))
            futures.append(thread_executor.submit(copy_file_using_process, file_path, dest_file_path, buffer_size,
                                                  process_executor))

        for future in as_completed(futures):
            try:
                result = future.result()
            except Exception as e:
                print(f"Exception occurred: {e}")


def copy_file_using_process(src_file_path, dest_file_path, buffer_size, process_executor):
    return process_executor.submit(copy_file, src_file_path, dest_file_path, buffer_size).result()


def copy_file(src_file_path, dest_file_path, buffer_size):
    with open(src_file_path, 'rb') as src_file, open(dest_file_path, 'wb') as dest_file:
        while True:
            data = src_file.read(buffer_size)
            if not data:
                break
            dest_file.write(data)
    return f"File {src_file_path} copied to {dest_file_path}."


if __name__ == '__main__':
    import time
    source_dir = r'C:\Users\user\Desktop\source'
    dest_dir = r'C:\Users\user\Desktop\destination'
    os.makedirs(dest_dir, exist_ok=True)
    start_time = time.time()
    parallel_copy(source_dir, dest_dir, buffer_size=16*1024*1024, num_threads=32,num_processes=16)
    end_time = time.time()

    time_diff = end_time - start_time
    time_diff_in_minutes = time_diff / 60
    print(f"Time taken: {time_diff_in_minutes:.2f} minutes")

三、调试性能比较

数据形式：图像; 数据量：27.6G

• 同一设备下将电脑数据拷贝到移动硬盘中，基于上述代码和参数设置，方式一拷贝需要大约50分钟，方式二需要20分钟。第二种方式相较于第一种方式的速度提升了60%。
• 当然实际拷贝速度取决于你的设备以及线程和进程参数的设置。不过第二拷贝方式明显会优于第一种。

四、完结

🌸🌸🌸撒花🌸🌸🌸

If you feel happy, clap your hands.👏👏👏
注：不建议转载本博客，实在想转载则需注明出处。