附加参数提升Rclone/Gclone/Fclone的上传速度

最新推荐文章于 2024-05-25 00:28:15 发布

csdn_life18

最新推荐文章于 2024-05-25 00:28:15 发布

阅读量3.2k

点赞数 1

文章标签： golang

原文链接：https://www.blueskyxn.com/202101/3233.html

版权

背景

以Rclone上传-本地toOD为例

我的自用上传命令是这个

rclone copy /up od:up

个人常用附加参数有

–ignore-existing -u -v -P --transfers=10 --ignore-errors --buffer-size=200M --check-first --checkers=10

–ignore-existing

Using this option will make rclone unconditionally skip all files that exist on the destination, no matter the content of these files.

While this isn’t a generally recommended option, it can be useful in cases where your files change due to encryption. However, it cannot correct partial transfers in case a transfer was interrupted.

使用此选项将使rclone无条件地跳过目标上存在的所有文件，无论这些文件的内容如何。

尽管通常不建议使用此选项，但在文件由于加密而更改的情况下很有用。但是，在传输中断的情况下，它无法纠正部分传输。

个人建议

和U/check一起时候需要注意冲突（虽然也能执行，效果不同），根据需求开启

–check-first --checkers=10

If this flag is set then in a sync, copy or move, rclone will do all the checks to see whether files need to be transferred before doing any of the transfers. Normally rclone would start running transfers as soon as possible.

This flag can be useful on IO limited systems where transfers interfere with checking.

Using this flag can use more memory as it effectively sets --max-backlog to infinite. This means that all the info on the objects to transfer is held in memory before the transfers start.

如果设置了此标志，则在一sync，copy或move，rclone会做所有的检查，看是否文件需要做任何的转移之前转移。通常，rclone将尽快开始运行传输。

该标志在传输受限于IO的IO受限系统上很有用。

使用此标志可以占用更多内存，因为它实际上设置 --max-backlog为无限。这意味着在传输开始之前，所有有关传输对象的信息都保存在内存中。

The number of checkers to run in parallel. Checkers do the equality checking of files during a sync. For some storage systems (eg S3, Swift, Dropbox) this can take a significant amount of time so they are run in parallel.

The default is to run 8 checkers in parallel.

要并行运行的检查程序的数量。校验器在同步期间对文件进行相等性检查。对于某些存储系统（例如S3，Swift，Dropbox），这可能会花费大量时间，因此它们可以并行运行。

默认为并行运行8个检查程序。

个人建议

如果你带宽跑不满，那就往上拉，小文件多，也往上拉，注意内存使用

另外Gclone不支持这个（至少我用的版本是这样），R/Fclone正常

-u

This forces rclone to skip any files which exist on the destination and have a modified time that is newer than the source file.

This can be useful when transferring to a remote which doesn’t support mod times directly (or when using --use-server-modtime to avoid extra API calls) as it is more accurate than a --size-only check and faster than using --checksum.

If an existing destination file has a modification time equal (within the computed modify window precision) to the source file’s, it will be updated if the sizes are different. If --checksum is set then rclone will update the destination if the checksums differ too.

If an existing destination file is older than the source file then it will be updated if the size or checksum differs from the source file.

On remotes which don’t support mod time directly (or when using --use-server-modtime) the time checked will be the uploaded time. This means that if uploading to one of these remotes, rclone will skip any files which exist on the destination and have an uploaded time that is newer than the modification time of the source file.

这将强制rclone跳过目标上存在的任何文件，并且修改后的时间比源文件新。

当直接传输到不支持mod时间的远程服务器（或--use-server-modtime用于避免额外的API调用）时，此方法很有用，因为它比--size-only检查更准确，比using更快--checksum。

如果现有目标文件的修改时间与源文件的修改时间相等（在计算的修改窗口精度内），则文件大小不同时将对其进行更新。如果--checksum设置为，则如果校验和也不同，则rclone将更新目标。

如果现有目标文件早于源文件，则如果大小或校验和与源文件不同，它将被更新。

在不直接（或使用--use-server-modtime）不支持mod时间的遥控器上，检查的时间将是上载的时间。这意味着，如果上载到这些远程控制器之一，则rclone将跳过目标上存在的任何文件，并且上载时间比源文件的修改时间新。

-v

With -v rclone will tell you about each file that is transferred and a small number of significant events.

With -vv rclone will become very verbose telling you about every file it considers and transfers. Please send bug reports with a log with this setting.

使用-vrclone可以告诉您有关传输的每个文件以及少量重要事件的信息。

使用-vvrclone会变得非常冗长，告诉您它考虑和传输的每个文件。请发送带有此设置的日志的错误报告。

-P

This flag makes rclone update the stats in a static block in the terminal providing a realtime overview of the transfer.

Any log messages will scroll above the static block. Log messages will push the static block down to the bottom of the terminal where it will stay.

Normally this is updated every 500mS but this period can be overridden with the --stats flag.

This can be used with the --stats-one-line flag for a simpler display.

Note: On Windows until this bug is fixed all non-ASCII characters will be replaced with . when --progress is in use.

该标志使rclone更新终端中静态块中的统计信息，从而提供传输的实时概览。

任何日志消息都将滚动到静态块上方。日志消息会将静态块向下推到将要停留的终端底部。

通常，此间隔每500mS更新一次，但是可以用该--stats标志覆盖该时间段。

可以与--stats-one-line标志一起使用，以简化显示。

注意：在Windows上，在修复此错误之前，所有非ASCII字符都将在使用.时替换--progress为。

–transfers=10

The number of file transfers to run in parallel. It can sometimes be useful to set this to a smaller number if the remote is giving a lot of timeouts or bigger if you have lots of bandwidth and a fast remote.

The default is to run 4 file transfers in parallel.

要并行运行的文件传输数。如果遥控器的超时时间很多，将其设置为较小的值有时会很有用；如果您的带宽较大且使用快速的遥控器，则将其设置为更大的值有时会很有用。

默认为并行运行4个文件传输。

–buffer-size=200M

Use this sized buffer to speed up file transfers. Each --transfer will use this much memory for buffering.

When using mount or cmount each open file descriptor will use this much memory for buffering. See the mount documentation for more details.

Set to 0 to disable the buffering for the minimum memory usage.

Note that the memory allocation of the buffers is influenced by the –use-mmap flag.

使用此大小的缓冲区可加快文件传输速度。每个人都--transfer 将使用这么多的内存进行缓冲。

当使用mount或cmount每个打开的文件描述符时，都将使用这么多的内存进行缓冲。有关更多详细信息，请参见安装文档。

设置为0禁用缓冲以最小化内存使用量。

请注意，缓冲区的内存分配受 –use-mmap标志的影响。

个人建议

默认据说是16M，内存占用量大概就是buffer-size和transfers的乘积

10X200M=2G，但是实际占用更大，请参考DEMO

–ignore-errors

跳过错误。比如 OneDrive 在传了某些特殊文件后会提示Failed to copy: failed to open source object: malwareDetected: Malware detected，这会导致后续的传输任务被终止掉，此时就可以加上这个参数跳过错误。但需要注意 RCLONE 的退出状态码不会为0。