Java分布式高性能文件服务

置顶 BossFriday

已于 2024-05-30 09:53:04 修改

阅读量1.9k

点赞数 3

分类专栏：分布式文章标签： java 分布式 rpc 中间件

于 2022-05-06 17:42:51 首次发布

本文链接：https://blog.csdn.net/camelials/article/details/124613041

版权

分布式专栏收录该内容

3 篇文章 1 订阅

订阅专栏

1. 背景

之前写了一个Java版Akka-Rpc：《Java版Akka-Rpc实现与应用》，当时就是做了简单的测试和一个性能测试：《Java版Akka-Rpc性能测试》，为了进一步的检验，想拿一个实际的项目去应用。心里想着搞个CRUD的项目也没有啥意思，正好5年前曾经用JAVA写过一个分布式文件服务，由于当时时间比较紧张（80%的代码是2周写完的），加之后续被一些不知道所以然的人维护的很乱（完成没有多久我转岗了，后来的代码我自己都不想去看，太乱了！），再后来由于维护不了，被人直接把存储部分替换为了FastDFS，只保留了HttpServer的壳了。让我最不可理解的是：100K文件上传的性能只有每秒150左右。于是最近利用业余时间基于之前写的Java版本akka写了一个分布式文件服务。当前100K文件上传和下载的性能可以达到两三千，折算IO为每秒二三百MB，基本复合零拷贝+顺序写盘的预期（由于压测环境使用云虚机，内网网络IO能力不详，因此这是否是文件服务本身的冒烟上限不得而知）。

2. 功能与设计初衷

2.1 设计初衷

【效率】使用netty实现http file server（不聚合httpRequest为FullHttpRequest）。–这些天又回顾了Netty的官方示例，在一些问题的解决中一度也是很苦恼（之前写的文件服务使用了HttpObjectAggregator聚合）
【效率】使用ActorRpc做为RPC组件。
【效率】文件存储落盘采用：零拷贝+顺序写盘，以最大化提升落盘速度。临时文件写入使用零拷贝、存储文件读取采用带环形缓存区RandomAccessFile操作（存储文件读取暂时排除使用MappedByteBuffer，因为MappedByteBuffer使用不当会导致JVM Crash，完成后看压测结果再决定）。–存储文件读取最没有使用带Buffer的RandomAccessFile，而是同样使用零拷贝。当前读写均使用零拷贝，写均保障顺序写盘。
【服务器资源占用】chunk读写机制保障内存占用较小：chunk by chunk发送下载数据、chunk by chunk处理上传数据。 --由于ActorDispatcher内部封装了线程池，保障不了chunkData的串行处理，目前是应用层面做了处理（详见：StorageDispatcher），有点纠结是否将该逻辑下层到RPC层面。好像有人吐槽这是Actor模型的一个弊端，其实根据资源ID去保障执行线程的一致性，这个问题就解了。
【功能】功能规划：普通上传、Base64上传（客户端截屏使用）、普通下载、文件删除、支持2G以上大文件上传下载（断点上传、下载）
【扩展】主要步骤使用接口束行，依据版本获取实例，如果找不到则使用默认实现。这样做目的是为了达到类似装饰者模式的效果。
【安全】文件下载地址防止暴力穷举。
【安全】文件内容以一定数据结构存储与落盘文件中，服务端无法直接还远原始文件。

2.2 主要功能

普通全量上传；
断点上传/下载（较大文件适用；由于数据元数据fileTotalSize用long表达，因此最大支持文件大小可以超过2G ）；
Base64字符串上传（截屏等无原始文件场景使用）；
文件删除（标记删除，因此文件删除后并不能释放磁盘空间）；
文件分桶存储（存储空间可以单独配置过期策略，例如：头像等文件业务上需要永不过期，而其他文件可以允许可以过期）；
过期清理（物理删除，可以释放磁盘空间）；
高性能下载（不管是全量下载还是断点下载均采用不断输出chunk流的方式）；
副本高可用（TODO，目前没有实现从节点副本，只是一个学习用途开源项目）；

2.3 Release Note

【2022-05-06】原型发布，普通全量上传：由于使用了chunk分片机制 + actorRPC，即使大文件不做断点上传，也不至于使文件服务炸线程和内存。chunked下载（完成）、压力测试（完成）。
【2022-11-10】Base64上传：客户端截屏等小文件上传使用（由于对不完整的部分base64字符串解码可能失败，因此需要先在httpServer层面自行进行聚合后再对完整信息进行base64解码，这是base64上传只适用于小文件上传的原因）。
【2022-11-15】断点上传：标记删除，因此不能释放磁盘空间。
【2022-11-21】文件删除
【2024-02-22】过期文件自动清理：物理删除，可以释放磁盘空间。
【2024-02-20】异常恢复：服务异常重启，断电宕机等情况下数据恢复（主要是未落盘的临时文件落盘，不过由于采用了零拷贝+顺序写的机制，临时文件落盘非常高效，一般情况很难撞上）；
【TBD暂不考虑实现】高可用：集群下的主从、文件副本等；主要思路为：由于文件都是按照时间和空间一直往后写，因此可以借鉴binlog的机制，当主发送变化时，立刻给对应的从发一个数据变更通知，从收到该通知之后立刻向主进行同步，从永远追主，只到追平。因此为了实现该机制，需要在落盘的同时去写一个类似的binlog：metaDataIndexLog。另外，为了支持更好的容灾，可以考虑多主多从或者一主多从；

备注：

高可用暂时不考虑实现的主要原因是实现了之后其实可以直接商用（目前只是考虑供学习交流使用）。几年前实现那版以上都进行了实现，并且经历了上百家私有部署IM的项目及公司内部使用的考验。当前的完成情况可以认为是一个准商用水平，因为在多节点部署的情况下，只要不是所有的节点都挂了，功能不受影响，唯一影响的是宕机节点的文件下载（可以通过手工的方式将宕机节点的数据复制到另外的服务器上，即可恢复）。
当前项目的完成度可以充当一个较好的OSS脚手架项目了，大家可以以此为二开的基础（据说七牛在存储层面的处理很多地方与之类似）。

源码地址

https://github.com/bossfriday/bossfriday-nubybear

3. 主要代码

3.1 StorageEngine

存储引擎主体代码，主要负责文件存储读写，例如：上传，下载，临时文件落盘等。

package cn.bossfriday.fileserver.engine;

import cn.bossfriday.common.exception.ServiceRuntimeException;
import cn.bossfriday.common.utils.FileUtil;
import cn.bossfriday.fileserver.actors.model.WriteTmpFileResult;
import cn.bossfriday.fileserver.common.conf.FileServerConfig;
import cn.bossfriday.fileserver.common.conf.FileServerConfigManager;
import cn.bossfriday.fileserver.common.conf.StorageNamespace;
import cn.bossfriday.fileserver.common.enums.OperationResult;
import cn.bossfriday.fileserver.engine.core.BaseStorageEngine;
import cn.bossfriday.fileserver.engine.core.IMetaDataHandler;
import cn.bossfriday.fileserver.engine.core.IStorageHandler;
import cn.bossfriday.fileserver.engine.core.ITmpFileHandler;
import cn.bossfriday.fileserver.engine.enums.StorageEngineVersion;
import cn.bossfriday.fileserver.engine.model.*;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections4.CollectionUtils;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;

import static cn.bossfriday.fileserver.common.FileServerConst.*;

/**
 * StorageEngine
 *
 * @author chenx
 */
@Slf4j
@SuppressWarnings("squid:S6548")
public class StorageEngine extends BaseStorageEngine {

    private static final int RECOVERABLE_TMP_FILE_WARNING_THRESHOLD = 1024 * 5;
    private static final String META_DATA_INDEX_IS_NULL = "MetaDataIndex is null!";
    private static final String META_DATA_IS_NULL = "MetaData is null!";

    private ConcurrentHashMap<String, StorageIndex> storageIndexMap;
    private ConcurrentHashMap<Long, RecoverableTmpFile> recoverableTmpFileHashMap = new ConcurrentHashMap<>();
    private StorageCleaner storageCleaner;

    @Getter
    private File baseDir;

    @Getter
    private File tmpDir;

    @Getter
    private HashMap<String, StorageNamespace> namespaceMap;

    private StorageEngine() {
        super(RECOVERABLE_TMP_FILE_WARNING_THRESHOLD * 2);
        this.init();
    }

    /**
     * getInstance
     */
    public static StorageEngine getInstance() {
        return SingletonHolder.INSTANCE;
    }

    @Override
    protected void startup() {
        try {
            // 未落盘临时文件异常恢复
            this.recoverTmpFile();

            // 加载存储指针
            this.loadStorageIndex();

            // 过期文件自动清理
            this.storageCleaner.startup();
        } catch (Exception ex) {
            log.error("StorageEngine.startup() error!", ex);
        }
    }

    @Override
    protected void shutdown() {
        try {
            this.recoverTmpFile();
            this.storageCleaner.shutdown();
        } catch (Exception ex) {
            log.error("StorageEngine.shutdown() error!", ex);
        }
    }

    @Override
    protected void onRecoverableTmpFileEvent(RecoverableTmpFile event) {
        String fileTransactionId = event.getFileTransactionId();
        try {
            // 存储文件落盘（Disruptor保障先进先出）
            IStorageHandler storageHandler = StorageHandlerFactory.getStorageHandler(event.getStoreEngineVersion());
            long metaDataIndexHash64 = storageHandler.apply(event);
            this.recoverableTmpFileHashMap.remove(metaDataIndexHash64);
            log.info("RecoverableTmpFile apply done, fileTransactionId: " + fileTransactionId + ", offset:" + event.getOffset());
        } catch (Exception ex) {
            log.error("onRecoverableTmpFileEvent() error!" + fileTransactionId, ex);
        }
    }

    /**
     * upload 文件上传
     *
     * @param data
     * @return
     * @throws IOException
     */
    public synchronized MetaDataIndex upload(WriteTmpFileResult data) throws IOException {
        if (data == null) {
            throw new ServiceRuntimeException("WriteTmpFileResult is null!");
        }

        // 申请存储空间
        String fileTransactionId = data.getFileTransactionId();
        int engineVersion = data.getStorageEngineVersion();
        IStorageHandler storageHandler = StorageHandlerFactory.getStorageHandler(engineVersion);
        IMetaDataHandler metaDataHandler = StorageHandlerFactory.getMetaDataHandler(engineVersion);
        ITmpFileHandler tmpFileHandler = StorageHandlerFactory.getTmpFileHandler(engineVersion);
        long metaDataTotalLength = metaDataHandler.getMetaDataTotalLength(data.getFileName(), data.getFileTotalSize());
        int metaDataLength = metaDataHandler.getMetaDataLength(data.getFileName());

        StorageIndex currentStorageIndex = this.getStorageIndex(data.getStorageNamespace(), engineVersion);
        StorageIndex resultIndex = storageHandler.ask(currentStorageIndex, metaDataTotalLength);

        if (resultIndex == null) {
            throw new ServiceRuntimeException("Result StorageIndex is null: " + data.getFileTransactionId());
        }

        long metaDataIndexOffset = resultIndex.getOffset() - metaDataTotalLength;
        if (metaDataIndexOffset < 0) {
            throw new ServiceRuntimeException("metaDataIndexOffset <0: " + data.getFileTransactionId());
        }

        MetaDataIndex metaDataIndex = MetaDataIndex.builder()
                .clusterNode(data.getClusterNodeName())
                .storeEngineVersion(engineVersion)
                .storageNamespace(data.getStorageNamespace())
                .time(resultIndex.getTime())
                .offset(metaDataIndexOffset)
                .metaDataLength(metaDataLength)
                .fileExtName(data.getFileExtName())
                .build();

        // 构建临时文件
        RecoverableTmpFile recoverableTmpFile = RecoverableTmpFile.builder()
                .fileTransactionId(fileTransactionId)
                .storeEngineVersion(data.getStorageEngineVersion())
                .storageNamespace(data.getStorageNamespace())
                .time(resultIndex.getTime())
                .offset(metaDataIndex.getOffset())
                .timestamp(data.getTimestamp())
                .fileName(data.getFileName())
                .fileTotalSize(data.getFileTotalSize())
                .build();
        String recoverableTmpFileName = storageHandler.getRecoverableTmpFileName(recoverableTmpFile);
        String recoverableTmpFilePath = tmpFileHandler.rename(data.getFilePath(), recoverableTmpFileName);
        recoverableTmpFile.setFilePath(recoverableTmpFilePath);

        // 入队落盘
        this.enqueue(metaDataIndex.hash64(), recoverableTmpFile);
        log.info("StorageEngine.upload() done:" + recoverableTmpFile);

        return metaDataIndex;
    }

    /**
     * chunkedDownload 文件分片下载
     *
     * @param metaDataIndex
     * @param metaData
     * @param offset
     * @param limit
     * @return
     * @throws IOException
     */
    public ChunkedMetaData chunkedDownload(MetaDataIndex metaDataIndex, MetaData metaData, long offset, int limit) throws IOException {
        if (metaDataIndex == null) {
            throw new ServiceRuntimeException(META_DATA_INDEX_IS_NULL);
        }

        if (metaData == null) {
            throw new ServiceRuntimeException(META_DATA_IS_NULL);
        }

        // 临时文件落盘采用零拷贝+顺序写盘方式非常高效，因此这里采用自旋等待的无锁方式
        for (int i = 0; ; i++) {
            if (!this.recoverableTmpFileHashMap.containsKey(metaDataIndex.hash64())) {
                break;
            }

            if (this.recoverableTmpFileHashMap.size() > RECOVERABLE_TMP_FILE_WARNING_THRESHOLD) {
                // 这种情况经常发生则建议横向扩容（先hardCode，可以考虑做成配置及对接业务监控等）
                log.error("StorageEngine.recoverableTmpFileHashMap.size() > RECOVERABLE_TMP_FILE_WARNING_THRESHOLD! (" + this.recoverableTmpFileHashMap.size() + ")");
            }

            // 如果临时文件没有落盘则开始自旋
            try {
                Thread.sleep(i);
            } catch (InterruptedException e) {
                log.error("StorageEngine.chunkedDownload() spinning error!", e);
                Thread.currentThread().interrupt();
            }
        }

        int version = metaDataIndex.getStoreEngineVersion();
        IStorageHandler storageHandler = StorageHandlerFactory.getStorageHandler(version);
        byte[] chunkedData = storageHandler.chunkedDownload(metaDataIndex, metaData.getFileTotalSize(), offset, limit);

        return ChunkedMetaData.builder()
                .offset(offset)
                .chunkedData(chunkedData)
                .build();
    }

    /**
     * getMetaData
     *
     * @param metaDataIndex
     * @return
     * @throws IOException
     */
    public MetaData getMetaData(MetaDataIndex metaDataIndex) throws IOException {
        if (metaDataIndex == null) {
            throw new ServiceRuntimeException(META_DATA_INDEX_IS_NULL);
        }

        int version = metaDataIndex.getStoreEngineVersion();
        IStorageHandler storageHandler = StorageHandlerFactory.getStorageHandler(version);

        return storageHandler.getMetaData(metaDataIndex);
    }

    /**
     * 文件删除
     *
     * @param metaDataIndex
     * @return
     * @throws IOException
     */
    public OperationResult delete(MetaDataIndex metaDataIndex) throws IOException {
        if (metaDataIndex == null) {
            throw new ServiceRuntimeException(META_DATA_INDEX_IS_NULL);
        }

        return StorageHandlerFactory.getStorageHandler(metaDataIndex.getStoreEngineVersion()).delete(metaDataIndex);
    }

    /**
     * init
     */
    private void init() {
        try {
            FileServerConfig config = FileServerConfigManager.getFileServerConfig();
            this.storageCleaner = new StorageCleaner(config);

            // 存储空间
            this.namespaceMap = new HashMap<>(16);
            config.getNamespaces().forEach(item -> {
                String key = item.getName().toLowerCase().trim();
                if (!this.namespaceMap.containsKey(item.getName())) {
                    this.namespaceMap.put(key, item);
                }
            });

            // 目录初始化
            this.baseDir = new File(config.getStorageRootPath(), FileServerConfigManager.getCurrentClusterNodeName());
            if (!this.baseDir.exists()) {
                this.baseDir.mkdirs();
            }

            for (String spaceName : this.namespaceMap.keySet()) {
                File storageNamespaceDir = new File(this.baseDir, spaceName);
                if (!storageNamespaceDir.exists()) {
                    storageNamespaceDir.mkdirs();
                }
            }

            this.tmpDir = new File(this.baseDir, FILE_PATH_TMP);
            if (!this.tmpDir.exists()) {
                this.tmpDir.mkdirs();
            }
        } catch (Exception e) {
            log.error("StorageEngine.init() error!", e);
        }
    }

    /**
     * loadStorageIndex
     *
     * @throws IOException
     */
    private void loadStorageIndex() throws IOException {
        this.storageIndexMap = new ConcurrentHashMap<>(16);
        StorageEngineVersion[] versions = StorageEngineVersion.class.getEnumConstants();
        for (String namespace : this.namespaceMap.keySet()) {
            for (StorageEngineVersion item : versions) {
                int version = item.getValue();
                IStorageHandler handler = StorageHandlerFactory.getStorageHandler(version);
                StorageIndex storageIndex = handler.getStorageIndex(namespace);
                this.storageIndexMap.put(getStorageIndexMapKey(namespace, version), storageIndex);
                log.info(storageIndex.toString());
            }
        }
    }

    /**
     * 未落盘临时文件异常恢复
     * <p>
     * 服务非正常停止可能导致临时文件未落盘，由于写盘采用零拷贝顺序写，因此实际中几乎撞不到。
     */
    private void recoverTmpFile() {
        IStorageHandler storageHandler = StorageHandlerFactory.getStorageHandler(DEFAULT_STORAGE_ENGINE_VERSION);
        File[] tmpFiles = this.tmpDir.listFiles();
        List<RecoverableTmpFile> recoverableTmpFiles = new ArrayList<>();
        for (File tmpFile : tmpFiles) {
            try {
                String tmpFileName = tmpFile.getName();
                String tmpFileExtName = FileUtil.getFileExt(tmpFileName);

                // 删除未完成不可恢复ing文件
                if (FILE_UPLOADING_TMP_FILE_EXT.equalsIgnoreCase(tmpFileExtName)) {
                    Files.delete(tmpFile.toPath());
                    log.info("Delete ing temp file done, {}", tmpFileName);

                    continue;
                }

                RecoverableTmpFile recoverableTmpFile = storageHandler.getRecoverableTmpFile(this.tmpDir.getAbsolutePath(), tmpFileName);
                recoverableTmpFiles.add(recoverableTmpFile);
            } catch (Exception ex) {
                log.error("StorageEngine.recoverTmpFile() error!", ex);
            }
        }

        if (CollectionUtils.isEmpty(recoverableTmpFiles)) {
            log.info("StorageEngine.recoverTmpFile() done, no RecoverableTmpFile need to recover.");
            return;
        }

        // 排序后（java无法直接向后跳写文件）入队落盘
        Collections.sort(recoverableTmpFiles);
        for (RecoverableTmpFile recoverableTmpFile : recoverableTmpFiles) {
            long hash = MetaDataIndex.hash64(recoverableTmpFile.getStorageNamespace(), recoverableTmpFile.getTime(), recoverableTmpFile.getOffset());
            this.enqueue(hash, recoverableTmpFile);
        }

        log.info("StorageEngine.recoverTmpFile() done.");
    }

    /**
     * getStorageIndexMapKey
     *
     * @param namespace
     * @param storageEngineVersion
     * @return
     */
    private static String getStorageIndexMapKey(String namespace, int storageEngineVersion) {
        return namespace + "-" + storageEngineVersion;
    }

    /**
     * getStorageIndex
     *
     * @param namespace
     * @param storageEngineVersion
     * @return
     */
    private StorageIndex getStorageIndex(String namespace, int storageEngineVersion) {
        String key = getStorageIndexMapKey(namespace, storageEngineVersion);
        if (!this.storageIndexMap.containsKey(key)) {
            throw new ServiceRuntimeException("StorageIndex not existed: " + key);
        }

        return this.storageIndexMap.get(key);
    }

    /**
     * enqueue
     *
     * @param hash
     * @param recoverableTmpFile
     */
    private void enqueue(long hash, RecoverableTmpFile recoverableTmpFile) {
        this.recoverableTmpFileHashMap.put(hash, recoverableTmpFile);
        this.publishEvent(recoverableTmpFile);
    }

    /**
     * SingletonHolder
     */
    private static class SingletonHolder {
        private static final StorageEngine INSTANCE = new StorageEngine();
    }
}

3.2 BaseStorageEngine

主要实现对Disruptor队列的包装，临时文件的落盘机制为消费Disruptor队列中的任务进行零拷贝顺序落盘。

package cn.bossfriday.fileserver.engine.core;

import cn.bossfriday.common.exception.ServiceRuntimeException;
import cn.bossfriday.common.utils.ThreadFactoryBuilder;
import cn.bossfriday.fileserver.common.conf.FileServerConfigManager;
import cn.bossfriday.fileserver.engine.model.RecoverableTmpFile;
import com.lmax.disruptor.*;
import com.lmax.disruptor.dsl.Disruptor;
import com.lmax.disruptor.dsl.ProducerType;
import lombok.extern.slf4j.Slf4j;

/**
 * BaseStorageEngine
 *
 * @author chenx
 */
@Slf4j
public abstract class BaseStorageEngine {

    protected Disruptor<RecoverableTmpFileEvent> queue;
    protected RingBuffer<RecoverableTmpFileEvent> ringBuffer;

    protected BaseStorageEngine(int capacity) {
        this.queue = this.getQueue(capacity);
        this.queue.handleEventsWithWorkerPool(new RecoverableTmpFileEventHandler());
    }

    /**
     * start
     */
    public void start() {
        this.ringBuffer = this.queue.start();

        if (this.ringBuffer == null) {
            throw new ServiceRuntimeException("BaseStorageEngine.start() error!");
        }

        this.startup();
        log.info("StorageEngine startup() done - " + FileServerConfigManager.getCurrentClusterNodeName());
    }

    /**
     * stop
     */
    public void stop() {
        this.queue.shutdown();
        this.shutdown();
        log.info("StorageEngine stop() done - " + FileServerConfigManager.getCurrentClusterNodeName());
    }

    /**
     * startup
     */
    protected abstract void startup();

    /**
     * shutdown
     */
    protected abstract void shutdown();

    /**
     * onRecoverableTmpFileEvent
     *
     * @param event
     */
    protected abstract void onRecoverableTmpFileEvent(RecoverableTmpFile event);

    /**
     * publishEvent
     *
     * @param msg
     */
    protected void publishEvent(RecoverableTmpFile msg) {
        EventTranslatorOneArg<RecoverableTmpFileEvent, RecoverableTmpFile> translator = new RecoverableTmpFileEventTranslator();
        this.ringBuffer.publishEvent(translator, msg);
    }

    /**
     * RecoverableTmpFileEventHandler
     */
    public class RecoverableTmpFileEventHandler implements WorkHandler<RecoverableTmpFileEvent> {

        @Override
        public void onEvent(RecoverableTmpFileEvent event) throws Exception {
            BaseStorageEngine.this.onRecoverableTmpFileEvent(event.getMsg());
        }
    }

    public class RecoverableTmpFileEvent {

        private RecoverableTmpFile msg;

        public RecoverableTmpFile getMsg() {
            return this.msg;
        }

        public void setMsg(RecoverableTmpFile msg) {
            this.msg = msg;
        }
    }

    public class RecoverableTmpFileEventFactory implements EventFactory<RecoverableTmpFileEvent> {

        @Override
        public RecoverableTmpFileEvent newInstance() {
            return new RecoverableTmpFileEvent();
        }
    }

    public class RecoverableTmpFileEventTranslator implements EventTranslatorOneArg<RecoverableTmpFileEvent, RecoverableTmpFile> {

        @Override
        public void translateTo(RecoverableTmpFileEvent event, long l, RecoverableTmpFile msg) {
            event.setMsg(msg);
        }
    }

    /**
     * getQueue
     *
     * @param capacity
     * @return
     */
    private Disruptor<RecoverableTmpFileEvent> getQueue(int capacity) {
        return new Disruptor<>(
                new RecoverableTmpFileEventFactory(),
                getRingBufferSize(getRingBufferSize(capacity)),
                new ThreadFactoryBuilder().setNameFormat("BaseStorageEngine-Disruptor-%d").setDaemon(true).build(),
                /**
                 * ProducerType.SINGLE：单生产者
                 * ProducerType.MULTI：多生产者
                 */
                ProducerType.MULTI,
                /**
                 * BlockingWaitStrategy：最低效的策略，但对CPU的消耗最小；
                 * SleepingWaitStrategy：与BlockingWaitStrategy类似，合用于异步日志类似的场景；
                 * YieldingWaitStrategy：性能最好，要求事件处理线数小于 CPU 逻辑核心数
                 */
                new SleepingWaitStrategy()
        );
    }

    /**
     * getRingBufferSize(保障ringBufferSize一定为2的次方)
     *
     * @param num
     * @return
     */
    private static int getRingBufferSize(int num) {
        int s = 2;
        while (s < num) {
            s <<= 1;
        }

        return s;
    }
}

3.3 主要元数据定义

3.3.1 MetaData

文件存储信息（不带fileData），只描述文件相关profile信息。

package cn.bossfriday.fileserver.engine.model;

import cn.bossfriday.fileserver.engine.core.ICodec;
import lombok.*;

import java.io.*;

/**
 * MetaData
 *
 * @author chenx
 */
@ToString
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class MetaData implements ICodec<MetaData> {

    public static final int STORE_ENGINE_VERSION_LENGTH = 1;
    public static final int FILE_STATUS_LENGTH = 1;
    public static final int TIMESTAMP_LENGTH = 8;
    public static final int UTF8_FIRST_SIGNIFICANT_LENGTH = 2;
    public static final int FILE_TOTAL_SIZE_LENGTH = 8;

    /**
     * 存储引擎版本（1字节）
     */
    private int storeEngineVersion;

    /**
     * 文件状态标识（1字节）：
     * 后续如果要扩展更多标志位（1字节最多表达8个标志位），
     * 可以新引入一个storageEngineVersion去实现（即新版本下fileStatus改为2字节）
     */
    private int fileStatus;

    /**
     * 上传时间戳（8字节）
     */
    private long timestamp;

    /**
     * 文件名（utf8字符串）
     */
    private String fileName;

    /**
     * 文件大小（8字节）
     */
    private long fileTotalSize;

    @Override
    public byte[] serialize() throws IOException {
        try (ByteArrayOutputStream out = new ByteArrayOutputStream();
             DataOutputStream dos = new DataOutputStream(out)) {
            dos.writeByte((byte) this.storeEngineVersion);
            dos.writeByte((byte) this.fileStatus);
            dos.writeLong(this.timestamp);
            dos.writeUTF(this.fileName);
            dos.writeLong(this.fileTotalSize);

            return out.toByteArray();
        }
    }

    @Override
    public MetaData deserialize(byte[] bytes) throws IOException {
        try (ByteArrayInputStream in = new ByteArrayInputStream(bytes);
             DataInputStream dis = new DataInputStream(in)) {
            int version = Byte.toUnsignedInt(dis.readByte());
            int status = Byte.toUnsignedInt(dis.readByte());
            long ts = dis.readLong();
            String fName = dis.readUTF();
            long fSize = dis.readLong();

            return MetaData.builder()
                    .storeEngineVersion(version)
                    .fileStatus(status)
                    .timestamp(ts)
                    .fileName(fName)
                    .fileTotalSize(fSize)
                    .build();
        }
    }
}

3.3.2 MetaDataIndex

主要表达文件在集群那个节点的哪个位置信息；

package cn.bossfriday.fileserver.engine.model;

import cn.bossfriday.common.utils.MurmurHashUtil;
import cn.bossfriday.fileserver.engine.core.ICodec;
import lombok.*;

import java.io.*;

/**
 * MetaDataIndex
 *
 * @author chenx
 */
@ToString
@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class MetaDataIndex implements ICodec<MetaDataIndex> {

    public static final int HASH_CODE_LENGTH = 4;

    /**
     * 集群节点
     */
    private String clusterNode;

    /**
     * 存储引擎版本
     */
    private int storeEngineVersion;

    /**
     * 存储空间
     */
    private String storageNamespace;

    /**
     * 上传时间戳
     */
    private int time;

    /**
     * 落盘文件偏移量
     */
    private long offset;

    /**
     * 元数据长度（不包含FileData）
     */
    private int metaDataLength;

    /**
     * 文件扩展名
     */
    private String fileExtName;

    @Override
    public byte[] serialize() throws IOException {
        try (ByteArrayOutputStream out = new ByteArrayOutputStream();
             DataOutputStream dos = new DataOutputStream(out)) {
            int hashInt = hash(this.clusterNode, this.storageNamespace, this.time, this.offset);
            dos.writeInt(hashInt);
            dos.writeUTF(this.clusterNode);
            dos.writeByte((byte) this.storeEngineVersion);
            dos.writeUTF(this.storageNamespace);
            dos.writeInt(this.time);
            dos.writeLong(this.offset);
            dos.writeInt(this.metaDataLength);
            dos.writeUTF(this.fileExtName);

            return out.toByteArray();
        }
    }

    @Override
    public MetaDataIndex deserialize(byte[] bytes) throws IOException {
        try (ByteArrayInputStream in = new ByteArrayInputStream(bytes);
             DataInputStream dis = new DataInputStream(in)) {
            /**
             * just read an int here
             * 严格来说这里还需要做防篡改校验（下载URL主要保障防暴力穷举即可，防篡改意愿并不强烈）
             */
            dis.readInt();
            String nodeValue = dis.readUTF();
            int versionValue = Byte.toUnsignedInt(dis.readByte());
            String namespaceValue = dis.readUTF();
            int timeValue = dis.readInt();
            long offsetValue = dis.readLong();
            int metaDataLengthValue = dis.readInt();
            String fileExtNameValue = dis.readUTF();

            return MetaDataIndex.builder()
                    .clusterNode(nodeValue)
                    .storeEngineVersion(versionValue)
                    .storageNamespace(namespaceValue)
                    .time(timeValue)
                    .offset(offsetValue)
                    .metaDataLength(metaDataLengthValue)
                    .fileExtName(fileExtNameValue)
                    .build();
        }
    }

    /**
     * hash64（相比hash32哈希碰撞几率进一步降低）
     */
    public long hash64() {
        return hash64(this.storageNamespace, this.time, this.offset);
    }

    /**
     * hash64（相比hash32哈希碰撞几率进一步降低）
     *
     * @param namespace
     * @param time
     * @param offset
     * @return
     */
    public static long hash64(String namespace, int time, long offset) {
        return MurmurHashUtil.hash64(namespace + time + offset);
    }

    /**
     * hash
     * 这里仅为使下载地址的生成散列更开
     *
     * @param clusterNode
     * @param namespace
     * @param time
     * @param offset
     * @return
     */
    private static int hash(String clusterNode, String namespace, int time, long offset) {
        String key = clusterNode + namespace + time + offset;
        return MurmurHashUtil.hash32(key);
    }
}

3.3.3 RecoverableTmpFile

可恢复临时文件，恢复策略为：根据文件名反序列化出所需要的相关信息；

package cn.bossfriday.fileserver.engine.model;

import cn.bossfriday.fileserver.engine.core.ICodec;
import lombok.*;

import java.io.*;

/**
 * RecoverableTmpFile
 *
 * @author chenx
 */
@ToString
@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
public class RecoverableTmpFile implements ICodec<RecoverableTmpFile>, Comparable<RecoverableTmpFile> {

    /**
     * fileTransactionId
     */
    private String fileTransactionId;

    /**
     * storeEngineVersion
     */
    private int storeEngineVersion;

    /**
     * storageNamespace
     */
    private String storageNamespace;

    /**
     * time
     */
    private int time;

    /**
     * offset
     */
    private long offset;

    /**
     * timestamp
     */
    private long timestamp;

    /**
     * fileName
     */
    private String fileName;

    /**
     * fileTotalSize
     */
    private long fileTotalSize;

    /**
     * filePath
     */
    private String filePath;

    @Override
    public byte[] serialize() throws IOException {
        try (ByteArrayOutputStream out = new ByteArrayOutputStream();
             DataOutputStream dos = new DataOutputStream(out)) {
            dos.writeUTF(this.fileTransactionId);
            dos.writeInt(this.storeEngineVersion);
            dos.writeUTF(this.storageNamespace);
            dos.writeInt(this.time);
            dos.writeLong(this.offset);
            dos.writeLong(this.timestamp);
            dos.writeUTF(this.fileName);
            dos.writeLong(this.fileTotalSize);
            dos.writeUTF(this.filePath);

            return out.toByteArray();
        }
    }

    @Override
    public RecoverableTmpFile deserialize(byte[] bytes) throws IOException {
        try (ByteArrayInputStream in = new ByteArrayInputStream(bytes);
             DataInputStream dis = new DataInputStream(in)) {
            String fileTransactionId = dis.readUTF();
            int storeEngineVersion = dis.readInt();
            String storageNamespace = dis.readUTF();
            int time = dis.readInt();
            long offset = dis.readLong();
            long timestamp = dis.readLong();
            String fileName = dis.readUTF();
            long fileTotalSize = dis.readLong();
            String filePath = dis.readUTF();

            return RecoverableTmpFile.builder()
                    .fileTransactionId(fileTransactionId)
                    .storeEngineVersion(storeEngineVersion)
                    .storageNamespace(storageNamespace)
                    .time(time)
                    .offset(offset)
                    .timestamp(timestamp)
                    .fileName(fileName)
                    .fileTotalSize(fileTotalSize)
                    .filePath(filePath)
                    .build();
        }
    }

    @Override
    public int compareTo(RecoverableTmpFile other) {
        int timeComparison = Integer.compare(this.getTime(), other.getTime());
        if (timeComparison != 0) {
            return timeComparison;
        }

        return Long.compare(this.getOffset(), other.getOffset());
    }
}

3.4 文件服务HttpServer处理

package cn.bossfriday.fileserver.http;

import cn.bossfriday.common.exception.ServiceRuntimeException;
import cn.bossfriday.common.http.RangeParser;
import cn.bossfriday.common.http.UrlParser;
import cn.bossfriday.common.http.model.Range;
import cn.bossfriday.fileserver.actors.model.FileDeleteMsg;
import cn.bossfriday.fileserver.actors.model.FileDownloadMsg;
import cn.bossfriday.fileserver.actors.model.WriteTmpFileMsg;
import cn.bossfriday.fileserver.common.enums.FileUploadType;
import cn.bossfriday.fileserver.context.FileTransactionContextManager;
import cn.bossfriday.fileserver.engine.StorageHandlerFactory;
import cn.bossfriday.fileserver.engine.StorageTracker;
import cn.bossfriday.fileserver.engine.core.IMetaDataHandler;
import cn.bossfriday.fileserver.engine.model.MetaDataIndex;
import cn.bossfriday.fileserver.utils.FileServerUtils;
import io.netty.buffer.ByteBuf;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelInboundHandlerAdapter;
import io.netty.handler.codec.http.*;
import io.netty.handler.codec.http.multipart.*;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.codec.binary.Base64;

import java.net.URI;
import java.net.URLDecoder;
import java.nio.charset.StandardCharsets;
import java.util.Map;

import static cn.bossfriday.fileserver.actors.model.FileDownloadMsg.FIRST_CHUNK_INDEX;
import static cn.bossfriday.fileserver.common.FileServerConst.*;

/**
 * HttpFileServerHandler
 * <p>
 * 备注：
 * 为了对服务端内存占用更加友好，不使用Http聚合（HttpObjectAggregator），
 * 如果使用HttpObjectAggregator则只需对一个FullHttpRequest进行读取即可，处理上会简单很多。
 * 不使用Http聚合一个完整的Http请求会进行1+N次读取：
 * 1、一次HttpRequest读取；
 * 2、N次HttpContent读取：后续处理中通过保障线程一致性去实现文件写入的零拷贝+顺序写
 *
 * @author chenx
 */
@Slf4j
public class HttpFileServerHandler extends ChannelInboundHandlerAdapter {

    private HttpRequest request;
    private HttpMethod httpMethod;
    private Map<String, String> pathArgsMap;
    private Map<String, String> queryArgsMap;
    private HttpPostRequestDecoder decoder;
    private String fileTransactionId;
    private String storageNamespace;
    private FileUploadType fileUploadType;
    private byte[] base64AggregatedData;
    private String metaDataIndexString;
    private Range range;

    private StringBuilder errorMsg = new StringBuilder();
    private int version = DEFAULT_STORAGE_ENGINE_VERSION;
    private long tempFilePartialDataOffset = 0;
    private long fileTotalSize = 0;
    private int base64AggregateIndex = 0;
    private boolean isKeepAlive = false;

    private static final HttpDataFactory HTTP_DATA_FACTORY = new DefaultHttpDataFactory(false);
    private static final UrlParser UPLOAD_URL_PARSER = new UrlParser("/{" + URI_ARGS_NAME_UPLOAD_TYPE + "}/{" + URI_ARGS_NAME_ENGINE_VERSION + "}/{" + URI_ARGS_NAME_STORAGE_NAMESPACE + "}");
    private static final UrlParser DOWNLOAD_URL_PARSER = new UrlParser("/" + URL_RESOURCE + "/{" + URI_ARGS_NAME_ENGINE_VERSION + "}/{" + URI_ARGS_NAME_META_DATA_INDEX_STRING + "}");

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) {
        try {
            if (msg instanceof HttpRequest) {
                this.httpRequestRead(ctx, (HttpRequest) msg);
            }

            if (msg instanceof HttpContent) {
                this.httpContentRead((HttpContent) msg);
            }
        } catch (Exception ex) {
            log.error("channelRead error: " + this.fileTransactionId, ex);
            this.errorMsg.append(ex.getMessage());
        } finally {
            if (msg instanceof LastHttpContent) {
                this.lastHttpContentChannelRead();
            }
        }
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
        log.error("exceptionCaught: " + this.fileTransactionId, cause);
        if (ctx.channel().isActive()) {
            ctx.channel().close();
        }

        // 异常情况临时文件删除
        FileServerUtils.abnormallyDeleteTmpFile(this.fileTransactionId, this.version);
    }

    /**
     * httpRequestRead
     *
     * @param ctx
     * @param httpRequest
     */
    private void httpRequestRead(ChannelHandlerContext ctx, HttpRequest httpRequest) {
        try {
            this.request = httpRequest;
            this.isKeepAlive = HttpUtil.isKeepAlive(httpRequest);
            this.fileTransactionId = FileServerUtils.getFileTransactionId(httpRequest);
            FileTransactionContextManager.getInstance().registerContext(this.fileTransactionId, ctx, this.isKeepAlive, this.request.headers().get("USER-AGENT"));

            if (HttpMethod.GET.equals(httpRequest.method())) {
                this.parseUrl(DOWNLOAD_URL_PARSER);
                this.httpMethod = HttpMethod.GET;
                this.metaDataIndexString = this.getUrlArgValue(this.pathArgsMap, URI_ARGS_NAME_META_DATA_INDEX_STRING);

                return;
            }

            if (HttpMethod.POST.equals(httpRequest.method())) {
                this.parseUrl(UPLOAD_URL_PARSER);
                this.httpMethod = HttpMethod.POST;
                this.storageNamespace = this.getUrlArgValue(this.pathArgsMap, URI_ARGS_NAME_STORAGE_NAMESPACE);
                this.fileUploadType = FileUploadType.getByName(this.getUrlArgValue(this.pathArgsMap, URI_ARGS_NAME_UPLOAD_TYPE));

                if (this.fileUploadType == FileUploadType.FULL_UPLOAD) {
                    // 全量上传
                    this.fileTotalSize = Long.parseLong(FileServerUtils.getHeaderValue(this.request, HEADER_FILE_TOTAL_SIZE));
                } else if (this.fileUploadType == FileUploadType.BASE_64_UPLOAD) {
                    // Base64上传
                    int contentLength = Integer.parseInt(FileServerUtils.getHeaderValue(this.request, String.valueOf(HttpHeaderNames.CONTENT_LENGTH)));
                    this.base64AggregatedData = new byte[contentLength];
                } else if (this.fileUploadType == FileUploadType.RANGE_UPLOAD) {
                    // 断点上传
                    this.fileTotalSize = Long.parseLong(FileServerUtils.getHeaderValue(this.request, HEADER_FILE_TOTAL_SIZE));
                    this.range = RangeParser.parseAndGetFirstRange(FileServerUtils.getHeaderValue(this.request, HttpHeaderNames.RANGE.toString()));
                }

                return;
            }

            if (HttpMethod.DELETE.equals(httpRequest.method())) {
                this.parseUrl(DOWNLOAD_URL_PARSER);
                this.httpMethod = HttpMethod.DELETE;
                this.metaDataIndexString = this.getUrlArgValue(this.pathArgsMap, URI_ARGS_NAME_META_DATA_INDEX_STRING);

                return;
            }

            throw new ServiceRuntimeException("unsupported HttpMethod!");
        } catch (Exception ex) {
            log.error("HttpRequest process error!", ex);
            this.errorMsg.append(ex.getMessage());
        } finally {
            if (this.httpMethod.equals(HttpMethod.POST)) {
                try {
                    this.decoder = new HttpPostRequestDecoder(HTTP_DATA_FACTORY, this.request);
                } catch (HttpPostRequestDecoder.ErrorDataDecoderException e1) {
                    log.warn("getHttpDecoder Error:" + e1.getMessage());
                }
            }
        }
    }

    /**
     * httpContentRead
     * Netty ByteBuf直接内存溢出问题需要重点关注，
     * 调试时可以通过增加：-Dio.netty.leakDetectionLevel=PARANOID来保障对每次请求都做内存溢出检测
     *
     * @param httpContent
     */
    private void httpContentRead(HttpContent httpContent) {
        try {
            if (this.httpMethod.equals(HttpMethod.POST)) {
                if (this.fileUploadType == FileUploadType.BASE_64_UPLOAD) {
                    this.base64Upload(httpContent);
                } else {
                    this.fileUpload(httpContent);
                }
            } else if (this.httpMethod.equals(HttpMethod.GET)) {
                this.fileDownload(httpContent);
            } else if (this.httpMethod.equals(HttpMethod.DELETE)) {
                this.deleteFile(httpContent);
            } else {
                if (httpContent instanceof LastHttpContent) {
                    this.errorMsg.append("unsupported http method");
                }
            }
        } finally {
            if (httpContent.refCnt() > 0) {
                httpContent.release();
            }
        }
    }

    /**
     * lastHttpContentChannelRead
     */
    private void lastHttpContentChannelRead() {
        this.resetHttpRequest();

        if (this.base64AggregatedData != null) {
            this.base64AggregatedData = null;
        }

        if (this.hasError()) {
            FileServerUtils.abnormallyDeleteTmpFile(this.fileTransactionId, this.version);
            FileServerUtils.sendResponse(this.fileTransactionId, HttpResponseStatus.INTERNAL_SERVER_ERROR, this.errorMsg.toString());
        }
    }

    /**
     * fileUpload
     */
    private void fileUpload(HttpContent httpContent) {
        if (this.decoder == null) {
            return;
        }

        try {
            /**
             * Initialized the internals from a new chunk
             * content – the new received chunk
             */
            this.decoder.offer(httpContent);
            if (!this.hasError()) {
                this.chunkedFileUpload();
            }
        } catch (Exception ex) {
            log.error("HttpFileServerHandler.fileUpload() error!", ex);
            this.errorMsg.append(ex.getMessage());
        }
    }

    /**
     * chunkedFileUpload（文件分片上传）
     */
    private void chunkedFileUpload() {
        try {
            while (this.decoder.hasNext()) {
                /**
                 * Returns the next available InterfaceHttpData or null if, at the time it is called,
                 * there is no more available InterfaceHttpData. A subsequent call to offer(httpChunk) could enable more data.
                 * Be sure to call ReferenceCounted.release() after you are done with processing to make sure to not leak any resources
                 */
                InterfaceHttpData data = this.decoder.next();
                if (data instanceof FileUpload) {
                    this.currentPartialHttpDataProcess((FileUpload) data);
                }
            }

            /**
             * Returns the current InterfaceHttpData if currently in decoding status,
             * meaning all data are not yet within, or null if there is no InterfaceHttpData currently in decoding status
             * (either because none yet decoded or none currently partially decoded).
             * Full decoded ones are accessible through hasNext() and next() methods.
             */
            HttpData data = (HttpData) this.decoder.currentPartialHttpData();
            if (data instanceof FileUpload) {
                this.currentPartialHttpDataProcess((FileUpload) data);
            }
        } catch (HttpPostRequestDecoder.EndOfDataDecoderException endOfDataDecoderException) {
            log.error("HttpFileServerHandler.chunkedFileUpload() EndOfDataDecoderException!");
        } catch (Exception ex) {
            log.error("HttpFileServerHandler.chunkedFileUpload() error!", ex);
            this.errorMsg.append(ex.getMessage());
        }
    }

    /**
     * currentPartialHttpDataProcess
     *
     * @param currentPartialData
     */
    private void currentPartialHttpDataProcess(FileUpload currentPartialData) {
        byte[] partialData = null;
        try {
            ByteBuf byteBuf = currentPartialData.getByteBuf();
            int readBytesCount = byteBuf.readableBytes();
            partialData = new byte[readBytesCount];
            byteBuf.readBytes(partialData);

            WriteTmpFileMsg msg = new WriteTmpFileMsg();
            msg.setStorageEngineVersion(this.version);
            msg.setFileTransactionId(this.fileTransactionId);
            msg.setStorageNamespace(this.storageNamespace);
            msg.setKeepAlive(this.isKeepAlive);
            msg.setFileName(URLDecoder.decode(currentPartialData.getFilename(), StandardCharsets.UTF_8.name()));
            msg.setRange(this.range);
            msg.setFileTotalSize(this.fileTotalSize);
            msg.setOffset(this.tempFilePartialDataOffset);
            msg.setData(partialData);
            StorageTracker.getInstance().onPartialUploadDataReceived(msg);
        } catch (Exception ex) {
            log.error("HttpFileServerHandler.chunkedProcessFileUpload() error!", ex);
            this.errorMsg.append(ex.getMessage());
        } finally {
            if (partialData != null) {
                this.tempFilePartialDataOffset += partialData.length;
            }

            if (currentPartialData.refCnt() > 0) {
                currentPartialData.release();
            }

            // just help GC
            partialData = null;
        }
    }

    /**
     * base64Upload
     * 由于对不完整Base64信息进行解码可能失败，因此Base64上传处理方式为聚合完成后进行Base64解码然后再进行全量上传
     * 这是base64Upload只能用于例如截屏等小文件的上传场景的原因。
     *
     * @param httpContent
     */
    private void base64Upload(HttpContent httpContent) {
        ByteBuf byteBuf = null;
        byte[] currentPartialData = null;
        byte[] decodedFullData = null;
        try {
            // 数据聚合
            byteBuf = httpContent.content();
            int currentPartialDataLength = byteBuf.readableBytes();
            currentPartialData = new byte[currentPartialDataLength];
            byteBuf.readBytes(currentPartialData);
            System.arraycopy(currentPartialData, 0, this.base64AggregatedData, this.base64AggregateIndex, currentPartialDataLength);
            this.base64AggregateIndex += currentPartialDataLength;

            // 聚合完成
            if (httpContent instanceof LastHttpContent) {
                decodedFullData = Base64.decodeBase64(this.base64AggregatedData);

                WriteTmpFileMsg msg = new WriteTmpFileMsg();
                msg.setStorageEngineVersion(this.version);
                msg.setFileTransactionId(this.fileTransactionId);
                msg.setStorageNamespace(this.storageNamespace);
                msg.setKeepAlive(this.isKeepAlive);
                msg.setFileName(this.fileTransactionId + "." + this.getUrlArgValue(this.queryArgsMap, URI_ARGS_NAME_EXT));
                msg.setRange(this.range);
                msg.setFileTotalSize(decodedFullData.length);
                msg.setOffset(this.tempFilePartialDataOffset);
                msg.setData(decodedFullData);
                StorageTracker.getInstance().onPartialUploadDataReceived(msg);
            }
        } finally {
            if (byteBuf != null) {
                byteBuf.release();
            }

            // just help GC
            currentPartialData = null;
            decodedFullData = null;
        }
    }

    /**
     * fileDownload
     *
     * @param httpContent
     */
    private void fileDownload(HttpContent httpContent) {
        if (httpContent instanceof LastHttpContent && !this.hasError()) {
            try {
                IMetaDataHandler metaDataHandler = StorageHandlerFactory.getMetaDataHandler(this.version);
                MetaDataIndex metaDataIndex = metaDataHandler.downloadUrlDecode(this.metaDataIndexString);
                FileDownloadMsg fileDownloadMsg = FileDownloadMsg.builder()
                        .fileTransactionId(this.fileTransactionId)
                        .metaDataIndex(metaDataIndex)
                        .chunkIndex(FIRST_CHUNK_INDEX)
                        .build();
                StorageTracker.getInstance().onDownloadRequestReceived(fileDownloadMsg);
            } catch (Exception ex) {
                log.error("HttpFileServerHandler.fileDownload() error!", ex);
                throw new ServiceRuntimeException("File download error!");
            }
        }
    }

    /**
     * deleteFile
     *
     * @param httpContent
     */
    private void deleteFile(HttpContent httpContent) {
        if (httpContent instanceof LastHttpContent && !this.hasError()) {
            try {
                IMetaDataHandler metaDataHandler = StorageHandlerFactory.getMetaDataHandler(this.version);
                MetaDataIndex metaDataIndex = metaDataHandler.downloadUrlDecode(this.metaDataIndexString);
                FileDeleteMsg msg = FileDeleteMsg.builder()
                        .fileTransactionId(this.fileTransactionId)
                        .metaDataIndex(metaDataIndex)
                        .build();
                StorageTracker.getInstance().onFileDeleteMsg(msg);
            } catch (Exception ex) {
                log.error("HttpFileServerHandler.deleteFile() error!", ex);
                throw new ServiceRuntimeException("delete file error!");
            }
        }
    }

    /**
     * resetHttpRequest
     */
    private void resetHttpRequest() {
        try {
            this.request = null;
            if (this.decoder != null) {
                this.decoder.destroy();
                this.decoder = null;
            }

            log.info("HttpFileServerHandler.resetHttpRequest() done: " + this.fileTransactionId);
        } catch (Exception e) {
            log.error("HttpFileServerHandler.resetHttpRequest() error!", e);
        }
    }

    /**
     * parseUrl
     *
     * @param urlParser
     */
    private void parseUrl(UrlParser urlParser) {
        try {
            URI uri = new URI(this.request.uri());
            this.pathArgsMap = urlParser.parsePath(uri);
            this.queryArgsMap = urlParser.parseQuery(uri);
            this.version = FileServerUtils.parserEngineVersionString(UrlParser.getArgsValue(this.pathArgsMap, URI_ARGS_NAME_ENGINE_VERSION));
        } catch (Exception ex) {
            log.error("HttpFileServerHandler.parseUrl() error!", ex);
            this.errorMsg.append(ex.getMessage());
        }
    }

    /**
     * getUrlArgValue
     *
     * @param argMap
     * @param key
     * @return
     */
    private String getUrlArgValue(Map<String, String> argMap, String key) {
        try {
            return UrlParser.getArgsValue(argMap, key);
        } catch (Exception ex) {
            log.error("HttpFileServerHandler.getUrlArgValue() error!", ex);
            this.errorMsg.append(ex.getMessage());
        }

        return null;
    }

    /**
     * hasError
     */
    private boolean hasError() {
        return this.errorMsg.length() > 0;
    }
}

4 服务启动及测试

4.1 服务运行

服务启动
1、service-config配置（resources\service-config.xml）说明
文件服务使用ZK做为集群状态管理，因此需要修改实际可达的ZK地址。

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <!--集群名称（取一个合适的名称即可，ZK根节点以此命名）-->
    <systemName>nubybear</systemName>

    <!--ZK地址（多个地址逗号分割）-->
    <zkAddress>localhost:2181</zkAddress>

	<!--当前节点-->
    <clusterNode>
        <!--节点名称-->
        <name>fs-node1</name>

        <!--节点IP-->
        <host>127.0.0.1</host>

        <!--节点RPC端口-->
        <port>18080</port>

        <!--节点虚拟节点数（即一致性哈希中的虚拟节点）-->
        <virtualNodesNum>100</virtualNodesNum>
    </clusterNode>
</config>

配置项	说明
systemName	集群名称（取一个合适的名称即可）
zkAddress	ZK地址
clusterNode.name	文件服务当前节点名称
clusterNode.host	文件服务当前节点IP
clusterNode.port	文件服务当前节点RPC端口
clusterNode.virtualNodesNum	文件服务当前节点虚拟节点数，即一致性哈希中的虚拟节点，如果每个节点配置的值一样则可以认为每个节点荷载平均；文件上传时以此为权重，下载优先走当时上传的节点；在做某些扩容时，可以将此配置为0，做到只读不写

2、file-config配置说明（resources\file-config.xml）
测试启动时一般无需修改。相关说明详见注释即可。

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <!--文件服务http端口-->
    <httpPort>18086</httpPort>

    <!--存储文件跟文件夹路径-->
    <storageRootPath>fileData</storageRootPath>

    <!--过期文件清理扫描时间间隔-->
    <cleanerScanInterval>36000</cleanerScanInterval>

    <!--存储空间：name：存储空间名称（存储空间根目录以此命名）；expireDay：存储空间文件过期时间，<=0表示永不过期-->
    <namespaces>
        <!--普通空间-->
        <namespace name="normal" expireDay="180"/>

        <!--永久存储空间-->
        <namespace name="forever" expireDay="-1"/>
    </namespaces>
</config>

3、运行n.bossfriday.fileserver.FileServerBootstrap中的main函数
之前做了一个优化：服务启动自动创建ZK根节点，因此当前直接启动即可；
在这里插入图片描述

4.2 上传及下载测试方式

1、上传
源码中包含了一个普通上传的http请求，服务启动成功后直接请求即可（不过由于idea中http请求不直接支持相对路径，因此文件的绝对路径需要各自根据实际情况调整）；如果要大家要用这个http文件测试上传其他文件，需要按照实际情况修改：X-File-Total-Size（文件大小/单位：字节）、filename（原始文件名，文件名如果带中文需要url编码）、文件绝对路径。
上传成功应答中：rc_url.path为文件下载地址的相对路径。
在这里插入图片描述
其他上类型上传测试可以使用：
cn.bossfriday.fileserver.test.FileUploadTest中的单元测试方法：

2、下载
直接浏览器输入：http://{ip}:{port}/{path}，例如：http://localhost:18086:/resource/v1/4WxoCNQMnYjoJXbpyiy2XVzNbACTXg2AEwNK85KcMZ4mqPocUEBeYjWt55S9

5. 性能

5.1 环境描述

服务器：8核16G 云服务器：1台；
文件大小：100K
压测工具：JMeter
网络环境：内网（使用1台与文件服务内网互通的打压机压测）

5.2 文件上传

秒吞吐量：2455.19
平均延时：27.53 毫秒

主逻辑如下，详见：cn.bossfriday.jmeter.sampler.FileUploadSampler，另外有兴趣的同学，可以下载代码了解下我这里如何基于JMeter去做的插件化的自定义采样器。

    @Override
    public SampleResult sample() {
        SampleResult result = new SampleResult();
        result.setSampleLabel(this.sampleLabel);
        if (isTestStartedError)
            return sampleFailedByTestStartedError();

        CloseableHttpClient httpClient = null;
        HttpPost httpPost = null;
        CloseableHttpResponse httpResponse = null;
        try {
            long currentSampleIndex = sampleIndex.incrementAndGet();
            httpClient = HttpApiHelper.getHttpClient(false);
            httpPost = new HttpPost(fileApiUri);
            httpPost.setConfig(httpRequestConfig);

            httpPost.addHeader("X-File-Total-Size", String.valueOf(localFile.length()));
            httpPost.addHeader("Connection", "Keep-Alive");
            MultipartEntityBuilder builder = MultipartEntityBuilder.create();
            builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
            builder.addBinaryBody("upfile", localFile, ContentType.create("application/x-zip-compressed"), URLEncoder.encode(localFile.getName(), "UTF-8"));
            HttpEntity entity = builder.build();
            httpPost.setEntity(entity);

            result.sampleStart();
            httpResponse = httpClient.execute(httpPost);
            result.sampleEnd();

            String statusCode = String.valueOf(httpResponse.getStatusLine().getStatusCode());
            if (!statusCode.equals("200")) {
                result.setSuccessful(false);
                result.setResponseCode(statusCode);
                result.setResponseMessage(statusCode);
                log.error("upload failed!(statusCode=" + statusCode + ")");

                return result;
            }

            result.setSuccessful(true);
            result.setResponseCode("200");
            result.setResponseMessage("OK");

            String downloadUrl = "";
            HttpEntity respEntity = httpResponse.getEntity();
            if (respEntity != null) {
                String responseBody = EntityUtils.toString(httpResponse.getEntity());
                if (!StringUtils.isEmpty(responseBody)) {
                    downloadUrl = getDownloadUrl(responseBody);
                }
            }

            String line = config.getLoalFileName() + "," + statusCode + "," + String.valueOf(result.getTime()) + "," + downloadUrl + "," + result.getStartTime();
            writeOutFile(bw, line);
            log.info(currentSampleIndex + "," + line);
        } catch (Exception ex) {
            BaseSampler.setSampleResult("500", "Exception:" + ex.getMessage(), false, result);
            log.error("FileUploadSampler.sample() error!", ex);
        } finally {
            if (httpPost != null)
                httpPost.releaseConnection();

            if (httpResponse != null)
                try {
                    httpResponse.close();
                } catch (Exception e) {
                    log.error("httpResponse close error!", e);
                }

            if (httpClient != null)
                try {
                    httpClient.close();
                } catch (Exception e) {
                    log.error("httpClient close error!", e);
                }
        }

        return result;
    }

5.3 文件下载

秒吞吐量：3513.70
响应平均延时：2.64 毫秒

主逻辑如下，详见：cn.bossfriday.jmeter.sampler.FileDownloadSampler

@Override
    public SampleResult sample() {
        SampleResult result = new SampleResult();
        result.setSampleLabel(this.sampleLabel);
        if (isTestStartedError)
            return sampleFailedByTestStartedError();

        CloseableHttpClient httpClient = null;
        HttpGet httpGet = null;
        CloseableHttpResponse httpResponse = null;
        InputStream in = null;
        try {
            String downUrl = getDownloadUrl();
            httpClient = HttpApiHelper.getHttpClient(false);
            httpGet = new HttpGet(fileApiUri + downUrl);
            httpGet.setConfig(httpRequestConfig);
            httpGet.addHeader("Connection", "Keep-Alive");

            result.sampleStart();
            httpResponse = httpClient.execute(httpGet);
            result.sampleEnd();

            int statusCode = httpResponse.getStatusLine().getStatusCode();
            if (statusCode != 200) {
                result.setSuccessful(false);
                result.setResponseCode(String.valueOf(statusCode));
                result.setResponseMessage("Failed");
                log.error("download failed!(statusCode=" + statusCode + ")");

                return result;
            }

            result.setSuccessful(true);
            result.setResponseCode("200");
            result.setResponseMessage("OK");

            HttpEntity entity = httpResponse.getEntity();
            if (entity != null) {
                in = entity.getContent();
                while (in.read() > 0) {
                    // 空跑（打压端不存储文件）
                }
            }

            log.info("down(" + sampleIndex.get() + ")  elapse " + result.getTime() + ", " + downUrl);
        } catch (Exception ex) {
            BaseSampler.setSampleResult("500", "Exception:" + ex.getMessage(), false, result);
            log.error("FileDownloadSampler.sample() error!", ex);
        } finally {
            try {
                if (in != null)
                    in.close();

            } catch (Exception ex) {
                log.warn("close stream error!(" + ex.getMessage() + ")");
            }

            if (httpGet != null)
                httpGet.releaseConnection();

            if (httpResponse != null)
                try {
                    httpResponse.close();
                } catch (Exception e) {
                    log.error("httpResponse close error!", e);
                }

            if (httpClient != null)
                try {
                    httpClient.close();
                } catch (Exception e) {
                    log.error("httpClient close error!", e);
                }
        }

        return result;
    }

5.4 压测结果备注

文件大小的梯度不够，没有做1MB、10MB……梯度（大家有兴趣可以自己做下）。从目前100K文件2455.19秒吞吐量来看，大文件下折算的每秒写入速率应该能更高，毕竟做法是：顺序写盘+零拷贝（FileChannel.transferFrom）。目前使用的云服务器，应该是台SSD磁盘的机器（不过顺序写，机械磁盘和SSD磁盘差异并不是那么的大）。
从上面贴的压测代码可以看出每次采样都是新构建一个HttpClient，结束时Close。如果复用HttpClient结果是否提升呢？
目前通过HttpPostRequestDecoder.currentPartialHttpData() 每次获得的ChunkedData只有8K还是16K（记不清楚了，大家可以加个日志去看下），因为不是关键问题，之前只是简单查了下：这个和Netty的内存分配机制有关，当前走的只是默认值。不过并没有查到如何去设置和更改，不知道哪位同学知道。回头可以试下通过调整该值后去减少每次上传的分片次数，看看这对性能的影响会有多少。

原创不易，请给作者打赏或点赞，您的支持是我坚持原创和分享的最大动力！
在这里插入图片描述

BossFriday

关注

3
点赞
踩
7

收藏

觉得还不错? 一键收藏
打赏
8
评论
Java分布式高性能文件服务

一个用Java开发的分布式高性能文件服务，主要功能为：上传（普通上传/断点续传/Base64上传）、下载、文件删除、文件定时清理（TODO）。主要技术手段：写盘机制：零拷贝+顺序写盘；RPC机制：ActorRpc；HttpServer机制：基于Netty的不FullHttpRequest聚合；
复制链接

扫一扫