新版JAVA对象存储（Minio）

小鱼干__

已于 2022-06-25 15:28:09 修改

阅读量3.2k

点赞数 2

分类专栏： java 文章标签： java linux spring boot

于 2022-06-25 15:25:07 首次发布

本文链接：https://blog.csdn.net/weixin_43905025/article/details/125459762

版权

java 专栏收录该内容

10 篇文章 0 订阅

订阅专栏

最近突然想搞一下对象存储，搜了搜发现博客上的都是旧写法，踩了好多坑，所以写了这篇博客。本人是菜鸟一个如果有错望指正。

文章目录

1.分布式文件系统应用场景
2. 环境搭建（单机部署）
3. java aip
- 版本查看
4. springBoot 整合

1.分布式文件系统应用场景

互联网海量非结构化数据的存储需求

电商网站：海量商品图片
视频网站：海量视频文件
网盘：海量文件
社交网站：海量图片

1.1 Minio介绍（http://www.minio.org.cn/）

MinIO是一个基于Apache License v2.O开源协议的对象存储服务。它兼容亚马逊s3云存储服务接口，非常适合于存储大容量非结构化的
数据，例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等，而一个对象文件可以是任意大小，从几kb到最大5T不等。

MinIO是一个非常轻量的服务，可以很简单的和其他应用的结合，类似NodeJS,Redis或者MySQL。

优点：高性能，可扩展性，开放全部源代码 + 企业级支持，云的原生支持，与Amazon S3 兼容，简单

1.2 基础概念

Object:存储到Minio的基本对象，如文件、字节流，Anything
Bucket:用来存储Object的逻辑空间。每个Bucket之间的数据是相互隔离的。对于客户端而言，就相当于一个存放文件的顶层文
件夹。
I
Drive:即存储数据的磁盘，在MinlO启动时，以参数的方式传入。Minio中所有的对象数据都会存储在Drive里。
Set：即一组Drive的集合，分布式部署根据集群规模自动划分一个或多个Set,每个Set中的Drive分布在不同位置。一个对象存
储在-个Set上。(For example:{1.64}is divided into4 sets each of size16.)
- 一个对象存储在一个Set上
- 一个集群划分为多个Set
- 一个Set包含的Drive数量是固定的，默认由系统根据集群规模自动计算得出
- 一个SET中的(Drive)尽可能分布在不同的节点上

1.3 纠删码EC(Erasure Code)

MinIO使用纠删码机制来保证高可靠性，使用highwayhash来处理数据损坏(Bit Rot Protection)。关于纠删码，简单来说就是可以
通过数学计算，把丢失的数据进行还原，它可以将n份原始数据，增加m份数据，并能通过n+m份中的任意n份数据，还原为原始数据。
即如果有任意小于等于份的数据失效，仍然能通过剩下的数据还原出来。

1.4 存储形式

文件对象上传到MilO,会在对应的数据存储磁盘中，以Bucket名称为目录，文件名称为下一级目录。

1.5 存储方案

2. 环境搭建（单机部署）

学习：win https://dl.minio.io/server/minio/release/windows-amd64/minio.exe

启动 .\minio.exe server D:.\data

3. java aip

https://docs.min.io/docs/java-client-api-reference.html

依赖

        <dependency>
            <groupId>io.minio</groupId>
            <artifactId>minio</artifactId>
            <version>8.4.2</version>
        </dependency>
        <dependency>
            <groupId>me.tongfei</groupId>
            <artifactId>progressbar</artifactId>
            <version>0.9.3</version>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13.2</version>
        </dependency>

常用API

package com.wx;


import io.minio.*;
import io.minio.errors.*;
import io.minio.messages.Bucket;
import io.minio.messages.Item;
import io.minio.messages.VersioningConfiguration;
import org.junit.Before;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;

import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.List;

@SpringBootTest
class SptingbootMinioApplicationTests {
    private static MinioClient minioClient;
    private static String bucketName = "text01";
    private static String filePath = "E:\\图\\wallhaven-vmxxm8.jpg";
    private static String endPoint = "http://192.168.2.248:9000 ";
    private static String accessKey = "minioadmin";
    private static String secretKey = "minioadmin";
    //上传文件大小限制5M
    private static long limitSize = 5242880L;

    private void init() {
        minioClient = MinioClient
                .builder()
                .endpoint(endPoint)
                .credentials(accessKey, secretKey)
                .build();

    }

    /**
     * 版本控制
     *
     * @throws Exception
     */
    private void version() throws Exception {
        init();
        // Enable versioning on 'my-bucketname'.
        minioClient.setBucketVersioning(
                SetBucketVersioningArgs.builder()
                        .bucket(bucketName)
                        .config(new VersioningConfiguration(VersioningConfiguration.Status.ENABLED, null))
                        .build());
        System.out.println("Bucket versioning is enabled successfully");
//
//        // Suspend versioning on 'my-bucketname'.
//        minioClient.setBucketVersioning(
//                SetBucketVersioningArgs.builder()
//                        .bucket("my-bucketname")
//                        .config(new VersioningConfiguration(VersioningConfiguration.Status.SUSPENDED, null))
//                        .build());
//        System.out.println("Bucket versioning is suspended successfully");
    }

    /**
     * 新建桶
     *
     * @throws Exception
     */
    @Test
    public void testCreateBucket() throws Exception {
        init();
        boolean isExist = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
        if (isExist) {
            System.out.println(bucketName + "已经存在！");
        } else {
            minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
            System.out.println("创建了一个名字是" + bucketName + "的bucket");
        }
    }

    /**
     * 查询桶列表
     *
     * @throws Exception
     */
    @Test
    public void testListBuckets() throws Exception {
        init();
        List<Bucket> bucketList = minioClient.listBuckets();
        bucketList.forEach(p -> {
            System.out.println(p.name());
        });
    }

    /**
     * 上传
     *
     * @throws Exception
     */
    @Test
    public void testUploadFile() throws Exception {
        init();
        version();
        for (int i = 0; i < 10; i++) {
            minioClient.uploadObject(
                    UploadObjectArgs
                            .builder()
                            .bucket(bucketName)
                            .object("text.jpg")//上传的名
                            .filename(filePath)//本地磁盘地址
                            .build());
        }


        System.out.println("上传完毕，请刷新minio的web页面，查看上传文件");
    }

    /**
     * 下载
     *
     * @throws Exception
     */
    @Test
    public void testDownloadFile() throws Exception {
        init();
        minioClient.downloadObject(
                DownloadObjectArgs
                        .builder()
                        .bucket(bucketName)
                        .object("text.jpg")//minio上的文件名
                        .filename("text.jpg")//下载的文件名
                        .build()
        );
        System.out.println("下载完毕");
    }


    /**
     * 删除
     *
     * @throws Exception
     */
    @Test
    public void testRemoveBucket() throws Exception {
        init();
        boolean isExist = minioClient.bucketExists(
                BucketExistsArgs
                        .builder()
                        .bucket(bucketName)
                        .build()
        );
        if (isExist) {
            //桶不空，删不掉，所以清桶的objects
            Iterable<Result<Item>> iterable = minioClient.listObjects(
                    ListObjectsArgs
                            .builder()
                            .bucket(bucketName)
                            .build()
            );
            for (Result<Item> o : iterable) {
                System.out.println("当前objectname---->>>>>>>" + o.get().objectName());
                minioClient.removeObject(
                        RemoveObjectArgs.builder()
                                .bucket(bucketName)
                                .object(o.get().objectName())
                                .versionId(o.get().versionId())
                                .build());
                System.out.println("清理---->>>>>>>" + o.get().objectName());
            }
            System.out.println("清理" + bucketName + "下的object完毕");
            minioClient.removeBucket(
                    RemoveBucketArgs
                            .builder()
                            .bucket(bucketName)
                            .build()
            );
            if (!minioClient.bucketExists(
                    BucketExistsArgs.
                            builder()
                            .bucket(bucketName)
                            .build()
            )) {
                System.out.println("删除" + bucketName + "完毕，刷新minio的web页面");
            }
        } else {
            System.out.println("没有这个bucket，无需操作");
        }
    }

}

版本查看

4. springBoot 整合

依赖

配置文件

server:
  port: 8082
spring:
  application:
    name: minio
  thymeleaf:
    cache: false #关闭缓存
  servlet:
    multipart:
      max-file-size: 10MB
      max-request-size: 100 MB

minio:
  endpoint: http://192.168.2.248:9000
  accesskey: minioadmin
  secretKey: minioadmin

连接 MinIO 配置(实体类) 与公共返回类

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Data
@ConfigurationProperties(prefix = "minio")
@Component
public class MinioProp {
    private String endpoint;
    private String accesskey;
    private String secretKey;
}

import lombok.AllArgsConstructor;
import lombok.NoArgsConstructor;

import java.io.Serializable;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Res implements Serializable {
    private static final long serialVersionUID = 1L;
    private Integer code;
    private Object data = "";
    private String message = "";
}

创建 MinioClient

@Configuration
public class MinioConfiguration {
    @Autowired
    private MinioProp minioProp;

    @Bean
    public MinioClient minioClient() throws Exception {
        MinioClient minioClient = MinioClient
                .builder()
                .endpoint(minioProp.getEndpoint())
                .credentials(minioProp.getAccesskey(), minioProp.getSecretKey())
                .build();
        return minioClient;
    }
}

MinIO 查看桶列表，存入，删除操作 MinioController

package com.wx.controller;

import com.alibaba.fastjson.JSON;
import com.wx.pojo.Res;
import io.minio.*;
import io.minio.messages.Item;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.IOUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.ui.ModelMap;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;

import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.InputStream;
import java.net.URLEncoder;
import java.text.DecimalFormat;
import java.util.*;

@Slf4j
@RestController
public class MinioController {
    @Autowired
    private MinioClient minioClient;

    private static final String MINIO_BUCKET = "text01";

    @GetMapping("/list")
    public List<Object> list(ModelMap map) throws Exception {
        Iterable<Result<Item>> myObjects = minioClient.listObjects(
                ListObjectsArgs
                        .builder()
                        .bucket(MINIO_BUCKET)
                        .build()
        );
        Iterator<Result<Item>> iterator = myObjects.iterator();
        List<Object> items = new ArrayList<>();
        String format = "{'fileName':'%s','fileSize':'%s'}";
        while (iterator.hasNext()) {
            Item item = iterator.next().get();
            items.add(JSON.parse(String.format(format, item.objectName(), formatFileSize(item.size()))));
        }
        return items;
    }

    @PostMapping("/upload")
    public Res upload(@RequestParam(name = "file", required = false) MultipartFile[] file) {
        Res res = new Res();
        res.setCode(500);

        if (file == null || file.length == 0) {
            res.setMessage("上传文件不能为空");
            return res;
        }

        List<String> orgfileNameList = new ArrayList<>(file.length);

        for (MultipartFile multipartFile : file) {
            String orgfileName = multipartFile.getOriginalFilename();
            orgfileNameList.add(orgfileName);

            try {
                InputStream in = multipartFile.getInputStream();
                minioClient.putObject(
                        PutObjectArgs
                                .builder()
                                .bucket(MINIO_BUCKET)
                                .object(orgfileName)
                                .stream(in, in.available(), -1)
                                .build()
                );
                in.close();
            } catch (Exception e) {
                log.error(e.getMessage());
                res.setMessage("上传失败");
                return res;
            }
        }

        Map<String, Object> data = new HashMap<String, Object>();
        data.put("bucketName", MINIO_BUCKET);
        data.put("fileName", orgfileNameList);
        res.setCode(200);
        res.setMessage("上传成功");
        res.setData(data);
        return res;
    }

    @RequestMapping("/download/{fileName}")
    public void download(HttpServletResponse response, @PathVariable("fileName") String fileName) {
        InputStream in = null;
        try {
            //获取元数据
            StatObjectResponse stat = minioClient.statObject(
                    StatObjectArgs
                            .builder()
                            .bucket(MINIO_BUCKET)
                            .object(fileName)
                            .build());

            response.setContentType(stat.contentType());
            response.setHeader("Content-Disposition", "attachment;filename=" + URLEncoder.encode(fileName, "UTF-8"));

            GetObjectResponse object = minioClient.getObject(
                    GetObjectArgs
                            .builder()
                            .bucket(MINIO_BUCKET)
                            .object(fileName)
                            .build()
            );
            IOUtils.copy(in, response.getOutputStream());
        } catch (Exception e) {
            log.error(e.getMessage());
        } finally {
            if (in != null) {
                try {
                    in.close();
                } catch (IOException e) {
                    log.error(e.getMessage());
                }
            }
        }
    }

    @DeleteMapping("/delete/{fileName}")
    public Res delete(@PathVariable("fileName") String fileName) {
        Res res = new Res();
        res.setCode(200);
        try {
            minioClient.removeObject(
                    RemoveObjectArgs
                            .builder()
                            .bucket("text01")
                            .object(fileName)
                            .build());
        } catch (Exception e) {
            res.setCode(500);
            log.error(e.getMessage());
        }
        return res;
    }

    private static String formatFileSize(long fileS) {
        DecimalFormat df = new DecimalFormat("#.00");
        String fileSizeString = "";
        String wrongSize = "0B";
        if (fileS == 0) {
            return wrongSize;
        }
        if (fileS < 1024) {
            fileSizeString = df.format((double) fileS) + " B";
        } else if (fileS < 1048576) {
            fileSizeString = df.format((double) fileS / 1024) + " KB";
        } else if (fileS < 1073741824) {
            fileSizeString = df.format((double) fileS / 1048576) + " MB";
        } else {
            fileSizeString = df.format((double) fileS / 1073741824) + " GB";
        }
        return fileSizeString;
    }
}

路由文件 RouterController

@Controller
public class RouterController {
    @GetMapping({"/", "/index.html"})
    public String index() {
        return "index";
    }

    @GetMapping({"/upload.html"})
    public String upload() {
        return "upload";
    }
}

前端列表页面 src\main\resources\templates\index.html

<!DOCTYPE html>
<html lang="zh-cn">
<head>
    <meta charset="utf-8"/>
    <title>图片列表</title>
    <link rel="stylesheet" href="http://cdn.staticfile.org/element-ui/2.13.1/theme-chalk/index.css">
</head>
<body>

<div id="app">

    <el-link icon="el-icon-upload" href="/upload.html">上传图片</el-link>
    <br/>

    <el-table :data="results" stripe style="width: 60%" @row-click="preview">
        <el-table-column type="index" width="50"></el-table-column>
        <el-table-column prop="fileName" label="文件名" width="180"></el-table-column>
        <el-table-column prop="fileSize" label="文件大小"></el-table-column>
        <el-table-column label="操作">
            <template slot-scope="scope">
                <a :href="'/download/' + scope.row.fileName + ''" class="el-icon-download">下载</a>
                <a :href="'/delete/' + scope.row.fileName + ''" @click.prevent="deleteFile($event,scope.$index,results)"
                   class="el-icon-delete">删除</a>
            </template>
        </el-table-column>
    </el-table>

    <br/>
    <el-link icon="el-icon-picture">预览图片</el-link>
    <br/>
    <div class="demo-image__preview" v-if="previewImg">
        <el-image style="width: 100px; height: 100px" :src="imgSrc" :preview-src-list="imgList"></el-image>
    </div>

</div>

<script src="http://cdn.staticfile.org/vue/2.6.11/vue.min.js"></script>
<script src="http://cdn.staticfile.org/axios/0.19.2/axios.min.js"></script>
<script src="http://cdn.staticfile.org/element-ui/2.13.1/index.js"></script>

<script>
    new Vue({
        el: '#app',
        data: {
            bucketURL: 'http://192.168.1.6:9000/mybucket/',
            previewImg: true,
            results: [],
            imgSrc: '',
            imgList: []
        },
        methods: {
            init() {
                axios.get('/list').then(response => {
                    this.results = response.data;
                    if (this.results.length == 0) {
                        this.imgSrc = '';
                        this.previewImg = false;
                    } else {
                        for (var i = 0; i < this.results.length; i++) {
                            this.imgList.push(this.bucketURL + this.results[i].fileName);
                            if (i == 0) {
                                this.imgSrc = this.bucketURL + this.results[0].fileName;
                            }
                        }
                    }
                });
            },
            preview(row, event, column) {
                this.imgSrc = this.bucketURL + row.fileName;
                this.previewImg = true;
            },
            deleteFile(e,index,list) {
                axios.delete(e.target.href, {}).then(res => {
                    if (res.data.code == 200) {
                        this.$message('删除成功！');
                        list.splice(index, 1);
                        this.previewImg = false;
                    } else {
                        this.$message('删除失败！');
                    }
                });
            }
        },
        mounted() {
            this.init();
        }
    });
</script>

</body>
</html>

前端上传页面 src\main\resources\templates\upload.html

<!DOCTYPE html>
<html lang="zh-CN">
<head>
    <meta charset="UTF-8">
    <title>图片上传</title>
    <link rel="stylesheet" type="text/css" href="http://cdn.staticfile.org/webuploader/0.1.5/webuploader.css">
    <script type="text/javascript" src="https://cdn.staticfile.org/jquery/3.5.0/jquery.min.js"></script>
    <script type="text/javascript" src="http://cdn.staticfile.org/webuploader/0.1.5/webuploader.min.js"></script>
</head>

<body>

<div id="uploader-demo">
    <div id="fileList" class="uploader-list"></div>
    <div id="filePicker">选择图片</div>
</div>
<br/>
<a href="/index.html">返回图片列表页面</a>

<script type="text/javascript">
    var uploader = WebUploader.create({
        auto: true,
        swf: 'http://cdn.staticfile.org/webuploader/0.1.5/Uploader.swf',
        server: '/upload',
        pick: '#filePicker',
        accept: {
            title: 'Images',
            extensions: 'gif,jpg,jpeg,bmp,png',
            mimeTypes: 'image/*'
        }
    });

    uploader.on('fileQueued', function (file) {
        var $li = $(
            '<div id="' + file.id + '" class="file-item thumbnail">' +
            '<img>' +
            '<div class="info">' + file.name + '</div>' +
            '</div>'
            ),
            $img = $li.find('img');

        var $list = $("#fileList");
        $list.append($li);

        uploader.makeThumb(file, function (error, src) {
            if (error) {
                $img.replaceWith('<span>不能预览</span>');
                return;
            }
            $img.attr('src', src);
        }, 100, 100);
    });

    uploader.on('uploadProgress', function (file, percentage) {
        var $li = $('#' + file.id),
            $percent = $li.find('.progress span');

        if (!$percent.length) {
            $percent = $('<p class="progress"><span></span></p>')
                .appendTo($li)
                .find('span');
        }
        $percent.css('width', percentage * 100 + '%');
    });

    uploader.on('uploadSuccess', function (file) {
        $('#' + file.id).addClass('upload-state-done');
    });

    uploader.on('uploadError', function (file) {
        var $li = $('#' + file.id),
            $error = $li.find('div.error');

        if (!$error.length) {
            $error = $('<div class="error"></div>').appendTo($li);
        }
        $error.text('上传失败');
    });

    uploader.on('uploadComplete', function (file) {
        $('#' + file.id).find('.progress').remove();
    });
</script>

</body>
</html>