java文件上传Minio——文件、文件分片上传

短命女 2023-10-13 17:48 139阅读 0赞

本文主要介绍了文件上传——单\\多文件分片上传；当然还有文件的md5值秒传、文件断点续传等内容，本文结合minio文件服务和redis做计数来实现的场景（下方地址是单机情况，即上传至本地demo）。有什么问题可以在留言哦！并在文章末尾附上demo源码下载！

大文件分片上传博文地址：[大文件分片上传][Link 1]

大文件(md5分片)上传博文地址：[大文件(md5分片)上传][md5]

大文件断点续传博文地址：[大文件断点续传][Link 2]

## 一、创建前后端项目及演示 ##

创建前后端项目，采用springboot项目结构，前端采用的vue框架搭建(前端完全是边用边百度，这里就不附前面项目结构，就一个页面)

![c1ee31c1670a4844b703c44b93beafc7.png][]![72553c20f59f4621bb4b2ddfbb700ba0.png][]后端控制台的输出日志![4ccd1af8ccb0424d9f2748094198548f.png][]

minio文件服务上面的结果

关于minio的配置与基本使用：[(37条消息) minio的基本使用——java\_java使用minio\_寒夜憨憨的博客-CSDN博客][37_ minio_java_java_minio_-CSDN]

![f3c78f468b9e456a83e998f66e916be1.png][]

## 二、文件上传minio核心代码 ##

其中包括了单文件上传minio和多文件分片上传minio;至于文件分片上传采用的是：先保存分片文件——>然后合并分片文件——>再删除分片文件;其实minio有自带的文件分片上传策略。

文件上传至minio，如果单纯的保存文件的话，其实可以直接采用前端去对接minio，然后当上传成功之后，返回上传至minio文件的相关信息给后端进行数据存储处理即可；这样上传文件就不要经过后端服务器了，减少数据处理环节。

### 1、minio配置信息核心代码 ###

在代码的注释都详细说明了每一步是做什么的，如果不懂的可以详细看看每一步的注释，当然也可以留言！

package com.jdh.fileUpload.config;
    
    import io.minio.*;
    import io.minio.errors.*;
    import io.minio.messages.Item;
    import lombok.*;
    import lombok.extern.slf4j.Slf4j;
    import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
    import org.springframework.context.annotation.Bean;
    import org.springframework.context.annotation.Configuration;
    import org.springframework.http.MediaType;
    import org.springframework.mock.web.MockMultipartFile;
    import org.springframework.util.StringUtils;
    import org.springframework.web.multipart.MultipartFile;
    import io.minio.MinioClient;
    
    import javax.annotation.Resource;
    import java.io.*;
    import java.util.Objects;
    import java.util.UUID;
    
    /**
     * @ClassName: minioConfig
     * @Author: jdh
     * @CreateTime: 2022-03-12
     * @Description:
     */
    @Slf4j
    @Data
    @Configuration
    @ConditionalOnProperty(prefix = "autoconfigure", value = "isMinio", matchIfMissing = false)
    public class MinioUtil {
    
        private MinioClient minioClient;
    
        @Resource
        private MinioProperties minioProperties;
    
    //    @Autowired
    //    private MyMinioClient myMinioClient;
    
    
        /**
         * 获取一个连接minio服务端的客户端
         *
         * @return MinioClient
         */
        @Bean
        public MinioClient getMinioClient() {
    
            try {
                String url = "http:" + minioProperties.getIp() + ":" + minioProperties.getPort();
                MinioClient minioClient = MinioClient.builder()
                        .endpoint(url)    //两种都可以,这种全路径的其实就是下面分开配置一样的
    //                        .endpoint(minioProperties.getIp(),minioProperties.getPort(),minioProperties.getSecure())
                        .credentials(minioProperties.getAccessKey(), minioProperties.getSecretKey())
                        .build();
                this.minioClient = minioClient;
                return minioClient;
            } catch (Exception e) {
                e.printStackTrace();
                log.info("创建minio客户端失败！", e);
                return null;
            }
        }
    
        /**
         * 创建桶
         *
         * @param bucketName 桶名称
         */
        public Boolean createBucket(String bucketName) throws Exception {
            if (!StringUtils.hasLength(bucketName)) {
                throw new RuntimeException("创建桶的时候，桶名不能为空！");
            }
            try {
                minioClient.makeBucket(MakeBucketArgs.builder()
                        .bucket(bucketName)
                        .build());
                return true;
            } catch (Exception e) {
                log.info("创建桶失败！", e);
                return false;
            }
    
        }
    
        /**
         * 检查桶是否存在
         *
         * @param bucketName 桶名称
         * @return boolean true-存在 false-不存在
         */
        public boolean checkBucketExist(String bucketName) throws Exception {
            if (!StringUtils.hasLength(bucketName)) {
                throw new RuntimeException("检测桶的时候，桶名不能为空！");
            }
    
            return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
        }
    
        /**
         * 检测某个桶内是否存在某个文件
         *
         * @param objectName 文件名称
         * @param bucketName 桶名称
         */
        public boolean getBucketFileExist(String objectName, String bucketName) throws Exception {
            if (!StringUtils.hasLength(objectName) || !StringUtils.hasLength(bucketName)) {
                throw new RuntimeException("检测文件的时候，文件名和桶名不能为空！");
            }
    
            try {
                // 判断文件是否存在
                return (minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build()) &&
                        minioClient.statObject(StatObjectArgs.builder().bucket(bucketName).object(objectName).build()) != null);
    
            } catch (ErrorResponseException e) {
                log.info("文件不存在 ! Object does not exist");
                return false;
            } catch (Exception e) {
                throw new Exception(e);
            }
        }
    
        /**
         * 删除文件夹
         *
         * @param bucketName 桶名
         * @param objectName 文件夹名
         * @param isDeep     是否递归删除
         * @return
         */
        public Boolean deleteBucketFolder(String bucketName, String objectName, Boolean isDeep) {
            if (!StringUtils.hasLength(bucketName) || !StringUtils.hasLength(objectName)) {
                throw new RuntimeException("删除文件夹的时候，桶名或文件名不能为空！");
            }
            try {
                ListObjectsArgs args = ListObjectsArgs.builder().bucket(bucketName).prefix(objectName + "/").recursive(isDeep).build();
                Iterable<Result<Item>> listObjects = minioClient.listObjects(args);
                listObjects.forEach(objectResult -> {
                    try {
                        Item item = objectResult.get();
                        System.out.println(item.objectName());
                        minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(item.objectName()).build());
                    } catch (Exception e) {
                        log.info("删除文件夹中的文件异常", e);
                    }
                });
                return true;
            } catch (Exception e) {
                log.info("删除文件夹失败");
                return false;
            }
        }
    
        /**
         * 文件上传文件
         *
         * @param file       文件
         * @param bucketName 桶名
         * @param objectName 文件名,如果有文件夹则格式为 "文件夹名/文件名"
         * @return
         */
        public Boolean uploadFile(MultipartFile file, String bucketName, String objectName) {
    
            if (Objects.isNull(file) || Objects.isNull(bucketName)) {
                throw new RuntimeException("文件或者桶名参数不全！");
            }
    
            try {
                //资源的媒体类型
                String contentType = MediaType.APPLICATION_OCTET_STREAM_VALUE;//默认未知二进制流
                InputStream inputStream = file.getInputStream();
                PutObjectArgs args = PutObjectArgs.builder()
                        .bucket(bucketName).object(objectName)
                        .stream(inputStream, file.getSize(), -1)
                        .contentType(file.getContentType())
                        .build();
                ObjectWriteResponse response = minioClient.putObject(args);
                inputStream.close();
                return response.etag() != null;
            } catch (Exception e) {
                log.info("单文件上传失败！", e);
                return false;
            }
        }
    
        /**
         * 分片合并
         *
         * @param bucketName
         * @param folderName
         * @param objectName
         * @param partNum
         * @return
         */
        public Boolean uploadFileComplete(String bucketName, String folderName, String objectName, Integer partNum) {
    
            try {
                //获取临时文件下的所有文件信息
                Iterable<Result<Item>> listObjects = minioClient.listObjects(ListObjectsArgs.builder().bucket(bucketName).prefix(folderName + "/").build());
    
                //计算minio中分片个数
                Integer num = 0;
                for (Result<Item> result : listObjects) {
                    num++;
                }
                //在依次校验实际分片数和预计分片数是否一致
                if (!num.equals(partNum)) {
                    log.info("文件 {} 分片合并的时候，检测到实际分片数 {} 和预计分片数 {} 不一致", folderName, num, partNum);
                    return false;
                }
    
                InputStream inputStream = null;
                log.info("开始合并文件 {} 分片合并，实际分片数 {} 和预计分片数 {}", folderName, num, partNum);
                for (int i = 0; i < num; i++) {
                    String tempName = folderName + "/" + objectName.substring(0, objectName.lastIndexOf(".")) + "_" + i + ".temp";
                    try {
                        //获取分片文件流
                        InputStream response = minioClient.getObject(
                                GetObjectArgs.builder().bucket(bucketName).object(tempName).build());
                        //流合并
                        if (inputStream == null) {
                            inputStream = response;
                        } else {
                            inputStream = new SequenceInputStream(inputStream, response);
                        }
                    } catch (Exception e) {
                        log.info("读取分片文件失败！", e);
                    }
                }
                if (inputStream == null) {
                    log.info("合并流数据为空！");
                    return false;
                }
                //转换为文件格式
                MockMultipartFile file = new MockMultipartFile(objectName, inputStream);
    
                //将合并的文件流写入到minio中
                PutObjectArgs args = PutObjectArgs.builder()
                        .bucket(bucketName).object(objectName)
                        .stream(file.getInputStream(), file.getSize(), -1)
    //                    .contentType(file.getContentType())//这里可以不知道类型
                        .build();
                String etag = minioClient.putObject(args).etag();
    
                // 删除临时文件
                if (etag != null) {
                    listObjects.forEach(objectResult -> {
                        try {
                            Item item = objectResult.get();
                            minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(item.objectName()).build());
                        } catch (Exception e) {
                            log.info("删除文件夹中的文件异常", e);
                        }
                    });
                    log.info("{}:临时文件夹文件已删除！", folderName);
                }
    
                inputStream.close();
                return etag != null;
            } catch (Exception e) {
                log.info("合并 {} - {} 文件失败！", folderName, objectName, e);
                return false;
            }
        }
    
        public static String generateUniqueId(int length) {
            UUID uuid = UUID.randomUUID();
            String hash = uuid.toString().replaceAll("-", "");
            return hash.substring(0, Math.min(length, hash.length()));
        }
    
    }

### 2、文件上传minio的实现核心代码 ###

package com.jdh.fileUpload.service.fileUpload;
    
    import com.alibaba.fastjson2.JSONObject;
    import com.jdh.fileUpload.config.MinioUtil;
    import com.jdh.fileUpload.utils.Result;
    import com.jdh.fileUpload.utils.Uuid;
    import com.jdh.fileUpload.vo.PartFileIndexVo;
    import lombok.extern.slf4j.Slf4j;
    import org.redisson.api.RedissonClient;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
    import org.springframework.data.redis.core.RedisTemplate;
    import org.springframework.stereotype.Service;
    import org.springframework.web.multipart.MultipartFile;
    
    import javax.annotation.Resource;
    
    /**
     * @ClassName: FileUploadMinioServiceImpl
     * @Author: jdh
     * @CreateTime: 2022-03-15
     * @Description:
     */
    @Slf4j
    @Service
    @ConditionalOnProperty(prefix = "autoconfigure", value = "isMinio", matchIfMissing = false)
    public class FileUploadMinioServiceImpl implements FileUploadMinioService {
    
        @Autowired
        private MinioUtil minioUtil;
    
        @Resource
        private RedissonClient redissonClient;
    
        @Autowired
        private RedisTemplate<String, Object> redisTemplate;
    
        @Override
        public Result<Boolean> fileUploadMinio(MultipartFile file) {
            String objectName = Uuid.generateUniqueId(8) + "_" + file.getOriginalFilename();
    
            try {
                Boolean aBoolean = minioUtil.uploadFile(file, "minio-file", objectName);
                log.info("文件上传是否完成：{}", aBoolean);
                return Result.success(aBoolean, "文件上传完成！");
            } catch (Exception e) {
                log.info("文件上传异常！", e);
                return Result.error(false, "文件上传异常！");
            }
        }
    
        @Override
        public Result<Boolean> filePartUploadMinio(MultipartFile file, String partFile) {
    
    //        String objectName = Uuid.generateUniqueId(8) + "_" + file.getOriginalFilename();
            try {
                PartFileIndexVo partFileIndexVo = JSONObject.parseObject(partFile, PartFileIndexVo.class);
                String objectName = "temp/" + partFileIndexVo.getFileUid() + "/"
                        + partFileIndexVo.getFileName().substring(0, partFileIndexVo.getFileName().lastIndexOf(".")) + "_"
                        + partFileIndexVo.getPartIndex() + ".temp";
                //上传文件分片
                Boolean partState = minioUtil.uploadFile(file, "minio-file", objectName);
    
                if (partState) {
    
                    Long result = redisTemplate.opsForValue().increment("partCount:" + partFileIndexVo.getFileUid());
    
                    if (result.equals(partFileIndexVo.getPartTotalNum().longValue())) {
    
                        log.info("{}:开始合并分片请求...", partFileIndexVo.getFileName());
                        //开始合并分片
                        Boolean complete = minioUtil.uploadFileComplete("minio-file", "temp/" + partFileIndexVo.getFileUid(),
                                partFileIndexVo.getFileName(), partFileIndexVo.getPartTotalNum());
                        //移除文件分片上传情况，这个也应该在redis中
                        Boolean delete = false;
                        if (complete) {
                            delete = redisTemplate.delete("partCount:" + partFileIndexVo.getFileUid());
                            log.info("{}:分片文件合并完成！", partFileIndexVo.getFileName());
                        }
                        return Result.success(complete && delete, partFileIndexVo.getFileUid());
                    }
                }
                return Result.success(partState, partFileIndexVo.getPartIndex().toString());
            } catch (Exception e) {
                log.info("文件上传异常！", e);
                return Result.error("文件上传异常！");
            }
        }
    
    }

### 3、关于相关前端核心代码 ###

页面代码

<div class="singleFileUploadMinio">
                    <el-upload  ref="upload" name="files" action="#"
                        :on-change="selectSingleFileMinio" :on-remove="removeSingleFileMinio" :file-list="singleFileMinio.fileList" :auto-upload="false">
                        <el-button slot="trigger" size="small" type="primary">选取文件</el-button>
                        <el-button style="margin-left: 10px;" size="small" type="success" @click="singleFileUploadMinio">点击进行单文件分片上传Minio</el-button>
                        <div slot="tip" class="el-upload__tip">主要用于测试单文件上传Minio</div>
                    </el-upload>
                </div>
                 
                 <div class="multipleFilePartUploadMinio">
                    <el-upload ref="upload" name="files" action="#"
                        :on-change="selectMultiplePartFileMinio" :on-remove="removeMultiplePartFileMinio" :file-list="multipleFilePartMinio.fileList" :auto-upload="false">
                        <el-button slot="trigger" size="small" type="primary">选取文件</el-button>
                        <el-button style="margin-left: 10px;" size="small" type="success" @click="multipleFilePartUploadMinio">点击进行多文件分片上传Minio</el-button>
                        <div slot="tip" class="el-upload__tip">主要用于测试多文件分片上传Minio</div>
                    </el-upload>
                </div>

数据定义组件代码

data() {
            return {
                urlPrefix: 'http://localhost:8082',
                headers: {
                    'Content-Type': 'multipart/form-data'
                },
                singleFileMinio: {
                    file: '',
                    fileList: []
                },
                multipleFilePartMinio: {
                    fileList: [],
                    fileData: []
                },
            };
        },

各方法代码

methods: {
        // 下面是单文件上传Minio的一些方法
        selectSingleFileMinio(file, fileList){
            this.singleFileMinio.file = file
            this.singleFileMinio.fileList = []
            this.singleFileMinio.fileList.push(file)
            console.log("单文件上传Minio->选中的文件：",this.singleFileMinio.file)
        },
        removeSingleFileMinio(file, fileList){
            this.singleFileMinio.file = ''
            console.log("单文件上传Minio->移除选中的文件：",this.singleFileMinio.file)
        },
        singleFileUploadMinio(){
            let url = this.urlPrefix + "/minio/singleFileUploadMinio";
            var fileParam = new FormData();
            fileParam.append("file",this.singleFileMinio.file.raw);
    
            this.$upFile(url,fileParam,null,resp => {
                let res = resp.data;
                if (res.data){
                    this.$message.success(res.msg)
                }else{
                    this.$message.error(res.msg)
                }
            })
        },
        // 下面多文件分片上传Minio的一些方法
        selectMultiplePartFileMinio(file, fileList){
            this.multipleFilePartMinio.fileList.push(file)
            console.log("多文件分片上传->选中的文件：",this.multipleFilePartMinio.fileList)
        },
        removeMultiplePartFileMinio(file, fileList){
            this.multipleFilePartMinio.fileList = this.multipleFilePartMinio.fileList.filter(item => item.uid != file.uid);
            console.log("多文件分片上传->移除选中的文件：",this.multipleFilePartMinio.fileList)
        },
        multipleFilePartUploadMinio(){
            let fileUploadCount = 0;
            let url = this.urlPrefix + "/minio/partFileUploadMinio";
            let partSize = 8; //MB
            let partFileUid = [];
            for(let i = 0; i < this.multipleFilePartMinio.fileList.length; i++ ){
                let partNum = Math.ceil(this.multipleFilePartMinio.fileList[i].size / 1024 / 1024 / partSize)
                let partFileIndex = {
                    fileName: this.multipleFilePartMinio.fileList[i].name,
                    fileUid: this.multipleFilePartMinio.fileList[i].uid,
                    partTotalNum: partNum,
                    partIndex: ''
                }
                partFileUid.push(this.multipleFilePartMinio.fileList[i].uid + '')
                for(let j = 0; j < partNum; j++ ){
                    partFileIndex.partIndex = j
                    var fileParam = new FormData();
                    fileParam.append('file', this.multipleFilePartMinio.fileList[i].raw.slice(j * 1024 * 1024 * partSize, (j + 1) * 1024 * 1024 * partSize))
                    fileParam.append('partFileIndex', JSON.stringify(partFileIndex))
                    
                    this.$upFile(url,fileParam,null,resp => {
                        let res = resp.data;
                        //判断最后一个分片文件传完之后，后台检验文件上传情况
                        if (res.code == 200){
    
                            if(!res.data){
                                var info = '当前分片上传Minio失败!uid:' + partFileUid + ';当前分片:' + j + ';返回分片:' + res.msg
                                console.log(info)
                            }
    
                            let isUid = partFileUid.includes(res.msg)
                            if(isUid){
                                fileUploadCount ++;
                                if(fileUploadCount == this.multipleFilePartMinio.fileList.length){
                                    console.log('多文件分片上传Minio完成')
                                    this.$message.success('多文件分片上传Minio完成')
                                }
                            }
                        }else{
                            var info = '多文件分片上传Minio失败!当前分片:' + j + ';当前uid:' + partFileUid
                            console.log(info)
                            this.$message.error(res.msg)
                        }
                    })
                }
            }
        }
      },

## 三、源码下载和结尾 ##

源码默认只有上传本地的各种demo方案，如果需要打开上传至minio的话，需要再配置文件修改配置文件，如下：

![521e04982ce14b9fbe6419cdc416f012.png][]

gitee后端链接：[java\_fileUpload\_demo: 关于文件上传的一些基本介绍（如文件上传、分片上传、md5值秒传、断点续传等）][java_fileUpload_demo_ _md5]

gitee前端链接：[https://gitee.com/java\_utils\_demo/vue2\_demo.git][https_gitee.com_java_utils_demo_vue2_demo.git]

**其实思路很简单也很明确，就是一个大文件分成n个小文件进行小文件依次单独上传，后端检测到所有分片上传完成之后，即对这些分片进行合并，并删除对应临时分片文件。**

再次说明下，如果需要上传至minio，那么需要先配置好minio和redis并启动这两个服务，然后再启动demo项目。

[Link 1]: https://blog.csdn.net/qq_45778079/article/details/131650471
[md5]: https://blog.csdn.net/qq_45778079/article/details/131668461
[Link 2]: https://blog.csdn.net/qq_45778079/article/details/131669816
[c1ee31c1670a4844b703c44b93beafc7.png]: https://img-blog.csdnimg.cn/c1ee31c1670a4844b703c44b93beafc7.png
[72553c20f59f4621bb4b2ddfbb700ba0.png]: https://img-blog.csdnimg.cn/72553c20f59f4621bb4b2ddfbb700ba0.png
[4ccd1af8ccb0424d9f2748094198548f.png]: https://img-blog.csdnimg.cn/4ccd1af8ccb0424d9f2748094198548f.png
[37_ minio_java_java_minio_-CSDN]: https://blog.csdn.net/qq_45778079/article/details/131572131
[f3c78f468b9e456a83e998f66e916be1.png]: https://img-blog.csdnimg.cn/f3c78f468b9e456a83e998f66e916be1.png
[521e04982ce14b9fbe6419cdc416f012.png]: https://img-blog.csdnimg.cn/521e04982ce14b9fbe6419cdc416f012.png
[java_fileUpload_demo_ _md5]: https://gitee.com/java_utils_demo/java_file-upload_demo.git
[https_gitee.com_java_utils_demo_vue2_demo.git]: https://gitee.com/java_utils_demo/vue2_demo.git