SpringBoot Mongodb文件存储服务器

﹏ヽ暗。殇╰゛Y 2022-02-28 03:58 433阅读 0赞

###### 1.Mongodb简介 ######

*  MongoDB是一个高性能，开源，无模式的文档型数据库
 *  Mongodb可以通过副本集、分片来扩展数据库性能
 *  **GridFS**：Mongo主要解决的是海量数据的访问效率问题。因为Mongo主要是支持海量数据存储的，所以Mongo还自带了一个出色的分布式文件系统GridFS，可以支持海量的数据存储。由于Mongo可以支持复杂的数据结构，而且带有强大的数据查询功能

###### 2.为什么使用Mongodb存储 ######

*  它有分布式文件系统GridFS
 *  可以通过Mongodb的分片来水平扩展存储容量，多台分片服务器组成一个大的文件存储服务集群
 *  通过副本集来备份文件，天然支持容灾
 *  通过对比上传文件的 MD5值，实现秒传功能
 *  备份还原很方便

###### 3.其他资料 ######

*  [Mongodb安装][Mongodb]
 *  [Mongodb设置副本集][Mongodb 1]
 *  [SpringBoot与Mongodb集成][SpringBoot_Mongodb]
 *  [Gridfs官方文档][Gridfs]

###### 4.要达到的目标，服务提供的接口 ######

*  GET /files/list : 参数：pageIndex、pageSize 分页查询文件列表
 *  GET /files/\{id\} : 下载某个文件
 *  GET /files/view/\{id\} : 在线预览某个文件。比如，显示图片
 *  POST /files/upload : 上传文件
 *  GET /files/delete/\{id\} : 删除文件

###### 5.开始实践 ######

SpringBoot与MongoDB上传文件的关键是 GridFsTemplate 、GridFSBucket

*  1.启动MongoDB，我使用了3台mongodb搭配的副本集集群  
    ![在这里插入图片描述][20190324151109923.png]
 *  2.新建SpringBoot应用，添加基础依赖：

<dependency>
    	<groupId>org.springframework.boot</groupId>
    	<artifactId>spring-boot-starter-data-mongodb</artifactId>
    </dependency>
    <dependency>
    	<groupId>org.springframework.boot</groupId>
    	<artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
    	<groupId>org.springframework.boot</groupId>
    	<artifactId>spring-boot-starter-thymeleaf</artifactId>
    </dependency>
    <dependency>
    	<groupId>cn.hutool</groupId>
    	<artifactId>hutool-all</artifactId>
    	<version>4.5.1</version>
    </dependency>

*  3.添加FileController控制器，提供上面说的接口：

@CrossOrigin(origins = "*", maxAge = 3600)
    @RestController
    @RequestMapping("files")
    public class FileController {
    
        @Autowired private IFileService fileService;
    
        /**
         * 上传文件列表
         * @param pageIndex
         * @param pageSize
         * @return
         */
        @RequestMapping("/list")
        public List<FileDocument> list(int pageIndex, int pageSize){
            return fileService.listFilesByPage(pageIndex,pageSize);
        }
    
        /**
         * 在线显示文件
         * @param id 文件id
         * @return
         */
        @GetMapping("/view/{id}")
        public ResponseEntity<Object> serveFileOnline(@PathVariable String id) {
            Optional<FileDocument> file = fileService.getById(id);
            if (file.isPresent()) {
                return ResponseEntity.ok()
                        .header(HttpHeaders.CONTENT_DISPOSITION, "fileName=" + file.get().getName())
                        .header(HttpHeaders.CONTENT_TYPE, file.get().getContentType())
                        .header(HttpHeaders.CONTENT_LENGTH, file.get().getSize() + "").header("Connection", "close")
                        .body(file.get().getContent());
            } else {
                return ResponseEntity.status(HttpStatus.NOT_FOUND).body("File was not found");
            }
        }
    
        /**
         * 下载文件
         * @param id
         * @return
         * @throws UnsupportedEncodingException
         */
        @GetMapping("/{id}")
        public ResponseEntity<Object> downloadFileById(@PathVariable String id) throws UnsupportedEncodingException {
            Optional<FileDocument> file = fileService.getById(id);
            if(file.isPresent()){
               return ResponseEntity.ok()
                        .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; fileName=" + URLEncoder.encode(file.get().getName() , "utf-8"))
                        .header(HttpHeaders.CONTENT_TYPE, "application/octet-stream")
                        .header(HttpHeaders.CONTENT_LENGTH, file.get().getSize() + "").header("Connection", "close")
                        .body(file.get().getContent());
            }else {
                return ResponseEntity.status(HttpStatus.NOT_FOUND).body("File was not found");
            }
        }
    
        /**
         * 表单上传文件
         * 当数据库中存在该md5值时，可以实现秒传功能
         * @param file 文件
         * @return
         */
        @PostMapping("/upload")
        public ResponseModel formUpload(@RequestParam("file") MultipartFile file){
            ResponseModel model = ResponseModel.getInstance();
            try {
                if(file != null && !file.isEmpty()){
                    String fileMd5 = SecureUtil.md5(file.getInputStream());
                    FileDocument fileDocument = fileService.saveFile(fileMd5 , file);
    
                    System.out.println(fileDocument);
                    model.setData(fileDocument.getId());
                    model.setCode(ResponseModel.Success);
                    model.setMessage("上传成功");
                }else {
                    model.setMessage("请传入文件");
                }
            }catch (IOException ex){
                ex.printStackTrace();
                model.setMessage(ex.getMessage());
            }
            return model;
        }
    
        /**
         * 删除附件
         * @param id
         * @return
         */
        @GetMapping("/delete/{id}")
        public ResponseModel deleteFileByGetMethod(@PathVariable String id){
            ResponseModel model = ResponseModel.getInstance();
            if(!StrUtil.isEmpty(id)){
                fileService.removeFile(id , true);
                model.setCode(ResponseModel.Success);
                model.setMessage("删除成功");
            }else {
                model.setMessage("请传入文件id");
            }
            return model;
        }
    }

*  4.FileSercieImpl上传服务实现类

private static String collectionName = "fileDatas";
    
        @Autowired private MongoTemplate mongoTemplate;
        @Autowired private GridFsTemplate gridFsTemplate;
        @Autowired private GridFSBucket gridFSBucket;
    
        /**
         * 表单上传附件
         * @param md5
         * @param file
         * @return
         */
        @Override
        public FileDocument saveFile(String md5, MultipartFile file) {
            //已存在该文件，则实现秒传
            FileDocument fileDocument = getByMd5(md5);
            if(fileDocument != null){
                return fileDocument;
            }
    
            fileDocument = new FileDocument();
            fileDocument.setName(file.getOriginalFilename());
            fileDocument.setSize(file.getSize());
            fileDocument.setContentType(file.getContentType());
            fileDocument.setUploadDate(new Date());
            fileDocument.setMd5(md5);
            String suffix = file.getOriginalFilename().substring(file.getOriginalFilename().lastIndexOf("."));
            fileDocument.setSuffix(suffix);
    
            try {
                //文件存入gridfs
                String gridfsId = uploadFileToGridFS(file.getInputStream() , file.getContentType());
                fileDocument.setGridfsId(gridfsId);
                //上传的信息保存在mongodb
                fileDocument = mongoTemplate.save(fileDocument , collectionName);
            }catch (IOException ex){
                ex.printStackTrace();
            }
            return fileDocument;
        }
    
        /**
         * 上传文件到Mongodb的GridFs中
         * @param in
         * @param contentType
         * @return
         */
        private String uploadFileToGridFS(InputStream in , String contentType){
            String gridfsId = IdUtil.simpleUUID();
            //文件，存储在GridFS中
            gridFsTemplate.store(in, gridfsId , contentType);
            return gridfsId;
        }
    
        /**
         * 删除附件
         * @param id 文件id
         * @param isDeleteFile 是否删除文件
         */
        @Override
        public void removeFile(String id, boolean isDeleteFile) {
            FileDocument fileDocument = mongoTemplate.findById(id , FileDocument.class , collectionName);
            if(fileDocument != null){
                Query query = new Query().addCriteria(Criteria.where("_id").is(id));
                DeleteResult result = mongoTemplate.remove(query , collectionName);
                System.out.println("result:" + result.getDeletedCount());
    
                if(isDeleteFile){
                    Query deleteQuery = new Query().addCriteria(Criteria.where("filename").is(fileDocument.getGridfsId()));
                    gridFsTemplate.delete(deleteQuery);
                }
            }
        }
    
        /**
         * 查询附件
         * @param id 文件id
         * @return
         * @throws IOException
         */
        @Override
        public Optional<FileDocument> getById(String id){
            FileDocument fileDocument = mongoTemplate.findById(id , FileDocument.class , collectionName);
            if(fileDocument != null){
                Query gridQuery = new Query().addCriteria(Criteria.where("filename").is(fileDocument.getGridfsId()));
                try {
                    GridFSFile fsFile = gridFsTemplate.findOne(gridQuery);
                    GridFSDownloadStream in = gridFSBucket.openDownloadStream(fsFile.getObjectId());
                    if(in.getGridFSFile().getLength() > 0){
                        GridFsResource resource = new GridFsResource(fsFile, in);
                        fileDocument.setContent(IoUtil.readBytes(resource.getInputStream()));
                        return Optional.of(fileDocument);
                    }else {
                        fileDocument = null;
                        return Optional.empty();
                    }
                }catch (IOException ex){
                    ex.printStackTrace();
                }
            }
            return Optional.empty();
        }
    
        /**
         * 根据md5获取文件对象
         * @param md5
         * @return
         */
        @Override
        public FileDocument getByMd5(String md5) {
            Query query = new Query().addCriteria(Criteria.where("md5").is(md5));
            FileDocument fileDocument = mongoTemplate.findOne(query , FileDocument.class , collectionName);
            return fileDocument;
        }
        
        //文件上传列表
        @Override
        public List<FileDocument> listFilesByPage(int pageIndex, int pageSize) {
            Query query = new Query().with(new Sort(Sort.Direction.DESC, "uploadDate"));
            long skip = (pageIndex -1) * pageSize;
            query.skip(skip);
            query.limit(pageSize);
            Field field = query.fields();
            field.exclude("content");
            List<FileDocument> files = mongoTemplate.find(query , FileDocument.class , collectionName);
            return files;
        }

*  5.添加 IndexController控制器，通过Thymeleaf模版渲染页面，在本项目内可以进行上传，列表查询，删除，预览，下载等

@Controller
    public class IndexController {
    
        @Autowired
        private IFileService fileService;
        //打开index页面
        @RequestMapping("/")
        public String index(ModelMap map , @RequestParam(value = "pageIndex" , defaultValue = "1") int pageIndex,@RequestParam(value = "pageSize" , defaultValue = "10") int pageSize){
            map.addAttribute("list" , fileService.listFilesByPage(pageIndex,pageSize));
            return "index";
        }
        //打开表单上传页面
        @RequestMapping("/upload")
        public String upload(){
            return "upload";
        }
       //打开js上传页面
        @RequestMapping("/jsupload")
        public String jsupload(){
            return "jsupload";
        }
    }

*  6.新建3个html页面，index.html、upload.html、jsupload.html  
    ![在这里插入图片描述][20190324152347847.png]
 *  7.项目配置

spring.application.name=mongodb-upload
    server.port=8081
    
    # thymeleaf配置，开发环境不启用缓存，正式环境下请启用缓存，提高性能
    spring.thymeleaf.cache=false
    # thymeleaf对html元素格式要求严格，设置它的mode为HTML，忘记结束标签后不会报错
    spring.thymeleaf.mode=HTML
    
    # 编码
    spring.http.encoding.charset=UTF-8
    spring.http.encoding.enabled=true
    
    # MongoDB 配置
    spring.data.mongodb.uri=mongodb://zhuyu:zhuyu@192.168.68.137:27017,192.168.68.137:27017,192.168.68.139:27017/ai?slaveOk=true&replicaSet=zypcy&write=1&readPreference=secondaryPreferred&connectTimeoutMS=300000
    
    #每个主机的连接数
    spring.data.mongodb.connections-per-host=20
    #线程队列数，它以上面connectionsPerHost值相乘的结果就是线程队列最大值
    spring.data.mongodb.threads-allowed-to-block-for-connection-multiplier=20
    spring.data.mongodb.connect-timeout=10000
    spring.data.mongodb.socket-timeout=10000
    spring.data.mongodb.max-wait-time=5000
    #控制是否在一个连接时，系统会自动重试
    spring.data.mongodb.auto-connect-retry=true
    spring.data.mongodb.socket-keep-alive=true
    
    # limit upload file size
    spring.servlet.multipart.max-file-size=30MB
    spring.servlet.multipart.max-request-size=50MB

*  8.启动项目，访问 localhost:8081 ，通过表单与js两种方法上传几个文件  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70]  
    预览图片与txt类型的文件  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70 1]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70 2]  
    下载文件  
    ![在这里插入图片描述][20190324154904233.png]
 *  9.通过Mongodb工具，查看MongoDB里面存储的数据，可以看到刚刚上传上去的文件被切分成一个个的chunks保存到fs.chunks中了  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70 3]  
    因篇幅原因只贴出了核心代码，代码被上传到码云上，[源码下载][Link 1]

上面的代码还可以更加完善，欢迎朋友们自行添加，如：鉴权—只有授权的用户才能上传文件到文件服务器

MongoDB数据超多之后请使用索引，真实案例，有张表数据超过300万，分页查询超时，设置合理的索引后秒查  
如，按时间倒序索引：db.Global\_Exception\_Info.createIndex(\{“createDate”:-1\})，正序是 1，倒序是 -1

[Mongodb]: https://blog.csdn.net/zhuyu19911016520/article/details/78740023
[Mongodb 1]: https://blog.csdn.net/zhuyu19911016520/article/details/79863226
[SpringBoot_Mongodb]: https://blog.csdn.net/zhuyu19911016520/article/details/81508183
[Gridfs]: https://docs.mongodb.com/manual/core/gridfs/
[20190324151109923.png]: /images/20220228/0b33aa81869a401c87606086a777b964.png
[20190324152347847.png]: /images/20220228/6f640048aec44cc7b16e10fc46ccd0c8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70]: /images/20220228/73c0422777644589b7ec4235fa3ab50c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70 1]: /images/20220228/7ce8c8832ea4456ba35ea74690486703.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70 2]: /images/20220228/6a60c3931b3c4ad495accccee4474761.png
[20190324154904233.png]: /images/20220228/11ff5300062a45b58165a425cf2232cb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3podXl1MTk5MTEwMTY1MjA_size_16_color_FFFFFF_t_70 3]: /images/20220228/5bd51c86d85c4cc181c27db86edb8d06.png
[Link 1]: https://gitee.com/zhuyu1991/spring-cloud/tree/master/mongo-upload