分布式文件存储FastDFS

妖狐艹你老母 2022-11-28 13:39 301阅读 0赞

**Table of Contents**

1.FastDFS简介

FastDFS服务端两个角色

结构图

上传流程

2.FastDFS安装

运行tracker

运行storage

配置Nginx

3.业务实现

--------------------

## 1.FastDFS简介 ##

FastDFS是一个开源的分布式文件系统，她对文件进行管理，功能包括：文件存储、文件同步、文件访问（文件上传、文件下载）等，解决了大容量存储和负载均衡的问题。特别适合以文件为载体的在线服务，如相册网站、视频网站等等。

### FastDFS服务端两个角色 ###

跟踪器（tracker）和存储节点（storage）。跟踪器主要做调度工作，在访问上起负载均衡的作用。

### 结构图 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 1][]

### 上传流程 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 2][]

## 2.FastDFS安装 ##

[https://hub.docker.com/r/morunchang/fastdfs][https_hub.docker.com_r_morunchang_fastdfs]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 3][]

docker pull morunchang/fastdfs

### **运行tracker** ###

docker run -d --name tracker --net=host --restart=always morunchang/fastdfs sh tracker.sh

### **运行storage** ###

docker run -d --name storage --net=host -e TRACKER_IP=192.168.71.128:22122 -e GROUP_NAME=group1 --restart=always morunchang/fastdfs sh storage.sh

如果想要增加新的storage服务器，再次运行该命令，注意更换 新组名

### **配置Nginx** ###

**默认已经配置好，可根据需求修改**

Nginx在这里主要提供对FastDFS图片访问的支持，fastdfs 容器中已经集成了Nginx，需要我们修改nginx的配置

进入storage的容器内部，修改nginx.conf

#进入容器
    docker exec -it storage  /bin/bash
    
    #修改文件
    vi /etc/nginx/conf/nginx.conf

**禁止缓存（可选）：**

add_header Cache-Control no-store;

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 4][]

**如修改，需重启storage容器**

## 3.业务实现 ##

源码：[https://gitee.com/akeung/fastfds-demo][https_gitee.com_akeung_fastfds-demo]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 5][]

pom

<dependency>
                <groupId>net.oschina.zcx7878</groupId>
                <artifactId>fastdfs-client-java</artifactId>
                <version>1.27.0.0</version>
            </dependency>

fdfs\_client.conf

#连接超时的世界 s
    connect_timeout=60
    #网络超时时间
    network_timeout=60
    #字符编码
    charset=UTF-8
    # tracker的http通信协议的端口
    http.tracker_http_port=8080
    # 22122 trackerserver的tcp 端口
    tracker_server=192.168.71.128:22122

application.yml

spring:
      servlet:
        multipart:
          max-file-size: 10MB
          max-request-size: 10MB
      application:
        name: file
    server:
      port: 18082

package com.example.api.utils;
    
    import com.example.api.bean.FastDFSFile;
    import com.fasterxml.jackson.core.JsonProcessingException;
    import com.fasterxml.jackson.databind.ObjectMapper;
    import org.csource.common.NameValuePair;
    import org.csource.fastdfs.*;
    import org.springframework.core.io.ClassPathResource;
    
    import java.io.*;
    
    /**
     * @author Akeung
     * 2020/8/18
     */
    public class FastDFSClient {
    
        /**
         *加载 tracker 链接信息
         */
        static {
            try {
                ClientGlobal.init(new ClassPathResource("fdfs_client.conf").getPath());
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    
        private static TrackerServer getTrackerServer() throws IOException {
            return new TrackerClient().getConnection();
        }
    
        private static StorageClient getStorageClient() throws IOException {
            return new StorageClient(getTrackerServer(), null);
        }
    
        /**
         * 上传
         *
         * @param file
         * @return
         */
        public static String upload(FastDFSFile file) {
    
            try {
                TrackerServer trackerServer = getTrackerServer();
                StorageClient storageClient = new StorageClient(trackerServer, null);
    
                //参数1 字节数组
                //参数2 扩展名(不带点)
                //参数3 元数据( 文件的大小,文件的作者,文件的创建时间戳)
                //byte cmd, String group_name, String local_filename, String file_ext_name, NameValuePair[] meta_list
                NameValuePair[] meta_list = new NameValuePair[]{
                        new NameValuePair("Author", file.getAuthor()),
                        new NameValuePair("Name", file.getName())
                };
    
                String[] strings = storageClient.upload_file(file.getContent(), file.getExt(), meta_list);
                /**
                 *  strings[0]==group1  strings[1]=M00/00/00/1.jpg
                 * <ul><li>results[0]: the group name to store the file</li></ul>
                 * <ul><li>results[1]: the new created filename</li></ul>
                 */
                String hostString = trackerServer.getInetSocketAddress().getHostString();
                int g_tracker_http_port = ClientGlobal.getG_tracker_http_port();
                //http://192.168.71.128:8080/group1/M00/00/00/wKhHgF87dTqAZk6kAAFJbAv-xZI615.jpg
                return "http://"+hostString+":"+g_tracker_http_port+"/"+strings[0]+"/"+strings[1];
            } catch (Exception e) {
                e.printStackTrace();
            }
            return null;
        }
    
        /**
         * 下载
         * @param groupName
         * @param remoteFileName
         * @return
         */
        public static InputStream download(String groupName, String remoteFileName) {
            ByteArrayInputStream byteArrayInputStream = null;
            try {
                StorageClient storageClient =getStorageClient();
                //参数1:指定组名
                //参数2 :指定远程的文件名
                byte[] bytes = storageClient.download_file(groupName, remoteFileName);
                byteArrayInputStream = new ByteArrayInputStream(bytes);
                return byteArrayInputStream;
            } catch (Exception e) {
                e.printStackTrace();
            } finally {
                try {
                    if (byteArrayInputStream != null) {
                        byteArrayInputStream.close();
                    }
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            return null;
        }
    
        /**
         * 删除
         * @param groupName
         * @param remoteFileName
         */
        public static void delete(String groupName, String remoteFileName) {
            try {
                StorageClient storageClient = getStorageClient();
                int i = storageClient.delete_file(groupName, remoteFileName);
                if (i == 0) {
                    System.out.println("删除成功");
                } else {
                    System.out.println("删除失败");
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    
        /**
         * 读取文件信息
         * @param groupName  group1
         * @param remoteFileName  M00/00/00/wKhHgF87dTqAZk6kAAFJbAv-xZI615.jpg
         * @return
         */
        public static FileInfo getFile(String groupName, String remoteFileName) {
            try {
                StorageClient storageClient = getStorageClient();
                //参数1 指定组名
                //参数2 指定文件的路径
                FileInfo fileInfo = storageClient.get_file_info(groupName, remoteFileName);
                return fileInfo;
            } catch (Exception e) {
                e.printStackTrace();
            }
            return null;
        }
    
        public static void main(String[] args) throws IOException {
    
            //test getFile
    //        ObjectMapper objectMapper = new ObjectMapper();
    //        FileInfo fileInfo = getFile("group1", "M00/00/00/wKhHgF87dTqAZk6kAAFJbAv-xZI615.jpg");
    //        System.out.println(objectMapper.writeValueAsString(fileInfo));
    
            //test download
    //        InputStream inputStream = download("group1", "M00/00/00/wKhHgF87jEKAd9qfAAFJbAv-xZI043.jpg");
    //        FileOutputStream outputStream = new FileOutputStream("F:/00.jpg");
    //
    //        byte[] bytes = new byte[1024];
    //        while (inputStream.read(bytes) != -1){
    //            outputStream.write(bytes);
    //        }
    //        outputStream.flush();
    //        outputStream.close();
    //        inputStream.close();
    
            //test delete
            delete("group1", "M00/00/00/wKhHgF87iqOADGD3AAFJbAv-xZI845.jpg");
    
        }
    
    
    }

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70]: /images/20221124/da388c5db27f4a618366ee968e0a89ec.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 1]: /images/20221124/0b196dc4cdc24ef7886204d6301c96d5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 2]: /images/20221124/66a6c497aca244e1a3ab886f53cd01f0.png
[https_hub.docker.com_r_morunchang_fastdfs]: https://hub.docker.com/r/morunchang/fastdfs
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 3]: /images/20221124/94e4140cbb774b58a39c080e9ffdb9ca.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 4]: /images/20221124/f39082fcd576418798f0c5cd187ad804.png
[https_gitee.com_akeung_fastfds-demo]: https://gitee.com/akeung/fastfds-demo
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjAzMjE5OQ_size_16_color_FFFFFF_t_70 5]: /images/20221124/cd5d1c7b95034dd88721ec487a14f0d0.png