分布式文件系统fastdfs

妖狐艹你老母 2022-04-11 08:16 407阅读 0赞

一。介绍fastdfs及原理

FastDFS是一款轻量级的开源分布式文件系统，它用纯C语言实现，支持Linux、FreeBSD、AIX等UNIX系统。只能通过 专有API对文件进行存取访问，不支持POSIX接口方式，不能mount使用.

原理介绍：

服务分为两大部件Tracker server和Storage server ：Tracker server负责调度 计算最适合的存储服务器 ，Storage server负责存储数组 Storage server被分为多个组 ，不同的文件上传分发到不同的组中，同一组之间可以形成高可用的主从备份，同一组之间的服务器之间是互为拷贝，数据完全相同

原理图：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70]  
**fastdfs操作顺序**

fastdfs 安装  
查询：docker search fastdfs  
下载：docker pull season/fastdfs  
本机方式启动注册中心tracker：docker run -itd --name trakcer -v ~/tracker\_data:  
/fastdfs/tracker/data --net=host season/fastdfs tracker  
默认启动22122端口  
禁用防火墙：vi /etc/sysconfig/selinux  
重启虚拟机：reboot

**防火墙禁用方法**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70 1]  
**启动仓库storage**：`docker run -itd --name storage -v ~/storage_data:/fastdfs/storage/data -v ~/store_path:/fastdfs/store_path --net=host -e TRACKER_SERVER:192.168.1.2:22122 season/fastdfs storage`

**进入容器**  
进入容器docker exec -it storage bash  
查看配置：more storage.conf  
复制出来，修改配置再拷贝回去：docker cp storage.conf:/fdfs\_conf ~/  
拷贝回去：docker cp ./storage.conf storage:/fdfs\_conf/  
因为docker是精简版的系统

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70 2]

**将ip修改为Linux系统Ip；**  
**重启storage**：docker stop storage → docker start storage  
**再次进入storage容器：** docker exec -it storage bash

**确认配置是否修改成功**

fdfs_monitor fdfs_conf/storage.conf

下图箭头为组名，默认group1,依次往上加，有组号说明tracker已启动  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70 3]

》》测试文件上传  
**查看/usr/bin下的所有fdfs工具**

[root@bogon fdfs]# ls /usr/bin | grep fdfs
    fdfs_appender_test
    fdfs_appender_test1
    fdfs_append_file
    fdfs_crc32
    fdfs_delete_file
    fdfs_download_file
    fdfs_file_info
    fdfs_monitor
    fdfs_storaged
    fdfs_test
    fdfs_test1
    fdfs_trackerd
    fdfs_upload_appender
    fdfs_upload_file  //上传文件

**启动客户端**

docker run -itd --name fdfs_sh --net=host season/fastdfs sh

创建a.txt文件同时赋值： echo  Hello>a.txt
    root@localhost:/#  cd fdfs_conf
    root@localhost:/fdfs_conf#  fdfs_upload_file storage.conf /a.txt
    group1/M00/00/00/wKiuglwGoJWARxYvAAAACZLaVGU072.txt  //上传成功 返回的地址
    //组名/地址/保证安全替换后的名字.后缀名
    -v 映射路径

下载文件到本地:（fdfs_download_file <config_file> <file_id> [local_filename]）
    	[root@bogon 00]# fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKg6klmyEG-AWCZjAAAADZDfcRo628.txt ~/bb.txt
    	[root@bogon 00]# ll ~
    	total 168836
    	-rw-------. 1 root root      2913 Apr  5  2016 anaconda-ks.cfg
    	-rw-r--r--  1 root root        13 Sep  7 20:35 a.txt
    	-rw-r--r--  1 root root        13 Sep  7 20:46 bb.tx

**删除文件（fdfs\_delete\_file <config\_file> <file\_id>）**

[root@bogon 00]# fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKg6klmyEG-AWCZjAAAADZDfcRo628.txt
        #删除了该文件 还可以查看到它的信息
        [root@bogon 00]# fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/wKg6klmyEG-AWCZjAAAADZDfcRo628.txt
        source storage id: 0
        source ip address: 192.168.58.146
        file create timestamp: 2017-09-07 20:37:19
        file size: 13
        file crc32: 2430562586 (0x90DF711A)
        #再次尝试删除 发现报错文件不存在
        [root@bogon 00]# fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKg6klmyEG-AWCZjAAAADZDfcRo628.txt
        [2017-09-07 20:48:23] ERROR - file: tracker_proto.c, line: 48, server: 192.168.58.146:23000, response status 2 != 0
        delete file fail, error no: 2, error info: No such file or directory

**上传下载代码：**  
**pom.xml文件：**

<repositories>
            <repository>
                <id>cn</id>
                <name>sn</name>
                <url>https://oss.sonatype.org/content/repositories/releases/</url>
            </repository>
        </repositories>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-web</artifactId>
            </dependency>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-data-jpa</artifactId>
            </dependency>
             <dependency>
                <groupId>com.luhuiguo</groupId>
                <artifactId>fastdfs-spring-boot-starter</artifactId>
                <version>0.2.0</version>
            </dependency>
            <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
            </dependency>
        </dependencies>

**controller层：**

@RestController
    public class UploadController {
        @Autowired
        private FastFileStorageClient storageClient;
        @Autowired
        JdbcTemplate jdbcTemplate;
        /**
         *
         * @param myFile 从浏览器提交过来
         * @return
         */
        @PostMapping("/fupload")
        public String uploud(@RequestParam("myFile") MultipartFile  myFile) throws IOException {
    
            String extName = FilenameUtils.getExtension(myFile.getOriginalFilename());
            StorePath sp=storageClient.uploadFile("group1",myFile.getInputStream(),myFile.getSize(),extName);
    
            String sql ="insert into myfile(filename,groupname,filepath) values(?,?,?)";
    
            jdbcTemplate.update(sql,myFile.getOriginalFilename(),sp.getGroup(),sp.getPath());
            return  sp.getFullPath();
    
        }
        
        @GetMapping("/fdownload/{id}")
        public void download(@PathVariable  String id , HttpServletResponse response) throws IOException {
    
            List list = jdbcTemplate.query("select  * from myfile where fileid ="+id,new ColumnMapRowMapper());
            Map map=(Map)list.get(0);
            
            //中文文件名要转码
            String fileName = URLEncoder.encode(map.get("filename").toString(),"UTF-8");
            String groupName= map.get("groupName").toString();
            String pathName = map.get("filePath").toString();
    
            //告诉浏览器下载的文件名
            response.setHeader("Content-Disposition","attachment;filename="+ fileName+"");
    
            //将文件内容输出到浏览器
            byte[] bytes = storageClient.downloadFile(groupName, pathName);
            response.getOutputStream().write(bytes);
    
        }
    }

**html页面：**  
上传：

<!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Title</title>
    </head>
    <body>
        <form method="post" enctype="multipart/form-data" action="fupload">
            文件<input type="file" name="myFile">
            <input type="submit" value="上传"/>
        </form>
    
    </body>
    </html>

下载：

<!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Title</title>
    </head>
    <body>
      <a href="fdownload/1">下载</a>
    </body>
    </html>

**配置文件application.yml：**

#window端口：
    server:
      port: 8899
    
    fdfs:
       #连接超时时间（毫秒）：
      connect-timeout: 10000
      
        #响应超时时间（毫秒）：
      so-timeout: 3000
      
      #Linux系统Docker地址：
      tracker-list:
        - 192.168.85.128:22122
    
    spring:
      datasource:
    	#连接数据库：
          url: jdbc:mysql://localhost/factory
          password: ps123456
          driver-class-name: com.mysql.jdbc.Driver
          username: root
    
      http:
         # 设置下载文件大小，单位byte
        multipart:
          max-file-size: 10485760

**返回数组 返回两个参数 第一个是存储的组名 第二是 文件的路径**

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70]: /images/20220411/ca23670affcf4bdebdbb127c8762c741.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70 1]: /images/20220411/6461ba93aa384ddbac92712a5f377f6a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70 2]: /images/20220411/39e0dfef59f1474cbd37e1af4212c0c5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjI0ODc2NA_size_16_color_FFFFFF_t_70 3]: /images/20220411/0fc3d9c72e7947628a758260162eed73.png