【FastDFS】分布式文件系统FastDFS之FastDHT文件去重

电玩女神 2023-10-05 14:44 56阅读 0赞

#### FastDHT去重 ####

*   *  概述
     *  FastDHT安装
     *  操作FastDHT服务
     *  Nginx集成FastDHT

### 概述 ###

FastDFS除了提供了与Nginx的集成，已提供了去重的文件解决方案。该解决方案FastDFS的作者**yuqing**也在github上以FastDHT分支贡献出来了。

> FastDHT is a high performance distributed hash table (DHT) which based key value pairs. It can store mass key value pairs such as filename mapping, session data and user related data.  
> FastDHT是基于键值对的高性能分布式哈希表（DHT）。它可以存储大量键值对，例如文件名映射，会话数据和与用户相关的数据。  
> ![在这里插入图片描述][2019071401062538.png]

### FastDHT安装 ###

由于github上下载来的安装包文件在安装时或多或少会出现bug，此处建议使用小编给大家处理过的安装包进行安装，以下所有安装包获取方式：

> **百度网盘：**  
> 链接：https://pan.baidu.com/s/1rnwVzlJDA0ykxBzbSOsDWQ  
> 提取码：thmn

> **蓝奏云：**  
> https://wws.lanzous.com/b01huf3ah  
> 密码:thmn

1.  安装BerkeleyDB 下载`db-4.7.25.tar.gz`

# 解压
    [root@CentOSX ~]# tar -zxf db-4.7.25.tar.gz
    [root@CentOSX ~]# cd db-4.7.25
    [root@CentOSX db-4.7.25]# cd build_unix/
    # 编译安装
    [root@CentOSX build_unix]# ./../dist/configure
    [root@CentOSX build_unix]# make
    [root@CentOSX build_unix]# make install

1.  安装FastDHT

[root@CentOSX ~]# tar zxf FastDHT_v2.01.tar.gz
    [root@CentOSX ~]# cd FastDHT
    [root@CentOSX FastDHT]# ./make.sh
    [root@CentOSX FastDHT]# ./make.sh install

> 安装结束后会在`/etc`目录下产生`/fdht`文件夹

[root@CentOSX FastDHT]# tree /etc/fdht/
    /etc/fdht/
    ├── fdht_client.conf
    ├── fdhtd.conf
    └── fdht_servers.conf

1.  修改`fdhtd.conf`

[root@CentOSX ~]# mkdir /data/fastdht
    [root@CentOSX ~]# vi /etc/fdht/fdhtd.conf
    base_path=/data/fastdht

1.  修改`fdht_servers.conf`

[root@CentOSX ~]# vi /etc/fdht/fdht_servers.conf
    group_count = 3
    group0 = CentOSA:11411
    group1 = CentOSB:11411
    group2 = CentOSC:11411

1.  修改`fdht_client.conf`配置文件

[root@CentOSX ~]# vi /etc/fdht/fdht_client.conf
    base_path=/tmp/

1.  启动FDHT服务

[root@CentOSX ~]# fdhtd /etc/fdht/fdhtd.conf start | stop | restart
    [root@CentOSX ~]# ps -axu| grep fd
    Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
    root      29127  0.0  0.8 195200 17504 ?        Sl   00:38   0:00 fdhtd /etc/fdht/fdhtd.conf start
    root      29193  0.0  0.0 103320   884 pts/0    S+   00:39   0:00 grep fd
    root     128672  0.2  0.1 275856  2204 ?        Sl   00:14   0:03 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
    root     128710  1.8  3.2  85584 67188 ?        Sl   00:14   0:27 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf

### 操作FastDHT服务 ###

*  设置值

[root@CentOSX ~]# fdht_set /etc/fdht/fdht_client.conf jiangzz:user001 name='jiangzz',age=18;
    This is FastDHT client test program v2.01
    
    Copyright (C) 2008, Happy Fish / YuQing
    
    FastDHT may be copied only under the terms of the GNU General
    Public License V3, which may be found in the FastDHT source kit.
    Please visit the FastDHT Home Page http://www.csource.org/
    for more detail.
    
    success set key count: 2, fail count: 0

*  读取值

[root@CentOSX ~]# fdht_get /etc/fdht/fdht_client.conf jiangzz:user001 name,age
    This is FastDHT client test program v2.01
    
    Copyright (C) 2008, Happy Fish / YuQing
    
    FastDHT may be copied only under the terms of the GNU General
    Public License V3, which may be found in the FastDHT source kit.
    Please visit the FastDHT Home Page http://www.csource.org/
    for more detail.
    
    name=jiangzz
    age=18
    
    success get key count: 2, fail count: 0

*  删除值

[root@CentOSX ~]# fdht_delete /etc/fdht/fdht_client.conf jiangzz:user001 name;
    This is FastDHT client test program v2.01
    
    Copyright (C) 2008, Happy Fish / YuQing
    
    FastDHT may be copied only under the terms of the GNU General
    Public License V3, which may be found in the FastDHT source kit.
    Please visit the FastDHT Home Page http://www.csource.org/
    for more detail.
    
    success delete keys: name
    
    success delete key count: 1, fail count: 0

### Nginx集成FastDHT ###

1.  修改`/etc/fdfs/storage.conf`配置文件

[root@CentOSX ~]# vi /etc/fdfs/storage.conf

check_file_duplicate=1
    keep_alive=1
    
    # 下面这句属于C语言的导包语句，必须加进去，否则报错
    #include /etc/fdht/fdht_servers.conf

1.  分别启动`fdhtd`服务、`fastfs`服务

[root@CentOSX usr]# /usr/local/bin/fdhtd /etc/fdht/fdhtd.conf restart
    [root@CentOSX usr]# /etc/init.d/fdfs_trackerd restart
    [root@CentOSX usr]# /etc/init.d/fdfs_storaged restart

*  上传文件测试

[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
    group2/M00/00/00/wKikgl0qC4KAErBTAAAixXWAIyY133.log
    [root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
    group2/M00/00/00/wKikgl0qDAqAa0XwAAAixWB5m1c851.log
    
    
    [root@CentOSB ~]# ls -l /data/fdfs/storage/store01/data/00/00/
    total 20
    lrwxrwxrwx. 1 root root   72 Jul 14 00:49 wKikgl0qC4KAErBTAAAixXWAIyY133.log -> /data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log
    -rw-r--r--. 1 root root 8901 Jul 14 00:49 wKikgl0qC4KARrYBAAAixZ60QfI755.log
    lrwxrwxrwx. 1 root root   72 Jul 14 00:51 wKikgl0qDAqAa0XwAAAixWB5m1c851.log -> /data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log

> 可以看出系统产生了`wKikgl0qC4KAErBTAAAixXWAIyY133.log`的两个链接

[2019071401062538.png]: https://img-blog.csdnimg.cn/2019071401062538.png