FastDFS分布式文件系统集群搭建

川长思鸟来 2022-02-02 05:43 447阅读 0赞

架构图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5NjY5MDU4_size_16_color_FFFFFF_t_70][]

**后面有脚本一键安装。。。。**

**第一步**：安装6个迷你版的Linux

**第二步**：由于迷你版Linux缺少一些常用的工具库，操作起来不方便，推荐安装如下的工具库：  
1、安装lrzsz， yum install lrzsz -y  
2、安装wget, yum install wget -y  
4、安装vim， yum install vim -y  
5、安装unzip，yum install unzip -y  
6、安装ifconfig，yum install net-tools -y

yum install lrzsz wget vim unzip net-tools -y

7、安装nginx及fastdfs需要的库依赖：

yum install gcc perl openssl openssl-devel pcre pcre-devel zlib zlib-devel libevent libevent-devel -y

**第三步**：在6个服务器节点上分别安装FastDFS，参考：[FastDFS分布式文件系统介绍及单机版安装][FastDFS]

**第四步**：在第1、2（133,135）这两台服务器节点配置tracker，并且启动  
配置tracker请修改tracker.conf文件，修改内容如下：

mkdir -p /opt/fastdfs/tracker
    cd /etc/fdfs
    cp tracker.conf.sample tracker.conf
    vim /etc/fdfs/tracker.conf
    base_path=/opt/fastdfs/tracker
    #测试一下tracker的负载均衡上传文件：修改tracker.conf文件如下的内容即可,该配置默认就为2，不用修改
    # the method of selecting group to upload files
    # 0: round robin
    # 1: specify group
    # 2: load balance, select the max free space group to upload file
    # 指定负载均衡类型
    store_lookup=2

注意要保证配置的文件路径要存在

**第五步**：在第3、4、5、6(136,137,138,139)这四台服务器节点上配置storage，并且启动  
配置storage请修改storage.conf文件，修改内容如下：

mkdir -p /opt/fastdfs/storage
    mkdir -p /opt/fastdfs/storage/files
    cd /etc/fdfs
    cp storage.conf.sample storage.conf
    vim storage.conf
    group_name=group1 #组名，根据实际情况修改(实际情况为：136,137的group_name为group1,138,139的group_name为group2)，值为 group1、group2、group3......
    base_path=/opt/fastdfs/storage
    store_path0=/opt/fastdfs/storage/files
    #两个tracke_server的地址端口
    tracker_server=192.168.184.133:22122
    tracker_server=192.168.117.135:22122

将指定的目录要创建好，然后就可以启动storage服务了。

至此一个FastDFS本身的集群就搭建好了，可以正常使用了。

> 注意：为了让storage服务能正常连接tracker，请关闭所有机器的防火墙：  
> systemctl status firewalld  
> systemctl disable firewalld  
> systemctl stop firewalld

启动133，135服务器上的tracker

fdfs_trackerd /etc/fdfs/tracker.conf

启动1136，137,138,139服务器上的storage

fdfs_storaged /etc/fdfs/storage.conf

在storage服务器上执行：

fdfs_monitor /etc/fdfs/storage.conf

状态都为ACTIV表示集群搭建成功，此处日志为只粘贴了136服务器的日志

[root@localhost opt]# fdfs_monitor /etc/fdfs/storage.conf
    [2019-05-13 15:26:00] DEBUG - base_path=/opt/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
    
    server_count=2, server_index=1
    
    tracker server is 192.168.184.135:22122
    
    group count: 2
    
    Group 1:
    group name = group1
    disk total space = 17394 MB
    disk free space = 13133 MB
    trunk free space = 0 MB
    storage server count = 2
    active server count = 2
    storage server port = 23000
    storage HTTP port = 8888
    store path count = 1
    subdir count per path = 256
    current write server index = 0
    current trunk file id = 0
    
    	Storage 1:
    		id = 192.168.184.136
    		ip_addr = 192.168.184.136 (localhost.localdomain)  ACTIVE
    		http domain = 
    		version = 5.11
    		join time = 2019-05-13 15:18:48
    		up time = 2019-05-13 15:18:48
    		total storage = 17394 MB
    		free storage = 13133 MB
    		upload priority = 10
    		store_path_count = 1
    		subdir_count_per_path = 256
    		storage_port = 23000
    		storage_http_port = 8888
    		current_write_path = 0
    		source storage id = 192.168.184.137
    		if_trunk_server = 0
    		connection.alloc_count = 256
    		connection.current_count = 1
    		connection.max_count = 1
    		total_upload_count = 0
    		success_upload_count = 0
    		total_append_count = 0
    		success_append_count = 0
    		total_modify_count = 0
    		success_modify_count = 0
    		total_truncate_count = 0
    		success_truncate_count = 0
    		total_set_meta_count = 0
    		success_set_meta_count = 0
    		total_delete_count = 0
    		success_delete_count = 0
    		total_download_count = 0
    		success_download_count = 0
    		total_get_meta_count = 0
    		success_get_meta_count = 0
    		total_create_link_count = 0
    		success_create_link_count = 0
    		total_delete_link_count = 0
    		success_delete_link_count = 0
    		total_upload_bytes = 0
    		success_upload_bytes = 0
    		total_append_bytes = 0
    		success_append_bytes = 0
    		total_modify_bytes = 0
    		success_modify_bytes = 0
    		stotal_download_bytes = 0
    		success_download_bytes = 0
    		total_sync_in_bytes = 0
    		success_sync_in_bytes = 0
    		total_sync_out_bytes = 0
    		success_sync_out_bytes = 0
    		total_file_open_count = 0
    		success_file_open_count = 0
    		total_file_read_count = 0
    		success_file_read_count = 0
    		total_file_write_count = 0
    		success_file_write_count = 0
    		last_heart_beat_time = 2019-05-13 15:26:30
    		last_source_update = 1970-01-01 08:00:00
    		last_sync_update = 1970-01-01 08:00:00
    		last_synced_timestamp = 1970-01-01 08:00:00 
    	Storage 2:
    		id = 192.168.184.137
    		ip_addr = 192.168.184.137  ACTIVE
    		http domain = 
    		version = 5.11
    		join time = 2019-05-13 15:19:51
    		up time = 2019-05-13 15:25:00
    		total storage = 17394 MB
    		free storage = 16039 MB
    		upload priority = 10
    		store_path_count = 1
    		subdir_count_per_path = 256
    		storage_port = 23000
    		storage_http_port = 8888
    		current_write_path = 0
    		source storage id = 
    		if_trunk_server = 0
    		connection.alloc_count = 256
    		connection.current_count = 1
    		connection.max_count = 1
    		total_upload_count = 0
    		success_upload_count = 0
    		total_append_count = 0
    		success_append_count = 0
    		total_modify_count = 0
    		success_modify_count = 0
    		total_truncate_count = 0
    		success_truncate_count = 0
    		total_set_meta_count = 0
    		success_set_meta_count = 0
    		total_delete_count = 0
    		success_delete_count = 0
    		total_download_count = 0
    		success_download_count = 0
    		total_get_meta_count = 0
    		success_get_meta_count = 0
    		total_create_link_count = 0
    		success_create_link_count = 0
    		total_delete_link_count = 0
    		success_delete_link_count = 0
    		total_upload_bytes = 0
    		success_upload_bytes = 0
    		total_append_bytes = 0
    		success_append_bytes = 0
    		total_modify_bytes = 0
    		success_modify_bytes = 0
    		stotal_download_bytes = 0
    		success_download_bytes = 0
    		total_sync_in_bytes = 0
    		success_sync_in_bytes = 0
    		total_sync_out_bytes = 0
    		success_sync_out_bytes = 0
    		total_file_open_count = 0
    		success_file_open_count = 0
    		total_file_read_count = 0
    		success_file_read_count = 0
    		total_file_write_count = 0
    		success_file_write_count = 0
    		last_heart_beat_time = 2019-05-13 15:26:29
    		last_source_update = 1970-01-01 08:00:00
    		last_sync_update = 1970-01-01 08:00:00
    		last_synced_timestamp = 1970-01-01 08:00:00 
    
    Group 2:
    group name = group2
    disk total space = 17394 MB
    disk free space = 16039 MB
    trunk free space = 0 MB
    storage server count = 2
    active server count = 2
    storage server port = 23000
    storage HTTP port = 8888
    store path count = 1
    subdir count per path = 256
    current write server index = 0
    current trunk file id = 0
    
    	Storage 1:
    		id = 192.168.184.138
    		ip_addr = 192.168.184.138  ACTIVE
    		http domain = 
    		version = 5.11
    		join time = 2019-05-13 15:19:52
    		up time = 2019-05-13 15:19:52
    		total storage = 17394 MB
    		free storage = 16039 MB
    		upload priority = 10
    		store_path_count = 1
    		subdir_count_per_path = 256
    		storage_port = 23000
    		storage_http_port = 8888
    		current_write_path = 0
    		source storage id = 192.168.184.139
    		if_trunk_server = 0
    		connection.alloc_count = 256
    		connection.current_count = 1
    		connection.max_count = 1
    		total_upload_count = 0
    		success_upload_count = 0
    		total_append_count = 0
    		success_append_count = 0
    		total_modify_count = 0
    		success_modify_count = 0
    		total_truncate_count = 0
    		success_truncate_count = 0
    		total_set_meta_count = 0
    		success_set_meta_count = 0
    		total_delete_count = 0
    		success_delete_count = 0
    		total_download_count = 0
    		success_download_count = 0
    		total_get_meta_count = 0
    		success_get_meta_count = 0
    		total_create_link_count = 0
    		success_create_link_count = 0
    		total_delete_link_count = 0
    		success_delete_link_count = 0
    		total_upload_bytes = 0
    		success_upload_bytes = 0
    		total_append_bytes = 0
    		success_append_bytes = 0
    		total_modify_bytes = 0
    		success_modify_bytes = 0
    		stotal_download_bytes = 0
    		success_download_bytes = 0
    		total_sync_in_bytes = 0
    		success_sync_in_bytes = 0
    		total_sync_out_bytes = 0
    		success_sync_out_bytes = 0
    		total_file_open_count = 0
    		success_file_open_count = 0
    		total_file_read_count = 0
    		success_file_read_count = 0
    		total_file_write_count = 0
    		success_file_write_count = 0
    		last_heart_beat_time = 2019-05-13 15:26:30
    		last_source_update = 1970-01-01 08:00:00
    		last_sync_update = 1970-01-01 08:00:00
    		last_synced_timestamp = 1970-01-01 08:00:00 
    	Storage 2:
    		id = 192.168.184.139
    		ip_addr = 192.168.184.139  ACTIVE
    		http domain = 
    		version = 5.11
    		join time = 2019-05-13 15:19:54
    		up time = 2019-05-13 15:25:43
    		total storage = 17394 MB
    		free storage = 16039 MB
    		upload priority = 10
    		store_path_count = 1
    		subdir_count_per_path = 256
    		storage_port = 23000
    		storage_http_port = 8888
    		current_write_path = 0
    		source storage id = 
    		if_trunk_server = 0
    		connection.alloc_count = 256
    		connection.current_count = 1
    		connection.max_count = 1
    		total_upload_count = 0
    		success_upload_count = 0
    		total_append_count = 0
    		success_append_count = 0
    		total_modify_count = 0
    		success_modify_count = 0
    		total_truncate_count = 0
    		success_truncate_count = 0
    		total_set_meta_count = 0
    		success_set_meta_count = 0
    		total_delete_count = 0
    		success_delete_count = 0
    		total_download_count = 0
    		success_download_count = 0
    		total_get_meta_count = 0
    		success_get_meta_count = 0
    		total_create_link_count = 0
    		success_create_link_count = 0
    		total_delete_link_count = 0
    		success_delete_link_count = 0
    		total_upload_bytes = 0
    		success_upload_bytes = 0
    		total_append_bytes = 0
    		success_append_bytes = 0
    		total_modify_bytes = 0
    		success_modify_bytes = 0
    		stotal_download_bytes = 0
    		success_download_bytes = 0
    		total_sync_in_bytes = 0
    		success_sync_in_bytes = 0
    		total_sync_out_bytes = 0
    		success_sync_out_bytes = 0
    		total_file_open_count = 0
    		success_file_open_count = 0
    		total_file_read_count = 0
    		success_file_read_count = 0
    		total_file_write_count = 0
    		success_file_write_count = 0
    		last_heart_beat_time = 2019-05-13 15:26:41
    		last_source_update = 1970-01-01 08:00:00
    		last_sync_update = 1970-01-01 08:00:00
    		last_synced_timestamp = 1970-01-01 08:00:00 
    [root@localhost opt]#

测试之前，需要修改client.conf配置文件，修改两个配置：

base_path=/opt/fastdfs/client
    tracker_server=192.168.184.133:22122
    tracker_server=192.168.184.135:22122
    新建文件夹：
    mkdir -p /opt/fastdfs/client

测试：

fdfs_test /etc/fdfs/client.conf upload /root/testfastfds.txt

显示：

[root@localhost fdfs]# fdfs_test /etc/fdfs/client.conf upload /root/testfastfds.txt
    This is FastDFS client test program v5.11
    
    Copyright (C) 2008, Happy Fish / YuQing
    
    FastDFS may be copied only under the terms of the GNU General
    Public License V3, which may be found in the FastDFS source kit.
    Please visit the FastDFS Home Page http://www.csource.org/ 
    for more detail.
    
    [2019-05-13 15:34:10] DEBUG - base_path=/opt/fastdfs/client, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
    
    tracker_query_storage_store_list_without_group: 
    	server 1. group_name=, ip_addr=192.168.184.138, port=23000
    	server 2. group_name=, ip_addr=192.168.184.139, port=23000
    
    group_name=group2, ip_addr=192.168.184.138, port=23000
    storage_upload_by_filename
    group_name=group2, remote_filename=M00/00/00/wKi4ilzZHfKAQbXhAAAAIIe1k0c677.txt
    source ip address: 192.168.184.138
    file timestamp=2019-05-13 15:34:10
    file size=32
    file crc32=2276823879
    example file url: http://192.168.184.138/group2/M00/00/00/wKi4ilzZHfKAQbXhAAAAIIe1k0c677.txt
    storage_upload_slave_by_filename
    group_name=group2, remote_filename=M00/00/00/wKi4ilzZHfKAQbXhAAAAIIe1k0c677_big.txt
    source ip address: 192.168.184.138
    file timestamp=2019-05-13 15:34:10
    file size=32
    file crc32=2276823879
    example file url: http://192.168.184.138/group2/M00/00/00/wKi4ilzZHfKAQbXhAAAAIIe1k0c677_big.txt

去group2（138,139）里面看到都存储了相同的内容，上传成功

![20190513154148282.png][]

![20190513154213502.png][]

测试结果：把group2的storage的所有节点都停了，再次上传文件则会负载到group1中额storage节点中，如果重启group2中的138节点，则会再次向group2节点中的138节点上传文件，这时候启动group2中的139节点，则会同步group2中的138节点中的文件。

到此FastDFS集群搭建完毕

脚本：只是安装，配置文件自行修改，在任意目录下执行该文件，sh installFastdfs.sh所需要的安装包在/usr/home/soft/fastdfs，安装路径在/usr/local/fastdfs

vim installFastdfs.sh

#安装一些工具库，方便操作
    yum install lrzsz wget vim unzip net-tools -y
    #安装nginx及fastdfs需要的库依赖
    yum install gcc perl openssl openssl-devel pcre pcre-devel zlib zlib-devel libevent libevent-devel -y
    mkdir -p /home/soft/fastdfs
    cd /home/soft/fastdfs
    wget https://github.com/happyfish100/libfastcommon/archive/V1.0.36.tar.gz -O libfastcommon-1.0.36.tar.gz
    wget https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz -O fastdfs-5.11.tar.gz
    mkdir -p /usr/local/fastdfs
    tar -zxvf libfastcommon-1.0.36.tar.gz -C /usr/local/fastdfs/
    cd /usr/local/fastdfs/libfastcommon-1.0.36/
    ./make.sh
    ./make.sh install
    #安装FastDFS
    cd /home/soft/fastdfs
    tar -zxvf fastdfs-5.11.tar.gz -C /usr/local/fastdfs/
    cd /usr/local/fastdfs/fastdfs-5.11/
    ./make.sh
    ./make.sh install
    #新建fastdfs配置文件中所需的路径
    mkdir -p /opt/fastdfs/tracker
    mkdir -p /opt/fastdfs/storage
    mkdir -p /opt/fastdfs/storage/files
    mkdir -p /opt/fastdfs/client

//一道亮丽的风景线//

接下搭建第二个维度的集群（通过Http访问FastDFS上的文件）

**第一步**：在第3、4、5、6（136,137,138,139）这四台服务器节点上安装带有FastDFS扩展模块的Nginx

参考：[FastDFS分布式文件系统介绍及单机版安装][FastDFS]：**十二、安装Nginx的FastDFS扩展模块**

第二步：配置第3、4、5、6（136,137,138,139）这四台服务器节点上的Nginx

vim /usr/local/fastdfs/nginx_fdfs/conf/nginx.conf
    #nginx拦截请求路径：
    location ~ /group[1-9]/M0[0-9] {   
        ngx_fastdfs_module; 
    }

ngx\_fastdfs\_module;指定依赖一个配置文件，所以需要配置mod\_fastdfs.conf，需要配置和修改的内容如下：

cp /home/soft/fastdfs/fastdfs-nginx-module-73869650b6171fd6e5bf81b5b8caa89f021d6c7b/src/mod_fastdfs.conf  /etc/fdfs/
    vim /etc/fdfs/mod_fastdfs.conf 
    
    base_path=/opt/fastdfs/mod_fastdfs #保存日志目录
    tracker_server=192.168.184.133:22122  #tracker服务器的IP地址以及端口号
    tracker_server=192.168.184.135:22122
    group_name=group1  #当前服务器的group名
    url_have_group_name=true     #文件url中是否有group名
    store_path0=/opt/fastdfs/storage/files    #存储路径
    group_count = 2                   #设置组的个数
    
    在末尾增加2个组的具体信息：
    [group1]
    group_name=group1
    storage_server_port=23000
    store_path_count=1
    store_path0=/opt/fastdfs/storage/files
    
    [group2]
    group_name=group2
    storage_server_port=23000
    store_path_count=1
    store_path0=/opt/fastdfs/storage/files

以上配置注意：文件路径要存在，所以请先创建好路径，否则会报错；测试验证第3、4、5、6这四台服务器节点上的Nginx，通过Nginx访问已经上传到FastDFS上的文件；  
启动4台服务器上的Nginx：  
通过浏览器访问：

http://192.168.184.136/group1/M00/00/00/wKi4iVzZIXmALSF\_AAAAJrfiD9E295.txt

http://192.168.184.137/group1/M00/00/00/wKi4iVzZIXmALSF\_AAAAJrfiD9E295.txt

http://192.168.184.138/group2/M00/00/00/wKi4ilzZIZ6AZ90CAAAAJrfiD9E276.txt

http://192.168.184.139/group2/M00/00/00/wKi4ilzZIZ6AZ90CAAAAJrfiD9E276.txt

这里只列出一个storage服务的访问。

![201905131737129.png][]

第三步：部署两个tracker server服务器上的Nginx，需要做的工作，**这个Nginx作用不是集成fastDFS模块，只是负载转发**

部署配置nginx负载均衡:
    upstream fastdfs_group_server {
    	server 192.168.184.136:80;
    	server 192.168.184.137:80;
    	server 192.168.184.138:80;
    	server 192.168.184.139:80;
       }
    #nginx拦截请求路径：
    location ~ /group[1-9]/M0[0-9] {   
    		proxy_pass http://fastdfs_group_server;
    	}

启动这两个Nginx，然后测试访问：  
http://192.168.184.133/group1/M00/00/00/wKi4iVzZIXmALSF\_AAAAJrfiD9E295.txt  
http://192.168.184.135/group2/M00/00/00/wKi4ilzZIZ6AZ90CAAAAJrfiD9E276.txt

![20190513191030943.png][]

![2019051319063027.png][]

第四步：部署前端用户访问入口Nginx服务器，即访问本地windows上的Nginx，该Nginx负载均衡到后端2个tracker server上的Nginx；

部署配置访问入口nginx负载均衡:
    upstream fastdfs_access_server {  
        server 192.168.184.133:80;  
        server 192.168.117.135:80;  
    }
    
    #nginx拦截请求路径：
    location ~ /group[1-9]/M0[0-9] {   
        proxy_pass http://fastdfs_group_server; 
    }

启动window上的nginx：  
测试访问：  
http://localhost/group1/M00/00/00/wKi4iVzZIXmALSF\_AAAAJrfiD9E295.txt  
http://localhost/group2/M00/00/00/wKi4ilzZIZ6AZ90CAAAAJrfiD9E276.txt

![2019051319381956.png][]

![20190513193829936.png][]

**到此FastDFS集群搭建完毕**

脚本：vim installFastdfsNginxMod.sh

cd /home/soft/fastdfs/
    wget https://github.com/happyfish100/fastdfs-nginx-module/archive/73869650b6171fd6e5bf81b5b8caa89f021d6c7b.zip -O fastdfs-nginx-module.zip
    wget http://nginx.org/download/nginx-1.12.2.tar.gz
    unzip fastdfs-nginx-module.zip
    tar -zxvf nginx-1.12.2.tar.gz
    cd nginx-1.12.2
    ./configure --prefix=/usr/local/fastdfs/nginx_fdfs --add-module=/home/soft/fastdfs/fastdfs-nginx-module-73869650b6171fd6e5bf81b5b8caa89f021d6c7b/src
    make && make install
    cp /home/soft/fastdfs/fastdfs-nginx-module-73869650b6171fd6e5bf81b5b8caa89f021d6c7b/src/mod_fastdfs.conf  /etc/fdfs/
    mkdir -p /opt/fastdfs/nginx_mod
    cd /usr/local/fastdfs/fastdfs-5.11/conf
    cp http.conf /etc/fdfs/
    cp mime.types /etc/fdfs/

**有了这两个脚本安装FastDFS就如此简单了。。。**

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5NjY5MDU4_size_16_color_FFFFFF_t_70]: /images/20220202/e8d0477c635c47f992ad903e1ed235bc.png
[FastDFS]: https://blog.csdn.net/qq_39669058/article/details/90052402
[20190513154148282.png]: /images/20220202/e67c7fa29f49451e9f59c3286e77d95c.png
[20190513154213502.png]: /images/20220202/d471b5982cc4431089595377b21fe132.png
[201905131737129.png]: /images/20220202/83b50f395b4248a79570fb5ea320ac0b.png
[20190513191030943.png]: /images/20220202/72815029df8748a193614da2b64e6870.png
[2019051319063027.png]: /images/20220202/3d3a344113b74c0db45b448cca97b5b2.png
[2019051319381956.png]: /images/20220202/fcf51305b3ff4709b0fd4c6edcaaa077.png
[20190513193829936.png]: /images/20220202/a25963768b494bc4b788301d1f2a69dd.png