搭建分散式檔案系統FastDFS叢集
FastDFS是為網際網路應用量身定做的一套分散式檔案儲存系統，非常適合用來儲存使用者圖片、視訊、文件等檔案。對於網際網路應用，和其他分散式檔案系統相比，優勢非常明顯。出於簡潔考慮，FastDFS沒有對檔案做分塊儲存，因此不太適合分散式計算場景。
在生產環境中往往資料儲存量比較大，因此會大部分會選擇分散式儲存來解決，主要解決以下幾個問題
- 海量資料儲存
- 資料高可用(冗餘備份)
- 較高讀寫效能和負載均衡
- 支援多平臺多語言
- 高併發問題
常見分散式儲存對比
FastDFS 相關元件及原理
FastDFS介紹
FastDFS是一個C語言實現的開源輕量級分散式檔案系統，支援Linux、FreeBSD、AID等Linux系統，解決了大量資料儲存和讀寫負載等問題，適合儲存4KB~500MB之間的小檔案，如圖片網站，短視訊網站，文件，APP下載站等，UC，京東，支付寶，迅雷，酷狗等都有使用，其中UC基於FastDFS向用戶提供網盤，廣告和應用下載的業務。FastDFS與MogileFS、HDFS、TFS等都不是系統級的分散式檔案系統，而是應用級的分散式檔案儲存服務
FastDFS架構
FastDFS服務有三個角色: 跟蹤服務(tracker server)、儲存服務(storage server)和客戶端(client)
tracker server:跟蹤服務，主要做排程工作，起到均衡的作用；負責管理所有的storage server和group，每個storage在啟動後會連線Tracker，告知自己所屬group等資訊，並保持週期性心跳，tracker根據storage心跳資訊，建立group --> [storage server list]的對映表；tracker管理的元資料很少，會直接存放在記憶體；tracker上的元資訊都是由storage彙報的資訊生成的，本身不需要持久化任何資料，tracker之間是對等關係，因此擴充套件tracker訪問非常容器，之間增加tracker訪問即可，所有tracker都接受storage心跳資訊，生成元資料資訊來提供讀寫訪問（與其他master-slave架構的優勢是沒有單點，tracker也不會成為瓶頸，最終資料是和一個可用的storage server進行傳輸）
storage server:儲存伺服器，主要提供容量和備份訪問；以group為單位，每個group內可以包含多個storage server，資料互為備份，儲存容量空間以group內容量最小的storage為準；建議group內的storage server配置相同；以group為單位組織儲存能夠方便的進行引用隔離、負載均衡和副本數定製；
缺點: group的容量受單機儲存容量的限制，同時group內機器壞掉，資料恢復只能依賴group內其他機器重新同步(硬碟替換，重新掛載重啟fdfs_storaged即可)
group儲存策略
- round robin (輪訓)
- load balance (選擇最大剩餘空間的組上傳檔案)
- specify group (指定group上傳)
group中storage儲存依賴本地檔案系統，storage可配置多個數據儲存目錄，磁碟不做raid，直接分別掛在到多個目錄，將這些目錄配置為storage的資料目錄即可
storage接受寫請求時，會根據配置好的規則，選擇其中一個儲存目錄來儲存檔案；為避免單個目錄下檔案過多，storage第一次啟動，會在每個資料儲存目錄裡建立2級子目錄，每級256個，總共65536個，新寫的檔案會以hash的方式被路由到其中某個子目錄下，然後將檔案資料直接作為一個本地檔案儲存到該目錄中
FastDFS工作流程
上傳
FastDFS提供基本的檔案訪問介面，如upload、download、append、delete等
選擇tracker server
叢集中tracker之間是對等關係，客戶端在上傳檔案時可用任意選擇一個tracker
選擇儲存group
當tracker接受到upload file的請求時，會為該檔案分配一個可以儲存的group，目前支援選擇group的規則為
1.Round Robin (所有group輪訓使用)
2.Specified group (指定某個確定的group)
3.Load balance (剩餘儲存空間較多的group優先)
選擇storage server
當選定group後，tracker會在group內選擇一個storage server給客戶端，目前支援選擇server的規則為
1.Round Robin (所有server輪訓使用)預設規則
2.根據IP地質進行排序選擇第一個伺服器 (IP地址最小者)
3.根據優先順序進行排序 (上傳優先順序由storage server來設定，引數為upload_priority)
選擇storage path (磁碟或者掛載點)
當分配好storage server後，客戶端將向storage傳送寫檔案請求，storage會將檔案分配一個數據儲存目錄，目前支援選擇儲存路徑的規則為:
1.round robin (輪訓)預設
2.load balance 選擇使用剩餘空間最大的儲存路徑
選擇下載伺服器
目前支援的規則為
1.輪訓方式，可以下載當前檔案的任一storage server
2.從源storage server下載
生成file_id
選擇儲存目錄後，storage會生成一個file_id，採用Base64編碼，包含欄位包括: storage server ip、檔案建立時間、檔案大小、檔案CRC32校驗碼和隨機數；每個儲存目錄下有兩個256*256個子目錄，storage會按檔案file_id進行兩次hash，路由到其中一個子目錄，然後將檔案file_id為檔名儲存在該子目錄下，最後生成檔案路徑: group名稱、虛擬磁碟路徑、資料兩級目錄、file_id
group1 /M00/02/44/wkgDRe348wAAAAGKYJK42378.sh
其中，組名: 上傳檔案後所在的儲存組的名稱，在檔案上傳成功後由儲存伺服器返回，需要客戶端自行儲存
虛擬磁碟路徑: 儲存伺服器配置的虛擬路徑，與磁碟選項store_path*引數對應
資料兩級目錄: 儲存伺服器在每個虛擬磁碟路徑下建立的兩級目錄，用於儲存資料檔案
同步機制
1.新增tracker伺服器資料同步
由於storage server上配置了所有的tracker server，storage server和tracker server之間的通訊是由storage server主動發起的，storage server為每臺tracker server 啟動一個執行緒進行通訊；在通訊過程中，若發現tracker server返回的本組storage server列表比本機記錄少，就會將該tracker server上沒有的storage server同步給該tracker，這樣的機制使得tracker之間是對等的關係，資料保持一致
2.組內新增storage資料同步
若新增storage server或者其狀態發生變化，tracker server都會將storage server列表同步給該組內所有storage server；以新增storage server為例，新加入storage server會主動連線tracker server，tracker server發現有新的storage server加入，就會將該組內所有的storage server返回給新加入的storage server，並重新將該組的storage server列表返回給該組內的其他storage server
3.組內storage資料同步
組內storage server之間是對等的，檔案上傳，刪除等操作可以在組內任意一臺storage server上進行。檔案同步只能在同組內的storage server之間進行，採用push方式，即源伺服器同步到目標伺服器
A. 只在同組內的storage server之間同步
B. 源資料才需要同步，備份資料不再同步
C. 特例: 新增storage server時，由其中一臺將已有的所有資料(包括源資料和備份資料)同步到新增伺服器
storage server 7種狀態
通過命令fdfs_monitor /etc/fdfs/client.conf可以檢視ip_addr選項顯示storage server當前狀態
INIT 初始化，尚未得到同步已有資料的源伺服器 WAIT_SYNC 等待同步，已得到同步已有資料的源伺服器 SYNCING 同步中 DELETE 已刪除，該伺服器從本組中摘除 OFFLINE 離線 ONLINE 線上，尚不能提供服務 ACTIVE 線上，可以提供服務
組內增加storage server A狀態變化過程:
1.storage server A主動連線tracker server，此時tracker server將storage serverA狀態設定為INIT
2.storage server A向tracker server詢問追加同步的源伺服器和追加同步截止時間點(當前時間)，若組內只有storage server A或者上傳檔案數為0，則告訴新主機不需要同步資料，storage serverA狀態為ONLINE；若組內沒有active狀態及其，就返回錯誤給新機器，新機器重新嘗試；否則tracker將其狀態設定為WAIT_SYNC
3.假如分配了storage server B為同步源伺服器和截止時間點，那麼storage serverB將會截止時間點之前的所有資料同步給storage server A，並請求tracker設定storage server A狀態為SYNCING；到了截止時間後，storage server B向storage server A的同步將由追加同步切換為正常binlog增量同步，當獲取不到更多binlog時，請求tracker將storage server A同步完所有資料，暫時沒有資料要同步時，storage server B請求tracker server將storage server A的狀態設定為ONLINE
4.storage server B向storage server A同步完所有資料，暫時沒有資料要同步時，storage server B請求tracker server將 storage server A的狀態設定為ONLINE
5.當storage server A向tracker server發起心跳時，tracker server將其狀態更改為ACTIVE，之後就是增量同步(binlog)
注:整個源同步班過程是源機器啟動弄一個同步執行緒，將資料Push到新機器，最大達到一個磁碟的IO，不能併發；由於源同步截止條件是獲取不到binlog，系統繁忙，不斷有新資料寫入的情況，將會導致一直無法完成源同步
下載
client傳送下載請求給某個tracker，必須帶上檔名資訊，tracker從檔名中解析出檔案的group、大小、建立時間等資訊，然後為該請求選擇一個storage用於讀請求；由於group內的檔案同步是非同步進行，可能出現檔案沒有同步到其他storage server上或者延遲的問題，可以使用nginx_fastdfs_module模組解決
關於檔案去重
由於FastDFS本身不能對重複上傳的檔案進行去重，而FastDHT可以做到去重。FastDHT是一個高效能的分散式雜湊系統，它是基於鍵值對儲存的，而且它需要依賴於Berkeley DB作為資料儲存的媒介，同時需要依賴於libfastcommon
由於業務需要，目前不存在檔案去重的時候，如果需要可以自己簡單瞭解一下
FastDHT
安裝FastDFS叢集
本次環境架構
針對tracker.conf && storage.conf && mod_fastdfs.conf有一篇單獨的文章介紹相關引數。有興趣的可以看一下，也可以直接看預設的配置檔案，對每個引數都有介紹
FastDFS 配置檔案詳解
網上大部分安裝FastDFS並沒有說清楚相關的配置檔案和引數，導致叢集無法啟動或者效能降低。 這篇文章主要整理一下以FastDFS 6.6版本為基礎的配置檔案引數，需要修改的地方已經提前說明清楚
環境說明
#nginx這裡可以部署2臺，加上keepliveed作高可用，由於我這裡機器不足，就使用單臺nginx進行代理 nginx 192.168.31.100 nginx tracker 節點 tracker 01:192.168.31.101 FastDFS,libfastcommon,nginx,ngx_cache_purge tracker 02:192.168.31.102 FastDFS,libfastcommon,nginx,ngx_cache_purge #其中tracker不提供儲存 Storage 節點 [group1] storage 01:192.168.31.103 FastDFS,libfastcommon,nginx,fastdfs-nginx-module storage 02:192.168.31.104 FastDFS,libfastcommon,nginx,fastdfs-nginx-module [group2] storage 03:192.168.31.105 FastDFS,libfastcommon,nginx,fastdfs-nginx-module storage 04:192.168.31.106 FastDFS,libfastcommon,nginx,fastdfs-nginx-module
1.所有的伺服器都需要安裝nginx，主要是用於訪問和上傳無關；
2.tracker安裝nginx主要為了提供http反向代理、負載均衡以及快取服務
3.每一臺storage伺服器部署Nginx及FastDFS擴充套件模組，主要用於對storage儲存的檔案提供http下載訪問，僅當前storage節點找不到檔案時會向源storage主機發送rediect或者proxy動作
所有節點安裝
關閉防火牆，selinux
systemctl stop firewalld systemctl disable firewalld iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat iptables -P FORWARD ACCEPT setenforce 0 sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
設定yum源
yum install -y wget wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo yum clean all yum makecache
溫馨提示：除了nginx節點(192.168.31.100)，其他節點都需要執行安裝fastdfs和nginx
安裝依賴包 (可解決99%的依賴問題)
yum -y install gcc gcc-c++ make autoconf libtool-ltdl-devel gd-devel freetype-devel libxml2-devel libjpeg-devel libpng-devel openssh-clients openssl-devel curl-devel bison patch libmcrypt-devel libmhash-devel ncurses-devel binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel glibc glibc-common glibc-devel libgcj libtiff pam-devel libicu libicu-devel gettext-devel libaio-devel libaio libgcc libstdc++ libstdc++-devel unixODBC unixODBC-devel numactl-devel glibc-headers sudo bzip2 mlocate flex lrzsz sysstat lsof setuptool system-config-network-tui system-config-firewall-tui ntsysv ntp pv lz4 dos2unix unix2dos rsync dstat iotop innotop mytop telnet iftop expect cmake nc gnuplot screen xorg-x11-utils xorg-x11-xinit rdate bc expat-devel compat-expat1 tcpdump sysstat man nmap curl lrzsz elinks finger bind-utils traceroute mtr ntpdate zip unzip vim wget net-tools
下載依賴包 （除了nginx節點，其他節點都要安裝）
mkdir /root/tools/ cd /root/tools wget http://nginx.org/download/nginx-1.18.0.tar.gz wget https://github.com/happyfish100/libfastcommon/archive/V1.0.43.tar.gz wget https://github.com/happyfish100/fastdfs/archive/V6.06.tar.gz wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.22.tar.gz #為了保證文章可用性，本次軟體包已經進行備份，下載地址如下 mkdir /root/tools/ cd /root/tools wget http://down.i4t.com/fdfs/v6.6/nginx-1.18.0.tar.gz wget http://down.i4t.com/fdfs/v6.6/V1.0.43.tar.gz wget http://down.i4t.com/fdfs/v6.6/V6.06.tar.gz wget http://down.i4t.com/fdfs/v6.6/V1.22.tar.gz #解壓 cd /root/tools tar xf nginx-1.18.0.tar.gz tar xf V1.0.43.tar.gz tar xf V1.22.tar.gz tar xf V6.06.tar.gz
安裝libfastcommon (除了nginx節點，其他節點都要安裝)
cd /root/tools/libfastcommon-1.0.43 ./make.sh ./make.sh install
安裝FastDFS (除了nginx節點，其他節點都要安裝)
cd /root/tools/fastdfs-6.06/ ./make.sh ./make.sh install
拷貝配置檔案 （tracker01 02節點）
[root@tracker01 fastdfs-6.06]# cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf #tracker節點 [root@01 fastdfs-6.06]# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf #客戶端檔案（測試使用） [root@01 fastdfs-6.06]# cp /root/tools/fastdfs-6.06/conf/http.conf /etc/fdfs/ #nginx配置檔案 [root@01 fastdfs-6.06]# cp /root/tools/fastdfs-6.06/conf/mime.types /etc/fdfs/ #nginx配置檔案
配置tracker 01節點
這裡可以先配置一臺節點，沒有問題在啟動另外的節點
建立tracker資料儲存及日誌目錄 （需要在tracker節點執行）
mkdir /data/tracker/ -p
修改配置檔案 （tracker 01節點執行）
cat >/etc/fdfs/tracker.conf <<EOF disabled = false bind_addr = port = 22122 connect_timeout = 5 network_timeout = 60 base_path = /data/tracker max_connections = 1024 accept_threads = 1 work_threads = 4 min_buff_size = 8KB max_buff_size = 128KB store_lookup = 0 store_server = 0 store_path = 0 download_server = 0 reserved_storage_space = 20% log_level = info run_by_group= run_by_user = allow_hosts = * sync_log_buff_interval = 1 check_active_interval = 120 thread_stack_size = 256KB storage_ip_changed_auto_adjust = true storage_sync_file_max_delay = 86400 storage_sync_file_max_time = 300 use_trunk_file = false slot_min_size = 256 slot_max_size = 1MB trunk_alloc_alignment_size = 256 trunk_free_space_merge = true delete_unused_trunk_files = false trunk_file_size = 64MB trunk_create_file_advance = false trunk_create_file_time_base = 02:00 trunk_create_file_interval = 86400 trunk_create_file_space_threshold = 20G trunk_init_check_occupying = false trunk_init_reload_from_binlog = false trunk_compress_binlog_min_interval = 86400 trunk_compress_binlog_interval = 86400 trunk_compress_binlog_time_base = 03:00 trunk_binlog_max_backups = 7 use_storage_id = false storage_ids_filename = storage_ids.conf id_type_in_filename = id store_slave_file_use_link = false rotate_error_log = false error_log_rotate_time = 00:00 compress_old_error_log = false compress_error_log_days_before = 7 rotate_error_log_size = 0 log_file_keep_days = 0 use_connection_pool = true connection_pool_max_idle_time = 3600 http.server_port = 8080 http.check_alive_interval = 30 http.check_alive_type = tcp http.check_alive_uri = /status.html EOF
啟動tracker
[root@01 fastdfs-6.06]# /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
配置tracker02節點
拷貝配置檔案
scp -r /etc/fdfs/tracker.conf [email protected]:/etc/fdfs/ ssh [email protected] mkdir /data/tracker/ -p
tracker02啟動tracker
[root@02 fastdfs-6.06]# /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
檢查啟動狀態
netstat -lntup|grep 22122 tcp 0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 108126/fdfs_tracker
如果啟動失敗可以檢視tracker報錯
tail -f /data/tracker/logs/trackerd.log
接下來編輯啟動指令碼
cat > /usr/lib/systemd/system/tracker.service <<EOF [Unit] Description=The FastDFS File server After=network.target remote-fs.target nss-lookup.target [Service] Type=forking ExecStart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start ExecStop=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf stop ExecRestart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart [Install] WantedBy=multi-user.target EOF $ systemctl daemon-reload $ systemctl start tracker $ systemctl enable tracker $ systemctl status tracker #需要先手動kill 掉tracker
storage 01-02節點配置
storage01節點和02節點配置相同，storage03和storage04配置相同
storage01和storage02節點屬於group1組
建立storage 資料儲存目錄
mkdir /data/fastdfs_data -p
修改配置檔案
cat >/etc/fdfs/storage.conf<<EOF disabled = false group_name = group1 bind_addr = client_bind = true port = 23000 connect_timeout = 5 network_timeout = 60 heart_beat_interval = 30 stat_report_interval = 60 base_path = /data/fastdfs_data max_connections = 1024 buff_size = 256KB accept_threads = 1 work_threads = 4 disk_rw_separated = true disk_reader_threads = 1 disk_writer_threads = 1 sync_wait_msec = 50 sync_interval = 0 sync_start_time = 00:00 sync_end_time = 23:59 write_mark_file_freq = 500 disk_recovery_threads = 3 store_path_count = 1 store_path0 = /data/fastdfs_data subdir_count_per_path = 256 tracker_server = 192.168.31.101:22122 tracker_server = 192.168.31.102:22122 log_level = info run_by_group = run_by_user = allow_hosts = * file_distribute_path_mode = 0 file_distribute_rotate_count = 100 fsync_after_written_bytes = 0 sync_log_buff_interval = 1 sync_binlog_buff_interval = 1 sync_stat_file_interval = 300 thread_stack_size = 512KB upload_priority = 10 if_alias_prefix = check_file_duplicate = 0 file_signature_method = hash key_namespace = FastDFS keep_alive = 0 use_access_log = false rotate_access_log = false access_log_rotate_time = 00:00 compress_old_access_log = false compress_access_log_days_before = 7 rotate_error_log = false error_log_rotate_time = 00:00 compress_old_error_log = false compress_error_log_days_before = 7 rotate_access_log_size = 0 rotate_error_log_size = 0 log_file_keep_days = 0 file_sync_skip_invalid_record = false use_connection_pool = true connection_pool_max_idle_time = 3600 compress_binlog = true compress_binlog_time = 01:30 check_store_path_mark = true http.domain_name = http.server_port = 80 EOF #注意: 需要修改tracker_server地址，多個節點多複製幾行，一個節點寫一行就可以。 不建議單節點使用localhost
配置啟動檔案
cat >/usr/lib/systemd/system/storage.service <<EOF [Unit] Description=The FastDFS File server After=network.target remote-fs.target nss-lookup.target [Service] Type=forking ExecStart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start ExecStop=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop ExecRestart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start storage systemctl status storage systemctl enable storage
檢查啟動狀態
netstat -lntup|grep 23000
如果出現啟動失敗，可以到設定的目錄檢視一下log
tail -f /data/fastdfs_data/logs/storaged.log
storage 02-03節點配置
storage03和storage04節點屬於group2組
基本流程不變這裡只說明需要修改的地方
在storage03-storage-04節點同步執行
建立storage 資料儲存目錄 mkdir /data/fastdfs_data -p 修改配置檔案 cat >/etc/fdfs/storage.conf<<EOF disabled = false group_name = group2 bind_addr = client_bind = true port = 23000 connect_timeout = 5 network_timeout = 60 heart_beat_interval = 30 stat_report_interval = 60 base_path = /data/fastdfs_data max_connections = 1024 buff_size = 256KB accept_threads = 1 work_threads = 4 disk_rw_separated = true disk_reader_threads = 1 disk_writer_threads = 1 sync_wait_msec = 50 sync_interval = 0 sync_start_time = 00:00 sync_end_time = 23:59 write_mark_file_freq = 500 disk_recovery_threads = 3 store_path_count = 1 store_path0 = /data/fastdfs_data subdir_count_per_path = 256 tracker_server = 192.168.31.101:22122 tracker_server = 192.168.31.102:22122 log_level = info run_by_group = run_by_user = allow_hosts = * file_distribute_path_mode = 0 file_distribute_rotate_count = 100 fsync_after_written_bytes = 0 sync_log_buff_interval = 1 sync_binlog_buff_interval = 1 sync_stat_file_interval = 300 thread_stack_size = 512KB upload_priority = 10 if_alias_prefix = check_file_duplicate = 0 file_signature_method = hash key_namespace = FastDFS keep_alive = 0 use_access_log = false rotate_access_log = false access_log_rotate_time = 00:00 compress_old_access_log = false compress_access_log_days_before = 7 rotate_error_log = false error_log_rotate_time = 00:00 compress_old_error_log = false compress_error_log_days_before = 7 rotate_access_log_size = 0 rotate_error_log_size = 0 log_file_keep_days = 0 file_sync_skip_invalid_record = false use_connection_pool = true connection_pool_max_idle_time = 3600 compress_binlog = true compress_binlog_time = 01:30 check_store_path_mark = true http.domain_name = http.server_port = 80 EOF #配置啟動檔案 cat >/usr/lib/systemd/system/storage.service <<EOF [Unit] Description=The FastDFS File server After=network.target remote-fs.target nss-lookup.target [Service] Type=forking ExecStart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start ExecStop=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop ExecRestart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl start storage systemctl status storage systemctl enable storage #檢查啟動狀態 netstat -lntup|grep 23000
如果出現systemctl啟動失敗，可以使用命令啟動，在根據日誌進行檢視。 大概啟動時間為10s
#storage啟動、停止、重啟命令 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf start /usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop /usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
所有節點storage啟動完畢後進行檢查，是否可以獲取到叢集資訊 (剛建立的叢集比較慢，稍等一會。需要等待狀態為ACTIVE即可)
#在任意節點storage節點執行命令都可以，獲取結果應該如下
[root@storage01 fdfs]# fdfs_monitor /etc/fdfs/storage.conf list
[2020-07-03 01:15:25] DEBUG - base_path=/data/fastdfs_data, connect_timeout=5, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=1, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
server_count=2, server_index=1
tracker server is 192.168.31.102:22122 #tracker server處理本次命令的節點
group count: 2 #group組數量
Group 1: #group1組資訊
group name = group1
disk total space = 17,394 MB
disk free space = 13,758 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 80
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
Storage 1: #storage1節點資訊
id = 192.168.31.103
ip_addr = 192.168.31.103 ACTIVE #storage節點狀態
http domain =
version = 6.06 #fdfs 版本
join time = 2020-07-03 01:08:29 #加入叢集時間
up time = 2020-07-03 01:08:29
total storage = 17,394 MB
free storage = 14,098 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 80
current_write_path = 0
source storage id = 192.168.31.104
if_trunk_server = 0
connection.alloc_count = 256
connection.current_count = 1
...............省略號............................
last_heart_beat_time = 2020-07-03 01:15:18
last_source_update = 1970-01-01 08:00:00
last_sync_update = 1970-01-01 08:00:00
last_synced_timestamp = 1970-01-01 08:00:00
Storage 2: #storage2節點資訊
id = 192.168.31.104 #storage 節點IP
ip_addr = 192.168.31.104 ACTIVE #storage 節點狀態
http domain =
version = 6.06 #storage 節點版本
join time = 2020-07-03 01:08:26 #加入時間
up time = 2020-07-03 01:08:26
total storage = 17,394 MB
free storage = 13,758 MB
upload priority = 10
store_path_count = 1
...............省略號............................
last_heart_beat_time = 2020-07-03 01:15:17
last_source_update = 1970-01-01 08:00:00
last_sync_update = 1970-01-01 08:00:00
last_synced_timestamp = 1970-01-01 08:00:00
Group 2: #group2叢集資訊
group name = group2
disk total space = 17,394 MB
disk free space = 15,538 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 80
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
Storage 1: #storage1節點資訊
id = 192.168.31.105
ip_addr = 192.168.31.105 ACTIVE
http domain =
version = 6.06
join time = 2020-07-03 01:13:42
up time = 2020-07-03 01:13:42
total storage = 17,394 MB
free storage = 15,538 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000 #storage埠
storage_http_port = 80
current_write_path = 0
source storage id =
if_trunk_server = 0
connection.alloc_count = 256
connection.current_count = 1
...............省略號............................
last_heart_beat_time = 2020-07-03 01:15:22
last_source_update = 1970-01-01 08:00:00
last_sync_update = 1970-01-01 08:00:00
last_synced_timestamp = 1970-01-01 08:00:00
Storage 2:
id = 192.168.31.106
ip_addr = 192.168.31.106 ACTIVE
http domain =
version = 6.06
join time = 2020-07-03 01:14:05
up time = 2020-07-03 01:14:05
total storage = 17,394 MB
free storage = 15,538 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 80
current_write_path = 0
source storage id = 192.168.31.105
if_trunk_server = 0
connection.alloc_count = 256
connection.current_count = 1
connection.max_count = 1
total_upload_count = 0
...............省略號............................
total_file_write_count = 0
success_file_write_count = 0
last_heart_beat_time = 2020-07-03 01:15:10
last_source_update = 1970-01-01 08:00:00
last_sync_update = 1970-01-01 08:00:00
last_synced_timestamp = 1970-01-01 08:00:00
配置client
這裡在tracker01節點配置celient客戶端 (其他節點可不配置,client.conf為可選配置)
mkdir -p /data/fdfs_client/logs #日誌存放路徑 cat >/etc/fdfs/client.conf <<EOF connect_timeout = 5 network_timeout = 60 base_path = /data/fdfs_client/logs tracker_server = 192.168.31.101:22122 tracker_server = 192.168.31.102:22122 log_level = info use_connection_pool = false connection_pool_max_idle_time = 3600 load_fdfs_parameters_from_tracker = false use_storage_id = false storage_ids_filename = storage_ids.conf http.tracker_server_port = 80 EOF #需要修改tracker_server地址
上傳檔案測試，這裡的檔案是init.yaml
[root@01 ~]# echo "test" >init.yaml [root@01 ~]# fdfs_upload_file /etc/fdfs/client.conf init.yaml group2/M00/00/00/wKgfaV7-GG2AQcpMAAAABTu5NcY98.yaml
Storage節點安裝Nginx
所有storage節點mod_fastdfs.conf配置如下
cat >/etc/fdfs/mod_fastdfs.conf <<EOF connect_timeout=2 network_timeout=30 base_path=/tmp load_fdfs_parameters_from_tracker=true storage_sync_file_max_delay = 86400 use_storage_id = false storage_ids_filename = storage_ids.conf tracker_server=192.168.31.101:22122 tracker_server=192.168.31.102:22122 storage_server_port=23000 url_have_group_name = true store_path_count=1 log_level=info log_filename= response_mode=proxy if_alias_prefix= flv_support = true flv_extension = flv group_count = 2 #include http.conf [group1] group_name=group1 storage_server_port=23000 store_path_count=1 store_path0=/data/fastdfs_data [group2] group_name=group2 storage_server_port=23000 store_path_count=1 store_path0=/data/fastdfs_data EOF
拷貝相關依賴 (以下是所有storage節點安裝)
cp /root/tools/fastdfs-6.06/conf/http.conf /etc/fdfs/ cp /root/tools/fastdfs-6.06/conf/mime.types /etc/fdfs/
安裝Nginx依賴包
yum install -y gcc glibc gcc-c++ prce-devel openssl-devel pcre-devel lua-devel libxml2 libxml2-devel libxslt-devel perl-ExtUtils-Embed GeoIP GeoIP-devel GeoIP-data zlib zlib-devel openssl pcre pcre-devel gcc g++ gcc-c++ gd-devel
建立nginx使用者
useradd -s /sbin/nologin nginx -M
編譯nginx
cd /root/tools/nginx-1.18.0 ./configure --prefix=/usr/local/nginx-1.18 --with-http_ssl_module --user=nginx --group=nginx --with-http_sub_module --add-module=/root/tools/fastdfs-nginx-module-1.22/src make && make install ln -s /usr/local/nginx-1.18 /usr/local/nginx
修改Nginx配置檔案
cat > /usr/local/nginx/conf/nginx.conf <<EOF
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 8888;
server_name localhost;
location ~/group[0-9]/M00 {
root /data/fastdfs_data;
ngx_fastdfs_module;
}
}
}
EOF
啟動nginx
/usr/local/nginx/sbin/nginx -t /usr/local/nginx/sbin/nginx
在storage節點上8888的請求且有group的都轉給ngx_fastdfs_module外掛處理
接下來我們手動上傳兩張圖片進行測試
[root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf abcdocker.png group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png [root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg group2/M00/00/00/wKgfal7-JySABmgLAABdMoE-LPo504.jpg
目前我們tracker屬於輪訓機制，會輪訓group1和group2；具體使用引數可以參考下面的文章
接下來我們可以通過瀏覽器訪問,不同的組對應不同的專案，FastDFS叢集可以有多個組，但是每臺機器只可以有一個storage
http://storage1節點:8888/group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png http://storage1節點:8888/group2/M00/00/00/wKgfal7-JySABmgLAABdMoE-LPo504.jpg
經過我的測試，即使我們把圖片上傳到group1中，在group2上面直接訪問也可以訪問成功，但是在group2的儲存目錄並沒有找到圖片檔案。原因如下
#Nginx日誌 192.168.31.174 - - [03/Jul/2020:03:03:31 +0800] "GET /group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0"
為何我們其他storage節點也可以訪問，原因是nginx中fastdfs-nginx-module模組可以重定向檔案連線到源伺服器取檔案
補充:FastDFS常用命令引數
#檢視叢集狀態 fdfs_monitor /etc/fdfs/storage.conf #上傳 fdfs_upload_file /etc/fdfs/client.conf abcdocker.png #下載 fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png #檢視檔案屬性 fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png #刪除檔案 fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png #刪除一個storage /usr/local/bin/fdfs_monitor /etc/fdfs/storage.conf delete group2 192.168.31.105
Tracker配置高可用
在tracker上安裝的nginx主要為了提供http訪問的反向代理、負載均衡和快取服務
這裡我們tracker01 02同時進行安裝即可
#下載nginx依賴包 mkdir /root/tools -p cd /root/tools wget http://down.i4t.com/ngx_cache_purge-2.3.tar.gz wget http://down.i4t.com/fdfs/v6.6/nginx-1.18.0.tar.gz tar xf ngx_cache_purge-2.3.tar.gz tar xf nginx-1.18.0.tar.gz #建立nginx使用者 useradd -s /sbin/nologin -M nginx #編譯nginx cd /root/tools/nginx-1.18.0 ./configure --prefix=/usr/local/nginx-1.18 --with-http_ssl_module --user=nginx --group=nginx --with-http_sub_module --add-module=/root/tools/ngx_cache_purge-2.3 make && make install ln -s /usr/local/nginx-1.18 /usr/local/nginx
nginx安裝完畢，接下來配置nginx.conf
tracker 中nginx節點可以不是80，我這裡以80位代表
mkdir /data/nginx_cache -p
$ vim /usr/local/nginx/conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
large_client_header_buffers 4 32k;
client_max_body_size 300m;
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 16k;
proxy_buffers 4 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;
proxy_cache_path /data/nginx_cache keys_zone=http-cache:100m;
upstream fdfs_group1 {
server 192.168.31.103:8888 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.80.104:8888 weight=1 max_fails=2 fail_timeout=30s;
}
upstream fdfs_group2 {
server 192.168.31.105:8888 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.31.106:8888 weight=1 max_fails=2 fail_timeout=30s;
}
server {
listen 80;
server_name localhost;
location /group1/M00 {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
proxy_pass http://fdfs_group1;
expires 30d;
}
location /group2/M00 {
proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
proxy_pass http://fdfs_group2;
expires 30d;
}
}
}
/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx
此時訪問tracker01節點和tracker02節點應該都沒有問題
http://192.168.31.101/group1/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg http://192.168.31.101/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg http://192.168.31.102/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg http://192.168.31.102/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg
效果圖如下
Nginx代理安裝
通過上面的步驟，已經可以使用storage節點和tracker節點進行訪問，但是為了解決統一管理和tracker高可用，我們還需要使用nginx在去代理tracker
#nginx安裝和上面一樣，我這裡就只更改nginx.conf檔案，nginx代理不需要快取模組，普通安裝即可
$ vim /usr/local/nginx/conf/nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream fastdfs_tracker {
server 192.168.31.101:80 weight=1 max_fails=2 fail_timeout=30s;
server 192.168.31.102:80 weight=1 max_fails=2 fail_timeout=30s;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://fastdfs_tracker/;
}
}
}
最後我們在tracker01節點上，測試nginx代理是否都可以訪問成功
[root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg group1/M00/00/00/wKgfZ17-Oa2AMuRqAABdMoE-LPo686.jpg [root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg group2/M00/00/00/wKgfaV7-Oa6ANXGLAABdMoE-LPo066.jpg 訪問檢視 [root@tracker01 ~]# curl 192.168.31.100/group1/M00/00/00/wKgfZ17-Oa2AMuRqAABdMoE-LPo686.jpg -I HTTP/1.1 200 OK Server: nginx/1.18.0 Date: Thu, 02 Jul 2020 19:49:49 GMT Content-Type: image/jpeg Content-Length: 23858 Connection: keep-alive Last-Modified: Thu, 02 Jul 2020 19:46:53 GMT Expires: Sat, 01 Aug 2020 19:49:49 GMT Cache-Control: max-age=2592000 Accept-Ranges: bytes [root@tracker01 ~]# curl 192.168.31.100/group2/M00/00/00/wKgfaV7-Oa6ANXGLAABdMoE-LPo066.jpg -I HTTP/1.1 200 OK Server: nginx/1.18.0 Date: Thu, 02 Jul 2020 19:50:17 GMT Content-Type: image/jpeg Content-Length: 23858 Connection: keep-alive Last-Modified: Thu, 02 Jul 2020 19:46:54 GMT Expires: Sat, 01 Aug 2020 19:50:17 GMT Cache-Control: max-age=2592000 Accept-Ranges: bytes