ELK使用redis做中间键，减少整个elk的压力

浅浅的花香味﹌ 2023-07-17 09:56 13阅读 0赞

1、安装redis，安装部署过程这里不介绍（这里redis的ip为192.168.0.197，端口为6379）

2、配置logstash的收集数并导入redis的配置文件和从redis中获取数据导入elasticsearch的两个配置文件

1、配置导入redis数据的配置文件，并启动logstash服务

  1. input {
  2.       file{
  3.         path => "/var/log/messages"             #指定要收集的日志文件
  4.   type => "system"  #指定类型为system，可以自定义，type值和output{ } 中的type对应即可
  5.         start_position => "beginning"       #从开始处收集
  6.         }
  7.       file{
  8.         path => "/home/otc/otc-web/logs/gxzx-otc-web.log"
  9.         type => "otc"
  10.         start_position => "beginning"
  11.          }
  12.        file{ 
  13.         path => "/home/deploy/financial-management/logs/gxzx-fin-web.log"
  14.         type => "financial"
  15.         start_position => "beginning"
  16.           }
  17.        file{
  18.         path => "/home/deploy/activity_service/logs/gxzx-act-web.log"
  19.         type => "act"
  20.         start_position => "beginning"
  21.        }     
  22.        file{
  23.         path => "/home/deploy/mining/logs/gxzx-min-web.log"
  24.         type => "mining"
  25.         start_position => "beginning"
  26. }     
  27.        }
  28. output {
  29.         if [type] == "system" {
  30.              redis {
  31.                  host => "192.168.0.197"
  32.                  password => '901Bcpct'
  33.                  port => "6379"
  34.                  db => "3"
  35.                  data_type => "list"
  36.                  key => 'logs_system'
  37.              }
  38.         }
  39.         if [type] == "otc" {
  40.              redis {
  41.                  host => "192.168.0.197"
  42.                  password => '901Bcpct'
  43.                  port => "6379"
  44.                  db => "3"
  45.                  data_type => "list"
  46.                  key => 'logs_otc'
  47.              }
  48.         }
  49.         if [type] == "financial" {
  50.              redis {
  51.                  host => "192.168.0.197"
  52.                  password => '901Bcpct'
  53.                  port => "6379"
  54.                  db => "3"
  55.                  data_type => "list"
  56.                  key => 'logs_financial'
  57.              }
  58.         }
  59.         if [type] == "act" {
  60.              redis {
  61.                  host => "192.168.0.197"
  62.                  password => '901Bcpct'
  63.                  port => "6379"
  64.                  db => "3"
  65.                  data_type => "list"
  66.                  key => 'logs_act'
  67.              }
  68.         }
  69.         if [type] == "mining" {
  70.              redis {
  71.                  host => "192.168.0.197"
  72.                  password => '901Bcpct'
  73.                  port => "6379"
  74.                  db => "3"
  75.                  data_type => "list"
  76.                  key => 'logs_mining'
  77.              }
  78.         }
  79. }

启动服务：

  1. 在源码安装的logstash的bin目录下执行，后面还要加一路径，默认是当前执行这的家目录下
  2.  ./logstash -f input_redis.conf &

2、配置从redis导出数据的配置文件

  1. input {
  2.         beats {
  3.             port => 5045
  4.         }
  5.         if [type] == "system" {
  6.              redis {
  7.                  host => "192.168.0.197"
  8.                  password => '901Bcpct'
  9.                  port => "6379"
  10.                  db => "3"
  11.                  data_type => "list"
  12.                  key => 'logs_system'
  13.         }
  14.         }
  15.         if [type] == "otc" {
  16.              redis {
  17.                  host => "192.168.0.197"
  18.                  password => '901Bcpct'
  19.                  port => "6379"
  20.                  db => "3"
  21.                  data_type => "list"
  22.                  key => 'logs_otc'
  23.              }
  24.         }
  25.         if [type] == "financial" {
  26.              redis {
  27.                  host => "192.168.0.197"
  28.                  password => '901Bcpct'
  29.                  port => "6379"
  30.                  db => "3"
  31.                  data_type => "list"
  32.                  key => 'logs_financial'
  33.              }
  34.         }
  35.         if [type] == "act" {
  36.              redis {
  37.                  host => "192.168.0.197"
  38.                  password => '901Bcpct'
  39.                  port => "6379"
  40.                  db => "3"
  41.                  data_type => "list"
  42.                  key => 'logs_act'
  43.              }
  44.         }
  45.         if [type] == "mining" {
  46.              redis {
  47.                  host => "192.168.0.197"
  48.                  password => '901Bcpct'
  49.                  port => "6379"
  50.                  db => "3"
  51.                  data_type => "list"
  52.                  key => 'logs_mining'
  53.              }
  54.         }
  55. }
  58. output {
  59.         if [type] == "system" {           #如果type为system，
  60.            elasticsearch {                  #就输出到Elasticsearch服务器
  61.              hosts => ["192.168.0.117:9200"]               #Elasticsearch监听地址及端口
  62.              index => "system-%{+YYYY.MM.dd}"           #指定索引格式
  63.                 }
  64.              }
  65.         if [type] == "otc" {
  66.   elasticsearch {
  67.              hosts => ["192.168.0.117:9200"]
  68.              index => "nginx_otc-%{+YYYY.MM.dd}"
  69.                }
  70.              } 
  71.         if [type] == "financial" {
  72.           elasticsearch {
  73.              hosts => ["192.168.0.117:9200"]
  74.              index => "nginx_financial-%{+YYYY.MM.dd}"
  75.              }
  76.           }  
  77.         if [type] == "act" {
  78.           elasticsearch {
  79.              hosts => ["192.168.0.117:9200"]
  80.              index => "act_log-%{+YYYY.MM.dd}"
  81.              }
  82.           }  
  83.         if [type] == "mining" {
  84.           elasticsearch {
  85.              hosts => ["192.168.0.117:9200"]
  86.              index => "mining_log-%{+YYYY.MM.dd}"
  87.              }
  88.           }  
  89.         }

同样启动服务

  1. ./logstash -f output_redis.conf --path.data=/home/elk/ &
  2. 此时我将路径改成了另一个路径了

此时我们就可以在redis上看到我们刚刚加的key和值

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70

此时我们的redis就加入到我们的elk当中

发表评论

表情：
评论列表 （有 0 条评论，13人围观）

还没有评论，来说两句吧...

相关阅读