如何查询生产环境logstash报Error parsing json

痛定思痛。 2022-09-16 04:57 626阅读 0赞

##### 1、生产环境有部分报错 #####

信息如下：

2021-10-15T15:49:28,932][WARN ][logstash.filters.json    ][main][7e17a3dc7e2c6df08ed7012ca6bfe17e3277d05d745f2c5bf55d2b01b151e25b] Error parsing json {:source=>"message", :raw=>"查询用户详情接口", :exception=>#<LogStash::Json::ParserError: Invalid UTF-8 start byte 0x9f
     at [Source: (byte[])"查询用户详情接口"; line: 1, column: 3]>}
    [2021-10-15T15:49:28,946][ERROR][logstash.outputs.elasticsearch][main][862f1cf74fcefb4312e0b0aa2e9fdf074e2e77c675e5c24e9b7a04d1054f1947] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{aName}-log_2021_10", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x6527baeb>], :response=>{"index"=>{"_index"=>"%{aName}-log_2021_10", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [%{aName}-log_2021_10], must be lowercase", "index_uuid"=>"_na_", "index"=>"%{aName}-log_2021_10"}}}}
    [2021-10-15T15:49:29,446][WARN ][logstash.filters.json    ][main][52e7421840cd9e07ac4081f9b005972562be1954f89c4f4bc5e30c5c19ebae78] Error parsing json {:source=>"message", :raw=>"spAuthAttributes ==========> {\"chainCodes\":\"60d9acb97314488689b899f4495e0857,3d01cb64635b46d4955606e6b356af70,c3283b04c2ad48a09d770b5a96b96a4c,850ff333c6664138b38344a75545ce8e\",\"customLoginStyle\":\"default\",\"samlEntityID\":\"portal\",\"securityLevel\":\"1\",\"spCode\":\"portal\",\"spLocalLogoutUrl\":\"\",\"spName\":\"portal\",\"spOsType\":0,\"tokenValidPeriod\":36000,\"url\":\"https://i.gt.cn\",\"userAttr\":\"\"}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'spAuthAttributes': was expecting ('true', 'false' or 'null')
     at [Source: (byte[])"spAuthAttributes ==========> {"chainCodes":"60d9acb97314488689b899f4495e0857,3d01cb64635b46d4955606e6b356af70,c3283b04c2ad48a09d770b5a96b96a4c,850ff333c6664138b38344a75545ce8e","customLoginStyle":"default","samlEntityID":"portal","securityLevel":"1","spCode":"portal","spLocalLogoutUrl":"","spName":"portal","spOsType":0,"tokenValidPeriod":36000,"url":"https://i.gt.cn","userAttr":""}"; line: 1, column: 18]>}
    [2021-10-15T15:49:29,447][WARN ][logstash.filters.json    ][main][52e7421840cd9e07ac4081f9b005972562be1954f89c4f4bc5e30c5c19ebae78] Error parsing json {:source=>"message", :raw=>"redis执行时间：{}1", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'redis执行时间': was expecting ('true', 'false' or 'null')
     at [Source: (byte[])"redis执行时间：{}1"; line: 1, column: 21]>}

2、处理方法  
这是在处理过程中有日志没有按照要进行打印，我们需要新开一个logstash把异常的日志打印出来；  
配置如下：

[root@k8s-master1 conf.d]# more aName1.conf 
    input{
        kafka{
               bootstrap_servers => "10.152.17.50:9092"
               group_id => "gt-scs-log-new"
               topics => "gt-scs-log"
               consumer_threads => 1
               decorate_events => true
               auto_offset_reset => "latest"
               type => "gt-scs"
        }
    }
    filter {
        if [type] == "gt-scs" {
          mutate { 
                    add_field => { "types" => "%{type}"}
                 }
          json {
                    source => "message"
              }
          json {
                    source => "message"
              }
    
        }
    }
    output { 
      if [tags] { 
            stdout {
                    codec => "rubydebug"
            }
      }
    }

##### 3、结果 #####

![在这里插入图片描述][watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA6L-Q57u05omT5oCq5pmL57qn5LmL6Lev_size_20_color_FFFFFF_t_70_g_se_x_16]

[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA6L-Q57u05omT5oCq5pmL57qn5LmL6Lev_size_20_color_FFFFFF_t_70_g_se_x_16]: /images/20220828/35166df4e22d489e826bc7ca08682640.png