如何查询生产环境logstash报Error parsing json-蒲公英云

如何查询生产环境logstash报Error parsing json

1、生产环境有部分报错

信息如下：

2021-10-15T15:49:28,932][WARN ][logstash.filters.json    ][main][7e17a3dc7e2c6df08ed7012ca6bfe17e3277d05d745f2c5bf55d2b01b151e25b] Error parsing json {:source=>"message", :raw=>"查询用户详情接口", :exception=>#<LogStash::Json::ParserError: Invalid UTF-8 start byte 0x9f
 at [Source: (byte[])"查询用户详情接口"; line: 1, column: 3]>}
[2021-10-15T15:49:28,946][ERROR][logstash.outputs.elasticsearch][main][862f1cf74fcefb4312e0b0aa2e9fdf074e2e77c675e5c24e9b7a04d1054f1947] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{aName}-log_2021_10", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x6527baeb>], :response=>{"index"=>{"_index"=>"%{aName}-log_2021_10", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [%{aName}-log_2021_10], must be lowercase", "index_uuid"=>"_na_", "index"=>"%{aName}-log_2021_10"}}}}
[2021-10-15T15:49:29,446][WARN ][logstash.filters.json    ][main][52e7421840cd9e07ac4081f9b005972562be1954f89c4f4bc5e30c5c19ebae78] Error parsing json {:source=>"message", :raw=>"spAuthAttributes ==========> {\"chainCodes\":\"60d9acb97314488689b899f4495e0857,3d01cb64635b46d4955606e6b356af70,c3283b04c2ad48a09d770b5a96b96a4c,850ff333c6664138b38344a75545ce8e\",\"customLoginStyle\":\"default\",\"samlEntityID\":\"portal\",\"securityLevel\":\"1\",\"spCode\":\"portal\",\"spLocalLogoutUrl\":\"\",\"spName\":\"portal\",\"spOsType\":0,\"tokenValidPeriod\":36000,\"url\":\"https://i.gt.cn\",\"userAttr\":\"\"}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'spAuthAttributes': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"spAuthAttributes ==========> {"chainCodes":"60d9acb97314488689b899f4495e0857,3d01cb64635b46d4955606e6b356af70,c3283b04c2ad48a09d770b5a96b96a4c,850ff333c6664138b38344a75545ce8e","customLoginStyle":"default","samlEntityID":"portal","securityLevel":"1","spCode":"portal","spLocalLogoutUrl":"","spName":"portal","spOsType":0,"tokenValidPeriod":36000,"url":"https://i.gt.cn","userAttr":""}"; line: 1, column: 18]>}
[2021-10-15T15:49:29,447][WARN ][logstash.filters.json    ][main][52e7421840cd9e07ac4081f9b005972562be1954f89c4f4bc5e30c5c19ebae78] Error parsing json {:source=>"message", :raw=>"redis执行时间：{}1", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'redis执行时间': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"redis执行时间：{}1"; line: 1, column: 21]>}

2、处理方法
这是在处理过程中有日志没有按照要进行打印，我们需要新开一个logstash把异常的日志打印出来；
配置如下：

[root@k8s-master1 conf.d]# more aName1.conf 
input{
    kafka{
           bootstrap_servers => "10.152.17.50:9092"
           group_id => "gt-scs-log-new"
           topics => "gt-scs-log"
           consumer_threads => 1
           decorate_events => true
           auto_offset_reset => "latest"
           type => "gt-scs"
    }
}
filter {
    if [type] == "gt-scs" {
      mutate { 
                add_field => { "types" => "%{type}"}
             }
      json {
                source => "message"
          }
      json {
                source => "message"
          }
    }
}
output { 
  if [tags] { 
        stdout {
                codec => "rubydebug"
        }
  }
}