ELK 系列六、logstash 的mutate过滤器功能介绍

约定不等于承诺〃 2022-04-25 05:48 253阅读 0赞

因为我在做相关elk项目的时候用的还是6.5版本，所以该文档就以6.5进行介绍，首先先了解下mutate是做什么的，该插件是集成至logstash过滤数据用的，在把数据写入到es之前进行简单的数据清洗和数据格式化。

mutate具有以下几个函数（sql用惯了）吧

<table>  函数介绍  
 <tbody> 
  <tr> 
   <td style="width:52px;">序号</td> 
   <td style="width:120px;">函数名</td> 
   <td style="width:93px;">输入类型</td> 
   <td style="width:633px;">备注</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">1</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-convert" rel="nofollow"><code>convert</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;">和关系型数据库的convert比较类似，都是做类型转换的，能转换的类型如下：integer、integer_eu、float、float_eu、string、boolean</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">2</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-copy" rel="nofollow"><code>copy</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;"> 
    <ul style="margin-left:0px;"> 
     <li><span style="color:#444444;">值类型是<a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">哈希值</a></span></li> 
     <li><span style="color:#444444;">此设置没有默认值。</span></li> 
    </ul><p style="margin-left:0px;"><span style="color:#444444;">将现有字段复制到另一个字段。将覆盖现有目标字段。</span></p> </td> 
  </tr> 
  <tr> 
   <td style="width:52px;">3</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-gsub" rel="nofollow"><code>gsub</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#array" rel="nofollow">array</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">4</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-join" rel="nofollow"><code>join</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">5</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-lowercase" rel="nofollow"><code>lowercase</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#array" rel="nofollow">array</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">6</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-merge" rel="nofollow"><code>merge</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">7</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-coerce" rel="nofollow"><code>coerce</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">8</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-rename" rel="nofollow"><code>rename</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">9</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-replace" rel="nofollow"><code>replace</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">10</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-split" rel="nofollow"><code>split</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;"> 
    <ul style="margin-left:0px;"> 
     <li><span style="color:#444444;">值类型是<a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">哈希值</a></span></li> 
     <li><span style="color:#444444;">此设置没有默认值。</span></li> 
    </ul><p style="margin-left:0px;"><span style="color:#444444;">使用分隔符将字段拆分为数组。仅适用于字符串字段。</span></p> </td> 
  </tr> 
  <tr> 
   <td style="width:52px;">11</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-strip" rel="nofollow"><code>strip</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#array" rel="nofollow">array</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">12</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-update" rel="nofollow"><code>update</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash" rel="nofollow">hash</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">13</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-uppercase" rel="nofollow"><code>uppercase</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#array" rel="nofollow">array</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:52px;">14</td> 
   <td style="width:120px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-capitalize" rel="nofollow"><code>capitalize</code></a></td> 
   <td style="width:93px;"><a href="https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#array" rel="nofollow">array</a></td> 
   <td style="width:633px;">&nbsp;</td> 
  </tr> 
 </tbody> 
</table>

# #

## 1、[`convert`][convert] ##

*  值类型是[哈希值][Link 1]
 *  此设置没有默认值。

将字段的值转换为其他类型，例如将字符串转换为整数。如果字段值是数组，则将转换所有成员。如果该字段是哈希，则不会采取任何操作。

有效的转换目标及其对不同输入的预期行为是：

*  `integer`：
    
     *  字符串被解析; 支持逗号分隔符（例如，字符串`"1,000"`生成一个值为1000的整数）; 当字符串有小数部分时，它们会被*截断*。
     *  浮点数和小数被*截断*（例如，`3.99`变为`3`，`-2.7`变为`-2`）
     *  布尔真和布尔假被转换为`1`和`0`分别
 *  `integer_eu`：
    
     *  相同`integer`，除了字符串值支持点分隔符和逗号小数（例如，`"1.000"`生成一个值为1000的整数）
 *  `float`：
    
     *  整数转换为浮点数
     *  字符串被解析; 支持逗号分隔符和点小数（例如，`"1,000.5"`生成一个值为一千零一半的整数）
     *  布尔真和布尔假被转换为`1.0`和`0.0`分别
 *  `float_eu`：
    
     *  相同`float`，除了字符串值支持点分隔符和逗号小数（例如，`"1.000,5"`生成一个值为一千零一半的整数）
 *  `string`：
    
     *  所有值都使用UTF-8进行字符串化和编码
 *  `boolean`：
    
     *  串`"true"`，`"t"`，`"yes"`，`"y"`，和`"1"`被转换成布尔`true`
     *  串`"false"`，`"f"`，`"no"`，`"n"`，和`"0"`被转换成布尔`false`
     *  空字符串转换为布尔值 `false`
     *  所有其他值在没有转换的情况下直接传递并记录警告消息

例子：

filter {
      mutate {
        convert => {
          "fieldname" => "integer"
          "booleanfield" => "boolean"
        }
      }
    }

2.[`copy`][copy]

*  值类型是[哈希值][Link 1]
 *  此设置没有默认值。

将现有字段复制到另一个字段。将覆盖现有目标字段。

例：

filter {
      mutate {
         copy => { "source_field" => "dest_field" }
      }
    }

10.[`split`][split]

*  值类型是[哈希值][Link 1]
 *  此设置没有默认值。

使用分隔符将字段拆分为数组。仅适用于字符串字段。

例：

filter {
      mutate {
         split => { "fieldname" => "," }
      }
    }

就先写到这里，后续会继续更新。。

参考资料：[https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html\#plugins-filters-mutate-convert][convert]

[convert]: https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-convert
[Link 1]: https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#hash
[copy]: https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-copy
[split]: https://www.elastic.co/guide/en/logstash/6.5/plugins-filters-mutate.html#plugins-filters-mutate-split