利用streamSet搭建一个简单的数据管道

落日映苍穹つ 2022-05-14 05:40 313阅读 0赞

本教程利用StreamSet搭建一个简单的数据管道，具体为：从本地目录中读取文件，并在分支中处理读取的文件数据，最后将处理的数据写入到本地目录。将通过数据预览来配置数据管道，并添加数据报警功能。

由于是初次尝试，难免存在错误，不足之处还请大神不吝赐教！！！

--------------------

**目录**

0 数据准备

1 配置管道属性

2 配置数据源（源点）

3 数据预览

4 通过流选择器路由数据

5 使用Jython处理信用卡类型

6 屏蔽信用卡号码

7 将数据写入到目标

8 利用Expression Evaluator添加相应字段

9 创建数据规则并报警

10 运行

--------------------

### 0 数据准备 ###

*  下载样本数据：[https://www.streamsets.com/documentation/datacollector/sample\_data/tutorial/nyc\_taxi\_data.csv][https_www.streamsets.com_documentation_datacollector_sample_data_tutorial_nyc_taxi_data.csv]
 *  创建数据收集器的本地目录，将样本数据放入origin目录下

/<base directory>/tutorial/origin
 /<base directory>/tutorial/destination
 /<base directory>/tutorial/error

备注：本教程中的<base directory>为StreamSet的安装目录，当然你也可以将数据放在其它位置，只要保证启动StreamSet的用户对相应目录有读写权限

### **1 配置管道属性** ###

**置管道属性，主要是对异常数据的处理**

*  在主界面点机 Create New Pipline；

![70][]

*  在弹出的对话框中输入 Pipline title 和 optional description，并选择run the pipeline on Data Collector，点击save进行保存；

![70 1][]

*  在下面的属性面板中，选择Error Records选项，配置Error Records属性为Write to File(将异常数据保存到文件中，这样就可以在不停止pipeline的情况下处理异常数据)

![70 2][]

*  Write to File的属性配置如下（没有的保持默认即可）：

<table> 
 <tbody> 
 <tr> 
 <td style="background-color:#e6e6e6;border-color:inherit;text-align:left;vertical-align:inherit;"> Write to File Property </td> 
 <td style="background-color:#e6e6e6;border-color:inherit;text-align:left;vertical-align:inherit;"> Description </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Directory </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Directory for error record files. Enter the directory that you set up for the tutorial. We recommended: /&lt;base directory&gt;/tutorial/error Note:&nbsp;To prevent validation errors, the directory must already exist. </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Files Prefix </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> This defines a prefix for error record files. By default, the files are prefixed with "SDC" and an expression that returns the&nbsp;Data Collector&nbsp;ID, but that's more than we need here. Delete the default and enter the following prefix:&nbsp;err_ </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Max File Size </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> For the tutorial, reduce the files size to something more manageable, such as 5 or 1 MB. </td> 
 </tr> 
 </tbody> 
</table>

![70 3][]

### **2 配置数据源（源点）** ###

数据源是整个管道的入口，在配置数据源的时候需要定义连接源点方式、要处理的数据类型以及其它的特殊属性等

*  选择 Select origin -> Directory 或者在最右侧的阶段库（Stage Library panel）中选择**Directory** origin: ![70 4][].
 *  在下面的属性面板中选择File，并配置相关属性如下（没有的保持默认即可）：

<table> 
 <tbody> 
 <tr> 
 <td style="background-color:#e6e6e6;border-color:inherit;text-align:left;vertical-align:inherit;"> Directory Property </td> 
 <td style="background-color:#e6e6e6;border-color:inherit;text-align:left;vertical-align:inherit;"> Value </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Files Directory </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Directory where you saved the sample file. Enter an absolute path. We recommended:&nbsp;/&lt;base directory&gt;/tutorial/origin. </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> File Name Pattern </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> The Directory origin processes only the files in the directory that match the file name pattern. The tutorial sample file name is&nbsp;nyc_taxi_data.csv. Since the file is the only file in the directory, you can use something generic, like the asterisk wild card (*) or *.csv. If you had other .csv files in the directory that you didn't want to process, you might be more specific, like this:&nbsp;nyc_taxi*.csv. Or if you want to process files with prefixes for other cities, you might use&nbsp;*taxi*.csv. </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Read Order </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> This determines the read order when the directory includes multiple files. You can read based on the last-modified timestamp or file name. Because it's simpler, let's use&nbsp;Last Modified Timestamp. </td> 
 </tr> 
 </tbody> 
</table>

*  选择Data Format配置数据格式属性如下

<table> 
 <tbody> 
 <tr> 
 <td style="background-color:#e6e6e6;border-color:inherit;text-align:left;vertical-align:inherit;"> Delimited Property </td> 
 <td style="background-color:#e6e6e6;border-color:inherit;text-align:left;vertical-align:inherit;"> Description </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Data Format </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> The data in the sample file is delimited, so select&nbsp;Delimited. </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Delimiter Format Type </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Since the sample file is a standard CSV file, use the default:&nbsp;Default CSV (ignores empty lines). </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Header Line </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> The sample file includes a header, so select&nbsp;With Header Line. </td> 
 </tr> 
 <tr> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> Root Field Type </td> 
 <td style="background-color:#ffffff;border-color:inherit;vertical-align:top;"> This property determines how the&nbsp;Data Collector&nbsp;processes delimited data. Use the default&nbsp;List-Map. This allows you to use standard functions to process delimited data. With the List root field type, you need to use delimited data functions. </td> 
 </tr> 
 </tbody> 
</table>

*  最后界面看起来像这样

![70 5][]

### 3 数据预览 ###

*  点击预览图标(**Preview** icon） ![70 6][]，按照如下进行预览配置

![70 7][]

<table> 
 <tbody> 
 <tr> 
 <td style="background-color:#e6e6e6;border-color:#cccccc;text-align:left;vertical-align:inherit;"> Data Preview Property </td> 
 <td style="background-color:#e6e6e6;border-color:#cccccc;text-align:left;vertical-align:inherit;"> Description </td> 
 </tr> 
 <tr> 
 <td style="border-color:#cccccc;vertical-align:top;"> Preview Source </td> 
 <td style="border-color:#cccccc;vertical-align:top;"> Use the default&nbsp;Configured Source&nbsp;to use the sample source data. </td> 
 </tr> 
 <tr> 
 <td style="border-color:#cccccc;vertical-align:top;"> Write to Destinations and Executors </td> 
 <td style="border-color:#cccccc;vertical-align:top;"> By default, this property is not selected. In general, you should keep this property clear to avoid writing data to destination systems or triggering tasks in destination systems. </td> 
 </tr> 
 <tr> 
 <td style="border-color:#cccccc;vertical-align:top;"> Show Field Type </td> 
 <td style="border-color:#cccccc;vertical-align:top;"> By default, this property is selected. Keep it selected to see the data type of fields in the record. </td> 
 </tr> 
 <tr> 
 <td style="border-color:#cccccc;vertical-align:top;"> Remember the Configuration </td> 
 <td style="border-color:#cccccc;vertical-align:top;"> Select to use these properties each time you run data preview. When you select this option, the UI enters data preview without showing this dialog box again. </td> 
 </tr> 
 </tbody> 
</table>

![70 8][]

![70 9][]

![70 10][]

注：如果无法预览数据，请检查前面步骤是否正确，解决除Validation\_0011问题以外的其它问题。Validation\_0011表示数据没有任何连接，不影响数据预览。

### 4 通过流选择器路由数据 ###

*  流选择器是基于用户定义的条件规则进行数据过滤，不符合条件的记录将会被路由到默认的流管道中
 *  选择Select Processor to Connect > Stream Selector或者在Stage Library panel中选择 Stream Selectorprocessor (![70 11][])，然后连接源数据

![70 12][]

*  点击General面板，选择Required Fields属性值为/payment\_type，如果之前预览数据没有问题，则会自动显示字段，如果没有，可以手动输入
 *  点击condition面板，配置过滤条件，通过点击➕号增加新的条件，满足过滤条件的将会被分流到管道1，其他的分流到管道2

![70 13][]

![70 14][]

### 5 使用Jython处理信用卡类型 ###

*  通过点击package manage 来 添加Jython Evaluator Processor，点击install即可开始安装，安装完成后点击重启数据收集器

![70 15][]

*  重启之后就可以选择Jython Evaluator，将其连接到管道1
 *  点击Jython属性面板，使用默认的批处理方式处理数据
 *  在Script的文本框中删除所有注释，然后添加如下代码

try:
      for record in records:
        cc = record.value['credit_card']
        if cc == '':
          error.write(record, "Payment type was CRD, but credit card was null")
          continue
        
        cc_type = ''
        if cc.startswith('4'):
          cc_type = 'Visa'
        elif cc.startswith(('51','52','53','54','55')):
          cc_type = 'MasterCard'
        elif cc.startswith(('34','37')):
          cc_type = 'AMEX'
        elif cc.startswith(('300','301','302','303','304','305','36','38')):
          cc_type = 'Diners Club'
        elif cc.startswith(('6011','65')):
          cc_type = 'Discover'
        elif cc.startswith(('2131','1800','35')):
          cc_type = 'JCB'
        else:
          cc_type = 'Other'
        
        record.value['credit_card_type'] = cc_type
        output.write(record)
    except Exception as e:
      error.write(record, e.message)

注：粘贴代码的时候可能由于格式问题（缩进等）会出错，请自行调整

*  出现以下问题又可能安装Jython的时候没有安装成功，缺少了某些库，建议卸载后重新安装

![70 16][]

### 6 屏蔽信用卡号码 ###

为了防止信用卡号码等敏感信息暴露出来，我们使用字段掩码来屏蔽信用卡号码

*  在Jython Evaluator之后增加一个Field Mask处理器
 *  然后点击属性面板中的Mask table，配置如下属性

<table> 
 <tbody> 
 <tr> 
 <td> Field Masker Property </td> 
 <td> Configuration </td> 
 </tr> 
 <tr> 
 <td> Mask Type </td> 
 <td> Regular Expression </td> 
 </tr> 
 <tr> 
 <td> Regular Expression </td> 
 <td> (.*)([0-9]{4}) </td> 
 </tr> 
 <tr> 
 <td> Groups to Show </td> 
 <td> 2 </td> 
 </tr> 
 </tbody> 
</table>

注：正则表达式(.\*)(\[0-9\]\{4\})可以屏蔽除最后四位数字以外的所有数字

![70 17][]

### 7 将数据写入到目标 ###

在此，我们将数据写入到本地文件中

*  在Field Mask之后连接一个Local FS
 *  点击Output Files 并配置如下属性值

<table> 
 <tbody> 
 <tr> 
 <td> Local FS Property </td> 
 <td> Configuration </td> 
 </tr> 
 <tr> 
 <td> Files Prefix </td> 
 <td> Defines a prefix for output file names. By default, the files are prefixed with "SDC" and an expression that returns the&nbsp;Data Collector&nbsp;ID, but that's more than we need here. Let's simplify and use "out_" instead. </td> 
 </tr> 
 <tr> 
 <td> Directory Template </td> 
 <td> By default, the directory template includes datetime variables to create a directory structure for output files. This is intended for writing large volumes of data. Since we only have the sample file to process, we don't need the datetime variables. Go ahead and delete the default and enter the directory where you want the files to be written. We suggested:&nbsp;/&lt;base directory&gt;/tutorial/destination. </td> 
 </tr> 
 <tr> 
 <td> Max File Size (MB) </td> 
 <td> For the tutorial, let's lower the file size to something manageable, like 5 or 1. </td> 
 </tr> 
 </tbody> 
</table>

![70 18][]

*  点击Data Format，配置如下：

<table> 
 <tbody> 
 <tr> 
 <td> DELIMITED PROPERTY </td> 
 <td> CONFIGURATION </td> 
 </tr> 
 <tr> 
 <td> Data Format </td> 
 <td> Delimited </td> 
 </tr> 
 <tr> 
 <td> Header Line </td> 
 <td> With Header Line </td> 
 </tr> 
 </tbody> 
</table>

![70 19][]

### 8 利用Expression Evaluator添加相应字段 ###

Jython Evaluator脚本在信用支付分支中添加了一个新字段。 为了确保所有记录具有相同的结构，我们将使用Expression Evaluator将相同的字段添加到非信用分支。这可确保所有记录在写入目标时具有相同的格式。为此，我们使用数据预览来验证Jython Evaluator如何将信用卡类型添加到记录中

*  点击Preview图标进行数据预览，如果此时报错，请检查错误原因
 *  点击Jython Evaluator即可显示数据经过Jython Evaluator时的输入输出信息，可以发信output data多了一个字段credit\_card\_type.

![70 20][]

*  关闭Preview，增加一个Expression Evaluator连接Stream Selector的第2个管道
 *  点击Expressions并配置如下属性：

<table> 
 <tbody> 
 <tr> 
 <td> OUTPUT FIELD </td> 
 <td> EXPRESSION </td> 
 </tr> 
 <tr> 
 <td> /credit_card_type </td> 
 <td> n/a </td> 
 </tr> 
 </tbody> 
</table>

![70 21][]

*  连接Expression Evaluator到FS destination

### 9 创建数据规则并报警 ###

现在，在运行基本管道之前，让我们添加一个数据规则和警报。数据规则是用户定义的规则，用于检查在两个阶段之间移动的数据。它们是寻找异常值和异常数据的有力方法。数据规则和警报需要对通过管道的数据进行详细的理解。对于更一般的管道监测信息，可以使用度量规则和警报。

*  在Stream Selector 和 Jython Evaluator, 点击图标 ![70 22][].然后在下面点击Add

![70 23][]

*  在数据规则对话框中配置如下信息：

<table> 
 <tbody> 
 <tr> 
 <td> DATA RULE PROPERTY </td> 
 <td> DESCRIPTION </td> 
 </tr> 
 <tr> 
 <td> Label </td> 
 <td> Missing Card Numbers </td> 
 </tr> 
 <tr> 
 <td> Condition </td> 
 <td> ${record:value("/credit_card") == ""} </td> 
 </tr> 
 <tr> 
 <td> Sampling Percentage </td> 
 <td> 35 </td> 
 </tr> 
 <tr> 
 <td> Alert Text </td> 
 <td> At least 10 missing credit card numbers! </td> 
 </tr> 
 <tr> 
 <td> Threshold Value </td> 
 <td> 10 </td> 
 </tr> 
 </tbody> 
</table>

*  点击Save并勾选Active

![70 24][]

### 10 运行 ###

点击开始按钮即可开启运行

![70 25][]

![70 26][]

参考：

[https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Tutorial/BasicTutorial.html\#concept\_phf\_cjt\_ls][https_streamsets.com_documentation_datacollector_latest_help_datacollector_UserGuide_Tutorial_BasicTutorial.html_concept_phf_cjt_ls]

[https_www.streamsets.com_documentation_datacollector_sample_data_tutorial_nyc_taxi_data.csv]: https://www.streamsets.com/documentation/datacollector/sample_data/tutorial/nyc_taxi_data.csv
[70]: /images/20220514/51fe2ab09c5b4f379a562ddefc7c2e1b.png
[70 1]: /images/20220514/8b8fa597d74547cc86ab3dd4f1d71b54.png
[70 2]: /images/20220514/b13d49b0e4d84fa49b24e32e639e0be5.png
[70 3]: /images/20220514/df971686eec4459caf02236155c893fc.png
[70 4]: /images/20220514/0d48a8c632724bb5b7920688deb7494d.png
[70 5]: /images/20220514/9de080d9bf8946599954d418f229c011.png
[70 6]: /images/20220514/a476db48cf594141834303fb275a28d7.png
[70 7]: /images/20220514/cafd5275446a43f1b94d1bacb0aefdcf.png
[70 8]: /images/20220514/247cc935f804403c95612bad4e928a29.png
[70 9]: /images/20220514/f92fbf22d7c146c99bc4d708755d9fd7.png
[70 10]: /images/20220514/659e206cbf2c4fd496d4aa0f86d19de3.png
[70 11]: /images/20220514/4423f9c628604a4c88ca7668789f592e.png
[70 12]: /images/20220514/273862395b514e3b82633799aa097e1b.png
[70 13]: /images/20220514/09dbc5c5bcbf47119b2df458b2586136.png
[70 14]: /images/20220514/24aef6fdc07c4290a98899cf42f22ecc.png
[70 15]: /images/20220514/fe9e1475d9d3465b810baf9e03dd0d9a.png
[70 16]: /images/20220514/6f45a54d3f3643488decea9bc8c2091b.png
[70 17]: /images/20220514/acde5db2688a487fb4f6aa599431669b.png
[70 18]: /images/20220514/804abf589d694b339024d726ae486d03.png
[70 19]: /images/20220514/33ad1ee0407d4ff6870d0f2344f821f7.png
[70 20]: /images/20220514/f8f16125b3ef4c5da84b6c7d1080030a.png
[70 21]: /images/20220514/59bf98a3f0b74e958419f5ed37db3b91.png
[70 22]: /images/20220514/abebb964da634a39807dcd437fd2ed93.png
[70 23]: /images/20220514/2b22a6b48f754df3a65d2e96c0a12351.png
[70 24]: /images/20220514/52f556f832cc4381a1e35ad189ed2846.png
[70 25]: /images/20220514/e8f54db5c1c2477e95b7dfffbc5cdc3c.png
[70 26]: /images/20220514/f4fc7eb4b7a14f4fa09e101e141a8995.png
[https_streamsets.com_documentation_datacollector_latest_help_datacollector_UserGuide_Tutorial_BasicTutorial.html_concept_phf_cjt_ls]: https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Tutorial/BasicTutorial.html#concept_phf_cjt_ls