seatunnel hive source 未设置分隔符导致多个字段合并成一个的问题定位解决

Myth丶恋晨 2024-03-17 23:17 70阅读 0赞

seatunnel hive source 未设置分隔符导致多个字段没有切分全保存在一个字段中了,翻看源码发现分隔符是是通过delimiter设置的,只要设置这个delimiter=","就可以了。  
![delimiter][]

#### 设置这个属性 delimiter=“,” ####

他的默认值是\\u0001,如果没有设置delimiter属性则会根据文件类型判断,如果是csv则使用”,”,其他的不会处理,也就是不会切分。  
![在这里插入图片描述][e17bc9b8a4ea4466bc7b5dba48830d0a.png]

env {
      spark.sql.catalogImplementation = "hive"
      spark.app.name = "SeaTunnel-spark3-hive"
      spark.executor.instances = 4
      spark.executor.cores = 2
      spark.executor.memory = "5g"
      spark.yarn.queue = "aiops"
      spark.ui.enabled = true
    }
    
    source {
    Hive {
      table_name = "default.zc_hive_500_id_int"
      metastore_uri = "thrift://cdh129130:9083"
      kerberos_principal = "hive/cdh129144@MYCDH"
      kerberos_keytab_path = "/home/aiops/keytab/hive.keytab"
      hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml"
      parallelism = 1
      read_columns = ["nid","date_id","mm","latn_id","email","ctime","address"],
      fetch_size = 10000
      delimiter=","
      
    }
    }
    
    transform {
    }
    
    sink {
    Console{
    
    }
    }

#### 源码分析过程: ####

根据抛异常的地方找打读记录行的位置![在这里插入图片描述][2ee5cb3a98ff453a9a8a27986b81582e.png]  
这里有多种实现  
![在这里插入图片描述][b7448749874a4ae3a5d334d2e1554db0.png]  
根据获取的字段列表转换行到SeaTunnelRow对象, 这里将line按separators\[level\]进行split切割为多列的值(这里很关键, 分隔符是什么?), 最后将字段index和value放到map,然后使用SeaTunnelRow对象封装map。  
![在这里插入图片描述][ccdd88b73f374f70a91a45c31c8e3aee.png]  
![在这里插入图片描述][e5a8e80c100d4578a63b40dcaf3e9546.png]  
分隔符在build中被赋值  
![在这里插入图片描述][74f9780860574ed49a65f1a8fa617068.png]  
Build中的fieldDelimiter来自于BaseSourceConfig.DELIMITER.key(),如果没有值则获取FILE\_FORMAT\_TYPE进行判断，如果是csv格式则是逗号分隔。  
![在这里插入图片描述][4225829659a94238afb29108436c7770.png]  
字段分隔符被设置到separators\[0\]  
![在这里插入图片描述][7b38f17ee97c40fe85d7bf538c8edbcf.png]  
![在这里插入图片描述][c2af601f0336419ea64651c7e4638dd3.png]  
如何能配置指定的分隔符呢？  
设置这个delimiter, 他是读取自这个属性的。

source {
    Hive {
      table_name = "default.zc_hive_500_id_int"
      metastore_uri = "thrift://cdh129130:9083"
      kerberos_principal = "hive/cdh129144@MYCDH"
      kerberos_keytab_path = "/home/aiops/keytab/hive.keytab"
      hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml"
      parallelism = 1
      read_columns = ["nid","date_id","mm","latn_id","email","ctime","address"],
      fetch_size = 10000
      delimiter=","
      
    }
    }

#### 参考 ####

https://github.com/apache/seatunnel/issues/4731  
本文链接 https://blog.csdn.net/shy\_snow/article/details/131845117

[delimiter]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/ec973c087d6245d39a621dce3d977ac4.png
[e17bc9b8a4ea4466bc7b5dba48830d0a.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/e91b9474627c422abf6236292f588d1f.png
[2ee5cb3a98ff453a9a8a27986b81582e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/52487c4c75d6476f851910d37424bfd8.png
[b7448749874a4ae3a5d334d2e1554db0.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/c3a933f681464fec904aaaadc5f7cc3f.png
[ccdd88b73f374f70a91a45c31c8e3aee.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/0bdb8a00a68f437690166346bb59a3e3.png
[e5a8e80c100d4578a63b40dcaf3e9546.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/3e9da45e1d7d448484452be705ae7379.png
[74f9780860574ed49a65f1a8fa617068.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/fe88e555cf254ac3acf4b1c41ef9129e.png
[4225829659a94238afb29108436c7770.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/bea2c113f7644510bd913a3eaa9d7bf5.png
[7b38f17ee97c40fe85d7bf538c8edbcf.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/cc88bd9bbf8c420bbfb6d2faac20f65f.png
[c2af601f0336419ea64651c7e4638dd3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/14/466f3c10d9464ebe9c5f66f00bad6d1a.png