Flink sql使用遇到的问题mark

亦凉 2022-11-10 05:33 674阅读 0赞

遇到了类似的问题：

报错信息：  
Caused by: org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'kafka' that implements 'org.apache.flink.table.factories.DynamicTableSourceFactory' in the classpath.  
Available factory identifiers are:  
datagen  
at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:240)  
at org.apache.flink.table.factories.FactoryUtil.getDynamicTableFactory(FactoryUtil.java:326)  
... 33 more

链接：[http://apache-flink.147419.n8.nabble.com/Could-not-find-any-factory-for-identifier-kafka-td5414.html][http_apache-flink.147419.n8.nabble.com_Could-not-find-any-factory-for-identifier-kafka-td5414.html]

我遇到这个的原因是将Flink  streaming 代码 跟 Flink sql代码放入了一个项目，解决办法就是拆开2个项目：

Flink  streaming 代码依赖：

> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70][]

Flink sql代码依赖：

> ![20210316153752180.png][]

在服务起Flink lib下面**不需要**放入 下面2个包，如果同时放入这2个包，只能执行sql代码，不能执行streaming代码

flink-connector-kafka\_2.11-1.11.2.jar

flink-sql-connector-kafka\_2.11-1.11.2.jar

==================================================================================================

我们在自定义table connector之后会遇到类似下面的问题 。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 1][]

![e82bb446814fb3fe268347dfb27cddde.png][]

原因：因为Flink 加载 table Factory 使用的时SPI机制，而正常的flink jar包是不包含META-INF.services 路径的，需要自己去添加

[![8aeb72376d39842d4944863d001adae3.png][]][8aeb72376d39842d4944863d001adae3.png 1]

org.apache.flink.table.factories.TableFactory

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 2][]

1，Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/Path

原因：Flink找不到hadoop环境，要么在环境变量配置：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 3][]  
要么在flink lib下添加包（1.11以后不推荐）：

flink-shaded-hadoop-3-uber

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 4][]

[http_apache-flink.147419.n8.nabble.com_Could-not-find-any-factory-for-identifier-kafka-td5414.html]: http://apache-flink.147419.n8.nabble.com/Could-not-find-any-factory-for-identifier-kafka-td5414.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70]: /images/20221023/c65da3b410144b23bf923c793d8e02c8.png
[20210316153752180.png]: /images/20221023/642bb5175cb941bd92c4c24fe81fb88c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 1]: /images/20221023/597ef2b3286b44d0a6dd5368597bd6e1.png
[e82bb446814fb3fe268347dfb27cddde.png]: /images/20221023/0df23327c627458e94b46964da60134c.png
[8aeb72376d39842d4944863d001adae3.png]: /images/20221023/86e5b1c798854d32b984d43438b545b3.png
[8aeb72376d39842d4944863d001adae3.png 1]: https://img2020.cnblogs.com/blog/1892627/202103/1892627-20210303120841195-1010498336.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 2]: /images/20221023/d891b4f027144388938b5bc33fe94f55.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 3]: /images/20221023/7dfad8c704b24a1a8c4239a8c5f8e78e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 4]: /images/20221023/6044024c58d342afbe5a9e34ef14c523.png