Could not find a file system implementation for scheme ‘hdfs‘.

偏执的太偏执、 2023-01-05 01:25 319阅读 0赞 
  1. /home/appleyuchi/Java/jdk1.8.0_131/bin/java -javaagent:/home/appleyuchi/下載/software/idea-IU-183.5912.21/lib/idea_rt.jar=41543:/home/appleyuchi/下載/software/idea-IU-183.5912.21/bin -Dfile.encoding=UTF-8 -classpath /home/appleyuchi/Java/jdk1.8.0_131/jre/lib/charsets.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/deploy.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/cldrdata.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/dnsns.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/jaccess.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/jfxrt.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/localedata.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/nashorn.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/sunec.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/sunjce_provider.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/sunpkcs11.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/ext/zipfs.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/javaws.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/jce.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/jfr.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/jfxswt.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/jsse.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/management-agent.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/plugin.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/resources.jar:/home/appleyuchi/Java/jdk1.8.0_131/jre/lib/rt.jar:/home/appleyuchi/桌面/Flink_Code/flink清洗数据案例/FlinkProj/DataClean/target/classes:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-java/1.6.0/flink-java-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-core/1.6.0/flink-core-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-annotations/1.6.0/flink-annotations-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-metrics-core/1.6.0/flink-metrics-core-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/esotericsoftware/kryo/kryo/2.24.0/kryo-2.24.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/tukaani/xz/1.0/xz-1.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-shaded-asm/5.0.4-4.0/flink-shaded-asm-5.0.4-4.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/commons/commons-math3/3.5/commons-math3-3.5.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/force-shading/1.6.0/force-shading-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-streaming-java_2.11/1.6.0/flink-streaming-java_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-runtime_2.11/1.6.0/flink-runtime_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-queryable-state-client-java_2.11/1.6.0/flink-queryable-state-client-java_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-hadoop-fs/1.6.0/flink-hadoop-fs-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/commons-io/commons-io/2.4/commons-io-2.4.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-shaded-netty/4.1.24.Final-4.0/flink-shaded-netty-4.1.24.Final-4.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-shaded-jackson/2.7.9-4.0/flink-shaded-jackson-2.7.9-4.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/javassist/javassist/3.19.0-GA/javassist-3.19.0-GA.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/typesafe/akka/akka-actor_2.11/2.4.20/akka-actor_2.11-2.4.20.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/typesafe/config/1.3.0/config-1.3.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/scala-lang/modules/scala-java8-compat_2.11/0.7.0/scala-java8-compat_2.11-0.7.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/typesafe/akka/akka-stream_2.11/2.4.20/akka-stream_2.11-2.4.20.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/reactivestreams/reactive-streams/1.0.0/reactive-streams-1.0.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/typesafe/ssl-config-core_2.11/0.2.1/ssl-config-core_2.11-0.2.1.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/typesafe/akka/akka-protobuf_2.11/2.4.20/akka-protobuf_2.11-2.4.20.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/typesafe/akka/akka-slf4j_2.11/2.4.20/akka-slf4j_2.11-2.4.20.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/clapper/grizzled-slf4j_2.11/1.0.2/grizzled-slf4j_2.11-1.0.2.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/github/scopt/scopt_2.11/3.5.0/scopt_2.11-3.5.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/twitter/chill_2.11/0.7.4/chill_2.11-0.7.4.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/twitter/chill-java/0.7.4/chill-java-0.7.4.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-shaded-guava/18.0-4.0/flink-shaded-guava-18.0-4.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-scala_2.11/1.6.0/flink-scala_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/scala-lang/scala-reflect/2.11.12/scala-reflect-2.11.12.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/scala-lang/scala-library/2.11.12/scala-library-2.11.12.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/scala-lang/scala-compiler/2.11.12/scala-compiler-2.11.12.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/scala-lang/modules/scala-xml_2.11/1.0.5/scala-xml_2.11-1.0.5.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-streaming-scala_2.11/1.6.0/flink-streaming-scala_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-clients_2.11/1.6.0/flink-clients_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-optimizer_2.11/1.6.0/flink-optimizer_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-connector-kafka-0.11_2.11/1.6.0/flink-connector-kafka-0.11_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-connector-kafka-0.10_2.11/1.6.0/flink-connector-kafka-0.10_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-connector-kafka-0.9_2.11/1.6.0/flink-connector-kafka-0.9_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-connector-kafka-base_2.11/1.6.0/flink-connector-kafka-base_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/kafka/kafka-clients/0.11.0.2/kafka-clients-0.11.0.2.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/bahir/flink-connector-redis_2.11/1.0/flink-connector-redis_2.11-1.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/redis/clients/jedis/2.9.0/jedis-2.9.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/commons/commons-pool2/2.4.2/commons-pool2-2.4.2.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/apache/flink/flink-statebackend-rocksdb_2.11/1.6.0/flink-statebackend-rocksdb_2.11-1.6.0.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/org/rocksdb/rocksdbjni/5.7.5/rocksdbjni-5.7.5.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/appleyuchi/bigdata/apache-maven-3.6.3/jar_warehouse/com/alibaba/fastjson/1.2.58/fastjson-1.2.58.jar henry.flink.DataClean
  2. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
  3. SLF4J: Defaulting to no-operation (NOP) logger implementation
  4. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
  5. Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Could not retrieve JobResult.
  6.     at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:617)
  7.     at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:123)
  8.     at henry.flink.DataClean.main(DataClean.java:126)
  9. Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
  10.     at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$submitJob$2(Dispatcher.java:256)
  11.     at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
  12.     at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
  13.     at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
  14.     at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
  15.     at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:690)
  16.     at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
  17.     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:332)
  18.     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:158)
  19.     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
  20.     at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
  21.     at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
  22.     at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
  23.     at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
  24.     at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
  25.     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
  26.     at akka.actor.ActorCell.invoke(ActorCell.scala:495)
  27.     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
  28.     at akka.dispatch.Mailbox.run(Mailbox.scala:224)
  29.     at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
  30.     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
  31.     at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
  32.     at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
  33.     at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  34. Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
  35.     at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
  36.     at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
  37.     at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:708)
  38.     at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:687)
  39.     ... 18 more
  40. Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
  41.     at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:199)
  42.     at org.apache.flink.util.function.ConsumerWithException.accept(ConsumerWithException.java:40)
  43.     at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$waitForTerminatingJobManager$29(Dispatcher.java:820)
  44.     at java.util.concurrent.CompletableFuture.uniRun(CompletableFuture.java:705)
  45.     ... 19 more
  46. Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
  47.     at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:176)
  48.     at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:936)
  49.     at org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:291)
  50.     at org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:281)
  51.     at org.apache.flink.runtime.dispatcher.Dispatcher.persistAndRunJob(Dispatcher.java:266)
  52.     at org.apache.flink.util.function.ConsumerWithException.accept(ConsumerWithException.java:38)
  53.     ... 21 more
  54. Caused by: java.lang.RuntimeException: Failed to start checkpoint ID counter: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
  55.     at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:254)
  56.     at org.apache.flink.runtime.executiongraph.ExecutionGraph.enableCheckpointing(ExecutionGraph.java:495)
  57.     at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:345)
  58.     at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
  59.     at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1152)
  60.     at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1132)
  61.     at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:295)
  62.     at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
  63.     ... 26 more
  64. Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
  65.     at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:403)
  66.     at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
  67.     at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
  68.     at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:61)
  69.     at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:443)
  70.     at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:399)
  71.     at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:248)
  72.     ... 33 more
  73. Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
  74.     at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:64)
  75.     at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:399)
  76.     ... 39 more
  78. Process finished with exit code 1

解决方案:

Flink1.12新增的jar是flink-shaded-hadoop-3-uber-3.1.1.7.0.3.0-79-7.0.jar

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FwcGxleXVjaGk_size_16_color_FFFFFF_t_70

发表评论

表情：
评论列表 （有 0 条评论，319人围观）

还没有评论，来说两句吧...

相关阅读