[CDH] CDH5集成Spark2.2后spark-shell启动报错解决

Love The Way You Lie 2022-04-17 05:39 192阅读 0赞

CDH5已集成Spark2.2，spark-shell启动报错：

[root@hadoop01 bin]# ./spark2-shell 
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
    	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
    	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
    	at scala.Option.getOrElse(Option.scala:121)
    	at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
    	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
    	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 7 more
    [root@hadoop01 bin]#

**原因分析：**Spark1.4以后，所有spark的编译都是没有将hadoop的classpath编译进去的。所以必须在spark-env.sh中指定hadoop中的所有jar包。

**解决：**去CM网页上修改Spark2.2配置，指定SPARK\_DIST\_CLASSPATH，然后重启过期配置。

export SPARK_DIST_CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath)

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhd2E4ODk5_size_16_color_FFFFFF_t_70][]

再次启动spark-shell，报错已经不存在了

[root@hadoop01 bin]# ./spark2-shell 
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    Spark context Web UI available at http://192.168.1.8:4040
    Spark context available as 'sc' (master = local[*], app id = local-1542104414574).
    Spark session available as 'spark'.
    Welcome to
          ____              __
         / __/__  ___ _____/ /__
        _\ \/ _ \/ _ `/ __/  '_/
       /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera2
          /_/
             
    Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_151)
    Type in expressions to have them evaluated.
    Type :help for more information.
    
    scala> :q
    [root@hadoop01 bin]#

参考官网： [https://spark.apache.org/docs/latest/hadoop-provided.html][https_spark.apache.org_docs_latest_hadoop-provided.html]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhd2E4ODk5_size_16_color_FFFFFF_t_70]: /images/20220417/caac04be45904172935b5b0c28c7b8a3.png
[https_spark.apache.org_docs_latest_hadoop-provided.html]: https://spark.apache.org/docs/latest/hadoop-provided.html