Windows 平台运行spark-shell 报"java.lang.NullPointerException, not found: value sqlContext" error 解决办法

朴灿烈づ我的快乐病毒、 2022-07-13 08:16 180阅读 0赞

#### **错误背景描述** ####

在spark 官网下载 spark的安装版：spark-1.6.3-bin-hadoop2.6.tgz，然后解压，配好环境变量，在window下CMD下运行spark-shell,启动spark失败（本地模式）。然后搜索了一下，发现中文基本找不到什么参考资料，不过在stackoverflow上找到了解决办法，遂记录下，供国内的同学参考下  
![这里写图片描述][SouthEast]

#### **错误信息** ####

ava.lang.NullPointerException (not found: value sqlContext)
    
    java.lang.RuntimeException: java.lang.NullPointerException
            at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
    at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
    at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
    at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
    at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
    at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
    at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
    at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
    at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
    at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
    at $iwC$$iwC.<init>(<console>:15)
    at $iwC.<init>(<console>:24)
    at <init>(<console>:26)
    at .<init>(<console>:30)
    at .<clinit>(<console>)
    at .<init>(<console>:7)
    at .<clinit>(<console>)
    at $print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
    at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132)
    at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
    at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
    at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
    at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
    at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
    at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
    at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
    at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.NullPointerException
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
    at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:582)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:557)
    at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    … 62 more
    
    <console>:16: error: not found: value sqlContext
    import sqlContext.implicits._
    ^
    <console>:16: error: not found: value sqlContext
    import sqlContext.sql

#### **错误解决办法** ####

出现这种错误通常是因为缺少 winutils.exe文件，而这个文件是Spark用来初始化Hive的上下文，而这又取决于Hadoop，这就需要Windows上的本机库正常工作。不幸的是，即使您在本地模式下使用Spark而不直接使用任何HDFS功能，也会发生这种情况。所以为了解决这个问题，你需要按如下步骤做。

1.  下载winutils.exe，注意32位要下载32位的,64位要下载64位的  
    [download the 64-bit winutils.exe ][download the 64-bit winutils.exe] 32位的自己去github上找找都能找到的
2.  在你的本地建立hadoop/bin 文件夹，然后把winutils.exe放进去。比如我的是C:\\hadoop\\bin,然后去建立一个系统环境变量HADOOP\_HOME，把你刚才的新建的文件夹的路径去掉bin ,例如：文件夹路径是C:\\hadoop\\bin那么HADOOP\_HOME就是C:\\hadoop，文件夹路径是C:\\spark\\hadoop\\bin那么  
    HADOOP\_HOME就是C:\\spark\\hadoop，建好系统变量以后去CMD下运行  
    echo %HADOOP\_HOME%,检查是否正确。  
    ![这里写图片描述][SouthEast 1]

3.检查你的C盘是否有C:\\tmp\\hive文件夹，没有新建之。一般都是有的，然后右键选择管理员获取所有权，最后在CMD下运行一下命令就可以了

`%HADOOP_HOME%\bin\winutils.exe chmod 777 \tmp\hive`这个命令的意思是让winutils.exe文件对 \\tmp\\hive有读写权限。

4.重新运行spark-shell 会出现一下画面就表示启动成功了  
![这里写图片描述][SouthEast 2]

如果不放心，在浏览器中运行[http://localhost:4040][http_localhost_4040]出现一下画面也表示启动成功。

![这里写图片描述][SouthEast 3]

#### **参考英文资料：** ####

1.  [stackoverflow][]
2.  [blogs.msdn.microsoft.com][]

[SouthEast]: /images/20220713/995a412dd49d48a6bbcd911040e7c1f2.png
[download the 64-bit winutils.exe]: https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin
[SouthEast 1]: /images/20220713/137cfa8368e6426c8cdc810ab0dd0654.png
[SouthEast 2]: /images/20220713/40da404028184abaa555ab4157eba2b1.png
[http_localhost_4040]: http://localhost:4040
[SouthEast 3]: /images/20220713/47ccf7da5c954d0fad5dc7bb0b500919.png
[stackoverflow]: http://stackoverflow.com/questions/32721647/how-to-start-spark-applications-on-windows-aka-why-spark-fails-with-nullpointer
[blogs.msdn.microsoft.com]: https://blogs.msdn.microsoft.com/arsen/2016/02/09/resolving-spark-1-6-0-java-lang-nullpointerexception-not-found-value-sqlcontext-error-when-running-spark-shell-on-windows-10-64-bit/