Spark在windows上默认使用IPython编辑器

喜欢ヅ旅行 2022-05-28 03:58 211阅读 0赞

#### Spark在windows上默认使用IPython编辑器 ####

在`%SPARK_HOME%\bin`目录下，有`pyspark`，`pyspark.cmd`，`pyspark2.cmd`文件。

其中pyspark是linux操作系统或Mac当中使用。

`pyspark.cmd`和`pyspark2.cmd`是windows操作系统中使用。

其中pyspark.cmd文件如下：

@echo off
    
    rem
    rem Licensed to the Apache Software Foundation (ASF) under one or more
    rem contributor license agreements.  See the NOTICE file distributed with
    rem this work for additional information regarding copyright ownership.
    rem The ASF licenses this file to You under the Apache License, Version 2.0
    rem (the "License"); you may not use this file except in compliance with
    rem the License.  You may obtain a copy of the License at
    rem
    rem    http://www.apache.org/licenses/LICENSE-2.0
    rem
    rem Unless required by applicable law or agreed to in writing, software
    rem distributed under the License is distributed on an "AS IS" BASIS,
    rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    rem See the License for the specific language governing permissions and
    rem limitations under the License.
    rem
    
    rem This is the entry point for running PySpark. To avoid polluting the
    rem environment, it just launches a new cmd to do the real work.
    
    rem The outermost quotes are used to prevent Windows command line parse error
    rem when there are some quotes in parameters, see SPARK-21877.
    cmd /V /E /C ""%~dp0pyspark2.cmd" %*"

实际上执行的是pyspark2.cmd文件。

--------------------

那么我们看下pyspark2.cmd文件内容：

@echo off
    
    rem
    rem Licensed to the Apache Software Foundation (ASF) under one or more
    rem contributor license agreements.  See the NOTICE file distributed with
    rem this work for additional information regarding copyright ownership.
    rem The ASF licenses this file to You under the Apache License, Version 2.0
    rem (the "License"); you may not use this file except in compliance with
    rem the License.  You may obtain a copy of the License at
    rem
    rem    http://www.apache.org/licenses/LICENSE-2.0
    rem
    rem Unless required by applicable law or agreed to in writing, software
    rem distributed under the License is distributed on an "AS IS" BASIS,
    rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    rem See the License for the specific language governing permissions and
    rem limitations under the License.
    rem
    
    rem Figure out where the Spark framework is installed
    call "%~dp0find-spark-home.cmd"
    
    call "%SPARK_HOME%\bin\load-spark-env.cmd"
    set _SPARK_CMD_USAGE=Usage: bin\pyspark.cmd [options]
    
    rem Figure out which Python to use.
    if "x%PYSPARK_DRIVER_PYTHON%"=="x" (
      set PYSPARK_DRIVER_PYTHON=python
      if not [%PYSPARK_PYTHON%] == [] set PYSPARK_DRIVER_PYTHON=%PYSPARK_PYTHON%
    )
    
    set PYTHONPATH=%SPARK_HOME%\python;%PYTHONPATH%
    set PYTHONPATH=%SPARK_HOME%\python\lib\py4j-0.10.6-src.zip;%PYTHONPATH%
    
    set OLD_PYTHONSTARTUP=%PYTHONSTARTUP%
    set PYTHONSTARTUP=%SPARK_HOME%\python\pyspark\shell.py
    
    call "%SPARK_HOME%\bin\spark-submit2.cmd" pyspark-shell-main --name "PySparkShell" %*

我们可以看到：`set PYSPARK_DRIVER_PYTHON=python`将python设置给driver驱动。

**我们想使用`IPython`编辑器，我们将值改为`ipython`就行**。

set PYSPARK_DRIVER_PYTHON=ipython

**设置好后，重新使用pyspark打开的dos窗口，就是IPython编辑器了**。

![image][]

[image]: https://raw.githubusercontent.com/zhang3550545/image_center/master/image/pyspark-windows.png