CDH 升级pyspark带来的问题

迈不过友情╰ 2022-10-06 05:45 83阅读 0赞

### 文章目录 ###

*  一.问题描述
 *  二.解决方案
 *   *  2.1 删除spark
     *  2.2 重启整个集群
     *  2.3 重新添加spark服务

# 一.问题描述 #

最近在自学pyspark，想本地通过pycharm来编辑pyspark脚本，运行的时候，提示没有pyspark模块，于是通过pycharm安装了pyspark(最新版本)。

**而且这个安装到了远程服务器上，真的坑**

奈何我安装的是CDH 6.3.1版本，然后spark版本是 2.4.0-cdh6.3.1，然后最新版本是3.0.2的  
![image.png][]

新的pyspak程序，都未记录到 History Server服务上。  
![image.png][image.png 1]

重新安装了pyspark 2.4.0版本，依旧未记录到 History Server服务上。

[root@hp1 software]# pyspark
    Python 2.7.5 (default, Apr  2 2020, 13:16:51) 
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    2021-04-09 17:22:21 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    Welcome to
          ____              __
         / __/__  ___ _____/ /__
        _\ \/ _ \/ _ `/ __/  '_/
       /__ / .__/\_,_/_/ /_/\_\   version 2.4.0
          /_/
    
    Using Python version 2.7.5 (default, Apr  2 2020 13:16:51)
    SparkSession available as 'spark'.
    >>>

# 二.解决方案 #

首先想到的问题是卸载spark，然后重新安装

## 2.1 删除spark ##

![image.png][image.png 2]

![image.png][image.png 3]

![image.png][image.png 4]

更改配置后重新先停止然后删除spark  
![image.png][image.png 5]

![image.png][image.png 6]

**界面删除了spark，依旧可以运行pyspark**

[root@hp2 ~]# pyspark
    Python 2.7.5 (default, Apr  2 2020, 13:16:51) 
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    Setting default log level to "WARN".
    To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
    Welcome to
          ____              __
         / __/__  ___ _____/ /__
        _\ \/ _ \/ _ `/ __/  '_/
       /__ / .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.3.1
          /_/
    
    Using Python version 2.7.5 (default, Apr  2 2020 13:16:51)
    SparkSession available as 'spark'.
    >>> exit()

## 2.2 重启整个集群 ##

![image.png][image.png 7]

## 2.3 重新添加spark服务 ##

![image.png][image.png 8]

![image.png][image.png 9]

![image.png][image.png 10]

**重启整个集群**

[image.png]: /images/20221005/ee0482e7b0d64197919a969360820dbd.png
[image.png 1]: /images/20221005/bd507cc2787541ad89fa5457a671d6a6.png
[image.png 2]: /images/20221005/fbfd8ad6258f4bbba81f44bb5c006931.png
[image.png 3]: /images/20221005/dc7a34ce3ef0400484b4072d1bf1c18d.png
[image.png 4]: /images/20221005/318c682bb0dd444d8f210873fb92e14f.png
[image.png 5]: /images/20221005/57915e09b41346a4870cc51df49cf694.png
[image.png 6]: /images/20221005/455044e5236041dbaaaaed6d24a6dd7e.png
[image.png 7]: /images/20221005/ebc8f7f1c9a54308b59b4f259a0dd120.png
[image.png 8]: /images/20221005/ea0ca7f1d63a44878354bbbed0fc5d2a.png
[image.png 9]: /images/20221005/5be31594d23349788e015778a1494124.png
[image.png 10]: /images/20221005/fb207e658f324217ae2ee9f3a14685fd.png