Spark入门（二）多主standalone安装

╰半夏微凉° 2023-07-02 08:27 114阅读 0赞

### 一、集群安装条件前置 ###

实验spark安装在【[Hadoop生态Zookeeper安装][Hadoop_Zookeeper]】机器上， 已完成安装zookeeper、jdk、hadoop和ssh、网络等配置环境等。

spark所依赖的虚拟机和操作系统配置

环境：ubuntu14 + spark-2.4.4-bin-hadoop2.6 + apache-zookeeper-3.5.6 + jdk1.8+ssh

虚拟机：（vmware10）

### 二、standalone安装环境设置 ###

**（1）下载spark**

[spark][]官网，由于本人使用hadoop2.6，所有下载[spark-2.4.4-bin-hadoop2.6][]。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70][]

**（2）上传到linux系统解压**

tar xvf spark-2.4.4-bin-hadoop2.6.tar.gz 
    #放在统一的软件目录下
    mv spark-2.4.4-bin-hadoop2.6 software/
    #别名
    ln -s software/spark-2.4.4-bin-hadoop2.6 spark-2.4.4

**（3） 配置spark-env.sh**

cd ~/spark-2.4.4/conf/
    cp spark-env.sh.template spark-env.sh
    vim spark-env.sh

编辑spark-env.sh添加以下内容

#与zk的adminServer端口8080冲突改为8082
    SPARK_MASTER_WEBUI_PORT="8082"
    #单机器work的数量
    SPARK_WORK_INSTANCES="1"
    #zk选spark主
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop01:2181,hadoop02:2181,hadoop03:2181 -Dspark.deploy.zooleeper.dir=/spark"
    
    export JAVA_HOME=/home/mk/jdk1.8

**（4）配置slaves**

cd ~/spark-2.4.4/conf/
    cp slaves.template slaves
    vim slaves

编辑slaves添加以下内容

hadoop01
    hadoop02
    hadoop03

**（5）复制到hadoop02、hadoop03**

scp -r /home/mk/software/spark-2.4.4-bin-hadoop2.6 mk@hadoop02:/home/mk/software/
    scp -r /home/mk/software/spark-2.4.4-bin-hadoop2.6 mk@hadoop03:/home/mk/software/
    
    #hadoop02机器别名
    ln -s software/spark-2.4.4-bin-hadoop2.6 spark-2.4.4
    
    #hadoop03机器别名
    ln -s software/spark-2.4.4-bin-hadoop2.6 spark-2.4.4

### 三、启动spark ###

**（1）启动zookeeper**

hadoop01、hadoop02、hadoop03分别启动zk

~/software/apache-zookeeper-3.5.6-bin/bin/zkServer.sh  start

**（2）启动spark master**

**hadoop01启动master**

~/spark-2.4.4/sbin/start-master.sh

![2020012816433599.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 3][]

**hadoop02启动master**

![20200128164945340.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 5][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 6][]

**hadoop03启动master**

![20200128165227419.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 7][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 9][]

master启动成功。

sparkUI浏览器进行访问[http://hadoop01:8080/][http_hadoop01_8080]

**（2）在集群主master启动worker**

注意：一定要在主节点上启动worker，否则worker不能注册到主节点。

应该在启动第一个master节点的时候使用start-all.sh代替start-master.sh+start-slaves.sh，避免worker在非主节点启动注册不上

![20200128165755719.png][]

**hadoop01检查**

![20200128165847550.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 10][]

**hadoop02检查**

![2020012817004027.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 11][]

**hadoop03检查**

![20200128170152622.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 12][]

**（3）运行spark计算PI例子**

~/spark-2.4.4/bin/run-example --master spark://hadoop01:7077,hadoop02:7077,hadoop03:7077 SparkPi

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 13][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 14][]

Pi is roughly 3.1402157010785055

**（4）停止hadoop01 master和worker检查主节点切换**

#hadoop01停止master和worker
    ~/software/spark-2.4.4-bin-hadoop2.6/sbin/stop-master.sh
    ~/software/spark-2.4.4-bin-hadoop2.6/sbin/stop-slave.sh
    
    #hadoop03监听主节点切换日志
    tail -f /home/mk/software/spark-2.4.4-bin-hadoop2.6/logs/spark-mk-org.apache.spark.deploy.worker.Worker-1-hadoop03.out

![20200128170919951.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 15][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 16][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 17][]

**（5）在hadoop01重新执行计算PI**

spark-2.4.4/bin/run-example --master spark://hadoop01:7077,hadoop02:7077,hadoop03:7077 SparkPi

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 18][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 19][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 20][]

**（6）关闭spark**

#主节点执行
    ~/spark-2.4.4/sbin/stop-all.sh
    #或者
    ~/spark-2.4.4/sbin/stop-master.sh
    ~/spark-2.4.4/sbin/stop-slaves.sh
    #其他master节点执行
    ~/spark-2.4.4/sbin/stop-master.sh

[Hadoop_Zookeeper]: https://blog.csdn.net/moakun/article/details/104098878
[spark]: http://spark.apache.org/downloads.html
[spark-2.4.4-bin-hadoop2.6]: http://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.6.tgz
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200127213504363.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[2020012816433599.png]: https://img-blog.csdnimg.cn/2020012816433599.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200128164436827.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200128164504485.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200128164909895.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[20200128164945340.png]: https://img-blog.csdnimg.cn/20200128164945340.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200128165044492.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/2020012816512119.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200128165155755.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[20200128165227419.png]: https://img-blog.csdnimg.cn/20200128165227419.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200128165408482.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200128165447246.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200128165557597.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[http_hadoop01_8080]: http://hadoop01:8080/
[20200128165755719.png]: https://img-blog.csdnimg.cn/20200128165755719.png
[20200128165847550.png]: https://img-blog.csdnimg.cn/20200128165847550.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200128165931682.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[2020012817004027.png]: https://img-blog.csdnimg.cn/2020012817004027.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200128170051497.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[20200128170152622.png]: https://img-blog.csdnimg.cn/20200128170152622.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200128170204283.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200128170344101.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200128170432159.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[20200128170919951.png]: https://img-blog.csdnimg.cn/20200128170919951.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200128171006971.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/2020012817103738.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20200128171422537.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20200128171933120.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20200128172029493.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20200128172130504.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9tYW9rdW4uYmxvZy5jc2RuLm5ldA==,size_16,color_FFFFFF,t_70