Spark——部署详解及问题解决

超、凢脫俗 2022-05-26 10:47 393阅读 0赞

本节目标：

掌握Spark的部署方式

所需软件：spark-2.2.1-bin-hadoop2.7

<table> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> NN </td> 
 <td style="vertical-align:top;"> DN </td> 
 <td style="vertical-align:top;"> JN </td> 
 <td style="vertical-align:top;"> ZK </td> 
 <td style="vertical-align:top;"> ZKFC </td> 
 <td style="vertical-align:top;"> RM </td> 
 <td style="vertical-align:top;"> NM </td> 
 <td style="vertical-align:top;"> Master </td> 
 <td style="vertical-align:top;"> Worker </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> master(192.168.85.10) </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> slave1(192.168.85.11) </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp;&nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> slave2(192.168.85.12) </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> slave3(192.168.85.13) </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 </tr> 
 </tbody> 
</table>

Spark部署方式主要有三种：local模式，standalone模式，spark on yarn 模式。

# 1、local单机模式： #

解压安装即可  
测试：./bin/spark-submit --class org.apache.spark.examples.SparkPi \--master local\[1\] ./examples/jars/spark-examples\_2.11-2.2.1.jar 100

# 2、standalone集群式： #

1.每台解压安装spark  
    注意：环境变量可不用配置，start-all.sh容易与hadoop命令冲突  
3.cp conf/spark-env.sh.template conf/spark-env.sh  
4.修改spark-env.sh  
export JAVA\_HOME=/usr/local/src/jdk1.8  
export SPARK\_MASTER\_IP=master  
export SPARK\_MASTER\_PORT=7077  
export SPARK\_WORKER\_CORES=1  
export SPARK\_WORKER\_INSTANCES=1  
export SPARK\_WORKER\_MEMORY=1g  
5.修改slaves

salve1

slave2

slave3

6.配置文件同步到其他节点

7.启动

./sbin/start-all.sh

8.webUI:

master:8080

9.HA配置  
spark-env.sh添加：

export SPARK\_DAEMON\_JAVA\_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=master:2181,slave1:2181,slave2:2181"

10.同步spark-env.sh配置文件

11.slave1热备节点spark-env.sh修改：  
export SPARK\_MASTER\_IP=slave1  
12.启动热备节点  
./sbin/start-master.sh

因为用到zk集群，启动spark前，先启动zk。

13.测试故障是否自动切换

![2018050110443128][]

![20180501104441221][]

14、测试任务提交

standalone模式分为两种运行方式：  
client方式:运行结果在客户端直接可见  
测试：./bin/spark-submit --class org.apache.spark.examples.SparkPi \--master spark://master:7077 ./examples/jars/spark-examples\_2.11-2.2.1.jar 100

cluster方式：运行结果不能直接可见,可在webUI中查到

测试：./bin/spark-submit --class org.apache.spark.examples.SparkPi \--master spark://master:7077 --deploy-mode cluster ./examples/jars/spark-examples\_2.11-2.2.1.jar 100

# 3、spark on yarn模式： #

基于Hadoop部署  
1.spark-env.sh增加  
export HADOOP\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop  
export YARN\_CONF\_DIR=$HADOOP\_HOME/etc/hadoop  
export SCALA\_HOME=/usr/local/src/scala-2.11.12  
export SPARK\_HOME=/usr/local/src/spark-2.2.1-bin-hadoop2.7  
export PATH=$SPARK\_HOME/bin:$PATH  
export SPARK\_JAR=$SPARK\_HOME/jars/\*  
2.同步配置文件  
3.启动hadoop，此时不应启动spark，否则会造成资源管理器冲突。  
4.测试  
client:运行结果在客户端直接可见

测试：./bin/spark-submit --class org.apache.spark.examples.SparkPi \--master yarn --deploy-mode client ./examples/jars/spark-examples\_2.11-2.2.1.jar 100

cluster：运行结果不能直接可见

测试：./bin/spark-submit --class org.apache.spark.examples.SparkPi \--master yarn \--deploy-mode cluster ./examples/jars/spark-examples\_2.11-2.2.1.jar 100

![20180501103044394][]

可能出现的报错：

ERROR spark.SparkContext: Error initializing SparkContext.
    org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

For more detailed output, check application tracking page:http://slave3:8088/cluster/app/application_1525099301044_0003Then, click on links to logs of each attempt.
    Diagnostics: Container [pid=2391,containerID=container_1525099301044_0003_02_000001] is running beyond virtual memory limits. Current usage: 40.8 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

该错误是YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。

解决方式：

1、修改yarn-site.xml

yarn配置文件中 containers 的默认属性有关,被强制限定了物理内存

![20180501100702589][]

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
 </property>
 <property>
     <name>yarn.nodemanager.pmem-check-enabled</name>
     <value>false</value>
 </property>
 <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
 </property>

2、SPARK\_HOME/conf下的spark-defaults.conf.template，改名为spark-defaults.conf，修改并同步

![20180501101006754][]

官方默认为512m

![2018050110475811][]

[2018050110443128]: /images/20220526/65ba09d6d5164c1cb7f24c3a869eebce.png
[20180501104441221]: /images/20220526/74da9a3d070f43f0a2c7532e353dc8d5.png
[20180501103044394]: /images/20220526/2a320226492a4287b119a835d766d101.png
[20180501100702589]: /images/20220526/2a03382392e04582993124c121fa1828.png
[20180501101006754]: /images/20220526/58f86510ca854fb0b418f9560ec736db.png
[2018050110475811]: /images/20220526/aca6685ecd474846990c36acbb84e4a7.png