大数据集群之spark2.4.0高可用安装配置

小灰灰 2023-02-20 08:28 11阅读 0赞

### 一、安装准备 ###

下载地址：[https://archive.apache.org/dist/spark/][https_archive.apache.org_dist_spark]

官方文档：[http://spark.apache.org/docs/latest/][http_spark.apache.org_docs_latest]

Scala2.12安装：[https://blog.csdn.net/qq262593421/article/details/105707274][https_blog.csdn.net_qq262593421_article_details_105707274]

Spark 2.4.0：[https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz][https_archive.apache.org_dist_spark_spark-2.4.0_spark-2.4.0-bin-hadoop2.7.tgz]

Spark 3.0.0：[https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz][https_archive.apache.org_dist_spark_spark-3.0.0_spark-3.0.0-bin-hadoop3.2.tgz]

注意：Spark2.4.0依赖Scala2.11环境，Spark3.0.0依赖Scala2.12环境，亲测本文适用于2.4.0和3.0.0两个版本

### 二、解压安装 ###

解压缩文件

cd /usr/local/hadoop
    tar zxpf spark-2.4.0-bin-hadoop2.7.tgz

2、创建软链接

ln -s spark-2.4.0-bin-hadoop2.7 spark

### 三、修改配置文件 ###

**slaves**

# conf/slaves
    hadoop003
    hadoop004
    hadoop005
    hadoop006

**spark-env.sh**

export JAVA_HOME=/usr/java/jdk1.8
    export SCALA_HOME=/usr/local/hadoop/scala
    export MYSQL_HOME=/usr/local/mysql
    export CLASSPATH=.:/usr/java/jdk1.8/lib/dt.jar:/usr/java/jdk1.8/lib/tools.jar
    export SPARK_HOME=/usr/local/hadoop/spark
    export HADOOP_HOME=/usr/local/hadoop/hadoop
    export HBASE_HOME=/usr/local/hadoop/hbase
    export GEOMESA_HBASE_HOME=/usr/local/hadoop/geomesa-hbase
    export ZOO_HOME=/usr/local/hadoop/zookeeper
    
    export SPARK_WORKING_MEMORY=16G
    # export SPARK_MASTER_IP=hadoop001
    export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop/etc/hadoop/
    export YARN_CONF_DIR=/usr/local/hadoop/hadoop/etc/hadoop/
    export SPARK_LOCAL_DIRS=/home/spark/tmp
    
    export SPARK_HISTORY_OPTS="
    -Dspark.history.ui.port=18080 
    -Dspark.history.fs.logDirectory=hdfs://ns1/spark/directory 
    -Dspark.history.retainedApplications=30"
    
    SPARK_MASTER_WEBUI_PORT=8989
    
    export SPARK_DAEMON_JAVA_OPTS="
    -Dspark.deploy.recoveryMode=ZOOKEEPER 
    -Dspark.deploy.zookeeper.url=hadoop001,hadoop002,hadoop003
    -Dspark.deploy.zookeeper.dir=/spark"

**metrics.properties**

*.sink.csv.directory=/home/spark/tmp/csv/

**spark-defaults.conf**

spark.local.dir				        /home/spark/tmp
    
    spark.eventLog.enabled 				true
    spark.eventLog.dir 				hdfs://ns1/spark/directory
    spark.yarn.jars					hdfs://ns1/spark/jars/*.jar
    spark.serializer                                org.apache.spark.serializer.KryoSerializer

### 四、环境变量配置 ###

编辑 /etc/profile 文件

vim /etc/profile

添加以下内容

export SPARK_HOME=/usr/local/hadoop/spark
    export PATH=$PATH:$SPARK_HOME

source /etc/profile

echo $SPARK_HOME

### 五、上传Spark jar包 ###

hadoop fs -mkdir /spark/jars

hadoop fs -put $SPARK_HOME/jars/* /spark/jars/

### 六、启动spark ###

在master上执行

$SPARK_HOME/sbin/start-all.sh

在备用master上执行

$SPARK_HOME/sbin/start-master.sh

在master上启动日志服务

$SPARK_HOME/sbin/start-history-server.sh

### 七、验证安装 ###

spark-shell

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70][]

本地模式运行

spark-submit \
    --class org.apache.spark.examples.SparkPi \
    --master local[*] \
    ${SPARK_HOME}/examples/jars/spark-examples_*.jar \
    10

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 1][]

spark-submit \
    --class org.apache.spark.examples.SparkPi \
    --master spark://hadoop001:7077 \
    ${SPARK_HOME}/examples/jars/spark-examples_*.jar \
    10

指定Mater运行

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 2][]

Spark On Yarn模式运行

spark-submit \
    --class org.apache.spark.examples.SparkPi \
    --master yarn \
    --deploy-mode cluster \
    ${SPARK_HOME}/examples/jars/spark-examples_*.jar \
    10

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 3][]

Spark Kill Application

yarn application -kill <applicationId>

查看Spark的两个Master和HistoryServer的WebUI

[http://hadoop001:8989/][http_hadoop001_8989]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 4][]

[http://hadoop002:8989/][http_hadoop002_8989]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 5][]

[http://hadoop001:18080/][http_hadoop001_18080]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 6][]

[https_archive.apache.org_dist_spark]: https://archive.apache.org/dist/spark/
[http_spark.apache.org_docs_latest]: http://spark.apache.org/docs/latest/
[https_blog.csdn.net_qq262593421_article_details_105707274]: https://blog.csdn.net/qq262593421/article/details/105707274
[https_archive.apache.org_dist_spark_spark-2.4.0_spark-2.4.0-bin-hadoop2.7.tgz]: https://archive.apache.org/dist/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
[https_archive.apache.org_dist_spark_spark-3.0.0_spark-3.0.0-bin-hadoop3.2.tgz]: https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20210520215204823.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20210520214534329.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20210520214750737.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20210520214442734.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx,size_16,color_FFFFFF,t_70
[http_hadoop001_8989]: http://hadoop001:8989/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/2020062521485743.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx,size_16,color_FFFFFF,t_70
[http_hadoop002_8989]: http://hadoop002:8989/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200625214833489.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx,size_16,color_FFFFFF,t_70
[http_hadoop001_18080]: http://hadoop001:18080/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20210520204412995.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxMjYyNTkzNDIx,size_16,color_FFFFFF,t_70