第6.1.4章 spark环境配置

秒速五厘米 2021-12-12 15:45 237阅读 0赞

spark与hive部署在一台服务器上。  
**1 hive-site.xml**  
这里可以看到spark和hive使用的是thrift协议通讯

<configuration>
      <property>
        <name>hive.metastore.uris</name>
        <value>thrift://bwsc79:9083,thrift://bwsc80:9083,thrift://bwsc81:9083</value>
        <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
      </property>
    </configuration>

**2 hbase-site.xml**  
从下面的配置可以看出，spark中的hbase的配置，是从hbase拷贝过来，可以看看[第3.1.1章 hbase环境准备][3.1.1_ hbase]

<configuration>
            <property>
                    <name>hbase.rootdir</name>
                    <value>hdfs://bwsc65:9000/hbase</value>
            </property>
            <property>
                    <name>hbase.cluster.distributed</name>
                    <value>true</value>
            </property>
            <property>
          <name>hbase.zookeeper.property.clientPort</name>
          <value>2181</value>
        </property>
            <property>
                    <name>hbase.zookeeper.property.dataDir</name>
            <value>/appdata/zookeeper/data</value>
            </property>
            <property>
                    <name>hbase.zookeeper.quorum</name>
            <value>bwsc65,bwsc66,bwsc67</value>
            </property>
            <property>
            <name>hbase.block.data.cachecompressed</name>
            <value>true</value>
            </property>
            <property>
                 <name>hbase.regionserver.codecs</name>
                 <value>snappy</value>
            </property>
    </configuration>

**3 [spark-env.sh][]**

vi /etc/profile
    export HIVE_HOME=/application/hive
    export PATH=$PATH:$HIVE_HOME/bin
    
    export SPARK_CLUSTER_HOME=/application/spark
    source /etc/profile

这里看到spark使用yarn做资源调度，这里配置了hadoop的jar，所以配置对应的classpath，但zk这里用的是真是的hadoop集群依赖的。

export LD_LIBRARY_PATH="$HADOOP_HOME/lib/native"
    export SPARK_DIST_CLASSPATH=$(/application/hadoop/bin/hadoop classpath)
    export SPARK_SSH_OPTS="-p 52113"
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=bwsc65:2181,bwsc66:2181,bwsc67:2181 -Dspark.deploy.zookeeper.dir=/spark"
    export SPARK_LOG_DIR=/appdata/spark-cluster/logs
    export SPARK_WORKER_CORES=6
    export SPARK_WORKER_MEMORY=6144m
    export SPARK_WORKER_PORT=9090
    export SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true"

**4 spark-defaults.conf**

spark.master spark://bwsc79:6066,bwsc80:6066,bwsc81:6066
    spark.submit.deployMode cluster
    spark.eventLog.enabled false
    spark.eventLog.dir hdfs://bwsc65:9000/spark-cluster/logs
    spark.serializer org.apache.spark.serializer.KryoSerializer
    spark.jars $SPARK_HOME/lib/*.jar
    
    #spark.cores.max 1
    spark.deploy.defaultCores 2
    spark.driver.cores 1
    spark.driver.memory 512m
    spark.driver.supervise true
    
    #executor cores = spark.max.cores / spark.executor.cores
    spark.executor.cores 1
    spark.executor.memory 512m
    
    #scheduler
    spark.scheduler.mode FAIR
    spark.task.maxFailures 4

**5 slaves**  
三个节点一样

bwsc79
    bwsc80
    bwsc81

[3.1.1_ hbase]: https://blog.csdn.net/warrah/article/details/71487541
[spark-env.sh]: http://spark-env.sh