详解：Spark程序的开始 SparkContext 源码走一走

绝地灬酷狼 2024-04-17 14:45 36阅读 0赞

## Spark程序的开始 SparkContext 源码走一走 ##

### 什么是SparkContext ###

IDEA中看看源码  
\*\*

*  Main entry point for Spark functionality. A SparkContext represents the connection to a Spark
 *  cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster.
 *  
 *  Only one SparkContext may be active per JVM. You must `stop()` the active SparkContext before
 *  creating a new one. This limitation may eventually be removed; see SPARK-2243 for more details.

### 什么是SparkConf ###

/\*\*

*  Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
 *  
 *  Most of the time, you would create a SparkConf object with `new SparkConf()`, which will load
 *  values from any `spark.*` Java system properties set in your application as well. In this case,
 *  parameters you set directly on the `SparkConf` object take priority over system properties.
 *  
 *  For unit tests, you can also call `new SparkConf(false)` to skip loading external settings and
 *  get the same configuration no matter what the system properties are.
 *  
 *  All setter methods in this class support chaining. For example, you can write
 *  `new SparkConf().setMaster("local").setAppName("My app")`.
 *

Spark的官网：http://spark.apache.org/docs/2.4.0/rdd-programming-guide.html  
使用的是spark2.4.0版本  
这里不得说一句：spark的官网写的是非常好的，还有就是源码虽然很枯燥，但是很有用处  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70]

### IDEA 上面操作一下吧 ###

ackage Spark.learning
    
    object SparkContextApp {
      def main(args: Array[String]): Unit = {
        import org.apache.spark.{SparkConf, SparkContext}
    
        val sparkConf = new SparkConf().setMaster("local[2]").setAppName("SparkContextApp")
        val sc =  new SparkContext(sparkConf)
    
    
        sc.stop()
      }
    
    }

### 其中 ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70 1]  
这里是Windows系统都存在的问题，可以忽略掉的  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70 2]  
下面Ok就可以了  
希望你有一个好的 开始

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/5714377d3cca41b5a776e1163853890b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/823d89e9a7f442b99e2187b154124494.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzNjg4NDcy_size_16_color_FFFFFF_t_70 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/8eae093f9fc4419b8bc54b5005cdb105.png