Spark distinct去重原理 (distinct会导致shuffle)

た入场券 2021-09-25 01:22 662阅读 0赞

**distinct算子原理：** 含有reduceByKey则会有shuffle

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p1b2NoYW5nX2xpdQ_size_16_color_FFFFFF_t_70][]

**贴上spark源码：**

/**
       * Return a new RDD containing the distinct elements in this RDD.
       */
      def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope {
        map(x => (x, null)).reduceByKey((x, y) => x, numPartitions).map(_._1)
      }

示例代码：

package com.wedoctor.utils.test
     
    import org.apache.log4j.{Level, Logger}
    import org.apache.spark.rdd.RDD
    import org.apache.spark.sql.SparkSession
     
    object Test {
      Logger.getLogger("org").setLevel(Level.ERROR)
      def main(args: Array[String]): Unit = {
        //本地环境需要加上
        System.setProperty("HADOOP_USER_NAME", "root")
        val session: SparkSession = SparkSession.builder()
           .master("local[*]")
           .appName(this.getClass.getSimpleName)
           .getOrCreate()
     
        val value: RDD[Int] = session.sparkContext.makeRDD(Array(3,3,4,5,5))
        value.distinct().foreach(println)
        //等价于
        value.map(x=>(x,null)).reduceByKey((x,y) => x).map(_._1).foreach(println)
        session.close()
      }
    }

转自：[https://blog.csdn.net/zuochang\_liu/article/details/105387704][https_blog.csdn.net_zuochang_liu_article_details_105387704]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p1b2NoYW5nX2xpdQ_size_16_color_FFFFFF_t_70]: /images/20210923/5d872656055f4aebaf12c3772cb04228.png
[https_blog.csdn.net_zuochang_liu_article_details_105387704]: https://blog.csdn.net/zuochang_liu/article/details/105387704