Spark pyspark rdd连接函数之join、leftOuterJoin、rightOuterJoin和fullOuterJoin介绍

旧城等待， 2022-05-31 13:50 364阅读 0赞

**Spark pyspark rdd连接函数之join、leftOuterJoin、rightOuterJoin和fullOuterJoin介绍**

**union用于组合两个rdd的元素，join用于内连接，而后三个函数(leftOuterJoin，rightOuterJoin，fullOuterJoin)用于类似于SQL的左、右、全连接。**

**针对key-value形式的RDD。**

例子：

1）数据初始化

>>> pp=(('cat', 2), ('cat', 5), ('book', 4), ('cat', 12))
    >>> pp
    (('cat', 2), ('cat', 5), ('book', 4), ('cat', 12))
    >>> qq=(("cat",2), ("cup", 5), ("mouse", 4),("cat", 12))
    >>> qq
    (('cat', 2), ('cup', 5), ('mouse', 4), ('cat', 12))
    >>> pairRDD1 = sc.parallelize(pp)
    >>> pairRDD2 = sc.parallelize(qq)
    >>> pairRDD1.collect()
    [('cat', 2), ('cat', 5), ('book', 4), ('cat', 12)]
    >>> pairRDD2.collect()
    [('cat', 2), ('cup', 5), ('mouse', 4), ('cat', 12)]

**2）Join内连接结果：**

>>> pairRDD1.join(pairRDD2).collect()
    [('cat', (2, 2)), ('cat', (2, 12)), ('cat', (5, 2)), ('cat', (5, 12)), ('cat', (12, 2)), ('cat', (12, 12))]

**3）leftOuterJoin结果：**

>>> pairRDD1.leftOuterJoin(pairRDD2).collect()
    [('book', (4, None)), ('cat', (2, 2)), ('cat', (2, 12)), ('cat', (5, 2)), ('cat', (5, 12)), ('cat', (12, 2)), ('cat', (12, 12))]

**4）rightOuterJoin结果：**

>>> pairRDD1.rightOuterJoin(pairRDD2).collect()
    [('cup', (None, 5)), ('mouse', (None, 4)), ('cat', (2, 2)), ('cat', (2, 12)), ('cat', (5, 2)), ('cat', (5, 12)), ('cat', (12, 2)), ('cat', (12, 12))]

**5）fullOuterJoin结果：**

>>> pairRDD1.fullOuterJoin(pairRDD2).collect()
    [('book', (4, None)), ('cup', (None, 5)), ('mouse', (None, 4)), ('cat', (2, 2)), ('cat', (2, 12)), ('cat', (5, 2)), ('cat', (5, 12)), ('cat', (12, 2)), ('cat', (12, 12))]

**6）union结果：**

>>> pairRDD1.union(pairRDD2).collect()
    [('cat', 2), ('cat', 5), ('book', 4), ('cat', 12), ('cat', 2), ('cup', 5), ('mouse', 4), ('cat', 12)]

参考:http://blog.cheyo.net/175.html