PySpark 统计所有词频并排序返回 DataFrame

心已赠人 2023-08-17 15:49 124阅读 0赞

上一篇文章已经实现了PySpark 统计词频并返回排名TOP N：[https://blog.csdn.net/sinat\_26811377/article/details/101208346][https_blog.csdn.net_sinat_26811377_article_details_101208346]

发现有两个小缺点需要改一下，因此对代码做了修改。

1. 统计所有词频。而不仅仅是统计TOP N的词频。

2. 返回的结果保存为DataFrame。而不是用list。

**最终实现结果如下。**

输入文本（list格式）：

what do you do
    how do you do
    how do you do
    how are you

输出结果（ DataFrame 格式）：

关键词  数量      词频占比
    0    do   6  0.400000
    1   you   4  0.266667
    2   how   3  0.200000
    3   are   1  0.066667
    4  what   1  0.066667

Pyspark 的实现代码：

from operator import add
    
    import pandas as pd
    from pyspark import SparkContext
    
    
    def WordCount():
        sc = SparkContext(appName="WordCount")
        data = sc.parallelize(words)
    
        result = data.flatMap(lambda x: x.split(" ")) \
            .map(lambda x: (x, 1)). \
            reduceByKey(add). \
            sortBy(lambda x: x[1], False)
        resultArray = result.collect()
    
        resultKeyList = []
        resultValueList = []
        resultDF = pd.DataFrame()
        for k, v in resultArray:
            resultKeyList.append(k)
            resultValueList.append(v)
        resultAllWords = sum(resultValueList)
    
        def division(list):
            return list / resultAllWords
    
        resultPercentList = list(map(division, resultValueList))
    
        resultDF['关键词'] = resultKeyList
        resultDF['数量'] = resultValueList
        resultDF['词频占比'] = resultPercentList
        return resultDF
    
    
    if __name__ == '__main__':
        words = ["what do you do", "how do you do", "how do you do", "how are you"]
        # top = 3
        df = WordCount()

[https_blog.csdn.net_sinat_26811377_article_details_101208346]: https://blog.csdn.net/sinat_26811377/article/details/101208346