Spark Window 代码片段整理

心已赠人 2022-05-10 23:18 140阅读 0赞

参考地址：

[http://spark.apache.org/docs/2.3.0/structured-streaming-programming-guide.html\#operations-on-streaming-dataframesdatasets][http_spark.apache.org_docs_2.3.0_structured-streaming-programming-guide.html_operations-on-streaming-dataframesdatasets]

### Window Operations on Event Time ###

![70][]

Dataset<Row> words = ... // streaming DataFrame of schema { timestamp: Timestamp, word: String }
    
    // Group the data by window and word and compute the count of each group
    Dataset<Row> windowedCounts = words.groupBy(
      functions.window(words.col("timestamp"), "10 minutes", "5 minutes"),
      words.col("word")
    ).count();

**Handling Late Data and Watermarking**

![70 1][]

Dataset<Row> words = ... // streaming DataFrame of schema { timestamp: Timestamp, word: String }
    
    // Group the data by window and word and compute the count of each group
    Dataset<Row> windowedCounts = words
        .withWatermark("timestamp", "10 minutes")
        .groupBy(
            functions.window(words.col("timestamp"), "10 minutes", "5 minutes"),
            words.col("word"))
        .count();

比较好的一个例子

[https://www.iteblog.com/archives/1705.html][https_www.iteblog.com_archives_1705.html] （[Spark 2.0介绍：Spark SQL中的Time Window使用][https_www.iteblog.com_archives_1705.html]）

[http_spark.apache.org_docs_2.3.0_structured-streaming-programming-guide.html_operations-on-streaming-dataframesdatasets]: http://spark.apache.org/docs/2.3.0/structured-streaming-programming-guide.html#operations-on-streaming-dataframesdatasets
[70]: /images/20220511/0f6a08bdab5b45c494f72777d436e0c2.png
[70 1]: /images/20220511/6631d663f7c24bf8b874e976f227dcc7.png
[https_www.iteblog.com_archives_1705.html]: https://www.iteblog.com/archives/1705.html