【spark内置函数】current_date、current_timestamp、date

【spark内置函数】current_date、current_timestamp、date_format

女爷i 2022-09-08 05:57 369阅读 0赞

一、目的

演示sparkSql三种内置函数current_date、current_timestamp、date_format的使用；

注意：current_timestamp输出的并不是长整型时间戳，而是2021-08-25 17:35:25.568格式；

二、代码

import org.apache.spark.sql.{DataFrame, SparkSession}
object Test {
  case class DateT(name:String,birthday:String)
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[*]").appName("testDate").getOrCreate()
    import spark.implicits._
    import org.apache.spark.sql.functions._
    val sc = spark.sparkContext
    sc.setLogLevel("WARN")
    val DateTest: DataFrame = Seq(
      DateT("aa","1995-12-11 12:12:13"),
      DateT("bb","2000-01-14 10:10:57")
    ).toDF()
    DateTest.withColumn("current_date",current_date())
      .withColumn("current_timestamp",current_timestamp())
      .withColumn("date",date_format(current_timestamp(),"yyyy-MM-dd HH:mm:ss"))
      .withColumn("date1",date_format(current_timestamp(),"yyyy-MM-dd hh:mm:ss"))
      .withColumn("date2",date_format(col("birthday"),"yyyy-MM-dd"))
      .show(false)
    spark.close()
  }
}

三、输出

+----+-------------------+------------+-----------------------+-------------------+-------------------+----------+
|name|birthday           |current_date|current_timestamp      |date               |date1              |date2     |
+----+-------------------+------------+-----------------------+-------------------+-------------------+----------+
|aa  |1995-12-11 12:12:13|2021-08-25  |2021-08-25 17:35:25.568|2021-08-25 17:35:25|2021-08-25 05:35:25|1995-12-11|
|bb  |2000-01-14 10:10:57|2021-08-25  |2021-08-25 17:35:25.568|2021-08-25 17:35:25|2021-08-25 05:35:25|2000-01-14|
+----+-------------------+------------+-----------------------+-------------------+-------------------+----------+