spark本地调试（idea）

谁借莪１个温暖的怀抱￠ 2022-12-17 10:56 109阅读 0赞

spark批处理程序通常是跑在类似yarn等集群环境中的，可是在进行开发的时候，不可避免的需要进行本地开发调试，记录下idea下调试本地spark程序方法。

demo代码：

/** * @description: 读取本地json文件，字符串解析出id并输出 */
    object CountPictureByDayTest { 
    
        def main(args: Array[String]): Unit = { 
            @transient
            val spark = SparkSession
              .builder()
              .appName(this.getClass.getSimpleName.replace("$", ""))
              .getOrCreate()
    
            // 输入的数据路径
            val input_path = "/Users/zhangbin19/Desktop/kela-input.json"
            // 输入的数据统计
            try { 
                val frame = spark.read.text(input_path)
                frame.createOrReplaceTempView("test")
    
                spark.udf.register("jason",(str: String) => { 
                    str.substring(str.indexOf("\"id\":\"") + 6,str.lastIndexOf("\",\"videoUrl\""))
                })
                spark.sql(
                    """ |select jason(value) as json from test where value like '%Sent message%' |""".stripMargin)
                  .rdd.collect()
                  .foreach(f => { 
                      println("id: " + f.getString(0))
                  })
    
            } catch { 
                case e: Exception => { 
                    println(e.getMessage)
                }
            }
            spark.stop()
        }
    }

执行运行程序报错如下：

/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/bin/java -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=61793:/Applications/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/lib/tools.jar:/Users/zhangbin19/weiboProject/spark_statistics/sparkjob1/target/classes:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-sql_2.11/2.0.2/spark-sql_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/com/univocity/univocity-parsers/2.1.1/univocity-parsers-2.1.1.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-sketch_2.11/2.0.2/spark-sketch_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-core_2.11/2.0.2/spark-core_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/org/apache/avro/avro-mapred/1.7.7/avro-mapred-1.7.7-hadoop2.jar:/Users/zhangbin19/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7.jar:/Users/zhangbin19/.m2/repository/org/apache/avro/avro/1.7.7/avro-1.7.7.jar:/Users/zhangbin19/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7-tests.jar:/Users/zhangbin19/.m2/repository/com/twitter/chill_2.11/0.8.0/chill_2.11-0.8.0.jar:/Users/zhangbin19/.m2/repository/com/esotericsoftware/kryo-shaded/3.0.3/kryo-shaded-3.0.3.jar:/Users/zhangbin19/.m2/repository/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/Users/zhangbin19/.m2/repository/com/twitter/chill-java/0.8.0/chill-java-0.8.0.jar:/Users/zhangbin19/.m2/repository/org/apache/xbean/xbean-asm5-shaded/4.4/xbean-asm5-shaded-4.4.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-client/2.2.0/hadoop-client-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-common/2.2.0/hadoop-common-2.2.0.jar:/Users/zhangbin19/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/zhangbin19/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/zhangbin19/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/zhangbin19/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/zhangbin19/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/zhangbin19/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-auth/2.2.0/hadoop-auth-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/Users/zhangbin19/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.2.0/hadoop-hdfs-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.2.0/hadoop-mapreduce-client-app-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.2.0/hadoop-mapreduce-client-common-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.2.0/hadoop-yarn-client-2.2.0.jar:/Users/zhangbin19/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/Users/zhangbin19/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.2.0/hadoop-yarn-server-common-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.2.0/hadoop-mapreduce-client-shuffle-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.2.0/hadoop-mapreduce-client-core-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.2.0/hadoop-yarn-common-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.2.0/hadoop-mapreduce-client-jobclient-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/hadoop/hadoop-annotations/2.2.0/hadoop-annotations-2.2.0.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-launcher_2.11/2.0.2/spark-launcher_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-network-common_2.11/2.0.2/spark-network-common_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-network-shuffle_2.11/2.0.2/spark-network-shuffle_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-unsafe_2.11/2.0.2/spark-unsafe_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar:/Users/zhangbin19/.m2/repository/org/apache/curator/curator-recipes/2.4.0/curator-recipes-2.4.0.jar:/Users/zhangbin19/.m2/repository/org/apache/curator/curator-framework/2.4.0/curator-framework-2.4.0.jar:/Users/zhangbin19/.m2/repository/org/apache/curator/curator-client/2.4.0/curator-client-2.4.0.jar:/Users/zhangbin19/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/Users/zhangbin19/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.jar:/Users/zhangbin19/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/zhangbin19/.m2/repository/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar:/Users/zhangbin19/.m2/repository/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/Users/zhangbin19/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/zhangbin19/.m2/repository/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar:/Users/zhangbin19/.m2/repository/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/Users/zhangbin19/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/Users/zhangbin19/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/zhangbin19/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar:/Users/zhangbin19/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/zhangbin19/.m2/repository/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar:/Users/zhangbin19/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar:/Users/zhangbin19/.m2/repository/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/Users/zhangbin19/.m2/repository/commons-net/commons-net/2.2/commons-net-2.2.jar:/Users/zhangbin19/.m2/repository/org/json4s/json4s-jackson_2.11/3.2.11/json4s-jackson_2.11-3.2.11.jar:/Users/zhangbin19/.m2/repository/org/json4s/json4s-core_2.11/3.2.11/json4s-core_2.11-3.2.11.jar:/Users/zhangbin19/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar:/Users/zhangbin19/.m2/repository/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar:/Users/zhangbin19/.m2/repository/org/scala-lang/scalap/2.11.0/scalap-2.11.0.jar:/Users/zhangbin19/.m2/repository/org/scala-lang/scala-compiler/2.11.0/scala-compiler-2.11.0.jar:/Users/zhangbin19/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/zhangbin19/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/zhangbin19/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/zhangbin19/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/zhangbin19/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/zhangbin19/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/zhangbin19/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/zhangbin19/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/zhangbin19/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/zhangbin19/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/zhangbin19/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/zhangbin19/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/zhangbin19/.m2/repository/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/Users/zhangbin19/.m2/repository/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/Users/zhangbin19/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/zhangbin19/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/Users/zhangbin19/.m2/repository/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/Users/zhangbin19/.m2/repository/org/apache/mesos/mesos/0.21.1/mesos-0.21.1-shaded-protobuf.jar:/Users/zhangbin19/.m2/repository/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar:/Users/zhangbin19/.m2/repository/io/netty/netty/3.8.0.Final/netty-3.8.0.Final.jar:/Users/zhangbin19/.m2/repository/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/Users/zhangbin19/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar:/Users/zhangbin19/.m2/repository/io/dropwizard/metrics/metrics-jvm/3.1.2/metrics-jvm-3.1.2.jar:/Users/zhangbin19/.m2/repository/io/dropwizard/metrics/metrics-json/3.1.2/metrics-json-3.1.2.jar:/Users/zhangbin19/.m2/repository/io/dropwizard/metrics/metrics-graphite/3.1.2/metrics-graphite-3.1.2.jar:/Users/zhangbin19/.m2/repository/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.5/jackson-module-scala_2.11-2.6.5.jar:/Users/zhangbin19/.m2/repository/com/fasterxml/jackson/module/jackson-module-paranamer/2.6.5/jackson-module-paranamer-2.6.5.jar:/Users/zhangbin19/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/zhangbin19/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:/Users/zhangbin19/.m2/repository/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/Users/zhangbin19/.m2/repository/net/sf/py4j/py4j/0.10.3/py4j-0.10.3.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-catalyst_2.11/2.0.2/spark-catalyst_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/Users/zhangbin19/.m2/repository/org/codehaus/janino/janino/2.7.8/janino-2.7.8.jar:/Users/zhangbin19/.m2/repository/org/codehaus/janino/commons-compiler/2.7.8/commons-compiler-2.7.8.jar:/Users/zhangbin19/.m2/repository/org/antlr/antlr4-runtime/4.5.3/antlr4-runtime-4.5.3.jar:/Users/zhangbin19/.m2/repository/org/apache/spark/spark-tags_2.11/2.0.2/spark-tags_2.11-2.0.2.jar:/Users/zhangbin19/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar:/Users/zhangbin19/.m2/repository/org/apache/parquet/parquet-column/1.7.0/parquet-column-1.7.0.jar:/Users/zhangbin19/.m2/repository/org/apache/parquet/parquet-common/1.7.0/parquet-common-1.7.0.jar:/Users/zhangbin19/.m2/repository/org/apache/parquet/parquet-encoding/1.7.0/parquet-encoding-1.7.0.jar:/Users/zhangbin19/.m2/repository/org/apache/parquet/parquet-generator/1.7.0/parquet-generator-1.7.0.jar:/Users/zhangbin19/.m2/repository/org/apache/parquet/parquet-hadoop/1.7.0/parquet-hadoop-1.7.0.jar:/Users/zhangbin19/.m2/repository/org/apache/parquet/parquet-format/2.3.0-incubating/parquet-format-2.3.0-incubating.jar:/Users/zhangbin19/.m2/repository/org/apache/parquet/parquet-jackson/1.7.0/parquet-jackson-1.7.0.jar:/Users/zhangbin19/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.11/jackson-mapper-asl-1.9.11.jar:/Users/zhangbin19/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.11/jackson-core-asl-1.9.11.jar:/Users/zhangbin19/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.5/jackson-databind-2.6.5.jar:/Users/zhangbin19/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar:/Users/zhangbin19/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.5/jackson-core-2.6.5.jar:/Users/zhangbin19/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/zhangbin19/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.2/scala-xml_2.11-1.0.2.jar:/Users/zhangbin19/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.2/scala-parser-combinators_2.11-1.0.2.jar:/Users/zhangbin19/.m2/repository/com/101tec/zkclient/0.3/zkclient-0.3.jar:/Users/zhangbin19/.m2/repository/com/alibaba/fastjson/1.2.51/fastjson-1.2.51.jar:/Users/zhangbin19/.m2/repository/org/apache/httpcomponents/httpclient/4.5.3/httpclient-4.5.3.jar:/Users/zhangbin19/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/zhangbin19/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/Users/zhangbin19/.m2/repository/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.jar:/Users/zhangbin19/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/zhangbin19/.m2/repository/org/scalaj/scalaj-http_2.10/2.2.2/scalaj-http_2.10-2.2.2.jar:/Users/zhangbin19/.m2/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/Users/zhangbin19/.m2/repository/mysql/mysql-connector-java/5.1.12/mysql-connector-java-5.1.12.jar:/Users/zhangbin19/.m2/repository/com/aliyun/oss/aliyun-sdk-oss/3.8.0/aliyun-sdk-oss-3.8.0.jar:/Users/zhangbin19/.m2/repository/org/jdom/jdom/1.1/jdom-1.1.jar:/Users/zhangbin19/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/zhangbin19/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/zhangbin19/.m2/repository/com/aliyun/aliyun-java-sdk-core/3.4.0/aliyun-java-sdk-core-3.4.0.jar:/Users/zhangbin19/.m2/repository/com/aliyun/aliyun-java-sdk-ram/3.0.0/aliyun-java-sdk-ram-3.0.0.jar:/Users/zhangbin19/.m2/repository/com/aliyun/aliyun-java-sdk-sts/3.0.0/aliyun-java-sdk-sts-3.0.0.jar:/Users/zhangbin19/.m2/repository/com/aliyun/aliyun-java-sdk-ecs/4.2.0/aliyun-java-sdk-ecs-4.2.0.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/motan-weibo-core/0.0.4/motan-weibo-core-0.0.4.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/weibo-trace/0.14/weibo-trace-0.14.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/api-commons/4.180/api-commons-4.180.jar:/Users/zhangbin19/.m2/repository/commons-fileupload/commons-fileupload/1.2.1/commons-fileupload-1.2.1.jar:/Users/zhangbin19/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/zhangbin19/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/Users/zhangbin19/.m2/repository/commons-pool/commons-pool/1.6/commons-pool-1.6.jar:/Users/zhangbin19/.m2/repository/org/apache/logging/log4j/log4j-core/2.6/log4j-core-2.6.jar:/Users/zhangbin19/.m2/repository/org/apache/logging/log4j/log4j-api/2.6/log4j-api-2.6.jar:/Users/zhangbin19/.m2/repository/dom4j/dom4j/1.6.1/dom4j-1.6.1.jar:/Users/zhangbin19/.m2/repository/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar:/Users/zhangbin19/.m2/repository/net/sf/json-lib/json-lib/2.4/json-lib-2.4-jdk15.jar:/Users/zhangbin19/.m2/repository/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar:/Users/zhangbin19/.m2/repository/net/sf/ezmorph/ezmorph/1.0.6/ezmorph-1.0.6.jar:/Users/zhangbin19/.m2/repository/xom/xom/1.1/xom-1.1.jar:/Users/zhangbin19/.m2/repository/org/codehaus/woodstox/woodstox-core-asl/4.1.0/woodstox-core-asl-4.1.0.jar:/Users/zhangbin19/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/zhangbin19/.m2/repository/org/codehaus/woodstox/stax2-api/3.1.0/stax2-api-3.1.0.jar:/Users/zhangbin19/.m2/repository/c3p0/c3p0/0.9.1/c3p0-0.9.1.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/3rd/hs4j/0.3/hs4j-0.3.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/common/clientBalancer/0.2.14/clientBalancer-0.2.14.jar:/Users/zhangbin19/.m2/repository/cglib/cglib-nodep/2.2/cglib-nodep-2.2.jar:/Users/zhangbin19/.m2/repository/org/objenesis/objenesis/1.2/objenesis-1.2.jar:/Users/zhangbin19/.m2/repository/redis/clients/jedis/2.1.5.weibo.23/jedis-2.1.5.weibo.23.jar:/Users/zhangbin19/.m2/repository/org/perf4j/perf4j/0.9.16/perf4j-0.9.16.jar:/Users/zhangbin19/.m2/repository/org/apache/thrift/libthrift/0.6.1/libthrift-0.6.1.jar:/Users/zhangbin19/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/Users/zhangbin19/.m2/repository/org/apache/thrift/libfb303/0.6.1/libfb303-0.6.1.jar:/Users/zhangbin19/.m2/repository/net/sf/ehcache/ehcache-core/1.7.2/ehcache-core-1.7.2.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-core/3.0.6.RELEASE/spring-core-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-jdbc/3.0.6.RELEASE/spring-jdbc-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-tx/3.0.6.RELEASE/spring-tx-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/commons-configservice/1.6/commons-configservice-1.6.jar:/Users/zhangbin19/.m2/repository/org/mapdb/mapdb/0.9.3/mapdb-0.9.3.jar:/Users/zhangbin19/.m2/repository/org/jboss/netty/netty/3.2.5.Final/netty-3.2.5.Final.jar:/Users/zhangbin19/.m2/repository/com/caucho/hessian/4.0.7.fix.2/hessian-4.0.7.fix.2.jar:/Users/zhangbin19/.m2/repository/com/codahale/metrics/metrics-core/3.0.1/metrics-core-3.0.1.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/common-jackson/1.4/common-jackson-1.4.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/weibo-api-core/5.49/weibo-api-core-5.49.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/commons-memcache/0.16/commons-memcache-0.16.jar:/Users/zhangbin19/.m2/repository/com/weibo/motan-core/1.1.1/motan-core-1.1.1.jar:/Users/zhangbin19/.m2/repository/com/squareup/javapoet/1.8.0/javapoet-1.8.0.jar:/Users/zhangbin19/.m2/repository/com/weibo/motan-transport-netty/1.1.1/motan-transport-netty-1.1.1.jar:/Users/zhangbin19/.m2/repository/com/weibo/motan-springsupport/1.1.1/motan-springsupport-1.1.1.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/motan-weibo-registry-vintage/0.0.4/motan-weibo-registry-vintage-0.0.4.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/vintage-client/1.10/vintage-client-1.10.jar:/Users/zhangbin19/.m2/repository/com/weibo/api/vintage-common/1.10/vintage-common-1.10.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-web/3.0.6.RELEASE/spring-web-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/commons-validator/commons-validator/1.4.0/commons-validator-1.4.0.jar:/Users/zhangbin19/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/zhangbin19/.m2/repository/com/weibo/datasys/alg-motan-client/1.7.81/alg-motan-client-1.7.81.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-context/3.0.6.RELEASE/spring-context-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-aop/3.0.6.RELEASE/spring-aop-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-beans/3.0.6.RELEASE/spring-beans-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-expression/3.0.6.RELEASE/spring-expression-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/org/springframework/spring-asm/3.0.6.RELEASE/spring-asm-3.0.6.RELEASE.jar:/Users/zhangbin19/.m2/repository/commons-io/commons-io/1.3.2/commons-io-1.3.2.jar com.weibo.mediafeatureplatform.CountPictureByDayTest
    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
    20/10/23 14:39:03 INFO SparkContext: Running Spark version 2.0.2
    20/10/23 14:39:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    20/10/23 14:39:03 ERROR SparkContext: Error initializing SparkContext.
    org.apache.spark.SparkException: A master URL must be set in your configuration
    	at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
    	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
    	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
    	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
    	at scala.Option.getOrElse(Option.scala:121)
    	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
    	at com.weibo.mediafeatureplatform.CountPictureByDayTest$.main(CountPictureByDayTest.scala:28)
    	at com.weibo.mediafeatureplatform.CountPictureByDayTest.main(CountPictureByDayTest.scala)
    20/10/23 14:39:03 INFO SparkContext: Successfully stopped SparkContext
    Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
    	at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
    	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
    	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
    	at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
    	at scala.Option.getOrElse(Option.scala:121)
    	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
    	at com.weibo.mediafeatureplatform.CountPictureByDayTest$.main(CountPictureByDayTest.scala:28)
    	at com.weibo.mediafeatureplatform.CountPictureByDayTest.main(CountPictureByDayTest.scala)
    
    Process finished with exit code 1

从报错信息看需要指定一个master配置，只需在Run->Edit Configurations->VM Options下添加配置`-Dspark.master=local`即可。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MDk1MDU1_size_16_color_FFFFFF_t_70_pic_center]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MDk1MDU1_size_16_color_FFFFFF_t_70_pic_center 1]  
再次执行程序，发现正常输出了

id: 1464300955888517188
    id: 1464300964478451840
    id: 1464300955888517188
    id: 1464300964478451840
    id: 1464300955888517188
    id: 1464300964478451840

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MDk1MDU1_size_16_color_FFFFFF_t_70_pic_center]: /images/20221122/f796751569ab46a1ba8dda70be3853fe.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MDk1MDU1_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221122/573cfa70dd07435fbb5129e2a89507af.png