用IDEA写spark单词统计

系统管理员 2022-05-26 22:40 208阅读 0赞

1.创建一个项目

![70][]

2.选择Maven项目，然后点击next

![70 1][]

3.填写maven的GAV，然后点击next

![70 2][]

4.填写项目名称，然后点击finish

5.创建好maven项目后，点击Enable Auto-Import

![70 3][]

创建maven后不能创建scala  按照下面的来

![70 4][]

注意自己的版本号

![70 5][]

注意！！！记得把以下几个选项勾上 不然打不了jar包

![70 6][]

![70 7][]

![70 8][]

这个不用选  因为自己选择的是默认的的c:\\\\user\\..\\.m2\\settings.xml

![70 9][]

6.配置Maven的pom.xml

<?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
    
        <groupId>cn.bw.spark</groupId>
        <artifactId>WordCount2</artifactId>
        <version>2.0</version>
    
    
        <properties>
            <maven.compiler.source>1.7</maven.compiler.source>
            <maven.compiler.target>1.7</maven.compiler.target>
            <encoding>UTF-8</encoding>
            <scala.version>2.10.6</scala.version>
            <scala.compat.version>2.10</scala.compat.version>
        </properties>
    
        <dependencies>
            <dependency>
                <groupId>org.scala-lang</groupId>
                <artifactId>scala-library</artifactId>
                <version>${scala.version}</version>
            </dependency>
    
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.10</artifactId>
                <version>1.6.1</version>
            </dependency>
    
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-streaming_2.10</artifactId>
                <version>1.6.1</version>
            </dependency>
    
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>2.6.4</version>
            </dependency>
            <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
                <version>5.1.38</version>
            </dependency>
        </dependencies>
    
        <build>
            <sourceDirectory>src/main/scala</sourceDirectory>
            <testSourceDirectory>src/test/scala</testSourceDirectory>
            <plugins>
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.0</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>compile</goal>
                                <goal>testCompile</goal>
                            </goals>
                            <configuration>
                                <args>
                                    <arg>-make:transitive</arg>
                                    <arg>-dependencyfile</arg>
                                    <arg>${project.build.directory}/.scala_dependencies</arg>
                                </args>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.18.1</version>
                    <configuration>
                        <useFile>false</useFile>
                        <disableXmlReport>true</disableXmlReport>
                        <includes>
                            <include>**/*Test.*</include>
                            <include>**/*Suite.*</include>
                        </includes>
                    </configuration>
                </plugin>
    
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-shade-plugin</artifactId>
                    <version>2.3</version>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>shade</goal>
                            </goals>
                            <configuration>
                                <filters>
                                    <filter>
                                        <artifact>*:*</artifact>
                                        <excludes>
                                            <exclude>META-INF/*.SF</exclude>
                                            <exclude>META-INF/*.DSA</exclude>
                                            <exclude>META-INF/*.RSA</exclude>
                                        </excludes>
                                    </filter>
                                </filters>
                                <transformers>
                                    <transformer
                                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                        <mainClass>cn.bw.spark.WordCountSpark</mainClass>
                                    </transformer>
                                </transformers>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    
    
    </project>

7.将src/main/java和src/test/java分别修改成src/main/scala和src/test/scala，与pom.xml中的配置保持一致

![70 10][]

![70 11][]

8.新建一个scala class，类型为Object

![70 12][]

9.编写spark程序

package cn.bw.spark
    
    import org.apache.spark.{SparkContext, SparkConf}
    
    object WordCount {
      def main(args: Array[String]) {
        //创建SparkConf()并设置App名称
        val conf = new SparkConf().setAppName("WC")
        //创建SparkContext，该对象是提交spark App的入口
        val sc = new SparkContext(conf)
        //使用sc创建RDD并执行相应的transformation和action
        sc.textFile(args(0)).flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_, 1).sortBy(_._2, false).saveAsTextFile(args(1))
        //停止sc，结束该任务
        sc.stop()
      }
    }

10.使用Maven打包：首先修改pom.xml中的main class

![70 13][]

![70 14][]

点击idea右侧的Maven Project选项

![70 15][]

点击Lifecycle,选择clean和package，然后点击Run Maven Build

![70 16][]

11.选择编译成功的jar包，并将该jar上传到Spark集群中的某个节点上

![70 17][]

12.首先启动hdfs和Spark集群

13.使用spark-submit命令提交Spark应用（注意参数的顺序）

/usr/local/spark-1.5.2-bin-hadoop2.6/bin/spark-submit \   //注意是在submit下
    --class cn.bw.spark.WordCount \                      //写的是你类的全路径

--master spark://node1.bw.cn:7077 \
    --executor-memory 512m \                               //指定运行内存
    --total-executor-cores 1 \                            //指定核数
    /root/spark-mvn-1.0-SNAPSHOT.jar \                   //jar的路径
    hdfs://node1.bw.cn:9000/words.txt \                //文件的路径
    hdfs://node1.bw.cn:9000/out                       //输出的路径

查看程序执行结果

hdfs dfs -cat hdfs://node1.bw.cn:9000/out/part-00000

[70]: /images/20220527/785ca690e7c949bc911cc76f530766eb.png
[70 1]: /images/20220527/22901f60788e4d54a22ff2ece3640243.png
[70 2]: /images/20220527/7e199311cb1f40328a512f69f6041107.png
[70 3]: /images/20220527/e91b15b34ea64554ba27a9772aa06fe9.png
[70 4]: /images/20220527/1e196268af8047e985ba196efa86a290.png
[70 5]: /images/20220527/ea1f5bdd31ae4df28a7486fd95ac2384.png
[70 6]: /images/20220527/a8c390b7e11c4db9b78cfb874cabda90.png
[70 7]: /images/20220527/161c7c466b654bd1bae9293d2bba3f82.png
[70 8]: /images/20220527/34c76962754446f6829ffd01d6c0f578.png
[70 9]: /images/20220527/9a8fa98f33c6457e84572de519006cf1.png
[70 10]: /images/20220527/d72c73091cd9408b985cf4cc70f4ff6a.png
[70 11]: /images/20220527/f53bc7d5ab0b4ad5ae0130c5a217fac9.png
[70 12]: /images/20220527/53245255bfe242b49ff93b539333afcd.png
[70 13]: /images/20220527/99e175385a0746f3b8b97382131ac4dd.png
[70 14]: /images/20220527/97fe408adece47a3acf28e140456c405.png
[70 15]: /images/20220527/25f40138874948b4966b06a5441e70c4.png
[70 16]: /images/20220527/c4a8f232a30843a0b39fa704f005f6e8.png
[70 17]: /images/20220527/2b25a7dd46274134af77acd935fa2cd2.png