Spark SQL 1.x之SQL Context使用

Dear 丶 2023-05-28 12:56 132阅读 0赞

# 创建Scala项目 #

使用Idea创建项目：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70]  
指定`<groupId>cn.ac.iie.spark</groupId>` 和 `<artifactId>sql</artifactId>`  
依赖包如下：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
      <modelVersion>4.0.0</modelVersion>
      <groupId>cn.ac.iie.spark</groupId>
      <artifactId>sql</artifactId>
      <version>1.0-SNAPSHOT</version>
      <inceptionYear>2008</inceptionYear>
      <properties>
        <scala.version>2.11.12</scala.version>
        <spark.version>2.4.5</spark.version>
      </properties>
    
      <dependencies>
        
        <dependency>
          <groupId>org.scala-lang</groupId>
          <artifactId>scala-library</artifactId>
          <version>${scala.version}</version>
        </dependency>
        
        <dependency>
          <groupId>org.apache.spark</groupId>
          <artifactId>spark-sql_2.11</artifactId>
          <version>${spark.version}</version>
        </dependency>
      </dependencies>
    
      <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
          <plugin>
            <groupId>org.scala-tools</groupId>
            <artifactId>maven-scala-plugin</artifactId>
            <version>2.15.0</version>
            <executions>
              <execution>
                <goals>
                  <goal>compile</goal>
                  <goal>testCompile</goal>
                </goals>
                <configuration>
                  <args>
                    <arg>-dependencyfile</arg>
                    <arg>${project.build.directory}/.scala_dependencies</arg>
                  </args>
                </configuration>
              </execution>
            </executions>
          </plugin>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.6</version>
            <configuration>
              <useFile>false</useFile>
              <disableXmlReport>true</disableXmlReport>
              
              
              <includes>
                <include>**/*Test.*</include>
                <include>**/*Suite.*</include>
              </includes>
            </configuration>
          </plugin>
    
          <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
              <archive>
                <manifest>
                  <mainClass></mainClass>
                </manifest>
              </archive>
              <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
              </descriptorRefs>
            </configuration>
          </plugin>
        </plugins>
      </build>
    
    </project>

# 创建Object #

创建一个Object，内容如下：

package cn.ac.iie.spark
    
    import org.apache.spark.{ SparkConf, SparkContext}
    import org.apache.spark.sql.SQLContext
    
    /** * SQLContextApp 的使用 * 注意：IDEA是在本地，而测试数据在服务器上，能不能在本地进行开发测试？可以 */
    object SQLContextApp { 
      def main(args: Array[String]): Unit = { 
    
        val path = args(0)
    
        // 1. 创建相应的Context
        val sparkConf= new SparkConf()
        // 在生产或测试环境中，APPName和Master是通过脚本指定的
        sparkConf.setAppName("SQLContextApp")
        sparkConf.setMaster("local[2]")
        val sc = new SparkContext(sparkConf)
        val sqlContext = new SQLContext(sc)
    
        // 2. 相关处理:json
        val people = sqlContext.read.format("json").load(path)
        people.printSchema()
        people.show()
        // 3. 关闭资源
        sc.stop()
      }
    }

因为需要传参数，我们设置`Edit Configuration`：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70 1]  
原始数据如下：

{"name":"Michael", "salary":3000}
    {"name":"Andy", "salary":4500}
    {"name":"Justin", "salary":3500}
    {"name":"Berta", "salary":4000}
    {"name":"vincent", "salary":90000}

控制台输出如下：

root
     |-- name: string (nullable = true)
     |-- salary: long (nullable = true)
    
    +-------+------+
    |   name|salary|
    +-------+------+
    |Michael|  3000|
    |   Andy|  4500|
    | Justin|  3500|
    |  Berta|  4000|
    |vincent| 90000|
    +-------+------+

可以看到正确打印出来了`schema`和数据  
在目录下使用maven进行打包：`mvn clean package -DskipTests`

# 提交到环境中运行 #

往往我们在生产或测试环境中，APPName和Master是通过脚本指定的，因此需要注释掉`sparkConf.setAppName("SQLContextApp") sparkConf.setMaster("local[2]")`  
在服务器中提交：

./spark-submit --name SQLContextApp --class cn.ac.iie.spark.SQLContextApp   --master local[2]    /home/iie4bu/lib/sql-1.0-SNAPSHOT.jar   /home/iie4bu/app/spark-2.4.5-bin-2.6.0-cdh5.15.1/examples/src/main/resources/people.json

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70 3]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70]: /images/20230528/5ff67c9cb45f4b068ba819cb018f7996.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70 1]: /images/20230528/d49709b2a4b94a24b4c834281a9d5ac8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70 2]: /images/20230528/61760a38a3e44f28908d9a780847645e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3ZpbmNlbnRfZHVhbg_size_16_color_FFFFFF_t_70 3]: /images/20230528/dad0e308cd3b4352b99aa84df414b7a0.png