简单上手版使用oozie提交spark程序(不带定时)

╰半夏微凉° 2023-03-13 06:48 17阅读 0赞

1.  做这个实验之前你的服务器上最好装了cdh集群，以及添加必要的组件，如hadoop，oozie，spark，yarn等。必须都启动成功。版本如下：  
    hadoop 3.0.0-cdh6.3.2  
    zookeeper 3.4.5-cdh6.3.2  
    oozie 5.1.0-cdh6.3.2  
    spark 2.4.0-cdh6.3.2
2.  需要准备一个关于spark的demo架包、job.propertie、workflow.xml。创建一个文件夹将三个放入其中，然后将文件夹put到HDFS指定的路径下面。(\***注意每次在linux下修改完后，记得put到HDFS中** \*)
3.  准备一个job.propertie和workflow.xml，内容如下  
    3.1**job.propertie**

#
    # Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    #      http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    #
    
    #HDFS文件系统的端口
    nameNode=hdfs://emg101.emapgo.com:8020 
    
    #resourceManager的端口,也有人写jobTracker
    resourceManager=emg104.emapgo.com:8032 
    
    #需要有，是否使用系统lib路径
    oozie.use.system.libpath=true 
    
    #需要有，workflow.xml在hdfs上的路径   
    oozie.wf.application.path=${nameNode}/user/${user.name}/examples/apps/spark/workflow.xml
    
    #workflow.xml中使用
    queueName=default
    
    #下面是spark的属性参数，具体参照自己的workflow.xml
    jobmaster=yarn-cluster
    jobmode=cluster
    jobname=oozie_missing
    jarclass=com.emg.etp.analysis.missing.MissingRoad
    #jar包所在HDFS的绝对路径
    jarpath=${nameNode}/user/root/examples/apps/spark/missing_road.jar
    #运行spark任务的参数，有的可能需要assembly相关的jar包，我的不需要
    sparkopts=--driver-memory 1G --executor-memory 2G --executor-cores 2 --num-executors 20
    #jar包所需要的参数
    input1=a.txt

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTgwNDA0OQ_size_16_color_FFFFFF_t_70]

3.2**workflow.xml**

<workflow-app xmlns="uri:oozie:workflow:1.0" name="spark-wf">
        <start to="spark-node"/>
        <action name="spark-node">
            <spark xmlns="uri:oozie:spark-action:1.0">
                <resource-manager>${resourceManager}</resource-manager>
                <name-node>${nameNode}</name-node>
                <configuration>
                    <property>
                        <name>queueName</name>
                        <value>${queueName}</value>
                    </property>
                </configuration>
                <master>${jobmaster}</master>
                <mode>${jobmode}</mode>
                <jar>${jarpath}</jar>
                <spark-opts>${sparkopts}</spark-opts>
                <arg>${input1}</arg>
            </spark>
            
            <ok to="end"/>
            <error to="fail"/>
        </action>
    
        <kill name="fail">
            <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
        </kill>
        <kill name="fail-output">
            <message>Incorrect output, expected [Hello Oozie] but was [${wf:actionData('shell-node')['my_output']}]</message>
        </kill>
        <end name="end"/>
    </workflow-app>

启动命令(注意 这里的job.properties是linux下的路径)
    bin/oozie job -oozie http://xxxx:11000/oozie -config oozie-apps/job.properties -run
    
    kill命令，jobid可以到oozie自带的web界面查看
    bin/oozie job -oozie http://xxxx:11000/oozie -kill 0000000-170724202624644-oozie-oozi-C

8088集群上看所运行的任务，可以看到先是启动Oozie Launcher，然后再调的spark任务。（oozieUI报错打不开，只能看这里了）  
![在这里插入图片描述][20200513102129673.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTgwNDA0OQ_size_16_color_FFFFFF_t_70 1]  
参考：  
https://blog.csdn.net/Abysscarry/article/details/81784179  
https://blog.csdn.net/pt707621764/article/details/52437567

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTgwNDA0OQ_size_16_color_FFFFFF_t_70]: /images/20230312/bf166525a12b48789a2514d58d1732e5.png
[20200513102129673.png]: https://img-blog.csdnimg.cn/20200513102129673.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTgwNDA0OQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200513101954808.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MTgwNDA0OQ==,size_16,color_FFFFFF,t_70