spark的shell脚本分析

忘是亡心i 2022-05-07 17:28 376阅读 0赞
  1. bin目录: { spark-shell , spark-sql } —> spark-submit–> spark-class
  2. sbin目录:

part1: bin目录

spark-shell

  1. function main() {
  2. export SPARK_SUBMIT_OPTS
  3. "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"
  4. }
  5. main "$@"

spark-sql

  1. exec "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver "$@"

spark-submit

  1. exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"

spark-class

  1. # For tests
  2. build_command() {
  3. "${JAVA_HOME}/bin/java" -Xmx128m -cp "="${SPARK_HOME}/jars/*" org.apache.spark.launcher.Main "$@"
  4. printf "%d\0" $?
  5. }
  6. CMD=()
  7. while IFS= read -d '' -r ARG; do
  8. CMD+=("$ARG")
  9. done < <(build_command "$@")
  10. COUNT=${#CMD[@]}
  11. LAST=$((COUNT - 1))
  12. LAUNCHER_EXIT_CODE=${CMD[$LAST]}
  13. CMD=("${CMD[@]:0:$LAST}")
  14. exec "${CMD[@]}"

part2: 调用spark的scala类

scala类org.apache.spark.deploy.SparkSubmit -main函数

  1. def main(args: Array[String]): Unit = {
  2. val appArgs = new SparkSubmitArguments(args)
  3. if (appArgs.verbose) {
  4. printStream.println(appArgs)
  5. }
  6. appArgs.action match {
  7. case SparkSubmitAction.SUBMIT => submit(appArgs)
  8. case SparkSubmitAction.KILL => kill(appArgs)
  9. case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
  10. }
  11. }

发表评论

表情:
评论列表 (有 0 条评论,376人围观)

还没有评论,来说两句吧...

相关阅读

    相关 Spark Shell 使用

    前言 前一章中我们介绍了Spark的`Standalone`模式的安装. 本章我们介绍下Spark Shell操作窗口的基本的安装. 基本启动与使用 基本算