hadoop yarn

电玩女神 2022-07-13 07:12 162阅读 0赞

# The fundamental idea of YARN is to split up the functionalities of resource management and job scheduling/monitoring into separate daemons. The idea is to have a global ResourceManager (*RM*) and per-application ApplicationMaster (*AM*). An application is either a single job or a DAG of jobs. #

yarn的核心思想是把资源管理和作业调度/监控分成两个独立的守护进程。

基于这个思想，设计了一个全局的资源管理器ResourceManager(RM)和为每个应用程序提供一个单应用程序控制器Application Master（AM）。

一个应用程序是一个单独的作业或者一组作业。

The ResourceManager and the NodeManager form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler.

资源管理器RM和节点管理器NM构成了数据计算框架。

资源管理器RM的天职是为系统中的所有应用程序仲裁资源。

节点管理器NM是单机器框架的代理，主要负责负责监控容器中资源（cpu、内存、磁盘、网络）的使用，报告相关情况给资源管理/调度器。

解释：也就是说，RM和NM组成了数据计算框架，RM管理系统中所有资源的框架，NM是管理容器的机器级别的框架（管理机器cpu内存硬盘网络资源）并汇报给

RM/Scheduler。

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.

单应用程序控制器实际是一个框架类库，用来与资源管理器RM协调资源，和节点管理器NM一起执行和监控任务。

![Center][]

The ResourceManager has two main components: Scheduler and ApplicationsManager.

资源管理器RM有个主要的组件：调度器和应用程序管理器。

The Scheduler is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is pure scheduler in the sense that it performs no monitoring or tracking of status for the application. Also, it offers no guarantees about restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based the resource requirements of the applications; it does so based on the abstract notion of a resource*Container* which incorporates elements such as memory, cpu, disk, network etc.

调度器主要负责为各种各样的正在运行的拥有相似约束（如：容量、队列约束等）的应用程序分配资源。

调度器仅仅是调度，并不监控、跟踪应用程序的状态。

此外，它提供不可靠的重启失败任务的服务。失败任务是应用程序失败或硬件失败导致的。

调度器基于应程序的资源要求执行它的调度功能。

当接收到资源容器（主要包含内存、cpu、磁盘、网络资源等）的抽象通知，它也会进行资源度。

The Scheduler has a pluggable policy which is responsible for partitioning the cluster resources among the various queues, applications etc. The current schedulers such as the [CapacityScheduler][] and the [FairScheduler][] would be some examples of plug-ins.

调度器有一个插接式连接策略，主要负责给各种各样的队列、应用程序等分配集群资源。现有的调度器如CapacityScheduler 和FairScheduler都是此类插件程序的例子。

The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.

应用程序管理器主要负责接收作业的提交，为执行应用程序指定的应用程序控制器协调容器，提供了失败重启应用程序控制器的服务。应用程序控制器有从调度器请求恰当的资源容器、跟踪容器的状态、监控进度的责任。

MapReduce in hadoop-2.x maintains **API compatibility** with previous stable release (hadoop-1.x). This means that all MapReduce jobs should still run unchanged on top of YARN with just a recompile.

在hadoop-2.X中MapReduce保持API对早期稳定版(hadoop-1.x)的兼容性。

这意味着所有的MapReduce作业只需要重新编译一下就可以应该仍然运行在YARN上端而不需要做出任何改变。

[Center]: /images/20220713/dd775ed04fb24a40ba972bbf626e4c7f.png
[CapacityScheduler]: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
[FairScheduler]: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html