Hadoop常用端口和定义方法

忘是亡心i 2023-06-27 03:53 4阅读 0赞

Hadoop常用端口和定义方法

Hadoop集群的各部分一般都会使用到多个端口，有些是daemon之间进行交互之用，有些是用于RPC访问以及HTTP访问。而随着Hadoop周边组件的增多，完全记不住哪个端口对应哪个应用，特收集记录如此，以便查询。

这里包含我们使用到的组件：HDFS, YARN, HBase, Hive, ZooKeeper:

<table> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;"> 组件 </td> 
 <td style="vertical-align:top;"> 节点 </td> 
 <td style="vertical-align:top;"> 默认端口 </td> 
 <td style="vertical-align:top;"> 配置 </td> 
 <td style="vertical-align:top;"> 用途说明 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> DataNode </td> 
 <td style="vertical-align:top;"> 50010 </td> 
 <td style="vertical-align:top;"> dfs.datanode.address </td> 
 <td style="vertical-align:top;"> datanode服务端口，用于数据传输 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> DataNode </td> 
 <td style="vertical-align:top;"> 50075 </td> 
 <td style="vertical-align:top;"> dfs.datanode.http.address </td> 
 <td style="vertical-align:top;"> http服务的端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> DataNode </td> 
 <td style="vertical-align:top;"> 50475 </td> 
 <td style="vertical-align:top;"> dfs.datanode.https.address </td> 
 <td style="vertical-align:top;"> https服务的端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> DataNode </td> 
 <td style="vertical-align:top;"> 50020 </td> 
 <td style="vertical-align:top;"> dfs.datanode.ipc.address </td> 
 <td style="vertical-align:top;"> ipc服务的端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> NameNode </td> 
 <td style="vertical-align:top;"> 50070 </td> 
 <td style="vertical-align:top;"> dfs.namenode.http-address </td> 
 <td style="vertical-align:top;"> http服务的端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> NameNode </td> 
 <td style="vertical-align:top;"> 50470 </td> 
 <td style="vertical-align:top;"> dfs.namenode.https-address </td> 
 <td style="vertical-align:top;"> https服务的端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> NameNode </td> 
 <td style="vertical-align:top;"> 8020 </td> 
 <td style="vertical-align:top;"> fs.defaultFS </td> 
 <td style="vertical-align:top;"> 接收Client连接的RPC端口，用于获取文件系统metadata信息。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> journalnode </td> 
 <td style="vertical-align:top;"> 8485 </td> 
 <td style="vertical-align:top;"> dfs.journalnode.rpc-address </td> 
 <td style="vertical-align:top;"> RPC服务 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> journalnode </td> 
 <td style="vertical-align:top;"> 8480 </td> 
 <td style="vertical-align:top;"> dfs.journalnode.http-address </td> 
 <td style="vertical-align:top;"> HTTP服务 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HDFS </td> 
 <td style="vertical-align:top;"> ZKFC </td> 
 <td style="vertical-align:top;"> 8019 </td> 
 <td style="vertical-align:top;"> dfs.ha.zkfc.port </td> 
 <td style="vertical-align:top;"> ZooKeeper FailoverController，用于NN HA </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> ResourceManager </td> 
 <td style="vertical-align:top;"> 8032 </td> 
 <td style="vertical-align:top;"> yarn.resourcemanager.address </td> 
 <td style="vertical-align:top;"> RM的applications manager(ASM)端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> ResourceManager </td> 
 <td style="vertical-align:top;"> 8030 </td> 
 <td style="vertical-align:top;"> yarn.resourcemanager.scheduler.address </td> 
 <td style="vertical-align:top;"> scheduler组件的IPC端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> ResourceManager </td> 
 <td style="vertical-align:top;"> 8031 </td> 
 <td style="vertical-align:top;"> yarn.resourcemanager.resource-tracker.address </td> 
 <td style="vertical-align:top;"> IPC </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> ResourceManager </td> 
 <td style="vertical-align:top;"> 8033 </td> 
 <td style="vertical-align:top;"> yarn.resourcemanager.admin.address </td> 
 <td style="vertical-align:top;"> IPC </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> ResourceManager </td> 
 <td style="vertical-align:top;"> 8088 </td> 
 <td style="vertical-align:top;"> yarn.resourcemanager.webapp.address </td> 
 <td style="vertical-align:top;"> http服务端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> NodeManager </td> 
 <td style="vertical-align:top;"> 8040 </td> 
 <td style="vertical-align:top;"> yarn.nodemanager.localizer.address </td> 
 <td style="vertical-align:top;"> localizer IPC </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> NodeManager </td> 
 <td style="vertical-align:top;"> 8042 </td> 
 <td style="vertical-align:top;"> yarn.nodemanager.webapp.address </td> 
 <td style="vertical-align:top;"> http服务端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> NodeManager </td> 
 <td style="vertical-align:top;"> 8041 </td> 
 <td style="vertical-align:top;"> yarn.nodemanager.address </td> 
 <td style="vertical-align:top;"> NM中container manager的端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> JobHistory Server </td> 
 <td style="vertical-align:top;"> 10020 </td> 
 <td style="vertical-align:top;"> mapreduce.jobhistory.address </td> 
 <td style="vertical-align:top;"> IPC </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> YARN </td> 
 <td style="vertical-align:top;"> JobHistory Server </td> 
 <td style="vertical-align:top;"> 19888 </td> 
 <td style="vertical-align:top;"> mapreduce.jobhistory.webapp.address </td> 
 <td style="vertical-align:top;"> http服务端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HBase </td> 
 <td style="vertical-align:top;"> Master </td> 
 <td style="vertical-align:top;"> 60000 </td> 
 <td style="vertical-align:top;"> hbase.master.port </td> 
 <td style="vertical-align:top;"> IPC </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HBase </td> 
 <td style="vertical-align:top;"> Master </td> 
 <td style="vertical-align:top;"> 60010 </td> 
 <td style="vertical-align:top;"> hbase.master.info.port </td> 
 <td style="vertical-align:top;"> http服务端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HBase </td> 
 <td style="vertical-align:top;"> RegionServer </td> 
 <td style="vertical-align:top;"> 60020 </td> 
 <td style="vertical-align:top;"> hbase.regionserver.port </td> 
 <td style="vertical-align:top;"> IPC </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HBase </td> 
 <td style="vertical-align:top;"> RegionServer </td> 
 <td style="vertical-align:top;"> 60030 </td> 
 <td style="vertical-align:top;"> hbase.regionserver.info.port </td> 
 <td style="vertical-align:top;"> http服务端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HBase </td> 
 <td style="vertical-align:top;"> HQuorumPeer </td> 
 <td style="vertical-align:top;"> 2181 </td> 
 <td style="vertical-align:top;"> hbase.zookeeper.property.clientPort </td> 
 <td style="vertical-align:top;"> HBase-managed ZK mode，使用独立的ZooKeeper集群则不会启用该端口。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HBase </td> 
 <td style="vertical-align:top;"> HQuorumPeer </td> 
 <td style="vertical-align:top;"> 2888 </td> 
 <td style="vertical-align:top;"> hbase.zookeeper.peerport </td> 
 <td style="vertical-align:top;"> HBase-managed ZK mode，使用独立的ZooKeeper集群则不会启用该端口。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> HBase </td> 
 <td style="vertical-align:top;"> HQuorumPeer </td> 
 <td style="vertical-align:top;"> 3888 </td> 
 <td style="vertical-align:top;"> hbase.zookeeper.leaderport </td> 
 <td style="vertical-align:top;"> HBase-managed ZK mode，使用独立的ZooKeeper集群则不会启用该端口。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> Hive </td> 
 <td style="vertical-align:top;"> Metastore </td> 
 <td style="vertical-align:top;"> 9083 </td> 
 <td style="vertical-align:top;"> /etc/default/hive-metastore中export PORT=来更新默认端口 </td> 
 <td style="vertical-align:top;">&nbsp;</td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> Hive </td> 
 <td style="vertical-align:top;"> HiveServer </td> 
 <td style="vertical-align:top;"> 10000 </td> 
 <td style="vertical-align:top;"> /etc/hive/conf/hive-env.sh中export HIVE_SERVER2_THRIFT_PORT=来更新默认端口 </td> 
 <td style="vertical-align:top;">&nbsp;</td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> ZooKeeper </td> 
 <td style="vertical-align:top;"> Server </td> 
 <td style="vertical-align:top;"> 2181 </td> 
 <td style="vertical-align:top;"> /etc/zookeeper/conf/zoo.cfg中clientPort= </td> 
 <td style="vertical-align:top;"> 对客户端提供服务的端口 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> ZooKeeper </td> 
 <td style="vertical-align:top;"> Server </td> 
 <td style="vertical-align:top;"> 2888 </td> 
 <td style="vertical-align:top;"> /etc/zookeeper/conf/zoo.cfg中server.x=[hostname]:nnnnn[:nnnnn]，标蓝部分 </td> 
 <td style="vertical-align:top;"> follower用来连接到leader，只在leader上监听该端口。 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> ZooKeeper </td> 
 <td style="vertical-align:top;"> Server </td> 
 <td style="vertical-align:top;"> 3888 </td> 
 <td style="vertical-align:top;"> /etc/zookeeper/conf/zoo.cfg中server.x=[hostname]:nnnnn[:nnnnn]，标蓝部分 </td> 
 <td style="vertical-align:top;"> 用于leader选举的。只在electionAlg是1,2或3(默认)时需要。 </td> 
 </tr> 
 </tbody> 
</table>

所有端口协议均基于TCP。

对于存在Web UI（HTTP服务）的所有hadoop daemon，有如下url：

**/logs**

日志文件列表，用于下载和查看

**/logLevel**

允许你设定log4j的日志记录级别，类似于hadoop daemonlog

**/stacks**

所有线程的stack trace，对于debug很有帮助

**/jmx**

服务端的Metrics，以JSON格式输出。

/jmx?qry=Hadoop:\*会返回所有hadoop相关指标。

/jmx?get=MXBeanName::AttributeName 查询指定bean指定属性的值，例如/jmx?get=Hadoop:service=NameNode,name=NameNodeInfo::ClusterId会返回ClusterId。