hadoop 详细配置

我不是女神ヾ 2024-02-18 19:51 19阅读 0赞

根据官网http://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-common/SingleCluster.html 去做hadoop相应的配置

![Center][]

(1)Prepare to Start the Hadoop Cluster

hadoop.env.sh

export JAVA_HOME=/home/hadoop/java/jdk1.8.0_161

(2) Pseudo-Distributed Operation

core-site.xml

<property>
            <name>fs.defaultFS</name>
            <value>hdfs://hlx:8020</value>
        </property>
    	 <property>
            <name>hadoop.tmp.dir</name>
            <value>/home/hadoop/java/hadoop-2.7.6/data/tmp</value>
        </property>

说明： 将localhost:修改成hlx(或是IP或是localhost)都可以！  
将端口号9000修改成8020；原因是hadoop1.X版本是9000,hadoop2.X版本是8020;也可以不修改哦！  
![Image 1][]

*查找默认的配置文件core-default.xml*

http://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-common/SingleCluster.html

![Image 1][] ![20180425163154703][]

hadoop.tmp.dir=/tmp/hadoop-$\{user.name\} 默认是缓存目录，会过段时间就会删除的！![20180425162933951][]

![Image 1][]

那么就必须修改这个目录哦！（data/tmp 必须要创建这两个目录哦！）

![20180425163254415][]

hadoop.tmp.dir=/home/hadoop/java/hadoop-2.7.6/data/tmp

hdfs-site.xml

<property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>

主要是设置备份数，默认的3份，伪分布式，只要1份就可以了！

Setup passphraseless ssh （已配置,忽略）

Excution

1)格式化文件系统

$ bin/hdfs namenode -format

![20180425163420404][]

![20180425163436942][]

2）启动文件系统

$ sbin/start-dfs.sh

![Image 1][] ![20180425163452467][]

有两个进程：

NameNode：存储元数据  
 DataNode： 存储数据

jps==>查看进程

![20180425163534824][]

也可以通过日志来查看：

/home/hadoop/java/hadoop-2.7.6/logs/

![20180425163552779][]

log=>日志； out=>标准输出；

![20180425163615184][]

3）hdfs监控界面（Linux）  
 NameNode - http://localhost:50070/

![Image 1][] ![20180425163633102][]

![20180425163653601][]

查看集群clusterID= /home/hadoop/java/hadoop-2.7.6/data/tmp/dfs/data/current/VERSION

![20180425163732589][]

**Yarn on a Single Node**

4）yarn资源调度 mapred-site.xml

<property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>

修改这个mapred-site.xml的文件名哦！

![201804251638204][]

![Image 1][]

5) yarn-site.xml

<property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>

有两个进程:  
 ResourceManage:管理总资源分配  
 NodeManager ：管理该台节点资源

启动yarn

$ sbin/start-yarn.sh

![20180425163901727][]

6)yarn监控界面:  
 ResourceManager - http://localhost:8088/

![Image 1][] ![20180425163920807][]

![Image 1][]![20180425163928661][]

停止所有的进程

$ sbin/stop-dfs.sh
    
    $ sbin/stop-yarn.sh

[Center]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/31ba420d17c64b19a3ae97ab45a50bd0.png
[Image 1]: 
[20180425163154703]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/0e9b8a2f89d0433d8a25c574acd70f34.png
[20180425162933951]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/7884aaf5175a436d8b967f9afa7e0b6c.png
[20180425163254415]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/69d3929014a948528fb1695a0c3b172a.png
[20180425163420404]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/46bf0ac97019457287c6752f34c00470.png
[20180425163436942]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/9167a84cbfc441cfba434de6454221e1.png
[20180425163452467]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/2a2da790214b4d82beef62f2fbd032eb.png
[20180425163534824]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/cbcb994b45d1456e890ac254007dbf40.png
[20180425163552779]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/4c7f9b08b0414655ab76c824d6bfbc44.png
[20180425163615184]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/edcac60b232640f6a451ead695c8915e.png
[20180425163633102]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/19d2be0b08bc43ec9590adc5e338e474.png
[20180425163653601]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/186527061d3f49f985ec4b0deea2e42b.png
[20180425163732589]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/6049385cfba5470583a1589d33020a9b.png
[201804251638204]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/6a8899be339041959fa4c442c9917bd6.png
[20180425163901727]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/5790609fed3e4b489dc16c9fb5e192f7.png
[20180425163920807]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/0ed5434441164150b79e3d4f1f9a3ff2.png
[20180425163928661]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/29/aed6f4fadb414c46a233f455314acadb.png