Hadoop 2.x配置及简单MapReduce案例

- 日理万妓 2022-05-24 23:17 241阅读 0赞

# 去官网下载压缩包 #

# 配置jdk(hadoop运行在java环境上) #

# Hadoop 2.x基本配置及运行MapReduce案例在本地模式下 #

在hadoop文件下

mkdir input
    cp etc/hadoop/*.xml input/
    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar grep input output 'dfs[a-z.]+'
    cat output/part-r-00000 
    
    
    ##小案例
    mkdir wcinput
    cd wcinput/
    touch wc.input
    vi wc.input 
        hadoop mapreduce
        hadoop hdfs
        yaen nodemanger
        hadoop resourcemanager
    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar woedcount wcinput wcoutputeduce-examples-2.5.0.jar wordcount wcinput wcoutput

![这里写图片描述][70]

# 配置部署启动HDFS及本地模式运行MapReduce案例（使用HDFS上数据） #

**修改主机名：**

vi /etc/sysconfig/network
    vi /etc/hosts
    /etc/rc.d/init.d/network restart

![这里写图片描述][70 1]

**修改/opt/modules/hadoop-2.5.0/etc/hadoop/core-site.xml**  
![这里写图片描述][70 2]  
**修改/opt/modules/hadoop-2.5.0/etc/hadoop/hdfs-site.xml**  
![这里写图片描述][70 3]

**格式化hdfs**

bin/hdfs namenode -format

**启动主节点**

sbin/hadoop-daemon.sh start namenode

**启动从节点**

sbin/hadoop-daemon.sh start datanode

**查看hadoop控制台**  
![这里写图片描述][70 4]

**新增hdfs文件**

bin/hdfs dfs -mkdir -p /user/beifeng/

**查询hdfs文件**

bin/hdfs dfs -ls -R /

**hdfs上传文件**

bin/hdfs dfs -put wcinput/wc.input /user/beifeng/mapreduce/wordcount/input/

**运行MapReduce程序**

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar woedcount /user/beifeng/mapreduce/wordcount/input/ /user/beifeng/mapreduce/wordcount/output

**查看运行结果**

bin/hdfs dfs -cat /user/beifeng/mapreduce/wordcount/output/part*

# 配置部署启动YARN及在YARN上运行MapReduce程序 #

**配置yarn-site.xml**

<configuration>
            <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
            </property>
    
            <property>
                    <name>yarn.resourcemanager.hostname</name>
                    <value>min</value>
            </property>
    
    </configuration>

**配置slaves**

vi etc/hadoop/slaves 
    
    min

**启动：**

sbin/yarn-daemon.sh start resourcemanager sbin/yarn-daemon.sh start nodemanager

**访问：**  
![这里写图片描述][70 5]

**配置mapred-env.sh**

vi etc/hadoop/mapred-env.sh 
    ##修改jdk配置路径
    export JAVA_HOME=/usr/java/jdk1.8.0_11

# 历史任务服务器启动 #

sbin/mr-jobhistory-daemon.sh start historyserver

![这里写图片描述][70 6]

# YARN的日志聚集功能配置使用 #

**日志聚集功能配置**  
![这里写图片描述][70 7]

**停止resourcemanager**

sbin/yarn-daemon.sh stop resourcemanager

**停止noderesourcemanager**

sbin/yarn-daemon.sh stop nodemanager

**停止历史任务服务器**

sbin//mr-jobhistory-daemon.sh stop historyserver

![这里写图片描述][70 8]

**启动resourcemanager、resourcemanager、历史任务服务器**  
![这里写图片描述][70 9]

**查看log日志**  
![这里写图片描述][70 10]

**配置HDFS垃圾回收**

vi etc/hadoop/core-site.xml

![这里写图片描述][70 11]

![这里写图片描述][70 12]

# Hadoop 2.x组件启动的三种方式及配置SSH无密码登录 #

![这里写图片描述][70 13]  
**不建议用下面这种**  
![这里写图片描述][70 14]

**配置SSH无密码登录**

cd
    cd .ssh
    ssh-keygen -t rsa
    ssh-copy-id min

![这里写图片描述][70 15]

# 配置namenode第二个节点 #

vi etc/hadoop/hdfs-site.xml

![这里写图片描述][70 16]

**配置历史服务器**

vi etc/hadoop/mapred-site.xml

![这里写图片描述][70 17]

[70]: /images/20220525/b9fc7a78adf2447787c3a8a47eb586aa.png
[70 1]: /images/20220525/437f374f97414030a107fe38298ecb1e.png
[70 2]: /images/20220525/147d213adffa4c039637f672645923c4.png
[70 3]: /images/20220525/e73f5aa90a744a71ae4242b7074b4ef4.png
[70 4]: /images/20220525/746a5055e64d4ca5a576f3958f70f9d9.png
[70 5]: /images/20220525/39328806eb4d4c259169d4f50e197cb5.png
[70 6]: /images/20220525/8e7924133fa64522957ae6bd25b22faf.png
[70 7]: /images/20220525/bb84c92c1111437ebbb73155751d8d0f.png
[70 8]: /images/20220525/2350f59c14174937bb64e72219a53f1e.png
[70 9]: /images/20220525/83f076f319a841fab5121a420f4a1f3d.png
[70 10]: /images/20220525/3af603ca6bc147bc8e543e43c87356f6.png
[70 11]: /images/20220525/dfc04b293333400b9e34daae33e7479e.png
[70 12]: /images/20220525/2580ffc4c71143fe9c2bd0e1dd08b0dd.png
[70 13]: /images/20220525/bfae66c675264be4b65e6c8a6ff6a5f1.png
[70 14]: /images/20220525/91c9cfe57d59470fb7efa8874d79803b.png
[70 15]: /images/20220525/248469c59f81442ab7da1fc2459f8765.png
[70 16]: /images/20220525/6851837a090b4427b0c5db813ba78a5c.png
[70 17]: /images/20220525/243dc4d98862471c8c876ef0ab5fc394.png