Hadoop 2.7.2版本单机搭建

蔚落 2023-02-23 12:28 57阅读 0赞

## 安装java环境 ##

这里安装的是`1.8.0_211`版本

## 下载hadoop压缩包 ##

各版本的Hadoop安装包下载地址：[http://archive.apache.org/dist/hadoop/common/][http_archive.apache.org_dist_hadoop_common]  
这里选择`2.7.2`。

mkdir /home/hadoop
    cd /home/hadoop
    wget http://archive.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
    tar -zxvf hadoop-2.7.2.tar.gz
    cd hadoop-2.7.2

## 设置环境变量 ##

vim /etc/profile
    # 在最后面增加如下两行代码
    export HADOOP_HOME=/home/hadoop/hadoop-2.7.2
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    # 保存退出
    source /etc/profile

## 修改运行文件的JAVA\_HOME ##

`vim etc/hadopp/hadoop-env.sh`  
然后修改`export JAVA_HOME=${JAVA_HOME}`为`export JAVA_HOME=/home/java/jdk1.8.0_211`

`vim etc/hadopp/yarn-env.sh`  
然后将`# export JAVA_HOME=/home/y/libexec/jdk1.6.0`为`export JAVA_HOME=/home/java/jdk1.8.0_211`

`vim etc/hadoop/mapred-env.sh`  
然后将`# export JAVA_HOME=/home/y/libexec/jdk1.6.0`为`export JAVA_HOME=/home/java/jdk1.8.0_211`

## 本地运行模式 ##

此模式下都是使用的本地目录

### grep测试用例 ###

mkdir input
    cp etc/hadoop/*.xml input
    ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+'
    cat output/*
    # 输出结果
    1	dfsadmin

### wordcount测试用例 ###

mkdir wcinput
    vim wcinput/wc.input
    # 下面为文件内容
    # a b
    # b c
    ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount wcinput wcoutput
    cat wcoutput/*
    # 输出结果
    a	1
    b	2
    c	1
    d	1

## 修改配置文件 ##

### hdfs配置文件 ###

#### core-site.xml ####

<configuration>
        
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://hostname:9000</value>
        </property>
    
        
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/home/hadoop/hadoop-2.7.2/data/tmp</value>
        </property>
    </configuration>

#### hdfs-site.xml ####

<configuration>
        
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
    </configuration>

### yarn配置文件 ###

#### yarn-site.xml ####

<configuration>
    
        
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    
        
        <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>nightwatch</value>
        </property>
    </configuration>

#### mapred-site.xml ####

`cp mapred-site.xml.template mapred-site.xml`

<configuration>
        
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>

## 启动服务 ##

### hdfs服务 ###

# 格式化namenode，只有首次启动namenode需要
    # 如果再次格式话则会导致datanode启动失败，则需要删除/home/hadoop/hadoop-2.7.2/data/tmp/data目录内容
    bin/hdfs namenode -format
    # 启动namenode
    hadoop-daemon.sh start namenode
    # 启动datanode
    hadoop-daemon.sh start datanode
    # 查看启动进程
    jps
    # 输出结果
    2844 Jps
    1327 DataNode
    1247 NameNode

启动完成之后，可以进入`http://hostname:50070/`，查看是否启动成功  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5ODAwNDM0_size_16_color_FFFFFF_t_70]

### yarn服务 ###

# 启动resource manager
    yarn-daemon.sh start resourcemanager
    # 启动node manager
    yarn-daemon.sh start nodemanager
    # 查看启动进程
    jps
    # 输出结果
    2012 NodeManager
    2844 Jps
    1774 ResourceManager
    1327 DataNode
    1247 NameNode

启动完成之后，可以进入`http://hostname:8088/cluster/nodes`，然后点击`node`  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5ODAwNDM0_size_16_color_FFFFFF_t_70 1]

## 伪分布式运行模式 ##

# 将本地文件上传至hdfs根目录下
    hadoop fs -put wcinput /
    # 查看是否上传成功
    hdfs dfs -ls /
    # 运行jar包
    ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /wcinput /wcoutput
    # 查看运行结果
    hdfs dfs -cat /wcoutput/*

[http_archive.apache.org_dist_hadoop_common]: http://archive.apache.org/dist/hadoop/common/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5ODAwNDM0_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200707185943735.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5ODAwNDM0,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5ODAwNDM0_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200707190938134.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM5ODAwNDM0,size_16,color_FFFFFF,t_70