Hadoop配置实验

左手的ㄟ右手 2022-01-20 23:13 187阅读 0赞

# hadoop配置实验 #

## 安装java jdk ##

服务器

cd /home
    mkdir software

在本地下载 [https://download.oracle.com/otn/java/jdk/8u212-b12/478a62b7d4e34b78b671c754eaaf38ab/jdk-8u211-linux-x64.tar.gz][https_download.oracle.com_otn_java_jdk_8u212-b12_478a62b7d4e34b78b671c754eaaf38ab_jdk-8u211-linux-x64.tar.gz]

本地终端

scp /Users/xujiaxing/Downloads/jdk-8u212-linux-x64.tar.gz root@10.251.254.72:./

服务器

cd ~
    mv jdk-8u212-linux-x64.tar.gz /home/software
    cd /home/software
    tar -zxvf jdk-8u212-linux-x64.tar.gz

vim /etc/profile

添加

export JAVA_HOME=/home/software/jdk1.8.0_212
    export PATH=$PATH:$JAVA_HOME/bin

刷新配置

source /etc/profile

检测是否成功安装：

java -version

![屏幕快照 2019-06-05 00.49.32.png][2019-06-05 00.49.32.png]

## 安装hadoop ##

服务器

mkdir /cloud

在本地下载 [https://archive.apache.org/dist/hadoop/core/hadoop-3.2.0/hadoop-3.2.0.tar.gz][https_archive.apache.org_dist_hadoop_core_hadoop-3.2.0_hadoop-3.2.0.tar.gz]

本地终端

scp /Users/xujiaxing/Downloads/hadoop-3.2.0.tar.gz root@10.251.254.72:./

服务器

mv ~/hadoop-3.2.0.tar.gz /cloud
    cd /cloud
    tar -zxvf hadoop-3.2.0.tar.gz -C /cloud/
    cd /cloud/hadoop3.2.0/etc/hadoop

修改配置文件

vim hadoop-env.sh

添加

export JAVA_HOME=/home/software/jdk1.8.0_212

修改core-site.xml

vim core-site.xml

配置内容如下

<configuration>
    <property>
            <name>hadoop.tmp.dir</name>
            <value>file:/usr/local/hadoop/tmp</value>
            <description>A base for other temporary directories.</description>
        </property>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
        </property>
    </configuration>

修改hdfs-site.xml

vim hdfs-site.xml

修改配置如下

<configuration>
    <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
        <property>
            <name>dfs.namenode.name.dir</name>
            <value>file:/usr/local/hadoop/tmp/dfs/name</value>
        </property>
        <property>
            <name>dfs.datanode.data.dir</name>
            <value>file:/usr/local/hadoop/tmp/dfs/data</value>
        </property>
    
    </configuration>

修改mapred-site.xml

mapred-site.xml

修改配置如下

<configuration>
    
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    </configuration>

修改yarn-site.xml

vim yarn-site.xml

修改内容如下

<configuration>
    
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>localhost</value>
    </property>
    </configuration>

将hadoop添加到环境变量

vim /etc/profile

添加

export HADOOP_HOME=/cloud/hadoop-3.2.0
    export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

然后更新一下环境变量

source /etc/profile

## 配置SSH无密码登录 ##

ssh-keygen -t rsa
    cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
    chmod 600 /root/.ssh/authorized_keys

测试

ssh localhost

![屏幕快照 2019-06-06 21.13.47.png][2019-06-06 21.13.47.png]

## 启动hadoop ##

进入Hadoop目录

cd /cloud/hadoop-3.2.0/etc/hadoop

执行

hdfs namenode -format

返回Hadoop根目录

cd /cloud/hadoop-3.2.0

启动yarn

sbin/start-yarn.sh

![屏幕快照 2019-06-06 21.08.01.png][2019-06-06 21.08.01.png]

![屏幕快照 2019-06-06 21.08.44.png][2019-06-06 21.08.44.png]

启动hdfs

sbin/start-dfs.sh

![屏幕快照 2019-06-06 21.08.56.png][2019-06-06 21.08.56.png]

## Reference ##

[https://www.cnblogs.com/lijingchn/p/5574476.html][https_www.cnblogs.com_lijingchn_p_5574476.html]

[https://www.cnblogs.com/gyouxu/p/4183417.html][https_www.cnblogs.com_gyouxu_p_4183417.html]

[https://www.cnblogs.com/xubing-613/p/6844564.html][https_www.cnblogs.com_xubing-613_p_6844564.html]

[https://segmentfault.com/q/1010000007718843][https_segmentfault.com_q_1010000007718843]

[https://blog.csdn.net/lglglgl/article/details/80553828][https_blog.csdn.net_lglglgl_article_details_80553828]

[https_download.oracle.com_otn_java_jdk_8u212-b12_478a62b7d4e34b78b671c754eaaf38ab_jdk-8u211-linux-x64.tar.gz]: https://download.oracle.com/otn/java/jdk/8u212-b12/478a62b7d4e34b78b671c754eaaf38ab/jdk-8u211-linux-x64.tar.gz
[2019-06-05 00.49.32.png]: https://i.loli.net/2019/06/05/5cf6a137518c812106.png
[https_archive.apache.org_dist_hadoop_core_hadoop-3.2.0_hadoop-3.2.0.tar.gz]: https://archive.apache.org/dist/hadoop/core/hadoop-3.2.0/hadoop-3.2.0.tar.gz
[2019-06-06 21.13.47.png]: https://i.loli.net/2019/06/06/5cf91194dfdbe97207.png
[2019-06-06 21.08.01.png]: https://i.loli.net/2019/06/06/5cf9107c5a9f193178.png
[2019-06-06 21.08.44.png]: https://i.loli.net/2019/06/06/5cf9107b8084873431.png
[2019-06-06 21.08.56.png]: https://i.loli.net/2019/06/06/5cf9107cc7b0631477.png
[https_www.cnblogs.com_lijingchn_p_5574476.html]: https://www.cnblogs.com/lijingchn/p/5574476.html
[https_www.cnblogs.com_gyouxu_p_4183417.html]: https://www.cnblogs.com/gyouxu/p/4183417.html
[https_www.cnblogs.com_xubing-613_p_6844564.html]: https://www.cnblogs.com/xubing-613/p/6844564.html
[https_segmentfault.com_q_1010000007718843]: https://segmentfault.com/q/1010000007718843
[https_blog.csdn.net_lglglgl_article_details_80553828]: https://blog.csdn.net/lglglgl/article/details/80553828