Hadoop Installation on Linux

墨蓝 2022-08-08 00:57 153阅读 0赞

1. Operation System

[hadoop@karei hadoop]$ uname -a
Linux karei 2.6.18-371.4.1.el5 #1 SMP Wed Jan 8 18:42:07 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
[hadoop@karei hadoop]$

2. Cluster Info

[root@karei ~]# clustat

3. Install JDK, and set environment variables for Java

a. Install

rpm -ihv jdk-8u45-linux-i586.rpm

b. Set environment variables

export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Note:

User: vi ~/.bash_profile —> source ~/.bash_profile

System: vi /etc/profile —> source /etc/profile

4. Download Hadoop from http://mirror.netinch.com/pub/apache/hadoop/common/

5. Decompression installation hadoop-2.7.0.tar.gz

a. gunzip hadoop-2.7.0.tar.gz

b. tar -xvf hadoop-2.7.0.tar.gz -C /usr/local/

c. cd /usr/local/

d. mv hadoop-2.7.0 hadoop

6. Set environment variables for Hadoop

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
#export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
#export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
#export HADOOP_OPTS=”$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib”

Note:

User: vi ~/.bash_profile —> source ~/.bash_profile

System: vi /etc/profile —> source /etc/profile

7. Create user and group for Hadoop for all nodes of cluster

a. Create group

groupadd bigdata

b. Create user

adduser hadoop

c. Add user into group

usermod -a -G bigdata hadoop

d. Set password for user

passwd hadoop

e. chown Hadoop home directory

chown hadoop:bigdata /usr/local/hadoop

8. Set RSA as Hadoop user for all nodes of cluster

a. ssh-keygen -t rsa
b. ssh-copy-id @

9. Configure nodes of cluster at file /etc/hosts

  1. [root@karei ~]# cat /etc/hosts
  2. 127.0.0.1 localhost.localdomain localhost
  3. ::1 localhost6.localdomain6 localhost6
  4. 100.99.167.53 gerra.bigdata.hadoop.net gerra
  5. 100.99.167.54 hemei.bigdata.hadoop.net hemei
  6. 100.99.167.55 karei.bigdata.hadoop.net karei
  7. 100.99.167.56 lephi.bigdata.hadoop.net lephi
  8. [root@karei ~]#

10. Configure masters and slaves under directory $HADOOP_HOME/etc/hadoop

[root@karei hadoop]#
[root@karei hadoop]# cat $HADOOP_HOME/etc/hadoop/masters
karei
hemei
[root@karei hadoop]#
[root@karei hadoop]#
[root@karei hadoop]# cat $HADOOP_HOME/etc/hadoop/slaves
gerra
lephi

[root@karei hadoop]#

11. Configure Hadoop

a. Configure $HADOOP_HOME/etc/hadoop/core-site.xml

  1. <?xml version="1.0"?>
  2. <configuration>
  3.   <property>
  4.     <name>fs.defaultFS</name>
  5.     <value>hdfs://karei:9000</value>
  6.   </property>
  7.   <property>
  8.     <name>io.file.buffer.size</name>
  9.     <value>131072</value>
  10.   </property>
  11.   <property>
  12.     <name>hadoop.tmp.dir</name>
  13.     <value>file:/usr/local/hadoop/tmp</value>
  14.     <description>Abase for other temporary directories.</description>
  15.   </property>
  16.   <property>
  17.     <name>hadoop.proxyuser.hadoop.hosts</name>
  18.     <value>*</value>
  19.   </property>
  20.   <property>
  21.     <name>hadoop.proxyuser.hadoop.groups</name>
  22.     <value>*</value>
  23.   </property>
  24. </configuration>

b. Configure $HADOOP_HOME/etc/hadoop/yarn-site.xml

  1. <?xml version="1.0"?>
  2. <configuration>
  3.   <property>
  4.     <name>yarn.nodemanager.aux-services</name>
  5.     <value>mapreduce_shuffle</value>
  6.   </property>
  7.   <property>
  8.     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  9.     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  10.   </property>
  11.   <property>
  12.     <name>yarn.resourcemanager.address</name>
  13.     <value>karei:8032</value>
  14.   </property>
  15.   <property>
  16.     <name>yarn.resourcemanager.scheduler.address</name>
  17.     <value>hemei:8030</value>
  18.   </property>
  19.   <property>
  20.     <name>yarn.resourcemanager.resource-tracker.address</name>
  21.     <value>karei:8031</value>
  22.   </property>
  23.   <property>
  24.     <name>yarn.resourcemanager.admin.address</name>
  25.     <value>karei:8033</value>
  26.   </property>
  27.   <property>
  28.     <name>yarn.resourcemanager.webapp.address</name>
  29.     <value>karei:8088</value>
  30.   </property>
  31. </configuration>

c. Configure $HADOOP_HOME/etc/hadoop/mapred-site.xml

  1. <?xml version="1.0"?>
  2. <configuration>
  3.   <property>
  4.     <name>mapreduce.framework.name</name>
  5.     <value>yarn</value>
  6.   </property>
  7.   <property>
  8.     <name>mapreduce.jobhistory.address</name>
  9.     <value>karei:10020</value>
  10.   </property>
  11.   <property>
  12.     <name>mapreduce.jobhistory.webapp.address</name>
  13.     <value>karei:19888</value>
  14.   </property>
  15. </configuration>

d. Configure $HADOOP_HOME/etc/hadoop/hdfs-site.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <configuration>
  4.   <property>
  5.     <name>dfs.namenode.secondary.http-address</name>
  6.     <value>hemei:9001</value>
  7.   </property>
  8.   <property>
  9.     <name>dfs.datanode.data.dir</name>
  10.     <value>file:/usr/local/hadoop/tmp/hdfs/datanode</value>
  11.     <description>DataNode directory for storing data chunks.</description>
  12.   </property>
  13.   <property>
  14.     <name>dfs.namenode.name.dir</name>
  15.     <value>file:/usr/local/hadoop/tmp/hdfs/namenode</value>
  16.     <description>NameNode directory for namespace and transaction logs storage.</description>
  17.   </property>
  18.   <property>
  19.     <name>dfs.replication</name>
  20.     <value>3</value>
  21.     <description>Number of replication for each chunk.</description>
  22.   </property>
  23.   <property>
  24.     <name>dfs.webhdfs.enabled</name>
  25.     <value>true</value>
  26.   </property>
  27. </configuration>

12. Format namenode

[root@karei hadoop]# ./bin/hadoop namenode -format

13. Start Hadoop

[hadoop@karei hadoop]$ ./sbin/start-all.sh 

  1. This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
  2. Java HotSpot(TM) Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
  3. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
  4. 15/04/28 21:46:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  5. Starting namenodes on [karei]
  6. karei: namenode running as process 10985. Stop it first.
  7. gerra: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-gerra.out
  8. lephi: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-lephi.out
  9. gerra: Java HotSpot(TM) Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
  10. gerra: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
  11. lephi: Java HotSpot(TM) Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
  12. lephi: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
  13. Starting secondary namenodes [hemei]
  14. hemei: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-hemei.out
  15. hemei: Java HotSpot(TM) Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
  16. hemei: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
  17. Java HotSpot(TM) Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
  18. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
  19. 15/04/28 21:47:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  20. starting yarn daemons
  21. starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-karei.out
  22. gerra: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-gerra.out
  23. lephi: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-lephi.out

Note: Java HotSpot(TM) Server VM warning

  1. Java HotSpot(TM) Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.  
  2. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

Solve: Set environment about HADOOP_COMMON_LIB_NATIVE_DIR and HADOOP_OPTS
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS=”$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib”

Note:

User: vi ~/.bash_profile —> source ~/.bash_profile

System: vi /etc/profile —> source /etc/profile

14. Check status

[hadoop@karei hadoop]$ jps

发表评论

表情：
评论列表 （有 0 条评论，153人围观）

还没有评论，来说两句吧...

相关阅读