大数据入门介绍一

青旅半醒 2024-02-19 17:24 56阅读 0赞

**Hadoop:**  
广义: 以apache hadoop软件为主的生态圈(hive zookeeper spark hbase)  
狭义: apache hadoop软件

[hadoop.apache.org][]  
[hive.apache.org][]  
[spark.apache.org][]

**hadoop软件:**  
1.x 企业不用  
2.x 主流  
3.x 没有企业敢用  
a.采坑  
b.**很多公司都是CDH5.x部署大数据环境 ([www.cloudera.com][])  
2.6.0-cdh5.7.0 =? apache hadoop2.6.0**

**hadoop软件:**  
hdfs：存储 分布式文件系统  
mapreduce：计算 job1 job2 编码 java but 企业不用(开发难度高 代码量大 计算慢)  
yarn：资源（CPU memory）和作业调度

上课: 2.6.0-cdh5.7.0  
2.6.0-cdh5.14.0

apache hadoop： [hadoop.apache.org][]  
cdh hadoop: [http://archive.cloudera.com/cdh5/cdh/5/][http_archive.cloudera.com_cdh5_cdh_5]

wget [http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz][http_archive.cloudera.com_cdh5_cdh_5_hadoop-2.6.0-cdh5.7.0.tar.gz]  
rz命令上传

学习: 伪分布式 1台

Ubuntu Linux:  
$ sudo apt-get install ssh  
$ sudo apt-get install rsync

CentOS：  
yum

--------------------

linux root用户  
mysql mysqladmin用户  
hadoop hadoop用户

### 1.创建用户和上传hadoop软件 ###

useradd hadoop
    su - hadoop
    [hadoop@hadoop002 ~]$ mkdir app
    [hadoop@hadoop002 ~]$ cd app/
    
    [hadoop@hadoop002 ~]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
    或
    [hadoop@hadoop002 ~]$ rz 上传

### 2.部署jdk ###

CDH环境:
    mkdir /usr/java   jdk部署  这里面目录就确定了，固定的
    mkdir /usr/share/java   部署CDH需要mysql jdbc jar包   
    
    rz上传 jdk-8u45-linux-x64.gz
    
    解压
    [root@hadoop002 java]# tar -xzvf jdk-8u45-linux-x64.gz
    [root@hadoop002 java]# ll
    解压之后，一定要ll看一下，你的用户和用户组是否发生变化，
    total 319156
    drwxr-xr-x 8 uucp  143      4096 Apr 11  2015 jdk1.8.0_45
    -rw-r--r-- 1 root root 153530841 Jul  8  2015 jdk-7u80-linux-x64.tar.gz
    -rw-r--r-- 1 root root 173271626 Sep 19 11:49 jdk-8u45-linux-x64.gz
    
    权限修正
    [root@hadoop002 java]# chown -R root:root jdk1.8.0_45
    [root@hadoop002 java]# ll
    total 319156
    drwxr-xr-x 8 root root      4096 Apr 11  2015 jdk1.8.0_45
    -rw-r--r-- 1 root root 153530841 Jul  8  2015 jdk-7u80-linux-x64.tar.gz
    -rw-r--r-- 1 root root 173271626 Sep 19 11:49 jdk-8u45-linux-x64.gz
    [root@hadoop002 java]# vi /etc/profile
    
    #env
    export JAVA_HOME=/usr/java/jdk1.8.0_45
    export JRE_HOME=$JAVA_HOME/jre
    export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH
    export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH
    
    
    [root@hadoop002 java]# source /etc/profile
    [root@hadoop002 java]# which java
    /usr/java/jdk1.8.0_45/bin/java
    [root@hadoop002 java]# 
    [root@hadoop002 java]#
    
    
    解释器 /bin/bash
    /sbin/nologin

### CDH: ###

组件 用户  
hdfs hdfs  
yarn yarn  
zookeeper zookeeper  
hbase hbase

su - zookeeper切不了：This account is currently not available.  
生产怎么做：/sbin/nologin --》 /bin/bash

### 3.解压hadoop ###

[hadoop@hadoop002 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
    [hadoop@hadoop002 app]$ cd hadoop-2.6.0-cdh5.7.0
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ll
    total 76
    drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin  可执行脚本
    drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1
    drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera
    drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc  配置目录(conf)
    drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples
    drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1
    drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include
    drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib  jar包目录
    drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec
    drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin hadoop组件的启动 停止脚本
    drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share
    drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

### 4.Configuration ###

Use the following:
    
    etc/hadoop/core-site.xml:
    
    <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
        </property>
    </configuration>
    
    
    etc/hadoop/hdfs-site.xml:
    <configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
    </configuration>

### 5.配置ssh localhost无密码信任关系 ###

[hadoop@hadoop002 ~]$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
    Created directory '/home/hadoop/.ssh'.
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /home/hadoop/.ssh/id_rsa.
    Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
    The key fingerprint is:
    ba:48:3d:ff:af:4d:da:74:67:31:d6:98:ad:a0:b3:76 hadoop@hadoop002
    The key's randomart image is:
    +--[ RSA 2048]----+
    |                 |
    |                 |
    |                 |
    |               +.|
    |        S   . o+o|
    |     . .   . ...o|
    |    . +   o o o o|
    |   . . +  .OE. o |
    |    . . .o=++    |
    +-----------------+
    [hadoop@hadoop002 ~]$ cd .ssh
    [hadoop@hadoop002 .ssh]$ ll
    total 8
    -rw------- 1 hadoop hadoop 1675 Feb 13 22:36 id_rsa     私钥
    -rw-r--r-- 1 hadoop hadoop  398 Feb 13 22:36 id_rsa.pub 公钥
    [hadoop@hadoop002 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    [hadoop@hadoop002 .ssh]$ 
    
    [hadoop@hadoop002 .ssh]$ ll
    total 12
    -rw-rw-r-- 1 hadoop hadoop  398 Feb 13 22:37 authorized_keys
    -rw------- 1 hadoop hadoop 1675 Feb 13 22:36 id_rsa
    -rw-r--r-- 1 hadoop hadoop  398 Feb 13 22:36 id_rsa.pub
    -rw-r--r-- 1 hadoop hadoop    0 Feb 13 22:39 known_hosts
    
    ssh localhost date 是需要输入密码，但是这个用户是没有配置密码。
    我们应该在没有配置密码情况下去完成无密码信任呢？
    
    改权限
    [hadoop@hadoop002 .ssh]$ chmod 600 authorized_keys
    [hadoop@hadoop002 .ssh]$ ssh localhost date
    The authenticity of host 'localhost (127.0.0.1)' can't be established.
    RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
    Wed Feb 13 22:41:17 CST 2019
    [hadoop@hadoop002 .ssh]$ 
    [hadoop@hadoop002 .ssh]$ 
    [hadoop@hadoop002 .ssh]$ ssh localhost date
    Wed Feb 13 22:41:22 CST 2019
    [hadoop@hadoop002 .ssh]$

### 6.格式化 ###

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format

### 7.启动 ###

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
    19/02/13 22:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    
    Starting namenodes on [localhost]
    localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
    localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out

### ssh 信任关系 是配置localhost ###

Starting secondary namenodes [0.0.0.0]
    The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
    RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
    Are you sure you want to continue connecting (yes/no)? yes
    0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
    0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
    19/02/13 22:49:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    
    ssh localhost date
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/stop-dfs.sh
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
    19/02/13 22:57:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Starting namenodes on [localhost]
    localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
    localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out
    Starting secondary namenodes [0.0.0.0]
    0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
    19/02/13 22:57:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ jps
    15059 Jps
    14948 SecondaryNameNode 第二名称节点 老二
    14783 DataNode  数据节点  小弟
    14655 NameNode  名称节点  老大 读写
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    
    open http://ip:50070

### 8.环境变量 ###

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ cat ~/.bash_profile
    
    # .bash_profile
    
    # Get the aliases and functions
    if [ -f ~/.bashrc ]; then
            . ~/.bashrc
    fi
    
    # User specific environment and startup programs
    export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
    export PATH=$HADOOP_PREFIX/bin:$PATH
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

### 9.命令: hdfs dfs操作命令和Linux命令极其相似 ###

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
    19/02/13 23:08:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
    19/02/13 23:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /
    bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
    boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -mkdir /ruozedata
    19/02/13 23:11:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
    19/02/13 23:11:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Found 1 items
    drwxr-xr-x   - hadoop supergroup          0 2019-02-13 23:11 /ruozedata
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /
    bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
    boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
    [hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

[hadoop.apache.org]: http://hadoop.apache.org
[hive.apache.org]: http://hive.apache.org
[spark.apache.org]: http://spark.apache.org
[www.cloudera.com]: http://www.cloudera.com
[http_archive.cloudera.com_cdh5_cdh_5]: http://archive.cloudera.com/cdh5/cdh/5/
[http_archive.cloudera.com_cdh5_cdh_5_hadoop-2.6.0-cdh5.7.0.tar.gz]: http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz