Centos7下Hadoop3.x源码编译(Zstd问题已解决)

素颜马尾好姑娘i 2022-02-19 03:51 1029阅读 0赞

## 工具 ##

<table> 
 <thead> 
  <tr> 
   <th>组件</th> 
   <th>版本</th> 
   <th>下载地址</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>JDK</td> 
   <td>1.8.0_131</td> 
   <td><a href="https://www.oracle.com/technetwork/java/javase/downloads/index.html" rel="nofollow">https://www.oracle.com/technetwork/java/javase/downloads/index.html</a></td> 
  </tr> 
  <tr> 
   <td>Hadoop</td> 
   <td>3.1.1</td> 
   <td><a href="http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz" rel="nofollow">http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz</a></td> 
  </tr> 
  <tr> 
   <td>Maven</td> 
   <td>3.6.0</td> 
   <td><a href="http://mirrors.hust.edu.cn/apache/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz" rel="nofollow">http://mirrors.hust.edu.cn/apache/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz</a></td> 
  </tr> 
  <tr> 
   <td>CMake</td> 
   <td>3.13.1</td> 
   <td><a href="https://cmake.org/files/v3.13/cmake-3.13.1.tar.gz" rel="nofollow">https://cmake.org/files/v3.13/cmake-3.13.1.tar.gz</a></td> 
  </tr> 
  <tr> 
   <td>ProtocolBuffer</td> 
   <td>2.5.0</td> 
   <td><a href="https://pan.baidu.com/s/179vRCDoK1vp1u7S8NgZD6Q" rel="nofollow">https://pan.baidu.com/s/179vRCDoK1vp1u7S8NgZD6Q</a></td> 
  </tr> 
 </tbody> 
</table>

编译完成生成的软件包

<table> 
 <thead> 
  <tr> 
   <th>软件包名</th> 
   <th>下载地址</th> 
   <th>描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>hadoop-3.1.1.tar.gz</td> 
   <td><a href="https://pan.baidu.com/s/1UFHq6dxoh5AiqO_cd85NTw" rel="nofollow">https://pan.baidu.com/s/1UFHq6dxoh5AiqO_cd85NTw</a></td> 
   <td>CentOS7.4下hadoop3.1.1源码编译</td> 
  </tr> 
  <tr> 
   <td>native-hadoop-3.1.1-with-sztd.tar.gz</td> 
   <td><a href="https://download.csdn.net/download/weixin_43858859/10832098" rel="nofollow">https://download.csdn.net/download/weixin_43858859/10832098</a></td> 
   <td>hadoop3.1.1源码编译生成的native，含zstd</td> 
  </tr> 
 </tbody> 
</table>

编译所需要工具和软件包  
下载hadoop-3.1.1-src.tar.gz源码包，解压并查看BUILDING.txt

[hadoop@hadoop src]$cat BUILDING.txt
    ...
    Requirements:
    * Unix System
    * JDK 1.8
    * Maven 3.3 or later
    * ProtocolBuffer 2.5.0
    * CMake 3.1 or newer (if compiling native code)
    * Zlib devel (if compiling native code)
    * openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
    * Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
    * Internet connection for first build (to fetch all Maven and Hadoop dependencies)
    * python (for releasedocs)
    * bats (for shell code testing)
    * Node.js / bower / Ember-cli (for YARN UI v2 building)
    ...

## 软件安装 ##

**安装JDK**  
下载jdk-8u131-linux-x64.tar.gz并上传至/home/hadoop/src  
使用root用户创建/usr/java目录，安装jdk

[root@hadoop ~]# mkdir /usr/local/maven
    [root@hadoop ~]#

解压jdk

[hadoop@hadoop src]$ tar -zxvf /home/hadoop/src/jdk-8u131-linux-x64.tar.gz -C /usr/java/

修改属主

[hadoop@hadoop ~]$ chown -R hadoop:hadoop /usr/java

配置环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
    ## ADD java environment
    export JAVA_HOME=/usr/java/jdk1.8.0_131
    export PATH=$JAVA_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证JDK

[hadoop@hadoop ~]$ java -version
    java version "1.8.0_131"
    Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
    [hadoop@hadoop ~]$

**安装Maven**  
下载apache-maven-3.6.0-bin.tar.gz并上传至/home/hadoop/src  
使用root用户创建/usr/local/maven目录，安装maven

[root@hadoop ~]# mkdir /usr/local/maven  ##创建maven安装目录
    [root@hadoop ~]# chown -R hadoop:hadoop /usr/local/maven  ##修改目录属主

解压maven

[hadoop@hadoop src]$ tar -zxvf apache-maven-3.6.0-bin.tar.gz -C /usr/local/maven/

配置maven环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
    ## ADD maven environment
    export MAVEN_HOME=/usr/local/maven/apache-maven-3.6.0
    export PATH=$MAVEN_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证Maven

[hadoop@hadoop ~]$ mvn -version
    Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-25T02:41:47+08:00)
    Maven home: /usr/local/maven/apache-maven-3.6.0
    Java version: 1.8.0_131, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_131/jre
    Default locale: en_US, platform encoding: UTF-8
    OS name: "linux", version: "3.10.0-693.el7.x86_64", arch: "amd64", family: "unix"
    [hadoop@hadoop ~]$

配置国内maven仓库源

[hadoop@hadoop conf]$ vim /usr/local/maven/apache-maven-3.6.0/conf/settings.xml 
    ...
    <mirrors>
        <mirror>
            <id>nexus-aliyun</id>
            <mirrorOf>central</mirrorOf>
            <name>Nexus aliyun</name>
         <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </mirror>
    </mirrors>
    ...

**安装工具包（root）**  
安装gcc make 工具

[root@hadoop ~]# yum install -y gcc* make

安装压缩工具

[root@hadoop ~]# yum -y install snappy*  bzip2* lzo* zlib*  lz4* gzip*

安装一些基本工具

[root@hadoop ~]# yum -y install openssl* svn ncurses* autoconf automake libtool

**安装openssl-devel**

[root@hadoop ~]# yum -y install openssl-devel

**安装zstd**  
安装扩展源 epel

[root@hadoop ~]# yum -y install epel-release

安装zstd

[root@hadoop ~]# yum -y install *zstd*

**安装CMake**  
CMake 3.1 或者更新版本，此处我们使用3.13.1（cmake-3.13.1.tar.gz）  
下载并上传cmake-3.13.1.tar.gz到/home/hadoop/src目录下  
解压cmake

[hadoop@hadoop src]$ tar -zxvf cmake-3.13.1.tar.gz

切换到源码包目录

[hadoop@hadoop src]$ cd cmake-3.13.1

检查依赖

[hadoop@hadoop cmake-3.13.1]$ ./bootstrap

**编译安装cmake（root）**  
使用root用户编译安装

[hadoop@hadoop cmake-3.13.1]$ su root
    Password: 
    [root@hadoop cmake-3.13.1]# make && make install

查看cmake版本

[hadoop@hadoop ~]$ cmake -version
    cmake version 3.13.1
    
    CMake suite maintained and supported by Kitware (kitware.com/cmake).
    [hadoop@hadoop ~]$

**安装ProtocolBuffer**  
下载protobuf-2.5.0.tar.gz并上传至/home/hadoop/src  
使用root用户创建/usr/local/protobuf目录，安装ProtocolBuffer

[root@hadoop ~]# mkdir /usr/local/protobuf  ##创建ProtocolBuffer安装目录
    [root@hadoop ~]# chown -R hadoop:hadoop /usr/local/protobuf  ##修改目录属主

解压ProtocolBuffer

[hadoop@hadoop src]$ tar -zxvf protobuf-2.5.0.tar.gz -C /home/hadoop/src/

切换到/home/hadoop/src/protobuf-2.5.0目录

[hadoop@hadoop ~]$ cd /home/hadoop/src/protobuf-2.5.0
    [hadoop@hadoop protobuf-2.5.0]$

检查库  
/usr/local/protobuf是protobuf安装目录

[hadoop@hadoop protobuf-2.5.0]$ ./configure --prefix=/usr/local/protobuf

编译安装protobuf

[hadoop@hadoop protobuf-2.5.0]$ make && make install
    ...
    make[3]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
    make[2]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
    make[1]: Leaving directory `/home/hadoop/src/protobuf-2.5.0/src'
    [hadoop@hadoop protobuf-2.5.0]$

配置protobuf环境变量

[root@hadoop jdk1.8.0_131]# vim /etc/profile
    ## ADD protobuf  environment
    export PROTOC_HOME=/usr/local/protobuf
    export PATH=$PROTOC_HOME/bin:$PATH

source使环境变量生效

[hadoop@hadoop ~]$ source /etc/profile

验证protobuf

[hadoop@hadoop protobuf]$ protoc --version
    libprotoc 2.5.0

## 编译Hadoop ##

下载hadoop-3.1.1-src.tar.gz并上传至/home/hadoop/src  
解压hadoop源码包

[hadoop@hadoop src]$ tar -zxvf hadoop-3.1.1-src.tar.gz

进入到hadoop-3.1.1-src目录

[hadoop@hadoop src]$ cd hadoop-3.1.1-src

修改pom文件（42 行）  
将hadoop-annotations的值由provide改成compile

[hadoop@hadoop src]$ vim hadoop-common-project/hadoop-auth/pom.xml 
    ...
    <dependencies>
        <dependency>
          
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-annotations</artifactId>
          <scope>compile</scope>
    </dependency>
    ...

务必保持网络畅通，如果出现缺少某个文件，则要先清理maven(使用命令 mvn clean) 再重新编译。

[root@hadoop hadoop-3.1.1-src]# mvn clean package -Pdist,native -DskipTests -Dtar  ##使用root编译

出现下面结果则成功编译  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjg2ODYzOA_size_16_color_FFFFFF_t_70]  
编译完成后的包文件位置  
/home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz

...
    [INFO] --- exec-maven-plugin:1.3.1:exec (tar) @ hadoop-dist ---
    $ tar cf hadoop-3.1.1.tar hadoop-3.1.1
    $ gzip -f hadoop-3.1.1.tar
    
    Hadoop dist tar available at: /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1.tar.gz
    
    [INFO] 
    ...

## hadoop native检查 ##

进入到编译好软件包目录下

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
    [hadoop@hadoop bin]$

检查native

[hadoop@hadoop bin]$ ./hadoop checknative -a
    2018-12-06 18:54:46,459 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
    2018-12-06 18:54:46,476 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
    2018-12-06 18:54:46,603 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
    Native library checking:
    hadoop:  true /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
    zlib:    true /lib64/libz.so.1
    zstd  :  true /lib64/libzstd.so.1
    snappy:  true /lib64/libsnappy.so.1
    lz4:     true revision:10301
    bzip2:   true /lib64/libbz2.so.1
    openssl: true /lib64/libcrypto.so
    ISA-L:   false libhadoop was built without ISA-L support
    2018-12-06 18:54:46,803 INFO util.ExitUtil: Exiting with status 1: ExitException

## 问题解决(zstd) ##

安装zstd需要先安装扩展源，否则会报找不到zstd包，如下：

[root@hadoop ~]# yum -y install *zstd*
    No package zstd available.
    Error: Nothing to do
    [root@hadoop ~]#

没有安装zstd出现的问题  
检查native时，缺失zstd

[hadoop@hadoop hadoop-3.1.1-src]$ cd /home/hadoop/src/hadoop-3.1.1-src/hadoop-dist/target/hadoop-3.1.1/bin
    [hadoop@hadoop bin]$ ./hadoop checknative -a
    2018-12-06 07:35:12,808 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
    2018-12-06 07:35:12,826 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
    2018-12-06 07:35:12,851 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
    Native library checking:
    hadoop:  true /home/hadoop/src/hadoop-3.1.1/lib/native/libhadoop.so.1.0.0
    zlib:    true /lib64/libz.so.1
    zstd  :  false 
    snappy:  true /lib64/libsnappy.so.1
    lz4:     true revision:10301
    bzip2:   true /lib64/libbz2.so.1
    openssl: true /lib64/libcrypto.so
    ISA-L:   false libhadoop was built without ISA-L support
    2018-12-06 07:35:12,967 INFO util.ExitUtil: Exiting with status 1: ExitException
    [hadoop@hadoop bin]$

转载：[https://blog.csdn.net/weixin\_43858859/article/details/84673249][https_blog.csdn.net_weixin_43858859_article_details_84673249]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjg2ODYzOA_size_16_color_FFFFFF_t_70]: /images/20220219/6c3e2faa980446deacb08ce4385b2b45.png
[https_blog.csdn.net_weixin_43858859_article_details_84673249]: https://blog.csdn.net/weixin_43858859/article/details/84673249