sqoop1.4.6相关操作及问题汇总

迷南。 2022-06-02 09:05 250阅读 0赞

**1 sqoop环境准备**  
[sqoop官网][sqoop]推荐使用稳定的1.4.6,之前在csdn误操作把之前写的删除掉了，现在重新补上  
**1.1 安装**

tar -zxvf sqoop-1.4.6.tar.gz
    ln -s sqoop-1.4.6 sqoop

文件可以从[sqoop-1.4.6.tar.gz][]下载  
**1.2 环境变量**

export SQOOP_HOME=/home/hadoop/application/sqoop
    export PATH=$PATH:$SQOOP_HOME/bin
    export HADOOP_MAPRED_HOME=${HADOOP_HOME}
    export HADOOP_COMMON_HOME=${HADOOP_HOME}

**1.3 更改$SQOOP\_HOME/sqoop**  
将

exec ${HADOOP_COMMON_HOME}/bin/hadoop  org.apache.sqoop.Sqoop "$@"

更改为

exec ${HADOOP_COMMON_HOME}/bin/hadoop jar $SQOOP_HOME/sqoop-1.4.6.jar org.apache.sqoop.Sqoop "$@"

其中`sqoop-1.4.6.jar`可以在[sqoop-1.4.6.bin\_\_hadoop-2.0.4-alpha.tar.gz][sqoop-1.4.6.bin_hadoop-2.0.4-alpha.tar.gz]中找到  
**1.4 注释$SQOOP\_HOME/configure-sqoop中部分**

#if [ ! -d "${HCAT_HOME}" ]; then
    #  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
    #  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
    #fi
    
    #if [ ! -d "${ACCUMULO_HOME}" ]; then
    #  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
    #  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
    #fi

**1.5 将mysql-connector-java-5.1.43.jar放入到$SQOOP\_HOME/lib**  
mysql-connector-java-5.1.43.jar可以从maven仓库中获取  
**2 sqoop 导入数据到mysql**  
参考[使用Sqoop从MySQL导入数据到Hive和HBase 及近期感悟][Sqoop_MySQL_Hive_HBase]

sqoop import \
    > --connect jdbc:mysql://192.168.5.135:3306/toolbox --username root -P \
    > --query "select CONCAT(id,'_',xhdwsbh,'_',kprq) row_key,tit.* from t_invoice_ticket tit where kprq>='20170401' and kprq<='20170430' and \$CONDITIONS " \
    > --split-by xhdwsbh -m 5 \
    > --hbase-table iw:test01 \
    > --column-family i \
    > --hbase-row-key row_key

参数说明[关于sqoop --split-by 及 -m的理解][sqoop --split-by _ -m]

[sqoop]: http://sqoop.apache.org/
[sqoop-1.4.6.tar.gz]: http://mirrors.shuosc.org/apache/sqoop/1.4.6/sqoop-1.4.6.tar.gz
[sqoop-1.4.6.bin_hadoop-2.0.4-alpha.tar.gz]: http://mirrors.shuosc.org/apache/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
[Sqoop_MySQL_Hive_HBase]: https://www.cnblogs.com/wgp13x/p/5028220.html
[sqoop --split-by _ -m]: http://blog.csdn.net/lookqlp/article/details/52096191