(二)presto集成Hive

Myth丶恋晨 2022-12-22 04:35 140阅读 0赞

# 一、集成之前 #

在presto集成Hive之前，要先启动Hive;由于Hive依赖HDFS,并且这里环境其元数据存储在mysql中，故在启动Hive之前要先启动HDFS和Mysql;

## 1、主节点启动hdfs ##

[root@master sbin]# pwd
    /opt/softWare/hadoop/hadoop-2.7.3/sbin
    [root@master sbin]# ./start-all.sh

## 2、启动Mysql ##

[root@master ~]# mysql -u root -p

## 3、启动hive metastore ##

[root@master bin]# nohup bin/hive --service metastore -p 9083 >/dev/null &

## 4、命令查看Hive表信息 ##

[root@master bin]# ./hive
    which: no hbase in (/opt/softWare/hive/apache-hive-2.1.1-bin/bin:/opt/softWare/hive/apache-hive-2.1.1-bin/conf:/opt/softWare/spark/spark-2.20/bin:/opt/softWare/scala/scala-2.13.1/bin:/opt/softWare/kafka/kafka_2.10-0.10.2.1/bin:/opt/softWare/zookeeper/zookeeper-3.5.2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/softWare/hadoop/hadoop-2.7.3/bin:/opt/softWare/java8_151/jdk1.8.0_151/bin:/opt/softWare/java8_151/jdk1.8.0_151/jre/bin :/opt/softWare/mysql/bin:/opt/softWare/kafkaEagle/kafka-eagle-bin-1.2.4/kafka-eagle/bin:/opt/softWare/sqoop/sqoop-1.4.7/bin:/root/bin)
    SLF4J: Class path contains multiple SLF4J bindings.
    SLF4J: Found binding in [jar:file:/opt/softWare/hive/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: Found binding in [jar:file:/opt/softWare/hadoop/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
    SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
    SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
    
    Logging initialized using configuration in jar:file:/opt/softWare/hive/apache-hive-2.1.1-bin/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
    Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
    hive> show databases;
    OK
    db_hive
    db_hive_test
    default
    Time taken: 0.991 seconds, Fetched: 3 row(s)
    hive>

# 二、集成Hive #

## 1、新建catalog目录 ##

在所有节点的/opt/softWare/presto/ln\_presto/etc/下新建catalog目录

[root@master etc]# mkdir catalog

## 2、创建hive.properties文件 ##

在所有节点的/opt/softWare/presto/ln\_presto/etc/catalog/目录下创建hive.properties文件

connector.name=hive-hadoop2
    hive.metastore.uri=thrift://master:9083

## 3、修改jvm.config文件 ##

在所有几点修改/opt/softWare/presto/ln\_presto/etc下的jvm.config文件，并增加hdfs的用户

-server
    -Xmx3G
    -XX:+UseG1GC
    -XX:G1HeapRegionSize=32M
    -XX:+UseGCOverheadLimit
    -XX:+ExplicitGCInvokesConcurrent
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:+ExitOnOutOfMemoryError
    -DHADOOP_USER_NAME=presto

## 4、创建hdfs用户 ##

为所有节点创建上述hdfs的presto用户

[root@master etc]# useradd presto

## 5、所有节点重启presto ##

.bin/launcher restart

# 三、CLI工具进行测试 #

## 1、将presto-cli-0.243.2-executable.jar移动到bin目录下并重新命名和复制权限 ##

此操作只在一个节点上执行即可，因为cli只是一个客户端；

[root@master presto]# mv presto-cli-0.243.2-executable.jar ln_presto/bin/

[root@master bin]# mv presto-cli-0.243.2-executable.jar presto

[root@master bin]# chmod a+x presto

## 2、连接访问Hive ##

[root@master bin]# ./presto --server 192.168.230.21:8099 --catalog=hive --schema=db_hive
    presto:db_hive> show tables;
     Table 
    -------
     stu   
     stu01 
    (2 rows)
    
    Query 20201117_072006_00008_xkr5d, FINISHED, 3 nodes
    Splits: 36 total, 36 done (100.00%)
    0:01 [2 rows, 42B] [3 rows/s, 82B/s]
    presto:db_hive>

db\_hive为hive数据库中已经存在的库；

presto:db_hive> select * from stu01;
     id | name 
    ----+------
    (0 rows)
    
    Query 20201117_074636_00013_xkr5d, FINISHED, 1 node
    Splits: 17 total, 17 done (100.00%)
    0:00 [0 rows, 0B] [0 rows/s, 0B/s]
    presto:db_hive> select count(*) from stu01;
     _col0 
    -------
         0 
    (1 row)
    
    Query 20201117_074709_00015_xkr5d, FINISHED, 1 node
    Splits: 18 total, 18 done (100.00%)
    0:01 [0 rows, 0B] [0 rows/s, 0B/s]
    
    presto:db_hive>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h5al9raW5n_size_16_color_FFFFFF_t_70][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2h5al9raW5n_size_16_color_FFFFFF_t_70]: /images/20221120/9ab06d68c18242a5a0b35a3db1891064.png