Kylin4.0 beta版部署

矫情吗；* 2023-01-22 05:55 1阅读 0赞

## Kylin4.0 beta版部署 ##

### 新特性 ###

*  Spark唯一构建引擎
 *  引入parquet, 正在踢出HBase
 *  可存储到HDFS

### 1. 准备机器 ###

*  准备三台机器, 系统CentOS7, 切记, 后续安装CDH版本为6.3.2, 高版本系统不支持
 *  改hosts, ssh, 免密登录
 *  改用户可打开文件数量

* vi /etc/security/limits.conf
    *    soft    nofile   32728
    *    hard    nofile   1024999
    *    soft    nproc    65535
    *    hard    noroc    unlimited
    *    soft    memlock    unlimited
    *    hard    memlock    unlimited
    * sysctl -p

*  禁用透明大页面压缩

echo never > /sys/kernel/mm/transparent_hugepage/defrag
    echo never > /sys/kernel/mm/transparent_hugepage/enabled

并将上面的两条命令写入开机自启动/etc/rc.local。

*  设置swap空间(所有节点)

echo "vm.swappiness = 0" >> /etc/sysctl.conf

### 2. 准备数据库 ###

*  主节点安装Mysql
 *  创建用户,数据库, 后续使用, 安装什么服务就建对应的数据库, 或者到后续配置前创建

-- cloudera-manager
    CREATE DATABASE scm DEFAULT CHARACTER SET utf8;
    CREATE USER 'scm'@'%'IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%' IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON *.* TO 'scm'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION; -- active_monitor CREATE DATABASE active_monitor DEFAULT CHARACTER SET utf8; CREATE USER 'active_monitor'@'%'IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON active_monitor.* TO 'active_monitor'@'%' IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON *.* TO 'active_monitor'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION; -- amon CREATE DATABASE amon DEFAULT CHARACTER SET utf8; CREATE USER 'amon'@'%'IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON *.* TO 'amon'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION; -- hive CREATE DATABASE hive DEFAULT CHARACTER SET utf8; CREATE USER 'hive'@'%'IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION; -- hue CREATE DATABASE hue DEFAULT CHARACTER SET utf8; CREATE USER 'hue'@'%'IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%' IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON *.* TO 'hue'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION; -- oozie CREATE DATABASE oozie DEFAULT CHARACTER SET utf8; CREATE USER 'oozie'@'%'IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'Kylin@2021!'; GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%' IDENTIFIED BY 'Kylin@2021!' WITH GRANT OPTION;
    -- 刷新mysql的权限列表 
    FLUSH PRIVILEGES;

*  主节点放置Mysql驱动包

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

### 3. 安装依赖 ###

*  安装jdk, 配置Java home

yum install java-1.8.0-openjdk-devel.x86_64
    ll /etc/alternatives/java

*  安装依赖

yum install -y bind-utils psmisc libxslt cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs httpd mod_ssl openssl-devel /lib/lsb/init-functions libpq.so.5python27postgresql-devel*postgresql-odbc.x86_64python2-develchkconfig zlib sqlite  redhat-lsb postgresql*   openssl  telnet pcre-devel gcc gcc-c++ MySQL-python

*  更新pip

pip install --upgrade pip
    pip install --upgrade setuptools
    pip2 install psycopg2

### 4. 部署CDH ###

CDH版本6.3.2, 官网停止下载了, 我还留了一份安装包, [csdn下载地址][csdn]

#### 安装CM ####

*  解压, 主节点安装

cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64/
    rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
    rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
    rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

*  其他节点

cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64/
    rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
    rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

*  所有节点修改agent配置, 指向master

/etc/cloudera-scm-agent/config.ini server_host=主机名

*  初始化数据库, 测试Mysql连接

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm Kylin@2021!
    All done, your SCM database is configured correctly! 成功

*  启动cm server

# 主节点启动server
    systemctl restart cloudera-scm-server
    # 查看server运行状态
    service cloudera-scm-server status
    # 查看server日志
    tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
    # 所有节点启动agent
    systemctl restart cloudera-scm-agent  
    # 查看agent运行状态
    service cloudera-scm-agent  status
    # 查看agent日志
    tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

*  启动完毕, 浏览器访问master ip:7180, 默认密码admin/admin

#### 安装CDH ####

*  开始离线安装CDH,准备parcel包

# 部署离线parcel源
    mkdir -p /var/www/html/cdh6_parcel
    # 将parcel包放在httpd的目录下
    cp cdh/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel /var/www/html/cdh6_parcel/
    cp cdh/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.1
    cp cdh/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1 /var/www/html/cdh6_parcel/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha
    cp cdh/manifest.json /var/www/html/cdh6_parcel/
    # 启动httpd服务
    systemctl start httpd

浏览器访问master ip/cdh6\_parcel

*  返回CM, 选择免费版下一步下一步, 选主机, 更多选项, 添加源继续, 等解压, 检查集群, 忽略,下一步, 选择安装的组件, 使用之前创建好的账号配置数据库, 配置默认下一步
 *  CDH部署完成

### 5. 部署Kylin ###

以上都顺利的话, 按照官方步骤执行, 即可启动成功  
[Deploy Kylin4 on CDH6][]

[csdn]: https://download.csdn.net/download/u012355401/18809481
[Deploy Kylin4 on CDH6]: https://cwiki.apache.org/confluence/display/KYLIN/Deploy+Kylin+4+on+CDH+6