Presto安装配置和集成Kerberos的Hive

九万里大数据 2021-07-21

2247

下载presto安装包

presto官网：https://prestodb.io

presto-server下载链接：https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.255/presto-server-0.255.tar.gz

presto-cli下载链接：https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.255/presto-cli-0.255-executable.jar

presto-jdbc下载链接: https://repo1.maven.org/maven2/com/facebook/presto/presto-jdbc/0.255/presto-jdbc-0.255.jar

配置presto

tar -zxf presto-server-0.255.tar.gz -C /opt
cd /opt/presto-server-0.255/etc

将presto-cli-0.255-executable.jar放到/opt/presto-server-0.255/bin下面

cd /opt/presto-server-0.255/bin
ln -s presto-cli-0.255-executable.jar presto
ln -s /opt/presto-server-0.255/bin/presto /usr/bin/presto

需要创建的配置文件

[root@jwldata01 etc]# pwd
/opt/presto-server-0.255/etc
[root@jwldata01 etc]# 
[root@jwldata01 etc]# tree
.
├── access-control.properties
├── catalog
│   └── hive.properties
├── config.properties
├── jvm.config
├── log.properties
└── node.properties

1 directory, 6 files

coordinator和worker共有的配置

jvm.config

-server
-Xmx8G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-Djava.security.krb5.conf=/etc/krb5.conf

log.properties

com.facebook.presto=INFO

node.properties

node.environment=presto_dev
node.id=presto-jwldata01
node.data-dir=/data01/presto

node.id每个节点需要修改，node.data-dir根据实际修改。

coordinator的配置

config.properties

coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=9900
query.max-memory=4GB
query.max-memory-per-node=1GB
discovery-server.enabled=true
discovery.uri=http://jwldata01:9900

（可选）如果是搭一个测试集群，单节点或者节点数比较少，可以让coordinator也充当worker，但生产环境还是建议角色分开。

node-scheduler.include-coordinator=true

（可选）如果需要presto只读 access-control.properties

access-control.name=read-only

worker的配置

config.properties

coordinator=false
http-server.http.port=9900
query.max-memory=4GB
query.max-memory-per-node=1GB
discovery.uri=http://jwldata01:9900

Hive Connector

catalog/hive.properties

connector.name=hive-hadoop2
hive.metastore.uri=thrift://jwldata01:9083
hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
 
#配置Presto访问HiveMetastore服务的Kerberos信息，该段配置可以只存在Presto的Coordinator节点
hive.metastore.authentication.type=KERBEROS
hive.metastore.service.principal=hive/_HOST@JWLDATA01
hive.metastore.client.principal=presto@JWLDATA01
hive.metastore.client.keytab=/home/presto/presto.keytab
 
#配置Presto访问HDFS的Kerberos信息，该段配置可以只存在Presto的Worker节点
hive.hdfs.authentication.type=KERBEROS
hive.hdfs.impersonation.enabled=true
hive.hdfs.presto.principal=presto@JWLDATA01
hive.hdfs.presto.keytab=/home/presto/presto.keytab

HDFS配置

Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml hadoop.proxyuser.presto.groups设置成* hadoop.proxyuser.presto.hosts设置成*

Kerberos配置

addprinc -randkey presto
xst -kt presto.keytab presto

将presto.keytab放到所有presto节点的/home/presto目录下

chown presto. /home/presto/presto.keytab
chmod 600 /home/presto/presto.keytab

LDAP配置

创建presto组和presto用户。根据需要，将presto用户加入有权限的几个LDAP组内。

重启脚本

cd /opt/presto-server-0.255
touch restart.sh
chmod u+x restart.sh
vi restart.sh

#!/bin/bash

./bin/launcher restart

echo "tail -f data01/presto/var/log/server.log"

相关报错

com.facebook.presto.spi.PrestoException: jwldata001:9083: java.net.SocketException: Connection reset
    at com.facebook.presto.hive.metastore.thrift.ThriftHiveMetastore.getAllDatabases(ThriftHiveMetastore.java:167)
    at com.facebook.presto.hive.metastore.thrift.BridgingHiveMetastore.getAllDatabases(BridgingHiveMetastore.java:86)
    at com.facebook.presto.hive.metastore.CachingHiveMetastore.loadAllDatabases(CachingHiveMetastore.java:342)
    at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
    at com.google.common.cache.CacheLoader$1.load(CacheLoader.java:188)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)

将presto用户加入到能访问hive metastore的LDAP组里，然后重启presto

com.facebook.presto.spi.PrestoException: Permission denied: user=presto, access=READ_EXECUTE, inode="/data/bank.db/account/ds=2020-09-21":hive:hive:drwxrwx--x
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkAccessAcl(DefaultAuthorizationProvider.java:363)
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:256)
    at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:168)
    at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194)
    at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)

将presto用户加入到能读取对应hive库的LDAP组里。如果希望presto用户能读取到所有的hive库，可以加入到hive组。

参考文档

https://cloud.tencent.com/developer/article/1158360 https://cloud.tencent.com/developer/article/1158362 https://www.cnblogs.com/erlou96/p/14592749.html https://prestodb.io/docs/current/connector/hive-security.html https://prestodb.io/docs/current/installation/deployment.html

欢迎关注我的微信公众号“九万里大数据”，原创技术文章第一时间推送。欢迎访问原创技术博客网站 jwldata.com^[1]，排版更清晰，阅读更爽快。

引用链接

[1]
jwldata.com: https://www.jwldata.com

数据库

文章转载自九万里大数据，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。