暂无图片
暂无图片
暂无图片
暂无图片
暂无图片
首页 / Presto On Yarn部署

Presto On Yarn部署

叁金大数据 2019-04-24
548

Presto是我们所熟知的数据查询引擎，它本身的部署也是非常的简单。但是有时候我们可能会有这样的需求，比如需要动态的对Presto集群进行扩容，需要对集群的资源进行统一的管理。我们需要一个Presto On Yarn的解决方案。

Yarn是一个通用资源管理系统，可为上层应用提供统一的资源管理和调度，它的引入为集群在利用率、资源统一管理和数据共享等方面带来了巨大好处。所以把Presto的应用提交到Yarn上可以帮助我们对Presto所需的资源进行统一管理，同时还可以根据业务需求对集群进行动态的容量控制。

需要注意的是Presto不像Spark那样默认就支持YARN，它需要借助于Slider才能实现Presto on Yarn。

Presto On Yarn 安装

根据官方文档整理步骤如下：

  1. 编译presto on yarn

    下载地址：https://github.com/prestodb/presto-yarn/

    编译时指定presto的版本即可，如mvn clean package -Dpresto.version=0.2.13
    ,在presto-yarn-package
    的target
    目录下找到presto-yarn-package-1.6-SNAPSHOT-0.213.zip
    ，在该压缩包下包含presto的压缩包

  2. 编译slider
    下载地址：https://archive.apache.org/dist/incubator/slider/

    选择最新的0.92版本，编译前需更改slider的pom
    文件，修改对应的java
    版本和hadoop
    版本，编译 mvn clean package -DskipTests
    ,在slider-assembly\target
    目录下找到编译完成后的包slider-0.92.0-incubating-all.tar.gz

  3. 配置slider

    将slider提交到相关机器，解压缩。需配置两个文件 slider-env.sh
    和slider-client.xml

    1# slider-env.sh
    2
    3export JAVA_HOME=    #如果未配置环境变量需配置该属性
    4export HADOOP_CONF_DIR=   #如果未配置环境变量需配置该属性

    配置slider-client.xml

     1<configuration>
     2   <property>
     3     <name>slider.client.resource.origin</name>
     4     <value>conf/slider-client.xml</value>
     5     <description>This is just for diagnostics</description>
     6   </property>
     7   <property>
     8     <name>slider.security.protocol.acl</name>
     9     <value>*</value>
    10     <description>When security is enabled, set appropriate acl. Default value means allow everyone.</description>
    11   </property>
    12   <property>
    13     <name>slider.yarn.queue</name>
    14     <value/>
    15     <description>the name of the YARN queue to use.</description>
    16   </property>
    17   <property>
    18     <name>slider.yarn.queue.priority</name>
    19     <value>1</value>
    20     <description>the priority of the application.</description>
    21   </property>
    22   <property>
    23     <name>slider.am.login.keytab.required</name>
    24     <value>false</value>
    25     <description>Declare that a keytab must be provided.</description>
    26   </property>
    27   <!-- 这里需注意，slider默认需要zookeeper的支持。所以这里必须要填 -->
    28   <property>
    29     <name>slider.zookeeper.quorum</name>
    30     <value>zookeeper.uri</value>
    31   </property>
    32   <property>
    33     <name>fs.defaultFS</name>
    34     <value>hdfs://mycluster/</value>
    35   </property>
    36</configuration>

  4. 配置presto
    解压presto-yarn-package
    ，获取其中的appConfig-default.json
    和resources-default.json
    。这两个配置为presto在yarn上相关的资源配置。相关配置参照官方配置说明：https://prestodb.io/presto-yarn/installation-yarn-configuration-options.html

     1# appConfig-default.json
     2{
     3 "schema": "http://example.org/specification/v2.0.0",
     4 "metadata": {
     5 },
     6 "global": {
     7   "site.global.app_user": "yarn",
     8   "site.global.user_group": "hadoop",
     9   "site.global.data_dir": "/export/data/soft/presto-yarn/data",# 需在每台节点上手动创建该目录,并对app_user赋权
    10   "site.global.config_dir": "/export/data/soft/presto-yarn/etc",# 需在每台节点上手动创建该目录,并对app_user赋权
    11   "site.global.app_name": "presto-server-0.213",
    12   "site.global.app_pkg_plugin": "¨E123EAGENT¨E95EWORK¨E95EROOT¨E125E/app/definition/package/plugins/""site.global.singlenode":"false""site.global.coordinator¨E95Ehost":"{AGENT_WORK_ROOT}/app/definition/package/plugins/",
    13   "site.global.singlenode": "false",
    14   "site.global.coordinator_host": "<span class="katex-html" aria-hidden="true" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:1em;vertical-align:-0.25em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">¨<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E123<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">A<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">G<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.10903em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">N<span class="mord mathit" style="margin-right:0.13889em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">T¨<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E95<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.13889em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">W<span class="mord mathit" style="margin-right:0.02778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">O<span class="mord mathit" style="margin-right:0.00773em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">R<span class="mord mathit" style="margin-right:0.07153em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">K¨<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E95<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.00773em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">R<span class="mord mathit" style="margin-right:0.02778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">O<span class="mord mathit" style="margin-right:0.02778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">O<span class="mord mathit" style="margin-right:0.13889em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">T¨<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E125<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E/<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">p<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">p/<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e<span class="mord mathit" style="margin-right:0.10764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">f<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">t<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n/<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">p<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">c<span class="mord mathit" style="margin-right:0.03148em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">k<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="margin-right:0.03588em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">g<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e/<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">p<span class="mord mathit" style="margin-right:0.01968em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="margin-right:0.03588em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">g<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s/"<span class="mspace" style="margin-right:0.16666666666666666em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">"<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">t<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e.<span class="mord mathit" style="margin-right:0.03588em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">g<span class="mord mathit" style="margin-right:0.01968em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="margin-right:0.01968em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l.<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="margin-right:0.03588em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">g<span class="mord mathit" style="margin-right:0.01968em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e"<span class="mspace" style="margin-right:0.2777777777777778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mspace" style="margin-right:0.2777777777777778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:0.8888799999999999em;vertical-align:-0.19444em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">"<span class="mord mathit" style="margin-right:0.10764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">f<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="margin-right:0.01968em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e"<span class="mspace" style="margin-right:0.16666666666666666em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">"<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">t<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e.<span class="mord mathit" style="margin-right:0.03588em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">g<span class="mord mathit" style="margin-right:0.01968em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="margin-right:0.01968em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l.<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">c<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="margin-right:0.02778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">r<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">t<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="margin-right:0.02778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">r¨<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E95<span class="mord mathit" style="margin-right:0.05764em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">h<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">t"<span class="mspace" style="margin-right:0.2777777777777778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mspace" style="margin-right:0.2777777777777778em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:0.69444em;vertical-align:0em;" style="box-sizing: border-box;font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">"{COORDINATOR_HOST}",
    15   "site.global.presto_query_max_memory": "3GB",# 自行指定相关配置
    16   "site.global.presto_query_max_memory_per_node": "600MB",
    17   "site.global.presto_server_port": "20770",# 指定coordinator端口
    18
    19   "site.global.catalog": "{'tpch': ['connector.name=tpch']}",# 指定初始的catalog
    20   "site.global.jvm_args": "['-server', '-Xmx1024M', '-XX:+UseG1GC', '-XX:G1HeapRegionSize=32M', '-XX:+UseGCOverheadLimit', '-XX:+ExplicitGCInvokesConcurrent', '-XX:+HeapDumpOnOutOfMemoryError', '-XX:OnOutOfMemoryError=kill -9 %p']",
    21   "site.global.log_properties": "['com.facebook.presto=INFO']",#指定日志信息
    22
    23   "application.def": ".slider/package/presto/presto-yarn-package-1.5-0.213.zip",#默认即可
    24   "java_home": "/export/data/soft/java"# 指定java home
    25 },
    26 "components": {
    27   "slider-appmaster": {
    28     "jvm.heapsize": "1024M"
    29   }
    30 }
    31}
    32</span class="strut" style="height:0.69444em;vertical-align:0em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.02778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.02778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.03588em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mspace" style="margin-right:0.16666666666666666em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.10764em;"></span class="strut" style="height:0.8888799999999999em;vertical-align:-0.19444em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.03588em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.03588em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mspace" style="margin-right:0.16666666666666666em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.03588em;"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.03588em;"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.03148em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.10764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.13889em;"></span class="mord mathit" style="margin-right:0.02778em;"></span class="mord mathit" style="margin-right:0.02778em;"></span class="mord mathit" style="margin-right:0.00773em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.07153em;"></span class="mord mathit" style="margin-right:0.00773em;"></span class="mord mathit" style="margin-right:0.02778em;"></span class="mord mathit" style="margin-right:0.13889em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.13889em;"></span class="mord mathit" style="margin-right:0.10903em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="strut" style="height:1em;vertical-align:-0.25em;"></span class="katex-html" aria-hidden="true">

     1# resources.json
     2{
     3 "schema": "http://example.org/specification/v2.0.0",
     4 "metadata": {
     5 },
     6 "global": {
     7   "yarn.vcores": "1"
     8 },
     9 "components": {
    10   "slider-appmaster": {
    11   },
    12   "COORDINATOR": {
    13     "yarn.role.priority": "1",
    14     "yarn.component.instances": "1",# coordinator节点启动时container个数
    15     "yarn.component.placement.policy": "1",
    16     "yarn.memory": "1500",# 指定yarn分配的资源
    17     "yarn.label.expression": "coordinator" # 指定在yarn上的label（此处可用于固定coordinator节点，便于对外提供服务。）
    18   },
    19   "WORKER": {
    20     "yarn.role.priority": "2",
    21     "yarn.component.instances": "3",# worker节点启动时container个数，需要注意的是，两者个数不能多余集群的数量，因为需要绑定端口，多余集群数据会绑定失败而退出
    22     "yarn.component.placement.policy": "1",
    23     "yarn.memory": "1500",# 指定yarn分配的资源
    24     "yarn.label.expression": "worker"
    25   }
    26 }
    27}

  5. 使用slider安装presto on yarn

    在hdfs创建/user/xxx/
    目录。
    执行slider package --install --name presto --package presto-yarn-package-1.6-SNAPSHOT-0.213.zip

  6. 启动Presto on Yarn
    slider create presto-query --template appConfig.json --resources resources.json

    在yarn的后台可见presto的application

由于coordinator的ip不固定，所以presto on yarn的部署还不算完成。

对于Presto on Yarn的coordinator 固定有两种解决方案：

  1. 通过对机器打lable的方式，将coordinator固定到某一台机器

  2. presto coordinator的发现依赖于discovery server。可以将discovery server单独部署到某台机器上。连接presto时先通过discovery server获得coordinator节点的地址

通过label的方式在上面的配置上已经展示了，下面介绍单独部署discovery server的方式。

Discovery Server 单独安装

  1. 下载并编译Discovery Server
    ：https://github.com/airlift/discovery

  2. 部署Discovery Server
    ，它的配置和presto的配置类似，需要手动创建etc文件夹和config.properties
    、node.properties
    、jvm.config
    、log.properties
    等。

  3. 通过bin/launcher start
    启动 discovery server
    ，可通过jps
    查看到相关进程

然后修改presto-on-yarn
的配置模板， 位于目录package\templates
中。有：

config.properties-COORDINATOR.j2

config.properties-WORKER.j2

node.properties.j2

queues.json.j2

其中我们需要关注config.properties
 和node.properties
，该文件的生成就依赖于我们前面的appConfig.json
。同样的我们也可以添加我们自定义的配置，需要更改配置读取的python文件。

示例如下：

1# config.properties-COORDINATOR.j2
2coordinator=true
3node-scheduler.include-coordinator={{singlenode}}
4discovery-server.enabled=true
5http-server.http.port={{presto_server_port}}
6query.max-memory={{presto_query_max_memory}}
7query.max-memory-per-node={{presto_query_max_memory_per_node}}
8query.queue-config-file={{conf_dir}}/queues.json
9discovery.uri=http://{{discovery_host}}:{{discovery_port}} #因为我们单独提出了discovery server所以需要添加discovery相关的配置

修改package/scripts/params.py，添加新增配置文件的获取

 1from resource_management import *
 2import uuid
 3
 4# server configurations
 5config = Script.get_config()
 6
 7java8_home = config['hostLevelParams']['java_home']
 8
 9app_root = config['configurations']['global']['app_root']
10app_name = config['configurations']['global']['app_name']
11
12presto_root = format("{app_root}/{app_name}")
13conf_dir = default('/configurations/global/config_dir', format("{presto_root}/etc"))
14
15catalog_dir = format("{conf_dir}/catalog")
16presto_plugin_dir = format("{presto_root}/plugin")
17source_plugin_dir = config['configurations']['global']['app_pkg_plugin']
18addon_plugins = default('/configurations/global/plugin', '')
19
20presto_user = config['configurations']['global']['app_user']
21user_group = config['configurations']['global']['user_group']
22
23data_dir = config['configurations']['global']['data_dir']
24pid_dir = format("{data_dir}/var/run")
25pid_file = format("{pid_dir}/slider_launcher.pid")
26log_dir = format("{data_dir}/var/log")
27log_file = format("{log_dir}/server.log")
28
29singlenode = config['configurations']['global']['singlenode']
30coordinator_host = config['configurations']['global']['coordinator_host']
31presto_query_max_memory = config['configurations']['global']['presto_query_max_memory']
32presto_query_max_memory_per_node = config['configurations']['global']['presto_query_max_memory_per_node']
33presto_server_port = config['configurations']['global']['presto_server_port']
34jvm_args = default('/configurations/global/jvm_args', '')
35log_properties = default('/configurations/global/log_properties', '')
36
37# 新增配置，对应appConfig.json中的site.global.discovery_host和site.global.discovery_port
38discovery_port = config['configurations']['global']['discovery_port']
39discovery_host= config['configurations']['global']['discovery_host']
40
41node_id = uuid.uuid1()
42
43catalog_properties = default('/configurations/global/catalog', '')
44additional_config_properties=default('/configurations/global/additional_config_properties', '')
45additional_node_properties=default('/configurations/global/additional_node_properties', '')

部署完成之后，我们就可以通过discover server
获取presto相关的服务了。可以通过http://discover_host:discovery_port/v1/service/presto获取到presto所有的节点信息，通过内容判断是否为coordinator
。

Slider命令示例

基于slider部署的presto集群可以通过slider相关的命令动态的调整集群容量，获取运行状态等等。可以通过./slider命令进行查看。

 1# 管理 applicaiton
 2bin/slider status presto-query 
 3bin/slider start presto-query
 4bin/slider stop presto-query
 5
 6# 查看 coordinator 部署在哪个节点，可以作为-- server 参数提供给 presto-cli 使用
 7bin/slider registry --name presto-query --getexp presto
 8
 9# 销毁 applicaiton
10bin/slider destroy presto-query --force
11
12# 删除 presto-yarn-package
13bin/slider package --delete --name PRESTO
14
15# 动态调整 applicaiton，增加或删除 WORKER 数
16bin/slider flex presto-query --component WORKER 2

到现在我们的Presto On Yarn已经部署完成了。

▼
往期精彩回顾
▼
SQL优化器执行过程之逻辑算子
Calcite 项目及使用介绍
SQL优化器简介


好久不见~


数据库
文章转载自叁金大数据，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

评论