大数据组件：工作流任务调度管理（二）AzKaban集群部署

• Azkaban source code
  

• git
  

• jdk

• gradle

  
  

这里我们下载最近的latest4.0.0版本源码。

sudo yum install -y git

gradle官网下载，这里我自己有6.6这个版本。

上传到hadoop-node01机器上。目录如下：

./gradlew build

# Build Azkaban
./gradlew build


# Clean the build
./gradlew clean


# Build and install distributions
./gradlew installDist


# Run tests
./gradlew test


# Build without running tests
./gradlew build -x test

下面我们拿之前编译好的的3.8.4这个版本，继续讲解。

mysql> create database azkaban;

#设置密码策略等级低模式以及长度
set global validate_password_length=6
set global validate_password_policy=0
#创建azkaban用户，并设置密码
CREATE USER 'azkaban'@'%' IDENTIFIED BY 'XXXXXX';
#赋予用户权限
GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'azkaban'@'%' WITH GRANT OPTION;

3、导入azkaban表

我们刚才上传azkaban-db的路径是/opt/module/azkaban

里面包含一个创建所有SQL的脚本

mysql> source /opt/module/azkaban/azkaban-db/create-all-sql-3.84.4.sql

mysql> show tables;
+-----------------------------+
| Tables_in_azkaban           |
+-----------------------------+
| QRTZ_BLOB_TRIGGERS          |
| QRTZ_CALENDARS              |
| QRTZ_CRON_TRIGGERS          |
| QRTZ_FIRED_TRIGGERS         |
| QRTZ_JOB_DETAILS            |
| QRTZ_LOCKS                  |
| QRTZ_PAUSED_TRIGGER_GRPS    |
| QRTZ_SCHEDULER_STATE        |
| QRTZ_SIMPLE_TRIGGERS        |
| QRTZ_SIMPROP_TRIGGERS       |
| QRTZ_TRIGGERS               |
| active_executing_flows      |
| active_sla                  |
| execution_dependencies      |
| execution_flows             |
| execution_jobs              |
| execution_logs              |
| executor_events             |
| executors                   |
| project_events              |
| project_files               |
| project_flow_files          |
| project_flows               |
| project_permissions         |
| project_properties          |
| project_versions            |
| projects                    |
| properties                  |
| ramp                        |
| ramp_dependency             |
| ramp_exceptional_flow_items |
| ramp_exceptional_job_items  |
| ramp_items                  |
| triggers                    |
| validated_dependencies      |
+-----------------------------+
35 rows in set (0.00 sec)

[qiusheng@node01 azkaban]$ sudo vim /etc/my.cnf
[mysqld]
 21 # Azkaban packet
 22 max_allowed_packet=1024M

[qiusheng@node01 azkaban]$ sudo systemctl restart mysqld

#1、修改时区
  7 #default.timezone.id=America/Los_Angeles
  8 default.timezone.id=Asia/Shanghai
#2、修改webserver.url
 22 #azkaban.webserver.url=http://localhost:8081
 23 azkaban.webserver.url=http://node01:8081
#3、修改mysql的host等
 46 #mysql.host=localhost
 47 mysql.host=node01
#4、添加metric属性
 55 executor.metric.reports=true
 56 executor.metric.milisecinterval.default=60000
#5、添加executor.port
 57 #add azkan Executor port
 58 executor.port=12333

[qiusheng@node01 azkaban-exec]$ xsync__nodes.sh opt/module/azkaban/azkaban-exec/
=============== node01 ===============
sending incremental file list
sent 1,937 bytes  received 23 bytes  3,920.00 bytes/sec
total size is 17,750,582  speedup is 9,056.42
=============== node02 ===============
sending incremental file list
sent 17,759,162 bytes  received 1,154 bytes  35,520,632.00 bytes/sec
total size is 17,750,582  speedup is 1.00
=============== node03 ===============
sending incremental file list
sent 17,759,162 bytes  received 1,154 bytes  11,840,210.67 bytes/sec
total size is 17,750,582  speedup is 1.00

[qiusheng@node01 azkaban-exec]$ bin/start-exec.sh

[qiusheng@node01 azkaban-exec]$ cat executor.port
12333

[qiusheng@node01 azkaban-exec]$ jps
10652 AzkabanExecutorServer
10733 Jps

mysql> select * from executors;
+----+--------+-------+--------+
| id | host   | port  | active |
+----+--------+-------+--------+
|  1 | node01 | 12333 |      0 |
+----+--------+-------+--------+
1 row in set (0.00 sec

[qiusheng@node01 azkaban-exec]$ curl -G "node01:12333/executor?action=activate" && echo
{"status":"success"}

[qiusheng@node02 azkaban-exec]$ bin/start-exec.sh
[qiusheng@node01 azkaban-exec]$ curl -G "node02:12333/executor?action=activate" && echo
{"status":"success"}

[qiusheng@node03 azkaban-exec]$ bin/start-exec.sh
[qiusheng@node01 azkaban-exec]$ curl -G "node03:12333/executor?action=activate" && echo
{"status":"success"}

mysql> select * from executors;
+----+--------+-------+--------+
| id | host   | port  | active |
+----+--------+-------+--------+
|  1 | node01 | 12333 |      1 |
|  2 | node02 | 12333 |      1 |
|  3 | node03 | 12333 |      1 |
+----+--------+-------+--------+
3 rows in set (0.00 sec)

#1、修改时区
  7 #default.timezone.id=America/Los_Angeles
  8 default.timezone.id=Asia/Shanghai
#2、修改mysql的host等
 46 #mysql.host=localhost
 47 mysql.host=node01

 49 azkaban.use.multiple.executors=true
 50 azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus

<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>
  <user password="tigerWang" roles="admin" username="tigerWang"/>
  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

[qiusheng@node01 bin]$ ./start-web.sh

[qiusheng@node01 azkaban-web]$ jps
11248 NodeManager
12402 Jps
11510 QuorumPeerMain
11944 Kafka
11385 JobHistoryServer
10890 DataNode
12266 AzkabanExecutorServer
10763 NameNode
12060 Application
12381 AzkabanWebServer

[qiusheng@node02 azkaban-exec]$ jps
12336 AzkabanExecutorServer
11633 QuorumPeerMain
11218 NodeManager
11091 ResourceManager
10886 DataNode
12040 Kafka
12155 Application
12447 Jps

[qiusheng@node03 azkaban-exec]$ jps
12649 Application
12985 Jps
11978 NodeManager
12154 QuorumPeerMain
12554 Kafka
11757 DataNode
11839 SecondaryNameNode
12847 AzkabanExecutorServer