案例|某电商如何构建Zabbix高可用监控平台？

下载heartbeat 依赖环境包
cd usr/src
wget http://hg.linux-ha.org/heartbeat-STABLE_3_0/archive/958e11be8686.tar.bz2 wget http://hg.linux-ha.org/glue/archive/0a7add1d9996.tar.bz2
wget https://github.com/ ClusterLabs/resource-agents/archive/v3.9.6.tar.gz安装相关依赖库
yum -y install gcc gcc-c++ autoconf automake libtool glib2-devel libxml2-devel bzip2 bzip2-devel e2fsprogs-devel libxslt-devel libtool-ltdl-devel asciidoc psmisc创建运行heartbeat 组及用户
groupadd haclient
useradd -g haclient hacluster -s sbin/nologin




部署heartbeat 编译安装依赖环境安装cluster-glue
cd usr/src
tar -jxvf 0a7add1d9996.tar.bz2
cd Reusable-Cluster-Components-glue--0a7add1d9996/
./autogen.sh
./configure --prefix=/usr/local/heartbeat --with-daemon-user=hacluster --with-daemon-group=haclient --enable-fatal-warnings=no LIBS='/lib64/libuuid.so.1' make && make install


安装resource-agents
tar -zxvf v3.9.6.tar.gz
cd resource-agents-3.9.6
./autogen.sh
./configure --prefix=/usr/local/heartbeat --with-daemon-user=hacluster --with-daemon-group=haclient --enable-fatal-warnings=no LIBS='/lib64/libuuid.so.1' make && make install


安装heartbeat
tar -jxvf 958e11be8686.tar.bz2
cd Heartbeat-3-0-958e11be8686/
./bootstrap


声明编译环境变量
export CFLAGS="$CFLAGS -I/usr/local/heartbeat/include -L/usr/local/heartbeat/lib"
./configure --prefix=/usr/local/heartbeat --with-daemon-user=hacluster --with-daemon-group=haclient --enable-fatal-warnings=no LIBS='/lib64/libuuid.so.1' make && make install


复制heartbeat 核心配置文件
cp doc/{ha.cf,haresources,authkeys} usr/local/heartbeat/etc/ha.d/授权认证文件600权限
chmod 600 usr/local/heartbeat/etc/ha.d/authkeys创建目录，配置网卡支持插件文件
mkdir -p usr/local/heartbeat/usr/lib/ocf/lib/heartbeat/
cp usr/lib/ocf/lib/heartbeat/ocf-* usr/local/heartbeat/usr/lib/ocf/lib/heartbeat/


注：一般启动时会报错因为ping 和ucast 这些配置都需要插件支持，需要将lib64下面的插件软连接到lib目录才不会抛出异常 ln -svf usr/local/heartbeat/lib64/heartbeat/plugins/RAExec/* usr/local/heartbeat/lib/heartbeat/ plugins/RAExec/
ln -svf usr/local/heartbeat/lib64/heartbeat/plugins/* usr/local/heartbeat/lib/heartbeat/plugins/


Heartbeat配置
sed -i 's/#auth 1/auth 1/g' usr/local/heartbeat/etc/ha.d/authkeys
sed -i 's/#1 crc/1 crc/g' usr/local/heartbeat/etc/ha.d/authkeys


2个节点分别复制drbd服务drbddisk脚本到heartbeat目录(编译安装heartbeat，默认没有该脚本)，实现DRBD主从节点资源组的挂载和卸载 cp -p /etc/ha.d/resource.d/drbddisk /usr/local/heartbeat/etc/ha.d/resource.d/


配置haresources资源文件，用于指定双机系统的主节点、 VIP、子网掩码、广播地址及启动的服务等集群资源设置Zabbix-HA1为主从节点角色中为主节点，同时2节点分别执行
echo 'Zabbix-HA1 IPaddr::192.168.8.4/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/opt::xfs' >> usr/local/heartbeat/etc/ha.d/haresources
注：drbd0为drbd创建资源名称，一定要与其一致，否则无法用drbddisk脚本实现DRBD主从节点资源组的挂载和卸载

分别在2节点配置heartbeat的主配置文件ha.cf
mv usr/local/heartbeat/etc/ha.d/ha.cf usr/local/heartbeat/etc/ha.d/ha.cfbak Zabbix-HA1节点执行
cat > usr/local/heartbeat/etc/ha.d/ha.cf <<EOF
debugfile var/log/ha-debug.log logfile var/log/heartbeat.log ucast eth1 172.16.38.2
keepalive 2
warntime 6
deadtime 10
initdead 120
udpport 694 auto_failback off node Zabbix-HA1 node Zabbix-HA2 ping 192.168.8.1
respawn hacluster usr/local/heartbeat/libexec/heartbeat/ipfail apiauth ipfail gid=haclient uid=hacluster
EOF




Zabbix-HA2节点执行
cat > usr/local/heartbeat/etc/ha.d/ha.cf <<EOF
debugfile var/log/ha-debug.log logfile var/log/heartbeat.log ucast eth1 172.16.38.1
keepalive 2
warntime 6
deadtime 10
initdead 120
udpport 694 auto_failback off node Zabbix-HA1 node Zabbix-HA2 ping 192.168.8.1
respawn hacluster usr/local/heartbeat/libexec/heartbeat/ipfail apiauth ipfail gid=haclient uid=hacluster
EOF
注：ucast eth1参数中配置IP地址为对方节点的心跳IP，否则heartbeat无法进行心跳检测和服务故障切换


两机器启动heartbeat服务systemctl enable heartbeat
systemctl start heartbeat  #重点：heartbeat开机启动顺序，先主节点启动，后从节点启动 systemctl stop heartbeat   #实现主从节点之间切换及VIP漂移


查看端口号
netstat -anup|grep 694

从节点停止zabbix-server服务和禁止开机自启动
systemctl stop zabbix-server  &&  systemctl disable zabbix-server


从节点移除zabbix程序和web
mkdir -p data/backup  &&  mv opt/zabbix/ data/backup  &&  mv /opt/www_zabbix /data/backup/


主节点禁用zabbix-server服务开机自启动，服务由heartbeat托管systemctl disable zabbix-server


主节点都执行以下操作
heartbeat接管zabbix-server服务启动停止
cat > /usr/local/heartbeat/etc/ha.d/resource.d/zabbix-server <<EOF #!/bin/bash


PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin export PATH


case "$1" in start)
systemctl start zabbix-server
;;
stop)
systemctl stop zabbix-server
;;
esac exit 0 EOF


赋予执行权限
chmod +x /usr/local/heartbeat/etc/ha.d/resource.d/zabbix-server


脚本拷贝到从节点
scp -P36091 -p /usr/local/heartbeat/etc/ha.d/resource.d/zabbix-server 192.168.8.187:/usr/local/heartbeat/etc/ha.d/resource.d


把zabbix-server脚本名称放入haresources文件中vim /usr/local/heartbeat/etc/ha.d/haresources
Zabbix-HA1 IPaddr::192.168.8.4/24/eth0 drbddisk::drbd0 Filesystem::/dev/drbd0::/opt::xfs zabbix-server


Zabbix-HA1主节点拷贝到Zabbix-HA2从节点
scp -P36091 /usr/local/heartbeat/etc/ha.d/haresources 192.168.8.187:/usr/local/heartbeat/ etc/ha.d/


防止mysql开机启动优先于禁用大页服务，导致tokudb 引擎加载失败(2个节点都要设置) systemctl disable mysql
echo 'systemctl start mysql' >> /etc/rc.local
echo 'systemctl start keepalived' >> /etc/rc.local

Zabbix-HA2节点备份zabbix相关配置表(有VIP地址节点导出)
注：拷贝sql文件到Zabbix-HA1节点，导入sql


Zabbix-HA1节点配置my.cnf 忽略要同步的历史大表
############Open GTID Mode#########
gtid_mode = on enforce_gtid_consistency =true log_slave_updates=true
master-info-repository=TABLE relay-log-info-repository=TABLE slave-parallel-workers=4
###########MySQL AB Replication##########
relay-log = /data/mysql/relay-log auto-increment-increment = 2
auto-increment-offset = 1
replicate-wild-ignore-table=zabbix.history replicate-wild-ignore-table=zabbix.history_uint replicate-wild-ignore-table=zabbix.history_str replicate-wild-ignore-table=zabbix.history_log replicate-wild-ignore-table=zabbix.history_text


Zabbix-HA2节点配置my.cnf
############Open GTID Mode#########
gtid_mode = on enforce_gtid_consistency =true log_slave_updates=true
master-info-repository=TABLE relay-log-info-repository=TABLE slave-parallel-workers=4
###########MySQL AB Replication##########
relay-log = /data/mysql/relay-log auto-increment-increment = 2
auto-increment-offset = 2
replicate-wild-ignore-table=zabbix.history replicate-wild-ignore-table=zabbix.history_uint replicate-wild-ignore-table=zabbix.history_str replicate-wild-ignore-table=zabbix.history_log replicate-wild-ignore-table=zabbix.history_text
注：主从复制各个实例server-id必须唯一，分别重启mysql服务生效


分别在2个主节点上设置主从复制
在Zabbix-HA1节点和Zabbix-HA2节点上分别创建具有复制权限的用户
grant replication slave on *.* to repl@'192.168.8.%' identified by 'yanghui';
flush privileges;


Zabbix-HA1节点授权允许Zabbix-HA2同步相关的表，才能开启start slave;)
change master to master_host='192.168.8.187',master_user='repl',master_password='yanghui',master_auto_position=1;
flush privileges;


Zabbix-HA2节点授权(允许从Zabbix-HA1同步相关的表，才能开启start slave;)
change master to master_host='192.168.8.186',master_user='repl',master_password='yanghui',master_auto_position=1;
flush privileges;


双节点分别授权允许通过VIP地址远程登陆MySQL Zabbix-HA1节点节点执行(8.186)
grant all on *.* to root@'192.168.8.187' identified by 'Zabbix@2021'; select user,host,password from mysql.user;


Zabbix-HA2节点执行(8.187)
grant all on *.* to root@'192.168.8.186' identified by 'Zabbix@2021'; select user,host,password from mysql.user;


分别登陆MySQL主从节点启动主从复制start slave;
show slave status\G;


drbd挂载的节点更改zabbix-web的IP地址为VIP地址
/opt/www_zabbix/conf/zabbix.conf.php
sed -i 's/192.168.8.186/192.168.8.5/g' /opt/www_zabbix/conf/zabbix.conf.php sed -i 's/192.168.8.186/192.168.8.5/g ' /opt/zabbix/etc/zabbix/zabbix_server.conf注：表示绿色为宿主机节点IP，红色为数据库的VIP，需重启zabbix-server服务

keepalived+mysql HA方案完善补充缘由：
使用脚本和定时任务监测keepalived 服务运行状态，实现mysql 故障切换后，keepalived 通过后台定时任务实现自动恢复服务。


keepalived脚本分别放到2个节点的/etc/keepalived目录
echo '*/2 * * * * root /etc/keepalived/monitor-keepalived.sh' >> /etc/crontab systemctl restart crond
注：修改定时任务调用脚本后，一定要重启crond任务才会生效


heartbeat+zabbix HA方案完善补充缘由：
heartbeat本身不对应用服务状态做判断，需要编写脚本来判断服务异常，弥补该架构的不足。


部署supervisor
yum install -y supervisor
mkdir -p /etc/supervisor/config.d
echo_supervisord_conf > /etc/supervisor/supervisord.conf


配置supervisord主配置文件
cat >> /etc/supervisor/supervisord.conf <<EOF [include]
files = /etc/supervisor/config.d/*.ini EOF


配置监视zabbix-server进程
cat > /etc/supervisor/config.d/zabbix-server.ini <<EOF [program:zabbix-heartbeat]
user=root
directory= /etc/supervisor
command= /bin/sh /etc/supervisor/heartbeat.sh numprocs=1
autostart=true autorestart=true startretries=3 EOF


启动supervisord服务
systemctl enable supervisord && systemctl start supervisord && systemctl status supervisord


补充：heartbeat编译参数注解：
vim /usr/local/heartbeat/etc/ha.d/ha.cf
debugfile /var/log/ha-debug  ##用于记录heartbeat的调试信息
logfile/var/log/ha-log  ##用于记录heartbeat的日志信息
logfacilitylocal0  ##设置heartbeat的日志,这里用的是系统日志
keepalive 2  ##设定心跳(监测)时间时间为2秒
deadtime 30  ##指定若备用节点在30秒内未收到主节点心跳信号,则接管主服务器资源
warntime 10  ##指定心跳延迟的时间为10秒,10秒内备节点不能接收主节点心跳信号，即往日志写入警告日志，但不会切换服务
initdead 60  ##系统启动或重启后预留的忽略时间段，取值至少为deadtime的两倍
udpport 694  ##广播/单播通讯使用的Udp端口
#bcast ens32  ##使用网卡eno32发送心跳检测
#mcast eth0 225.0.0.1 694 1 0  ##采用网卡eth0的Udp多播来组织心跳，一般在备用节点
ucast ens32 192.168.1.64


##采用网卡eth32的udp单播来组织心跳，后面跟的IP地址为双机对方IP地址
auto_failback on  ##定义当主节点恢复后，是否将服务自动切回，争抢VIP地址
node xuegod63.cn node xuegod64.cn ping 192.168.1.1


##主节点名称##备用节点名称
##通过ping网关检测心跳是否正常，仅用来测试网络


apiauth ipfail gid=haclient uid=hacluster  ##设置启动IPfail的用户和组
respawn hacluster /usr/local/heartbeat/libexec/heartbeat/ipfail  ##指定和heartbeat一起启动、关闭的进程注：Bcast、ucast和mcast分别代表广播、单播和多播，是组织心跳的的方式，任选其一