Redis Sentinel 主从配置

云时代IT运维 2019-03-18

219

redis主从配置

1. redis sentinel介绍

Sentinel是Redis的高可用性（HA）解决方案，由一个或多个Sentinel实例组成的Sentinel系统可以监视任意多个主服务器，以及这些主服务器属下的所有从服务器，并在被监视的主服务器进行下线状态时，自动将下线主服务器属下的某个从服务器升级为新的主服务器，然后由新的主服务器代替已下线的主服务器继续处理命令请求。Redis提供的sentinel（哨兵）机制，通过sentinel模式启动redis后，自动监控master/slave的运行状态，基本原理是：心跳机制+投票裁决。
sentinel组件的核心功能

监控：Sentinel 会不断地检查你的主服务器和从服务器是否运作正常。
提醒：当被监控的某个 Redis 服务器出现问题时， Sentinel 可以通过 API 向管理员或者其他应用程序发送通知。
自动故障迁移：当一个主服务器不能正常工作时， Sentinel 会开始一次自动故障迁移操作，它会将失效主服务器的其中一个从服务器升级为新的主服务器，并让失效主服务器的其他从服务器改为复制新的主服务器；
通知：当客户端试图连接失效的主服务器时，集群也会向客户端返回新主服务器的地址，使得集群可以使用新主服务器代替失效服务器。

2. redis主从架构图

3. redis主从架构部署

服务器用途	OS版本	软件版本	IP地址
Redis Server 01（主）	Centos-6.8_64bit	redis-3.2.8	10.83.32.138
Redis Server 02（从）	Centos-6.8_64bit	redis-3.2.8	10.83.32.133
Redis Sentinel 01	Centos-6.8_64bit	redis-3.2.8	10.83.32.138
Redis Sentinel 02	Centos-6.8_64bit	redis-3.2.8	10.83.32.133
Redis Sentinel 03	Centos-6.8_64bit	redis-3.2.8	10.83.32.144

4. Redis 主从安装步骤：

cd /software
wget http://download.redis.io/releases/redis-3.2.8.tar.gz
tar -xf redis-3.2.8.tar.gz -C /usr/local/
cd /usr/local/
ln -sv redis-3.2.8/ redis
‘redis’ -> ‘redis-3.2.8/’
cd redis
make
make install
#配置环境变量
vim /etc/profile.d/redis.sh
REDIS_HOME=/usr/local/redis/src
PATH=$PATH:$REDIS_HOME
export PATH
source /etc/profile
#创建redis需要的目录
[root@kubenode1 redis]# mkdir -p /data/redis/sentinel
[root@kubenode1 redis]# mkdir -p /var/run/redis
[root@kubenode1 redis]# mkdir -p /var/log/redis
[root@kubenode1 redis]# mkdir -p /var/redis/redis_6379
[root@kubenode1 redis]# mkdir -p /etc/redis
#修改配置文件
vim /etc/redis/redis_6379.conf
daemonize yes
pidfile "/var/run/redis/redis_6379.pid"
port 6379
bind 0.0.0.0
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/var/log/redis/redis_6379.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump_6379.rdb"
dir "/var/redis/redis_6379"
slave-read-only yes
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
requirepass test123
maxmemory 1953125kb
appendonly yes
appendfilename "appendonly_6379.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
# Generated by CONFIG REWRITE
[root@kubenode1 redis]#
#编写redis服务启动配置文件
#!/bin/sh
#
# Simple Redis init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
# chkconfig: 2345 10 90
# description: redis_6379 ....
REDISPORT=6379
EXEC=/usr/local/redis/src/redis-server
CLIEXEC=/usr/local/redis/src/redis-cli
PIDFILE=/var/run/redis/redis_${REDISPORT}.pid
CONF="/etc/redis/redis_${REDISPORT}.conf"
REDISPASS=test123
case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running or crashed"
        else
                echo "Starting Redis server..."
                $EXEC $CONF
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                $CLIEXEC -p $REDISPORT -a $REDISPASS shutdown
                while [ -x /proc/${PID} ]
                do
                    echo "Waiting for Redis to shutdown ..."
                    sleep 1
                done
                echo "Redis stopped"
        fi
        ;;
    status)
        PID=$(cat $PIDFILE)
        if [ ! -x /proc/${PID} ]
        then
            echo ‘Redis is not running‘
        else
            echo "Redis is running ($PID)"
        fi
        ;;
    *)
        echo "Please use start or stop or status as first argument"
        ;;
esac
chmod +x /etc/init.d/redis6379
chkconfig redis6379 on
iptables -I INPUT 2 -p tcp -m state --state NEW -m TCP -m multiport --dports 6379 -j ACCEPT
service iptables save
iptables --line-numbers -L -n
#如果服务器开启了iptables，还需要增加iptables的策略，增加6379端口；
[root@kubenode1 ~]# service redis6379 start
Starting Redis server...
[root@kubenode1 ~]# ps aux |grep redis
root     17831  0.0  0.0  33336  7448 ?        Ssl  13:11   0:00 /usr/local/redis/src/redis-server 0.0.0.0:6379
root     17845  0.0  0.0 112660   948 pts/0    R+   13:11   0:00 grep --color=auto redis
[root@kubenode1 ~]# redis-cli -h 10.83.32.138 -p 6379 -a test123
10.83.32.138:6379> keys *
(empty list or set)
10.83.32.138:6379> set test 123
OK
10.83.32.138:6379> keys *
1) "test"
10.83.32.138:6379> exit
#启动redis服务，同时查看redis进程是否已经启动，登录redis测试等；
[root@kubenode1 ~]# service redis6379 stop
Stopping ...
Redis stopped
[root@kubenode1 ~]# ps aux |grep redis
root     18137  0.0  0.0 112660   948 pts/0    S+   13:13   0:00 grep --color=auto redis
#停止redis服务
#由于redis的安装步骤比较多，我将redis的安装步骤总结成了一个shell脚本，执行shell脚本即可完成单机redis的安装；
#!/bin/bash
#
ip=$(ip addr|grep inet|grep -v 127.0.0.1|awk '{print $2}'|awk -F '/' '{print $1}')
passwd=$1
echo -e "\033[31m 开始安装redis......\033[0m"
if [ $(netstat -antlp|grep 6379|grep -v grep|wc -l) -ne 1 ];then
cd /software
wget http://download.redis.io/releases/redis-3.2.8.tar.gz
tar xzvf redis-3.2.8.tar.gz -C /usr/local/ >/dev/null 2>&1
cd /usr/local/
ln -sv redis-3.2.8/ redis
cd /usr/local/redis
make
make install
fi
if [ ! -d /data/redis/data ];then
    mkdir -p /data/redis/data
fi
if [ ! -d /confs/redis ];then
    mkdir -p /confs/redis
fi
if [ ! -d /logs/redis ];then
    mkdir -p /logs/redis
fi
if [ ! -f /confs/redis/redis_6379.conf ];then
    wget -O /confs/redis/redis_6379.conf ftp://bqjrftp:Pass123$%^@10.83.20.27:9020/confs/redis_6379.conf
    sed -i 's/# bind 127.0.0.1/bind '${ip}'/g' /confs/redis/redis_6379.conf
    sed -i 's#requirepass "bqpms"#requirepass '${passwd}'#g' /confs/redis/redis_6379.conf
fi
if [ ! -f /etc/init.d/redis ];then
    wget -O /etc/init.d/redis ftp://bqjrftp:Pass123$%^@10.83.20.27:9020/confs/redis
    chmod +x /etc/init.d/redis
    chkconfig --add redis
    chkconfig redis on
    sed -i 's/REDISPW=bqpms/REDISPW='${passwd}'/g' /etc/init.d/redis
fi
systemctl daemon-reload
systemctl start redis

5. redis主从配置

按照第4步的脚本可以继续安装第二台redis服务器。第二台redis服务器与第一台redis服务器的配置文件不同之处在于增加了如下配置:

slaveof 10.83.32.138 6379  #在从节点的redis机器上面配置slaveof redis主节点IP  端口
masterauth "test123"  #在从节点的redis机器上面配置认证密码，这个密码就是主redis服务器里面配置的requirepass "test123"
systemctl restart redis

重新启动redis服务,然后通过命令行查看redis主从信息：

[root@kubenode1 ~]# redis-cli -p 6379 -h 10.83.32.138 -a test123 info|awk -F ":" '/role/{print $2}'
master
[root@kubenode2 software]# redis-cli -p 6379 -h 10.83.32.133 -a test123 info|awk -F ":" '/role/{print $2}'
slave

6. Redis Sentinel配置（配置3个监视节点）

三台sentinel机器的配置文件是一样的，具体的配置如下：
vim /etc/redis/sentinel.conf
port 26379
dir "/data/redis/sentinel"
daemonize yes
protected-mode no
logfile "/var/log/redis/sentinel.log"
sentinel myid 8b8858ef01acd3917a1a0f0e12b85f5221bd8b2f
sentinel monitor MyMaster6379 10.83.32.138 6379 2
sentinel down-after-milliseconds MyMaster6379 2000
sentinel failover-timeout MyMaster6379 6000
# Generated by CONFIG REWRITE
sentinel config-epoch MyMaster6379 1
sentinel leader-epoch MyMaster6379 0
sentinel current-epoch 1
sentinel auth-pass MyMaster6379 test123 #sentinel需要配置认证密码
vim /etc/init.d/sentinel
#!/bin/sh
#
# Simple Redis_sentinel init.d script conceived to work on Linux systems
# as it does use of the /proc filesystem.
# chkconfig: 2345 10 90
# description: sentinel ....
SENTINEL_PORT=26379
EXEC=/usr/local/redis/src/redis-sentinel
PIDFILE=/var/run/redis/redis_${SENTINEL_PORT}.pid
CONF="/etc/redis/sentinel.conf"
start() {
   if [ -f $PIDFILE ]
   then
     echo "$PIDFILE exists, process is already running or crashed"
     else
     echo "Starting Redis server..."
     touch ${PIDFILE}
     $EXEC $CONF
   fi
}
stop() {
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                PID=$(cat $PIDFILE)
                echo "Stopping ..."
                pkill redis-sentinel
                rm -rf ${PIDFILE}
                while [ -f ${PIDFILE} ]
                do
                    echo "Waiting for Redis_sentinel to shutdown ..."
                    sleep 1
                done
                echo "Redis_sentinel stopped"
        fi
}
status() {
        PID=$(cat $PIDFILE)
        if [ ! -x /proc/${PID} ]
        then
            echo ‘Redis_sentinel is not running‘
        else
            echo "Redis_sentinel is running"
        fi
}
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
        status
        ;;
    restart)
        stop
        sleep 2
        start
        ;;
      *)
        echo "echo "USAGE: $0 start|stop|restart|status""
        ;;
esac
chmod +x /etc/init.d/sentinel
chkconfig sentinel on
systemctl start sentinel
#通过命令行检查sentinel

7. Redis Sentinel 验证

以上六步已经成功安装了redis服务器和sentinel服务器。我们的设计架构是redis两台服务器，一台是主服务器，一台是从服务器；sentinel是三台服务器（sentinel服务器一般建议是奇数台，因为偶数台容易产生脑裂），其中两台sentinel服务器是和redis服务器服用的。
第一个验证步骤是检查redis服务的主从状态和sentinel服务的状态；

[root@kubenode2 redis]# redis-cli -p 6379 -a test123 info|awk -F ":" '/role/{print $2}'
master
[root@kubenode1 init.d]# redis-cli -p 6379 -h 10.83.32.138 -a test123 info|awk -F ":" '/role/{print $2}'
slave
#在redis slave的节点上面查看配置文件里面有一行
slaveof 10.83.32.133 6379
并且这一行的配置是动态生成的。slave节点在哪一台机器上面，就会在哪一台机器上面出现如上的slaveof 配置
[root@kubenode1 init.d]# redis-cli -h 10.83.32.138 -p 26379 info|grep sentinel
redis_mode:sentinel
executable:/usr/local/redis/src/redis-sentinel
config_file:/etc/redis/sentinel.conf
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=MyMaster6379,status=ok,address=10.83.32.133:6379,slaves=1,sentinels=3
[root@kubenode2 redis]# redis-cli -h 10.83.32.138 -p 26379 info|grep sentinel
redis_mode:sentinel
executable:/usr/local/redis/src/redis-sentinel
config_file:/etc/redis/sentinel.conf
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=MyMaster6379,status=ok,address=10.83.32.133:6379,slaves=1,sentinels=3
[root@kubenode3 redis]# redis-cli -h 10.83.32.144 -p 26379 info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=MyMaster6379,status=ok,address=10.83.32.133:6379,slaves=1,sentinels=3
#从三台安装了sentinel的节点上面执行命令查看sentinel服务状态，可以确认运行sentinel服务的节点是3台，主master redis服务器是10.83.32.133
#我们现在模拟停止掉redis master节点的redis服务，然后等上一分钟再启动redis master节点上面的redis服务，查看redis master是否完成了切换；
[root@kubenode2 redis]# /etc/init.d/redis stop
Stopping ...
Redis stopped
[root@kubenode2 redis]# #停止master  节点redis
[root@kubenode2 redis]# /etc/init.d/redis start
Starting Redis server...
[root@kubenode2 redis]#   #启动master  节点redis
[root@kubenode1 init.d]# redis-cli -p 6379 -h 10.83.32.138 -a test123 info|awk -F ":" '/role/{print $2}'
master
#另外一台机器的角色变成了master
[root@kubenode2 redis]# redis-cli -p 6379 -a test123 info|awk -F ":" '/role/{print $2}'
slave
#原来的master变成了slave
[root@kubenode1 init.d]# redis-cli -h 10.83.32.138 -p 26379 info|grep sentinel
redis_mode:sentinel
executable:/usr/local/redis/src/redis-sentinel
config_file:/etc/redis/sentinel.conf
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=MyMaster6379,status=ok,address=10.83.32.138:6379,slaves=1,sentinels=3
#sentinel服务结果查看的master redis为新的redis master

8. 利用keepalived实现统一的redis连接地址

两台redis主从服务器分别安装keepalived组件，并且配置VIP地址，配置监控脚本。实现的最终目标为：

客户端连接redis只需要一个ip地址；
如果主redis服务器的redis出现故障，vip自动迁移到从redis服务器，并且从redis服务器升级为master；
如果主redis服务器的keepalived服务出现故障，vip自动迁移到从redis服务器，并且主redis服务器的redis进程自动停止，从redis服务器升级为master;

安装和配置如下

yum -y install keepalived
#主redis master服务器的keepalived配置
[root@kubenode3 ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
lvs_id LVS_redis01
vrrp_mcast_group4 224.0.0.22
}
vrrp_script chk_redis {
  script "/etc/keepalived/scripts/redis_check.sh"
    weight -20
    interval 12                                  
  }
vrrp_instance VI_1 {
  state backup                            
  interface eth0                          
  virtual_router_id 52
  nopreempt
  priority 200    
  advert_int 4                   
  virtual_ipaddress {
    10.83.32.199                      
  }
  track_script {
    chk_redis                    
  }
   notify_stop  /etc/keepalived/scripts/keepalived_stop.sh
}
# 从redis服务器的keepalived 配置
[root@kubenode2 ~]# cat /etc/keepalived/keepalived.conf 
global_defs {
lvs_id LVS_redis02
vrrp_mcast_group4 224.0.0.22 
}
vrrp_script chk_redis {
  script "/etc/keepalived/scripts/redis_check.sh"  #redis检查脚本
  weight -20
  interval 12                               
}
vrrp_instance VI_1 {
  state backup                           
  interface eth0                         
  virtual_router_id 52
  priority 100   #从redis服务器优先级低
  nopreempt
  advert_int 4                  
  virtual_ipaddress {
   10.83.32.199  # vip地址
}
  track_script {
   chk_redis                   
  }
   notify_stop  /etc/keepalived/scripts/keepalived_stop.sh  #停止keepalived的时候也会执行这个脚本
}
mkdir -p /etc/keepalived/scripts  #在两台redis服务器上面创建脚本目录

脚本1：redis_check.sh
主要用途是用于keepalived服务运行时根据固定间隔时间执行的脚本。原理是判断本机是不是VIP地址的所有者，如果是判断redis是不是为master，如果不是master就会停止掉keepalived服务。如果本机不是VIP地址的所有者，判断本机的redis是否能够连接通，如果连接不通也停止掉keepalived服务；

#!/bin/bash
Vip=10.83.32.199
Port=6379
VipValue=`/sbin/ip add|grep $Vip`
RedisCli_Cmd="/usr/local/redis/src/redis-cli"
Redispass="test123"
case_value(){
  case $1 in
     RedisRole)
        Value=`$RedisCli_Cmd -p $Port -a $Redispass info|awk -F'[:|\r]' '/role/{print $2}'`
        ;;
     Alive)
        Value=`$RedisCli_Cmd -p $Port -a $Redispass PING`
  esac
}
sub_value(){
    case_value $1
    if [ $Value = "$2" ];then
    exit 0 
    else
          `which pkill` keepalived
          exit 1
     fi
}
if [ -n "$VipValue" ];then
   while true
   do
   Value=`$RedisCli_Cmd -p $Port -a $Redispass info|awk -F'[:|\r]' '/role/{print $2}'`
      if [ "$Value" == "slave" ];then
      sleep 1
      echo "the host redis is $Value"
      else
      break
      fi
   done
  sub_value RedisRole master
else
  sub_value Alive PONG
fi

脚本2：keepalived_stop.sh

keepalived的停止脚本，配合keepalived配置文件一起来进行。如果keepalived服务被停止掉的话，就会同步执行停止redis服务的命令。保证vip漂移到另外一台redis机器上的同时redis的master角色也会漂移过去；

vim keepalived_stop.sh
 #!/bin/bash
`which pkill` redis-server

脚本3： keepalived_start.sh

keepalived的启动脚本，用于/etc/rc.d/rc.local开机启动。在 /etc/rc.d/rc.local里面先启动redis，然后再启动keepalived.启动keepalived的时候，先检查本机的redis是不是master，如果是master就启动,如果不是master是slave，再判断vip地址是不是能够ping通，如果能够ping通再启动keepalived

#!/bin/bash
RedisRole=`/usr/local/bin/redis-cli -p 6379 -a test123 info|awk -F'[:|\r]' '/role/{print $2}'`
KeepalivedStartCmd="/etc/init.d/keepalived start"
if [ $RedisRole = "master" ];then
        $KeepalivedStartCmd
else
        while true;
        do
                sleep 1
                ping 10.83.22.241 -c 1 >/dev/null 2>&1
                if [ $? -eq 0 ];then $KeepalivedStartCmd;break;fi
        done
fi

4. 验证VIP地址登录：

systemctl start keepalived
ip address |grep 10.83.32.199 
[root@kubenode3 ~]# redis-cli -h 10.83.32.199 -p 6379 -a test123
10.83.32.199:6379> keys *
1) "test"
10.83.32.199:6379> exit
[root@kubenode1 ~]# redis-cli -p 6379 -h 10.83.32.138 -a test123 info|awk -F ":" '/role/{print $2}'
master
[root@kubenode1 ~]# /etc/init.d/redis6379 stop
Stopping ...
Redis stopped
[root@kubenode1 ~]#
#在master节点停止redis服务，查看vip地址
[root@kubenode2 ~]# ip addr|grep 10.83.32.199
    inet 10.83.32.199/32 scope global eth0
[root@kubenode2 ~]#
[root@kubenode2 ~]# redis-cli -p 6379 -a test123 info|awk -F ":" '/role/{print $2}'
master
[root@kubenode2 ~]#
#发现slave节点变成了redis 的master，并且vip迁移到这个节点
[root@kubenode3 ~]# redis-cli -h 10.83.32.199 -p 6379 -a test123
10.83.32.199:6379> keys *
1) "test"
10.83.32.199:6379> exit
[root@kubenode3 ~]
#测试登录vip地址，发现可以正常登录
[root@kubenode2 ~]# systemctl stop keepalived
[root@kubenode2 ~]#
#现在停止master节点的keepalived

# 推荐关注我的个人微信公众号 “云时代IT运维”,周期性更新最新的应用运维类技术文档。关注虚拟化和容器技术、CI/CD、自动化运维等最新前沿运维技术和趋势；

redis

文章转载自云时代IT运维，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

Redis Sentinel 主从配置

redis主从配置

评论