Table of Contents

• 1. 环境准备
• 1.1. 硬件配置
• 1.2. 操作系统软件包&第三方工具
• 1.3. 关闭SELINUX
• 1.4. 配置防火墙
• 1.5. 优化系统参数
• 1.6. 修改hugepage配置
• 2. 部署Redis实例
• 2.1. 节点信息
• 2.2. 创建安装目录
• 2.3. 安装Redis实例
• 2.3.1. 安装依赖包
• 2.3.2. 下载安装包、编译安装
• 2.4. 配置Redis实例的参数
• 2.4.1. 编辑示例文件
• 2.4.2. 批量生成配置文件
• 2.5. 配置环境变量
• 2.6. 启动Redis实例
• 3. 部署Redis Cluster
• 3.1. Ruby环境准备
• 3.2. 搭建集群
• 3.3. 检查集群
• 4.附录
• 4.1. Redis基本参数参考
• 4.2. Redis复制参数参考
• 4.3. Redis Cluster参数参考

1. 环境准备

1.1. 硬件配置

1.2. 操作系统软件包&第三方工具

^: 若Redis版本为Redis 5.x及以上版本时，可不需要安装ruby。

1.3. 关闭SELINUX

临时禁用selinux：

# setenforce 0
# more /etc/selinux/config | grep -Ev "$|[#;]"

永久禁用selinux：

# vi /etc/selinux/config

将
SELINUX= enforcing
修改为
SELINUX=disabled 
:x保存退出

1.4. 配置防火墙

如需要禁用防火墙，可执行如下命令：

1）若是CentOS 7.x系统，可执行如下命令禁用防火墙：
关闭防火墙：
# systemctl stop firewalld.service
禁用防火墙开机自启：
# systemctl disable firewalld.service
查看防火墙状态：
# systemctl status firewalld.service

2）若是Ubuntu系统，可执行如下命令禁用防火墙：
# iptables -P INPUT ACCEPT

3）其他系统禁用防火墙命令，请自行查找。

如需配置防火墙规则，可参照如下示例：

# vim /etc/iptables.rules

*filter
:INPUT ACCEPT [3715:523951]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1925:209933]
-A INPUT -i lo -j ACCEPT
#允许ssh端口通过
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
#允许icmp
-A INPUT -p icmp -j ACCEPT
#允许Redis服务器间互通，即Redis服务器间可互相访问Redis端口为2011~2014的Redis实例
-A INPUT -s 192.168.111.111/24 -p tcp -m state --state NEW -m tcp --dport 2011:2014 -j ACCEPT
-A INPUT -s 192.168.111.112/24 -p tcp -m state --state NEW -m tcp --dport 2011:2014 -j ACCEPT
-A INPUT -s 192.168.111.113/24 -p tcp -m state --state NEW -m tcp --dport 2011:2014 -j ACCEPT
#运行其他服务器访问Redis，如允许172.172.122.222服务器访问本地Redis端口为2011~2014的Redis实例
-A INPUT -s 172.172.122.222/32 -p tcp -m state --state NEW -m tcp --dport 2011:2014 -j ACCEPT
#暴露本地22201服务端口
-A INPUT -p tcp -m tcp --dport 22201 -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT
COMMIT

防火墙规则配置完毕后，将配置文件导入：

# iptables-restore < /etc/iptables.rules

导入防火墙规则配置文件后，打开防火墙：

# iptables -P INPUT DROP

如需要开机自动恢复规则，把恢复命令添加到启动脚本：

# echo '/sbin/iptables-restore < /etc/iptables.rules' >>/etc/rc.local
# echo '/sbin/iptables -P INPUT DROP' >>/etc/rc.local

# cat /etc/rc.local

1.5. 优化系统参数

编辑系统配置文件sysctl.conf：

# vim /etc/sysctl.conf

vm.overcommit_memory=1
vm.swappiness = 0
net.core.somaxconn = 1024
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_fin_timeout = 30
net.nf_conntrack_max = 655350
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_local_reserved_ports = 2011-2014,12011

^: net.ipv4.tcp_tw_reuse设置为1表示开启重用，允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭。

装载ip_conntrack模块：

# modprobe ip_conntrack

载入sysctl配置文件：

# sysctl -p

1.6. 修改hugepage配置

临时禁用hugepage：

# echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled

为了使机器重启后THP配置依然生效，可以在/etc/rc.local中追加：

# echo '/bin/echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

2. 部署Redis实例

2.1. 节点信息

2.2. 创建安装目录

规划目录：

# mkdir -p /data/third_party/redis/redis_cluster/2011/{bin,data,run,logs,conf} 
# mkdir -p /data/third_party/redis/redis_cluster/2012/{bin,data,run,logs,conf} 
# mkdir -p /data/third_party/redis/redis_cluster/2013/{bin,data,run,logs,conf} 
# mkdir -p /data/third_party/redis/redis_cluster/2014/{bin,data,run,logs,conf} 

2.3. 安装Redis实例

2.3.1. 安装依赖包

# yum -y install gcc

2.3.2. 下载安装包、编译安装

解压
# cd /opt/app/softwares
# wget http://download.redis.io/releases/redis-3.2.12.tar.gz
# tar -zxvf redis-3.2.12.tar.gz -C /var/lib

编译依赖
# cd /var/lib
# mv redis-3.2.12/ redis-online
# cd redis-online/deps/
# make geohash-int hiredis jemalloc linenoise lua

编译Redis
# cd ..
# make && make install

2.4. 配置Redis实例的参数

2.4.1. 编辑示例文件

# vim /data/third_party/redis/redis_cluster/2011/conf/6379.conf


daemonize yes 

pidfile "/data/third_party/redis/redis_cluster/2011/run/2011.pid" 
logfile "/data/third_party/redis/redis_cluster/2011/logs/2011.log" 

port 2011 

dbfilename "dump-2011.rdb" 
dir "/data/third_party/redis/redis_cluster/2011/data" 

tcp-backlog 511 

timeout 0 
tcp-keepalive 0 
loglevel notice 

databases 16 

repl-backlog-size 100mb

min-slaves-max-lag 10

#save 900 1 
#save 300 10 
#save 60 10000 
save "" 

maxmemory 31005464653 
maxmemory-policy allkeys-lru 

protected-mode no 
bind 0.0.0.0 

stop-writes-on-bgsave-error yes 
rdbcompression yes 

rdbchecksum yes 

slave-serve-stale-data yes 

slave-read-only yes 

repl-diskless-sync no 

repl-diskless-sync-delay 5 

repl-disable-tcp-nodelay no 

slave-priority 100 

appendonly no 

appendfilename "appendonly-2011.aof" 

appendfsync no 

no-appendfsync-on-rewrite no 

auto-aof-rewrite-percentage 100 
auto-aof-rewrite-min-size 64mb 

aof-load-truncated yes 

lua-time-limit 5000 

slowlog-log-slower-than 10000 
slowlog-max-len 128 

latency-monitor-threshold 0 

notify-keyspace-events "" 

hash-max-ziplist-entries 512 
hash-max-ziplist-value 64 

list-max-ziplist-entries 512 
list-max-ziplist-value 64 

set-max-intset-entries 512 

zset-max-ziplist-entries 128 
zset-max-ziplist-value 64 

hll-sparse-max-bytes 3000 

activerehashing yes 

client-output-buffer-limit normal 0 0 0 
client-output-buffer-limit slave 1024mb 512mb 60 
client-output-buffer-limit pubsub 32mb 8mb 60 
hz 10 

aof-rewrite-incremental-fsync yes 
maxclients 10000 

# masterauth "hust4400" 
# requirepass "hust4400"

rename-command KEYS XXXX-KEYS
rename-command FLUSHALL XXXX-FLUSHALL
rename-command FLUSHDB XXXX-FLUSHDB

2.4.2. 批量生成配置文件

编辑脚本cp_conf.sh：

# vim /data/third_party/redis/redis_cluster/2011/conf/cp_conf.sh 

#!/bin/bash

for ((i=2011;$i <= 2012; i=$i+1))
{
       cd /data/third_party/redis/redis_cluster/2011/conf/
       cp 6379.conf $i.conf 
       sleep 1
       sed -i "s,2011,$i,g" $i.conf
       echo "cluster-enabled yes" >> $i.conf
       echo "cluster-node-timeout 15000" >> $i.conf
       echo "cluster-require-full-coverage no " >> $i.conf
       echo "cluster-config-file \"nodes-$i.conf\"" >> $i.conf
       sleep 1
}

执行脚本，批量生成多个配置文件：

# chmod +x /data/third_party/redis/redis_cluster/2011/conf/cp_conf.sh 
# sh /data/third_party/redis/redis_cluster/2011/conf/cp_conf.sh 

删除单机配置文件6379.conf和脚本：

# rm -rf /data/third_party/redis/redis_cluster/2011/conf/6379.conf
# rm -rf /data/third_party/redis/redis_cluster/2011/conf/cp_conf.sh

检查配置文件参数：

# for((i=2011;i<=2012;i++));do grep -E 'tcp-keepalive|appendfsync' /data/third_party/redis/redis_cluster/2011/conf/$i.conf;done

2.5. 配置环境变量

编辑profile配置文件

# vim /etc/profile

...
unset i
unset -f pathmunge
export LD_LIBRARY_PATH=./
ulimit -SHn 655350
REDIS_HOME=/data/third_party/redis
export REDIS_HOME
PATH=PATH:REDIS_HOME/bin

执行如下命令，使配置的环境变量在当前终端立即生效：

# source /etc/profile

2.6. 启动Redis实例

启动Redis实例：

# redis-server /data/third_party/redis/redis_cluster/2011/conf/2011.conf
# redis-server /data/third_party/redis/redis_cluster/2012/conf/2012.conf
# redis-server /data/third_party/redis/redis_cluster/2013/conf/2013.conf
# redis-server /data/third_party/redis/redis_cluster/2014/conf/2014.conf

查看Redis实例进程信息：

# ps -ef|grep -v grep|grep "redis-server 0.0.0.0:201" 

3. 部署Redis Cluster

3.1. Ruby环境准备

在任意节点部署即可。

创建安装目录

# mkdir -p /opt/app/softwares

下载安装包

# wget https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.gz

编译ruby

# cd /opt/app/softwares
# tar xvf ruby-2.3.1.tar.gz
# cd ruby-2.3.1
# ./configure -prefix=/usr/local/ruby

安装ruby

# make
# make install
# cd /usr/local/ruby
# cp bin/ruby /usr/local/bin
# cp bin/gem /usr/local/bin

安装rubygem redis依赖

# cd /opt/app/softwares
# wget http://rubygems.org/downloads/redis-3.3.0.gem
# gem install -l redis-3.3.0.gem

^: 若是Ubuntu系统，则通过此种方式部署ruby环境，安装ruby：apt-get update apt-get install ruby-full -y 安装rubygem redis依赖：gem install redis

3.2. 搭建集群

使用redis-trib搭建集群，在部署ruby的节点下，执行如下命令：
# cp /data/PaaS/redis/redis_cluster/bin/redis-4.0.14/src/redis-trib.rb /usr/local/bin/
# redis-trib.rb create --replicas 1 192.168.111.111:2011 192.168.111.111:2012 192.168.111.111:2013 192.168.111.111:2014 192.168.111.112:2011 192.168.111.112:2012 192.168.111.112:2013 192.168.111.112:2014 192.168.111.113:2011 192.168.111.113:2012 192.168.111.113:2013 192.168.111.113:2014

^: 若是Redis 5.x及以上版本，可采用redis-cli工具部署，因为Redis 5.x及以上版本彻底抛弃了ruby，将集群管理功能集成到了redis-cli工具中，

创建语法同redis-trib。

3.3. 检查集群

通过redis-trib检查集群状态

# redis-trib.rb check 192.168.111.111:2011

Performing Cluster Check (using node 192.168.111.111:2011)
M: 7b5f21f124373b7d99edafaab01b90afdcfe8592 192.168.111.111:2011
slots:0-2730 (2731 slots) master
1 additional replica(s)
M: 81757eeb491059021c8e2468f361245764800c05 192.168.111.112:2011
slots:2731-5460 (2730 slots) master
1 additional replica(s)
M: 1e94fb4e201b9e7c4eccc207a6f8bae883d1dff1 192.168.111.111:2012
slots:8192-10922 (2731 slots) master
1 additional replica(s)
M: cf118efd950061ae013c52539f66f7ff1779100f 192.168.111.112:2012
slots:10923-13652 (2730 slots) master
1 additional replica(s)
S: b9de42bb24a48cc6a8b6fc72f2c38f43057bf459 192.168.111.113:2014
slots: (0 slots) slave
replicates cf118efd950061ae013c52539f66f7ff1779100f
S: 79187b202bdb436e26d5a7b16e1e91bff0584dc1 192.168.111.111:2014
slots: (0 slots) slave
replicates 8bcaa25b6d0c61fedf1b181971f5eb7f4da7380d
S: 12874ba078047ad747e6123f68c32ddd94c752bc 192.168.111.112:2013
slots: (0 slots) slave
replicates 7b5f21f124373b7d99edafaab01b90afdcfe8592
S: 722ee8a5d3508df3eb6afeff60550de9b5b524be 192.168.111.113:2013
slots: (0 slots) slave
replicates 81757eeb491059021c8e2468f361245764800c05
M: 8bcaa25b6d0c61fedf1b181971f5eb7f4da7380d 192.168.111.113:2011
slots:5461-8191 (2731 slots) master
1 additional replica(s)
S: 7a5c454dfecc3ed53c6bd8cc29a01c3114ee0067 192.168.111.111:2013
slots: (0 slots) slave
replicates 15595bdf3cccc20eb25faf7993608b20967a8673
S: 2c24ba35f85e185b98c66ead9f3ce1a444839fb8 192.168.111.112:2014
slots: (0 slots) slave
replicates 1e94fb4e201b9e7c4eccc207a6f8bae883d1dff1
M: 15595bdf3cccc20eb25faf7993608b20967a8673 192.168.111.113:2012
slots:13653-16383 (2731 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
Check for open slots...
Check slots coverage...
[OK] All 16384 slots covered.

通过redis-cli命令行检查集群状态：

# redis-cli -h 192.168.111.111 -p 2011 cluster info 

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:12
cluster_size:6
cluster_current_epoch:12
cluster_my_epoch:1
cluster_stats_messages_ping_sent:57
cluster_stats_messages_pong_sent:77
cluster_stats_messages_sent:134
cluster_stats_messages_ping_received:66
cluster_stats_messages_pong_received:57
cluster_stats_messages_meet_received:11
cluster_stats_messages_received:134

通过redis-cli命令行检查集群节点状态：

# redis-cli -h 192.168.111.111 -p 2011 cluster nodes 

81757eeb491059021c8e2468f361245764800c05 192.168.111.112:2011@12011 master - 0 1615276888103 5 connected 2731-5460
1e94fb4e201b9e7c4eccc207a6f8bae883d1dff1 192.168.111.111:2012@12012 master - 0 1615276889104 2 connected 8192-10922
cf118efd950061ae013c52539f66f7ff1779100f 192.168.111.112:2012@12012 master - 0 1615276889000 6 connected 10923-13652
b9de42bb24a48cc6a8b6fc72f2c38f43057bf459 192.168.111.113:2014@12014 slave cf118efd950061ae013c52539f66f7ff1779100f 0 1615276888000 12 connected
79187b202bdb436e26d5a7b16e1e91bff0584dc1 192.168.111.111:2014@12014 slave 8bcaa25b6d0c61fedf1b181971f5eb7f4da7380d 0 1615276890105 9 connected
7b5f21f124373b7d99edafaab01b90afdcfe8592 192.168.111.111:2011@12011 myself,master - 0 1615276887000 1 connected 0-2730
12874ba078047ad747e6123f68c32ddd94c752bc 192.168.111.112:2013@12013 slave 7b5f21f124373b7d99edafaab01b90afdcfe8592 0 1615276887101 7 connected
722ee8a5d3508df3eb6afeff60550de9b5b524be 192.168.111.113:2013@12013 slave 81757eeb491059021c8e2468f361245764800c05 0 1615276886000 11 connected
8bcaa25b6d0c61fedf1b181971f5eb7f4da7380d 192.168.111.113:2011@12011 master - 0 1615276885000 9 connected 5461-8191
7a5c454dfecc3ed53c6bd8cc29a01c3114ee0067 192.168.111.111:2013@12013 slave 15595bdf3cccc20eb25faf7993608b20967a8673 0 1615276889000 10 connected
2c24ba35f85e185b98c66ead9f3ce1a444839fb8 192.168.111.112:2014@12014 slave 1e94fb4e201b9e7c4eccc207a6f8bae883d1dff1 0 1615276888000 8 connected
15595bdf3cccc20eb25faf7993608b20967a8673 192.168.111.113:2012@12012 master - 0 1615276884000 10 connected 13653-16383

4.附录

4.1. Redis基本参数参考

4.2. Redis复制参数参考

4.3. Redis Cluster参数参考