背景

Redis的默认 maxclients 的大小是 10000
之前有同事问过, 应用服务器比较多的情况下 这个值会比较高. 
当时设置到了 15000 
最近突然想, redis 的最大值能够大多少. 
昨天验证了 nginx的一些情况, 今天想继续验证redis的情况

结论

Redis 可以超过10万个以上的clients
他并不会收到 64k个端口号的限制.
因为跟nginx一样, 限制是基于TCP四元组来的. 

但是clients 多了之后, 压测试CPU的性能消耗是非常高的. 

当使用 8万个client进行压测时.
发现 rps最高合计到了 12万. 
单独使用 一个机器的2万个client进行压测时
发现 rps 可以到 7万多. 
但是同时 8万个 clients 时 CPU 长时间 100% 平均响应时间 300ms
如果是   2万个 clients 时 CPU 大概是在 60% 平均响应时间:70ms 

参数配置

maxcliets 120000
requirepass Somepassword
daemonize yes 

多个机器进行验证

将 redis-benchmark 放到 测试机器的 usr/bin 目录下
然后执行命令: 
redis-benchmark -h 172.24.110.186 -p 6379 -a MyPassword -n 1000000 -c 30000 -d 256  -q

然后查看redsi的client信息

[root@ALLDB redis]# redis-cli -a MyPassword info clients
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Clients
connected_clients:115749
cluster_connections:0
maxclients:120000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

也就是说可以超过 65536 的限制. 
但是性能可能会下降比较厉害 

默认参数设置

sysctl -w net.ipv4.ip_local_port_range="15000 64000"
sysctl -w net.ipv4.tcp_max_tw_buckets=5000
sysctl -w net.ipv4.tcp_tw_reuse=1

降低极限进行测试

四台机器同事进行测试

redis-benchmark -h 172.24.110.186 -p 6379 -a MyPassword -n 1000000000 -c 20000 -d 256  -q

CPU使用情况

top - 09:25:44 up 1 day, 16:26,  1 user,  load average: 0.23, 0.15, 0.10
Threads:   6 total,   1 running,   5 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.6 us,  9.6 sy,  0.0 ni, 72.0 id,  0.0 wa,  0.0 hi, 16.8 si,  0.0 st
KiB Mem : 64757916 total,  1879036 free, 25873224 used, 37005656 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 30984120 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
25572 root      20   0 1671136 445244   2224 R 99.9  0.7   1:20.88 redis-server
25573 root      20   0 1671136 445244   2224 S  0.0  0.7   0:00.00 bio_close_file
25574 root      20   0 1671136 445244   2224 S  0.0  0.7   0:00.00 bio_aof
25575 root      20   0 1671136 445244   2224 S  0.0  0.7   0:00.00 bio_lazy_free
25576 root      20   0 1671136 445244   2224 S  0.0  0.7   0:00.09 jemalloc_bg_thd
25577 root      20   0 1671136 445244   2224 S  0.0  0.7   0:00.00 jemalloc_bg_thd

网络连接情况:
netstat -ano |grep 6379 |awk '{print $6}' |sort |uniq -c
  78601 ESTABLISHED
      2 LISTEN

四台服务器 8万个client的压测数据

PING_INLINE: rps=28660.0 (overall: 32883.0) avg_msec=308.951 (overall: 307.907)
PING_INLINE: rps=40567.5 (overall: 32775.8) avg_msec=387.327 (overall: 312.077)
PING_INLINE: rps=52361.5 (overall: 32663.5) avg_msec=177.727 (overall: 312.621)
PING_INLINE: rps=73640.3 (overall: 32782.0) avg_msec=69.453 (overall: 308.377)

如果设置到一台机器上面只有 20000个clients 时
PING_INLINE: rps=72059.9 (overall: 72334.7) avg_msec=70.371 (overall: 70.676)

一个压力机器时的情况:

[root@ALLDB redis]# top -Hp 25572 -bn 1
top - 09:43:03 up 1 day, 16:43,  1 user,  load average: 0.22, 0.66, 0.50
Threads:   6 total,   1 running,   5 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.5 us,  4.2 sy,  0.0 ni, 86.6 id,  0.0 wa,  0.0 hi,  6.7 si,  0.0 st
KiB Mem : 64757916 total,  2043384 free, 25663996 used, 37050536 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 31193388 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
25572 root      20   0 1943520 490168   2396 R 53.3  0.8  12:24.22 redis-server
25573 root      20   0 1943520 490168   2396 S  0.0  0.8   0:00.00 bio_close_file
25574 root      20   0 1943520 490168   2396 S  0.0  0.8   0:00.00 bio_aof
25575 root      20   0 1943520 490168   2396 S  0.0  0.8   0:00.00 bio_lazy_free
25576 root      20   0 1943520 490168   2396 S  0.0  0.8   0:00.39 jemalloc_bg_thd
25577 root      20   0 1943520 490168   2396 S  0.0  0.8   0:00.00 jemalloc_bg_thd

Redis的整体监控情况

# Server
redis_version:7.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:14927caef46752a5
redis_mode:standalone
os:Linux 3.10.0-1160.el7.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:25572
process_supervised:no
run_id:215cc7957b47242f0590b1600f862f275391696e
tcp_port:6379
server_time_usec:1718501727403450
uptime_in_seconds:1276
uptime_in_days:0
hz:500
configured_hz:10
lru_clock:7225695
executable:/redis/./redis-server
config_file:/redis/redis.conf
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6379

如上是基本信息. 

# Clients
connected_clients:78602
cluster_connections:0
maxclients:120000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

这些事 client的基本信息

# Memory
used_memory:1670916880
used_memory_human:1.56G
used_memory_rss:776372224
used_memory_rss_human:740.41M
used_memory_peak:1692626856
used_memory_peak_human:1.58G
used_memory_peak_perc:98.72%
used_memory_overhead:900971680
used_memory_startup:7141008
used_memory_dataset:769945200
used_memory_dataset_perc:46.28%
allocator_allocated:1673267288
allocator_active:1673748480
allocator_resident:1703129088
total_system_memory:66312105984
total_system_memory_human:61.76G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:481192
allocator_rss_ratio:1.02
allocator_rss_bytes:29380608
rss_overhead_ratio:0.46
rss_overhead_bytes:-926756864
mem_fragmentation_ratio:0.46
mem_fragmentation_bytes:-896829440
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:893830192
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

默认没有驱逐策略, 其实是需要修改一下的. 

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1718501626
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:-1
rdb_saves:9
rdb_last_cow_size:84361216
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:5
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1094144
total_commands_processed:43051884
instantaneous_ops_per_sec:96746
total_net_input_bytes:2287393296
total_net_output_bytes:815541572
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:592.46
instantaneous_output_kbps:651.64
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:34
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:2010123
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:10069
total_forks:9
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:43654130
total_writes_processed:42597790
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:997287
reply_buffer_expands:0
eventloop_cycles:6032609
eventloop_duration_sum:337303802
eventloop_duration_cmd_sum:10437452
instantaneous_eventloop_cycles_per_sec:10870
instantaneous_eventloop_duration_usec:124601
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:4a04cf5a7396791ae7361b7fdec40bc906b24d71
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:287.111681
used_cpu_user:87.725018
used_cpu_sys_children:0.715498
used_cpu_user_children:2.248074
used_cpu_sys_main_thread:286.880444
used_cpu_user_main_thread:87.613899

# Modules

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace