Redis内存碎片清理

数据库干货铺 2020-01-11

1727

当Redis中清理了大量的Key之后原先Redis申请的内存（used_memory_rss）将继续持有而不会释放，此时查看内存信息将会看到存在大量的内存碎片。那么，Redis的内存碎片可以清理么，该如何清理呢？

翻看了Redis的相关资料发现，Redis4版本之后开始支持内存碎片的清理，于是进行了一次测试，内容如下：

1. 搭建Redis

搭建一个Redis，版本为4.0.14.搭建步骤参考历史博文或微信公众号，步骤相对简单，没有太多幺蛾子，很快便可以搭建成功。

2. 插入一堆Key,使其内存占用很大

可以批量写一个循环，插入大量key。

3. 删除90%以上的key

循环删除key或在创建key时设置过期时间，待key删除或过期之后，可以查看内存的情况。

127.0.0.1:6379> info memory 
# Memory
used_memory:137040696
used_memory_human:130.69M
used_memory_rss:11705876480
used_memory_rss_human:10.90G
used_memory_peak:12091169848
used_memory_peak_human:11.26G
used_memory_peak_perc:1.13%
used_memory_overhead:3473184
used_memory_startup:786648
used_memory_dataset:133567512
used_memory_dataset_perc:98.03%
total_system_memory:16862617600
total_system_memory_human:15.70G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:12000000000
maxmemory_human:11.18G
maxmemory_policy:noeviction
mem_fragmentation_ratio:85.42
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

可以发现实际使用内存为130.69M，而Redis申请的内存为10.90G，碎片率mem_fragmentation_ratio为85.42，相当高了

4. 清理内存碎片

默认情况下自动清理碎片的参数是关闭的，可以按如下命令查看

127.0.0.1:6379> config get activedefrag 
1) "activedefrag"
2) "no"

启动自动清理内存碎片

127.0.0.1:6379> config set  activedefrag yes
OK

开启后再查看内存信息

127.0.0.1:6379> info memory
# Memory
used_memory:138029408
used_memory_human:131.64M
used_memory_rss:5052907520
used_memory_rss_human:4.71G
used_memory_peak:12091169848
used_memory_peak_human:11.26G
used_memory_peak_perc:1.14%
used_memory_overhead:3752728
used_memory_startup:786648
used_memory_dataset:134276680
used_memory_dataset_perc:97.84%
total_system_memory:16862617600
total_system_memory_human:15.70G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:12000000000
maxmemory_human:11.18G
maxmemory_policy:noeviction
mem_fragmentation_ratio:36.61
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

此时redis占用的内存used_memory_rss已降低至4.71G了，内存碎片为36.61

5. 查看内存分配情况

此时也可以查看内存分配情况,其中重要的指标是查看bins里的util,此时可以发现当前最大的已达到0.998（1除外）

127.0.0.1:6379> memory malloc-stats
___ Begin jemalloc statistics ___
Version: 4.0.3-0-ge9192eacf8935e29fc62fddc2701f7942b1cc02c
Assertions disabled
Run-time option settings:
  opt.abort: false
  opt.lg_chunk: 21
  opt.dss: "secondary"
  opt.narenas: 8
  opt.lg_dirty_mult: 3 (arenas.lg_dirty_mult: 3)
  opt.stats_print: false
  opt.junk: "false"
  opt.quarantine: 0
  opt.redzone: false
  opt.zero: false
  opt.tcache: true
  opt.lg_tcache_max: 15
CPUs: 2
Arenas: 8
Pointer size: 8
Quantum size: 8
Page size: 4096
Min active:dirty page ratio per arena: 8:1
Maximum thread-cached size class: 32768
Chunk size: 2097152 (2^21)
Allocated: 138983464, active: 149237760, metadata: 133846144, resident: 299532288, mapped: 5274861568
Current active ceiling: 153092096


arenas[0]:
assigned threads: 1
dss allocation precedence: secondary
min active:dirty page ratio: 8:1
dirty pages: 36435:4043 active:dirty, 348100 sweeps, 737670 madvises, 4686933 purged
                            allocated      nmalloc      ndalloc    nrequests
small:                       51173416     27946257     27800354    215995984
large:                       20701184      6397067      6396713      7682104
huge:                        67108864          754          753          754
total:                      138983464     34344078     34197820    223678842
active:                     149237760
mapped:                    5270667264
metadata: mapped: 130830336, allocated: 115776
bins:           size ind    allocated      nmalloc      ndalloc    nrequests      curregs      curruns regs pgs  util       nfills     nflushes      newruns       reruns
                   8   0         2952         4753         4384     18226389          369            1  512   1 0.720         2600         2566            1            0
                  16   1       699600      5702102      5658377     34777916        43725          392  256   1 0.435        61712        64807        22136          251
                  24   2       824592      9648532      9614174     75492339        34358          441  512   3 0.152       102398       100732        16561      1083971
                  32   3          800         1828         1803     11077167           25            1  128   1 0.195         1031          883            1            0
                  40   4          400          893          883      1285046           10            1  512   5 0.019          787          791            1            0
                  48   5         1824        10123        10085     23813168           38            1  256   3 0.148         6390         7039           18            9
                  56   6        11200       665763       665563      3654283          200           60  512   7 0.006        11325        18434         1302          135
                  64   7        44480      2221901      2221206      8140167          695           62   64   1 0.175        38634        43809        34702         1105
                  80   8      2243120      3241050      3213011     12640545        28039          154  256   5 0.711        36308        40673        12556          268
                  96   9      1248384      1173523      1160519      5632612        13004          116  128   3 0.875        17234        22442         9049          828
                 112  10       257152       497094       494798       378079         2296            9  256   7 0.996        10382        20700         1942          529
                 128  11       173056       241148       239796      1500956         1352           43   32   1 0.982        11994        19268         7491         1149
                 160  12       510880       676589       673396      1867045         3193           25  128   5 0.997        12036        20344         5279          974
                 192  13        86784       163154       162702       104853          452            8   64   3 0.882         8095        19909         2538          885
                 224  14      1050784       937414       932723       882636         4691           37  128   7 0.990        14534        21327         7237          653
                 256  15       177920       122377       121682       108334          695           44   16   1 0.987        11216        18094         7648         1025
                 320  16       158720       160687       160191      1395080          496           10   64   5 0.775         8282        19932         2458          806
                 384  17       108288        87166        86884        56781          282           10   32   3 0.881         7364        18594         2691          830
                 448  18       520576       311865       310703       235276         1162           19   64   7 0.955         9885        20257         4849         1277
                 512  19       366080       139077       138362       136708          715           91    8   1 0.982        16677        21203        17307          758
                 640  20       296320       136674       136211      1387569          463           15   32   5 0.964         8781        19327         4294         1194
                 768  21       754176       164055       163073       154637          982           62   16   3 0.989        13751        19514        10232          853
                 896  22       507136       142840       142274       104081          566           19   32   7 0.930         8831        19297         4488         1232
                1024  23       460800        81240        80790        71715          450          113    4   1 0.995        11411        17538        20204         1932
                1280  24       967680       142375       141619      3997592          756           48   16   5 0.984        12880        19162         8852          974
                1536  25       806400        83240        82715      1363965          525           66    8   3 0.994        11744        17875        10296         1083
                1792  26       507136        70931        70648        50310          283           18   16   7 0.982         8140        17915         4403         1138
                2048  27       540672        52915        52651        40011          264          132    2   1 1             8604        16582        26425         1888
                2560  28      1487360       115947       115366      2688774          581           73    8   5 0.994        15012        19969        14410          962
                3072  29      1022976        70646        70313        58822          333           84    4   3 0.991        10445        17189        17546         2050
                3584  30      1634304        76807        76351        66030          456           57    8   7 1            11190        17542         9538         1175
                4096  31       487424        56959        56840        43668          119          119    1   1 1             9176        16863        56959            0
                5120  32      6154240       145461       144259      2675097         1202          301    4   5 0.998        17704        22138        36076         1483
                6144  33      1198080        45380        45185        83315          195           98    2   3 0.994         8050        16148        22721         1759
                7168  34       824320        32518        32403      1310845          115           29    4   7 0.991         6598        14416         8063         1715
                8192  35     18800640       355210       352915       362972         2295         2295    1   2 1            39103        42128       355210            0
               10240  36      2232320        76168        75950        66295          218          109    2   5 1            11960        17361        37917         1769
               12288  37      2039808        52860        52694        40296          166          166    1   3 1             8941        16632        52860            0
               14336  38      1964032        36992        36855        24610          137           69    2   7 0.992         6879        15285        18422         1777
large:          size ind    allocated      nmalloc      ndalloc    nrequests      curruns
               16384  39       688128         7548         7506         7760           42
               20480  40       880640         9391         9348      1293561           43
               24576  41       688128         6279         6251         6367           28
               28672  42       602112         4187         4166         4261           21
               32768  43      2818048        17937        17851        18430           86
               40960  44      1433600      6306450      6306415      6306450           35
               49152  45      1277952         5117         5091         5117           26
               57344  46       860160         5916         5901         5916           15
               65536  47       393216         2953         2947         2953            6
               81920  48      1310720         8244         8228         8244           16
               98304  49       294912         5037         5034         5037            3
              114688  50       573440         4224         4219         4224            5
              131072  51       393216         2233         2230         2233            3
              163840  52      1638400         2798         2788         2798           10
              196608  53      1179648         1740         1734         1740            6
              229376  54       229376         1162         1161         1162            1
              262144  55       524288          872          870          872            2
              327680  56       327680         1290         1289         1290            1
              393216  57            0          782          782          782            0
              458752  58            0          520          520          520            0
              524288  59       524288          400          399          400            1
              655360  60       655360          606          605          606            1
              786432  61            0          371          371          371            0
              917504  62            0          196          196          196            0
             1048576  63      2097152          224          222          224            2
             1310720  64      1310720          328          327          328            1
             1572864  65            0          179          179          179            0
             1835008  66            0           83           83           83            0
huge:           size ind    allocated      nmalloc      ndalloc    nrequests   curhchunks
             2097152  67            0          110          110          110            0
             2621440  68            0          263          263          263            0
             3145728  69            0           86           86           86            0
             3670016  70            0           91           91           91            0
             4194304  71            0           45           45           45            0
             5242880  72            0           55           55           55            0
             6291456  73            0           26           26           26            0
             7340032  74            0           15           15           15            0
             8388608  75            0           13           13           13            0
            10485760  76            0            8            8            8            0
            12582912  77            0           13           13           13            0
            14680064  78            0            2            2            2            0
            16777216  79            0            4            4            4            0
            20971520  80            0            2            2            2            0
            25165824  81            0            2            2            2            0
            29360128  82            0            2            2            2            0
            33554432  83            0            4            4            4            0
            41943040  84            0            2            2            2            0
            50331648  85            0            2            2            2            0
            58720256  86            0            2            2            2            0
            67108864  87     67108864            7            6            7            1
                     ---
--- End jemalloc statistics ---

6. 手动清理

其实还可以手动清理，可以采用如下命令

127.0.0.1:6379> memory purge
OK

7. 相关参数配置说明

内存清理相关参数如下，可以使用config get的方式查看对应的值

# Enabled active defragmentation
# 碎片整理总开关
# activedefrag yes


# Minimum amount of fragmentation waste to start active defrag
# 内存碎片达到多少的时候开启整理
active-defrag-ignore-bytes 100mb


# Minimum percentage of fragmentation to start active defrag
# 碎片率达到百分之多少开启整理
active-defrag-threshold-lower 10


# Maximum percentage of fragmentation at which we use maximum effort
# 碎片率小余多少百分比开启整理
active-defrag-threshold-upper 100


# Minimal effort for defrag in CPU percentage
active-defrag-cycle-min 25


# Maximal effort for defrag in CPU percentage
active-defrag-cycle-max 75