kafka的__consumer_offsets日志如何清理

对__consumer_offsets进行数据迁移；迁移速度相当的忙；消费者出现数小时无法消费现象

经排查发现__consumer_offsets的日志高达几百G

server.properties 配置文件中cleanup 为delete ,日志保存时间为1天

log.cleaner.enable=false

log.cleanup.policy=delete

log.retention.hours=24

但__consumer_offsets 日志没有进行自动删除

各种google后发现__consumer_offsets的确与普通topic在清理策略上不同

运行如下命名

./kafka-configs.sh --zookeeper 127.0.0.1:2181 --entity-type topics --entity-name __consumer_offsets --describe

Configs for topic ‘__consumer_offsets’ are segment.bytes=104857600,cleanup.policy=compact,compression.type=producer

发现__consumer_offsets使用的是compact压缩策略；

又因为server.properties 配置文件中将log.cleaner.enable设置为false导致compact也未起到效果

解决办法

1.将server.properties 中的log.cleaner.enable 设置为true

2.将__consumer_offsets的cleanup.policy多添加一个delete策略

./kafka-configs.sh --bootstrap-server 127.0.0.1:9092 --entity-type topics --entity-name __consumer_offsets --alter --add-config cleanup.policy=[delete,compact]

kafka __consumer_offsets日志清理 - 知乎 (zhihu.com)

┌──(root💀mymaster2)-[/opt/kafka/bin]
└─# ./kafka-configs.sh --bootstrap-server 127.0.0.1:9092 --entity-type topics --entity-name __consumer_offsets --describe
Dynamic configs for topic __consumer_offsets are:
compression.type=producer sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=producer, DEFAULT_CONFIG:compression.type=producer}
cleanup.policy=compact sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:cleanup.policy=compact, STATIC_BROKER_CONFIG:log.cleanup.policy=delete, DEFAULT_CONFIG:log.cleanup.policy=delete}
segment.bytes=104857600 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:segment.bytes=104857600, STATIC_BROKER_CONFIG:log.segment.bytes=209715200, DEFAULT_CONFIG:log.segment.bytes=1073741824}

┌──(root💀mymaster2)-[/opt/kafka/bin]
└─# ./kafka-configs.sh --bootstrap-server 127.0.0.1:9092 --entity-type topics --entity-name __consumer_offsets --alter --add-config cleanup.policy=[delete,compact]
Completed updating config for topic __consumer_offsets.

┌──(root💀mymaster2)-[/opt/kafka/bin]
└─# ./kafka-configs.sh --bootstrap-server 127.0.0.1:9092 --entity-type topics --entity-name __consumer_offsets --describe
Dynamic configs for topic __consumer_offsets are:
compression.type=producer sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:compression.type=producer, DEFAULT_CONFIG:compression.type=producer}
cleanup.policy=delete,compact sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:cleanup.policy=delete,compact, STATIC_BROKER_CONFIG:log.cleanup.policy=delete, DEFAULT_CONFIG:log.cleanup.policy=delete}
segment.bytes=104857600 sensitive=false synonyms={DYNAMIC_TOPIC_CONFIG:segment.bytes=104857600, STATIC_BROKER_CONFIG:log.segment.bytes=209715200, DEFAULT_CONFIG:log.segment.bytes=1073741824}