Kafka mirrormaker2 did not sync all messages - Stack Overflow

admin管理员组

文章数量:1302958

We using kafka 3.9 in kraft mode and mirrormaker2 to sync main cluster with backup cluster in Active/Passive mode. The problem is, when we turn on sync(mm2) on backup cluster, mm2 sync only a 1/3 of messages and then stop syncing old messages and sync only new one. It feels like mm2 skipping some data for some reason. For example, we got topic with size of 30GB(~20 million messages), but only 12GB(~1.5 million) synced. We think that problem can be with auto.offset.reset and we set it to auto.offset.reset=earliest but no luck. Here is our full config of mm2:

clusters = MAIN, RES

MAIN.bootstrap.servers = kaf2-p01a:9092,kaf2-p02a:9092,kaf2-p03a:9092

RES.bootstrap.servers = kaf2-p01b:9092,kaf2-p02b:9092,kaf2-p03b:9092

MAIN->RES.enabled = true

topics = .*
groups = .*

replication.factor=3

checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1
offset-syncs.topic.replication.factor=1

replication.policy.class=.apache.kafka.connect.mirror.IdentityReplicationPolicy

emit.checkpoints.enabled=true
refresh.groups.interval.seconds=60
sync.group.offsets.enabled=true
sync.group.offsets.interval.seconds=60
emit.checkpoints.interval.seconds=60

tasks.max=6

RES.producer.max.request.size=15000000
RES.producerpression.type=gzip

auto.offset.reset=earliest
consumer.auto.offset.reset=earliest
RES.consumer.auto.offset.reset=earliest
MAIN.consumer.auto.offset.reset=earliest

We try auto.offset.reset=earliest and expect to sync all data instead of half.

本文标签： Kafka mirrormaker2 did not sync all messagesStack Overflow