Does a kafka broker keep persistent tcp connections with all of the consumers in all the groups? - Stack Overflow

admin管理员组

文章数量:1416625

So Kafka client's for consuming messages uses poll() which along with the heartbeat is what makes the broker decide if a client is still alive.
What is not clear to me is if the poll() and heartbeat are sent over persistent network connections or each poll() or heartbeat are creating a new tcp connection each time.
I would assume the connections would be persistent but would that mean that for a broker with several thousands of consumers there would be several thousands of open tcp connections (most of them potentially idle if the consumer is doing some event processing)?

So Kafka client's for consuming messages uses poll() which along with the heartbeat is what makes the broker decide if a client is still alive.
What is not clear to me is if the poll() and heartbeat are sent over persistent network connections or each poll() or heartbeat are creating a new tcp connection each time.
I would assume the connections would be persistent but would that mean that for a broker with several thousands of consumers there would be several thousands of open tcp connections (most of them potentially idle if the consumer is doing some event processing)?

• apache-kafka
• kafka-consumer-api
• kafka-partition
• rebalancing

Share Improve this question Follow asked Feb 2 at 20:58 JimJim 4,44144 gold badges3030 silver badges6060 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

Consumers keep persistent connections to all brokers they are fetching data and keep a separate connection to their consumer group coordinator, even if it's also a broker they are already fetching data from. In the node metrics the connection to the coordinator has the node attribute set to -1.

In most cases establishing a new connection is expensive, so Kafka clients try to reuse existing connections and keep connections they are actively using.

Connections that are idle are automatically closed after connections.max.idle.ms which is 9 minutes by default.

Consumers use the coordination connection to send their heartbeats. While heartbeats are small, they happen every heartbeat.interval.ms which is 3 seconds by default.

So yes, if you have thousands of consumers (or other Kafka clients) connected to your cluster, each broker can have thousands of network connections.

It's often recommended to monitor the number of connections (and the creation, closure rate) of your clusters. Both Kafka clients and brokers expose these metrics. See https://kafka.apache./documentation/#selector_monitoring

本文标签：