admin管理员组文章数量:1122846
I am testing a program to verify if compression helps to reduce the size of the topic message. My sample topic has config 'max.message.bytes=1024000' which is ~1MB and on the producer side config I have set the same value to 'max.request.size' and then I attempted to send across a string that is of size 1573015 which is ~1.5MB and this throws below error which is expected.
org.apache.kafkamon.errors.RecordTooLargeException: The message is 1573015 bytes when
serialized which is larger than 1048576, which is the value of the max.request.size configuration.
Next, since I want the compression responsibility at producer level, I set compression config at producer with the property 'compression.type' as 'zstd' (I have also tried gzip) but producer throws the same error. I expected the compression config to reduce the message size to <1MB on the producer before it is being sent.
I also observed same behaviour when I test 'compression.type' at topic level or producer level or compression.type property set at both topic and producer (I would like to avoid setting this property at broker level thought since I want this to take effect only for a specific topic or producers of that topic).
I wanted to understand whether compression.type actually reduces the message size that is sent across from producer to Kafka broker where broker unpacks it and verifies the size of uncompressed message and throws this error ? or is it that there may be configuration error on producer due to which the compression is not happening in the first place ?
Much appreciated if anyone can shed some light into the inner workings of property max.request.size with regard to compression.type.
Using a standalone program I did verify that the message sample I used for this test can be compressed to <1MB using gzip and zstd. The kafka version that I used for this test is Confluent Kafka Platform 8.0 which is running on a single node cluster locally on Ubuntu WSL.
Compression property reference: .html#compression-type
I am testing a program to verify if compression helps to reduce the size of the topic message. My sample topic has config 'max.message.bytes=1024000' which is ~1MB and on the producer side config I have set the same value to 'max.request.size' and then I attempted to send across a string that is of size 1573015 which is ~1.5MB and this throws below error which is expected.
org.apache.kafka.common.errors.RecordTooLargeException: The message is 1573015 bytes when
serialized which is larger than 1048576, which is the value of the max.request.size configuration.
Next, since I want the compression responsibility at producer level, I set compression config at producer with the property 'compression.type' as 'zstd' (I have also tried gzip) but producer throws the same error. I expected the compression config to reduce the message size to <1MB on the producer before it is being sent.
I also observed same behaviour when I test 'compression.type' at topic level or producer level or compression.type property set at both topic and producer (I would like to avoid setting this property at broker level thought since I want this to take effect only for a specific topic or producers of that topic).
I wanted to understand whether compression.type actually reduces the message size that is sent across from producer to Kafka broker where broker unpacks it and verifies the size of uncompressed message and throws this error ? or is it that there may be configuration error on producer due to which the compression is not happening in the first place ?
Much appreciated if anyone can shed some light into the inner workings of property max.request.size with regard to compression.type.
Using a standalone program I did verify that the message sample I used for this test can be compressed to <1MB using gzip and zstd. The kafka version that I used for this test is Confluent Kafka Platform 8.0 which is running on a single node cluster locally on Ubuntu WSL.
Compression property reference: https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#compression-type
Share Improve this question asked Jan 2 at 10:38 vick_4444vick_4444 3351 gold badge5 silver badges21 bronze badges 9- Is your topic values are Json's? If yes, you should consider using Avro/Protobuf schemas, which reduces message sizes greatly. – Vytautas Šerėnas Commented Jan 2 at 10:45
- "Because the schema is provided at decoding time, metadata such as the field names don’t have to be explicitly encoded in the data. This makes the binary encoding of Avro data very compact.", - docs.confluent.io/platform/current/schema-registry/fundamentals/… – Vytautas Šerėnas Commented Jan 2 at 10:51
- I have incorporated Avro schema with Avro message serialization. – vick_4444 Commented Jan 2 at 10:55
- I would say then that - Compressing already compact Avro binary data may result in diminishing returns. – Vytautas Šerėnas Commented Jan 2 at 11:13
- 1 Thanks for pointing that out. I did another test, this time I set 'message.max.bytes' set at 1MB and set 'max.request.size' property to 2MB with no compression, this led to error and then I did second test where I added compression property config and it worked. I mistook that max.request.size overrides message.max.bytes. Can you please post your last comment as separate post so that I can mark that as answer. Much appreciated for explaining the difference between the two. – vick_4444 Commented Jan 2 at 15:31
1 Answer
Reset to default 0Message is never sent to broker, because your Producer will do a size validation first.
There is completely different configuration for a broker - message.max.bytes "Sets the maximum size for a message that can be accepted by the broker. The default is 1 MB.". In both cases if you go over the limit RecordTooLargeException
is thrown, but yours contains max.request.size which indicates Producer configuration.
Documentation clearly states that producer is responsible for compression, but broker can do recompression, if topics compression.type is different than producer compression.type.
https://www.confluent.io/blog/apache-kafka-message-compression/#configuring-compression-type
本文标签: confluent platformKafka maxrequestsize VS compressiontypeStack Overflow
版权声明:本文标题:confluent platform - Kafka max.request.size VS compression.type - Stack Overflow 内容由网友自发贡献,该文观点仅代表作者本人, 转载请联系作者并注明出处:http://www.betaflare.com/web/1736320625a1936192.html, 本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容,一经查实,本站将立刻删除。
发表评论