How to guarantee message ordering while maintaining high throughput in RabbitMQ - Stack Overflow

admin管理员组

文章数量:1404619

I have 20000 devices which send the messages that I push to RabbitMQ, on the other side I have 1 or more worker services that process the message and reply to a custom response queue on RabbitMQ. I need to ensure strict message ordering in RabbitMQ while also achieving high throughput.

I've searched the web and these are the options that I came up with:

1. Single queue, single consumer
   Guarantees order but limits throughput since only one consumer can process messages at a time.
2. Single queue, single consumer with ConcurrentMessageLimit > 1
   Risks breaking order due to concurrent processing within the consumer.
3. Queue per device, single consumer per queue
   Could work, but would need to create a lot of queues. Get's complicated if user adds a new device or removes one.
4. RabbitMQ Super Streams with Single Active Consumer
   This option looks promising, however in the end it really looks like option 3 with added throughput benefits. Or am I missing something? As far as I understand I still need to have 20000 streams and handle the cases when device is added or remove to do the cleanup of individual streams.
   For example, if I have 3 queues as in the official example this would still mean that I have 3 active consumers but with concurrency limit = 1. Which means I can achieve high throughput by adding as many streams as necessary to suffice for my workload. Did I get it right? What if workload increases - does this mean I need to be able to add streams dynamically, is this even supported by super streams?

本文标签： How to guarantee message ordering while maintaining high throughput in RabbitMQStack Overflow