javascript - Why is AWS SQS so slow? - Stack Overflow

admin管理员组

文章数量:1325000

I've been testing the amount of time it takes to send a message and receive it from an SQS queue. It takes an average of 800-1200 ms, which seems like a ridiculously long time. Here is my code for testing, please tell me if I'm doing something wrong.

var t0;
sendMessage('hello');

function sendMessage(message){

    var params = {
        MessageBody: message,
        QueueUrl: queueUrl
    };
    t0 = now();
    sqs.sendMessage(params, function(err,data){
        if(err){
            throw err;
        } else {
            console.log("Message Send Confirmation");
        }
    });
    unbatch();
}

async function unbatch(){

    var params = {
        QueueUrl: queueUrl,
        MaxNumberOfMessages: 10
    };
    var go = true;
    while(go){
        console.log("Polling...");
        sqs.receiveMessage(params, function(err, data){
            if(data.Messages){
                console.log("Message Received");
                console.log("Total Time: " + ((now() - t0)/1000));
                go = false;
                var deleteParams = {
                    QueueUrl: queueUrl,
                    ReceiptHandle: data.Messages[0].ReceiptHandle
                };
                sqs.deleteMessage(deleteParams, function(err, data) {
                    if (err) {
                        console.log("Delete Error", err);
                    } else {
                        console.log("Message Deleted");
                    }
                });
            }
        });
        await sleep(1);
    }
}

function sleep(ms){
    return new Promise(resolve => setTimeout(resolve, ms));
}

It sends the message and immediately begins trying to recieve a message every millisecond. Once received, it calculates the time. Shouldn't this be taking significantly less time?

I've been testing the amount of time it takes to send a message and receive it from an SQS queue. It takes an average of 800-1200 ms, which seems like a ridiculously long time. Here is my code for testing, please tell me if I'm doing something wrong.

var t0;
sendMessage('hello');

function sendMessage(message){

    var params = {
        MessageBody: message,
        QueueUrl: queueUrl
    };
    t0 = now();
    sqs.sendMessage(params, function(err,data){
        if(err){
            throw err;
        } else {
            console.log("Message Send Confirmation");
        }
    });
    unbatch();
}

async function unbatch(){

    var params = {
        QueueUrl: queueUrl,
        MaxNumberOfMessages: 10
    };
    var go = true;
    while(go){
        console.log("Polling...");
        sqs.receiveMessage(params, function(err, data){
            if(data.Messages){
                console.log("Message Received");
                console.log("Total Time: " + ((now() - t0)/1000));
                go = false;
                var deleteParams = {
                    QueueUrl: queueUrl,
                    ReceiptHandle: data.Messages[0].ReceiptHandle
                };
                sqs.deleteMessage(deleteParams, function(err, data) {
                    if (err) {
                        console.log("Delete Error", err);
                    } else {
                        console.log("Message Deleted");
                    }
                });
            }
        });
        await sleep(1);
    }
}

function sleep(ms){
    return new Promise(resolve => setTimeout(resolve, ms));
}

It sends the message and immediately begins trying to recieve a message every millisecond. Once received, it calculates the time. Shouldn't this be taking significantly less time?

• javascript
• node.js
• amazon-web-services
• amazon-sqs

Share Improve this question Follow asked Oct 19, 2017 at 23:23 MattMatt 30511 gold badge55 silver badges1212 bronze badges 4

• I wrote a "queue drainer" tool that reads from SQS and stored the in a MySQL database... It has some cleverness built-in, like deleting the previous batch asynchronously while fetching the next batch, but it still has some optimization that could be done, and I forget the specific numbers... but I'm almost certain it can consume close to 200 messages per second... so your times do seem off, a bit and of course with 10 messages in queue vs. just 1, you'd have a significant trx/sec improvement. Where are you running this code? In EC2 in the same region as the queue, or somewhere else? – Michael - sqlbot Commented Oct 20, 2017 at 3:13
• I'm not really concerned with how many messages per second it can do. I'm more concerned with the round trip time of a single message. Also, I was running this from my local system. – Matt Commented Oct 20, 2017 at 19:34
• 2 Running from your local system will introduce significant latency, because you're speaking HTTP over TLS to the service, so multiple round trips of overhead are there. The point of mentioning messages per second is not the speed of the service -- it's much faster than that -- the point is that if I am processing in excess of 100 messages per second, then I am necessarily processing each message in under 10ms. I only have one thread running. If I set max number of messages to 1, I still blaze through about 20 messages/sec, or ~50ms. Time to do just 1 from a cold start reveals little. – Michael - sqlbot Commented Oct 20, 2017 at 20:25
• 2 It looks like you're dealing with only a single message at a time, and including all latency to send/receive message from your local puter (instead of an EC2 instance in the same region). SQS is not optimized for this use case (single message) - it's optimized for many message throughput & resiliency (at least once delivery). – Krease Commented Nov 1, 2017 at 14:24

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 5

The reason you would use any queue is not for performance but rather for resilience.

Queues solve many problems, they provide async munications between disconnected systems, they allow you to scale systems really well, and provide enhanced resilience ensuring that messages do not get "lost" in the event of a system failure.

When working with queues you should rather consider designing your system around the concept of eventual consistency which means your message will eventually get there and processed but maybe not when or even in the order you expect.

Capabilities such speed, ordering, retrying etc. will vary between queue implementations (SQS, Kafka, RabbitMq etc.)

If you are looking for super high IOPS then perhaps queues aren't what you want.

本文标签： javascriptWhy is AWS SQS so slowStack Overflow