javascript - Is Mongoose not scalable with document array editing and version control? - Stack Overflow

admin管理员组

文章数量:1332339

I am developing a web application with Node.js and MongoDB/Mongoose. Our most used Model, Record, has many subdocument arrays. Some of these, for instance, include "Comment", "Bookings", and "Subscribers".

In the client side application, whenever the user hits the "delete" button it fires off an AJAX request to the delete route for that specific ment. The problem I am running into is that, when many of these AJAX calls e in at once, Mongoose fails with a "Document not found" error on some (but not all) of the calls.

This only happens when the calls are made rapidly and many at a time. I think this is due to the version in Mongoose causing document conflicts. Our current process for a delete is:

1. Fetch the document using Record.findById()
2. Remove the subdocument from the appropriate array (using, say, ment.remove())
3. Call record.save()

I have found a solution where I can manually update the collection using Record.findByIdAndUpdate and then using the $pull operator. However, this means we can't use any of mongoose's middleware and loose the version control entirely. And the more I think about it, the more I realize situations where this would happen and I would have to use Mongoose's wrapper functions like findByIdAndUpdate or findAndRemove. The only other solution I can think of would be to put the removal attempt into a while loop and hope it works, which seems like a very poor fix.

Using the Mongoose wrappers doesn't really solve my problem as it won't allow me to use any sort of Middleware or hooks at all then, which is basically one of the huge benefits of using Mongoose.

Does this mean that Mongoose is essentially useless for anything of with rapid editing and I might as well just use native MongoDB drivers? Am I misunderstanding Mongoose's limitations? How could I solve this problem?

I am developing a web application with Node.js and MongoDB/Mongoose. Our most used Model, Record, has many subdocument arrays. Some of these, for instance, include "Comment", "Bookings", and "Subscribers".

In the client side application, whenever the user hits the "delete" button it fires off an AJAX request to the delete route for that specific ment. The problem I am running into is that, when many of these AJAX calls e in at once, Mongoose fails with a "Document not found" error on some (but not all) of the calls.

This only happens when the calls are made rapidly and many at a time. I think this is due to the version in Mongoose causing document conflicts. Our current process for a delete is:

1. Fetch the document using Record.findById()
2. Remove the subdocument from the appropriate array (using, say, ment.remove())
3. Call record.save()

I have found a solution where I can manually update the collection using Record.findByIdAndUpdate and then using the $pull operator. However, this means we can't use any of mongoose's middleware and loose the version control entirely. And the more I think about it, the more I realize situations where this would happen and I would have to use Mongoose's wrapper functions like findByIdAndUpdate or findAndRemove. The only other solution I can think of would be to put the removal attempt into a while loop and hope it works, which seems like a very poor fix.

Using the Mongoose wrappers doesn't really solve my problem as it won't allow me to use any sort of Middleware or hooks at all then, which is basically one of the huge benefits of using Mongoose.

Does this mean that Mongoose is essentially useless for anything of with rapid editing and I might as well just use native MongoDB drivers? Am I misunderstanding Mongoose's limitations? How could I solve this problem?

• javascript
• node.js
• mongodb
• express
• mongoose

Share Improve this question Follow edited Feb 10, 2014 at 6:30 Asya Kamsky 42.4k55 gold badges112112 silver badges133133 bronze badges asked Mar 22, 2013 at 18:53 Chris FosterChris Foster 2,64922 gold badges2323 silver badges3030 bronze badges 0

Add a ment  | 

4 Answers 4

Sorted by: Reset to default 7 +250

Mongoose's versioned document array editing is not scalable for the simple reason that it's not an atomic operation. As a result, the more array edit activity you have, the more likely it is that two edits will collide and you'll suffer the overhead of retry/recovery from that in your code.

For scalable document array manipulation, you have to use update with the atomic array update operators: $pull[All], $push[All], $pop, $addToSet, and $. Of course, you can also use these operators with the atomic findAndModify-based methods of findByIdAndUpdate and findOneAndUpdate if you also need the original or resulting doc.

As you mentioned, the big downside of using update instead of findOne+save is that none of your Mongoose middleware and validation is executed during an update. But I don't see that you have any choice if you want a scalable system. I'd much rather manually duplicate some middleware and validation logic for the update case than have to suffer the scalability penalties of using Mongoose's versioned document array editing. Hey, at least you still get the benefits of Mongoose's schema-based type casting on updates!

I think, from our own experiences, the answer to your question is "yes". Mongoose is not scalable for rapid array-based updates.

Background

We're experiencing the same issue at HabitRPG. After a recent surge in user growth (bringing our DB to 6gb), we started experiencing VersionError for many array-based updates (background on VersionError). ensureIndex({_id:1,__v1:1}) helped a bit, but that tapered as yet more users joined. It would appear to me Mongoose is indeed not scalable for array-based updates. You can see our whole investigation process here.

Solution

If you can afford moving from an array to an object, do that. Eg, ments: Schema.Types.Array => ments: Schema.Types.Mixed, and sort by post.ments.{ID}.date, or even a manual post.ments.{ID}.position if necessary.

If you're stuck with arrays:

1. db.collection.ensureIndex({_id:1,__v:1})
2. Use your methods described above. You won't benefit from hooks and validations, but there are worse things.

I would strongly suggest pulling those arrays out into new collections. For example, a Comments collection where each document has a record ID to indicate where it belongs. This is a much more scalable solution.

You are correct, Mongoose's array operations are not atomic and therefore do not scale well.

I thought of another idea, which I'm not certain about but seems worth offering: soft-delete.

Mongoose is very concerned about array-structure changes because they make future changes ambiguous. But if you were to just tag a ment subdocument with ment.deleted=true then you might be able to do more such operations without encountering conflicts. Then you could have a cron task that goes through and actually removes those ments.

Oh, an additional idea is to use some sort of memory cache, so if an record has been accessed/edited in the last few minutes, it's available without having to pull it from the server, which means that two requests ing in at the same time are going to be modifying the same object.

Note: I'm not actually sure that either of these are good ideas in general or that they'll solve your problem, so go ahead and edit/ment/downvote if they're bad :)

本文标签： javascriptIs Mongoose not scalable with document array editing and version controlStack Overflow