Mongoose: findOneAndUpdate ~500k docs

mongoose bulkwrite
mongoose index

As I want to update a whole collection of documents (some may exists, some not) with 500k entries, I loop over my docs (which are plain JS objects that come from a CSV) and update all my docs via findOneAndUpdate method.

Here is a simplified JS code:

const mongoose = require("mongoose");
const { mongoUrl } = require("../config.js");
const MyModel = require("../models/MyModel");

async function run() {
  mongoose.set("useCreateIndex", true);
  await mongoose.connect(mongoUrl, { useNewUrlParser: true });

  const r = [/* */]; // <-- Something with 500k objects

  const total = r.length;
  for (let i in r) {
    const e = r[i];
    e.REF = e.REF.trim();
    await MyModel.findOneAndUpdate({ REF: e.REF }, e, { upsert: true, new: true });
    if (i % 500 === 0) {
      console.log((i / total) * 100 + "%");
    }
  }
}

run();

First insertions are fast (more than 500 each second), but when it comes to 25%, it start to be really slow (10 seconds or more for 500 entries). I think it would be slower (frozen?) near 80%.

Is there a better approach to do such a thing? How can I speed up things?

I mean, it's only 500k entires (with a lot of properties, but I don't think it matters).


Posting comment as answer:

Do you have an index on REF? Otherwise the database will have to scan through the entire collection to figure out whether there is a model with that same REF.

How to Use `findOneAndUpdate()` in Mongoose, By default, findOneAndUpdate() returns the document as it was before update was applied. const Character = mongoose.model('Character', new mongoose. collection.findOneAndUpdate ¶ Update a single document in a collection or view based on a query filter and return the document in either its pre-update or post-update form. Unlike collection.updateOne() , this action allows you to atomically find, update, and return a document with the same command.


there are 2 ways you can do this more effective that i can think of.

  1. make sure you have index on REF .

  2. create batch updates instead of waiting 1 by 1 bulkWrite

Mongoose v5.9.27: API docs, Set to false to make findOneAndUpdate() and findOneAndRemove() use native findOneAndUpdate() rather than findAndModify() . [options.reconnectTries=30]� To use db.collection.findOneAndUpdate() to update the shard key: You must run on a mongos either in a transaction or as a retryable write. Do not issue the operation directly on the shard. You must include an equality condition on the full shard key in the query filter.


I used to do these kind of operations for database model migrations, and I found out that .bulkWrite() really makes a difference here. What I would do is the following:

const mongoose = require("mongoose");
const { mongoUrl } = require("../config.js");
const MyModel = require("../models/MyModel");

async function run() {
  mongoose.set("useCreateIndex", true);
  await mongoose.connect(mongoUrl, { useNewUrlParser: true });

  const r = [/* */]; // <-- Something with 500k objects

  const total = r.length;
  // Set empty array of bulk write operations
  let bulkWriteOps = [];

  for (let i in r) {
    const e = r[i];
    e.REF = e.REF.trim();
    /* As every update operation is different because it depends on the iteration 
    variable, you have to push each element individually */
    bulkWriteOps.push({
        updateOne: {
            'filter': { REF: e.REF },
            'update': e,
            'upsert': true
            // As you are not using the result of the update, you don't need the new: true flag
        }
    })

    if (i % 500 === 0) {
      console.log((i / total) * 100 + "%");
    }
  }
}

if (bulkWriteOps.length) await MyModel.bulkWrite(bulkWriteOps)

run();

In your case, in every iteration you are awaiting for the document to be queried and updated. Bulk write was designed to avoid these kind of situations and let the DB do the hard work behind the scenes.

If you want to read more in depth about the topic, take a look at MongoDB docs, here it tells you about the batch size and order of operations.

Hope this somehow helps you with your issue

javascript, The above works just fine, however when I try and do a findOneAndUpdate, From 100k downloads per week, to 500k downloads per week. It supports translation of short texts and real HTML documents in any of 100+ world languages. DeprecationWarning: Mongoose: findOneAndUpdate() and findOneAndDelete() without the useFindAndModify option set to false are deprecated const mConnect = await mongoose.connect(mongoUri, { useFindAndModify: false, useCreateIndex: true, us


Mongoose findOneAndUpdate can't take an object? : node, Mongoose: findOneAndUpdate ~500k документы и обновляю все мои документы с помощью метода findOneAndUpdate . Если вы хотите более подробно ознакомиться с этой темой, взгляните на документ MongoDB docs , здесь� A Computer Science portal for geeks. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.


Mongoose: findOneAndUpdate ~500k документы, Mongoose's findOneAndUpdate() function finds the first document that matches a given filter , applies an update , and returns the document. await User.findOneAndUpdate(filter, update, { new: true }); // Equivalent await User.findOneAndUpdate(filter, update, { returnOriginal: false }); See Mongoose findOneAndUpdate() docs and this tutorial on updating documents in Mongoose.


findOneAndUpdate() in Mongoose, This page documents the mongo shell method, and does not refer to the MongoDB Node.js driver (or any other driver) method. For corresponding MongoDB driver� Mongoose constructor. The exports object of the mongoose module is an instance of this class. Most apps will only use this one instance. Example: const mongoose = require ('mongoose'); mongoose instanceof mongoose.Mongoose; // true // Create a new Mongoose instance with its own `connect()`, `set()`, `model()`, etc. const m = new mongoose.Mongoose();