Mongoose virtual populate and aggregates

mongoose virtual populate not working
mongoose aggregate
mongoose aggregate populate
mongoose populate match
mongoose populate multiple
mongoose populate subdocument
mongoose find virtual field
deep populate mongoose

I'm trying to do aggregations on Mongoose schemas that use the newer virtual populate functionality (using Mongoose 4.13, Mongo 3.6).

Lets say I have the following (simplified for illustration purposes) schemas:

const ProjectSchema = new mongoose.Schema({
  projectId: Number,
  description: String
});

ProjectSchema.virtual('tasks', {
  ref: 'Task',
  localField: 'projectId',
  foreignField: 'projectId' 
  justOne: false
]);

const TaskSchema = new mongoose.Schema({
  taskId: Number,
  projectId: Number
  hours: Number
});

const Project = mongoose.model('Project', ProjectSchema);
const Task = mongoose.model('Task', TaskSchema);

Querying on Project and populating relating tasks is working fine using .populate() like:

Project.find({projectId: <id>}).populate('tasks');

But now I would like to sum the hours on tasks by project (left the $sum part out below btw..). I'm only getting empty arrays back no matter what. Isn't it possible to aggregate with virtual populate or?

const result = await Project.aggregate([
   { $match : { projectId: <id> } },
   { $lookup: { 
       from: 'tasks',
       localField: 'projectId',
       foreignField: 'projectId',
       as: 'tasks'
     },
     {
        $unwind: '$tasks' 
     }
   }
 ]);

Virtual populate won't work with aggregate, because "The documents returned are plain javascript objects, not mongoose documents (since any shape of document can be returned)."(c), more information in the docs - Mongoose aggregate

Aggregate data in Populate Virtual � Issue #8344 � Automattic , Populate virtuals only allow for a simple query with a few conditions, but not for an aggregate query What is the expected behavior? It would be What are the versions of Node.js, Mongoose and MongoDB you are using? The documents returned are plain javascript objects, not mongoose documents (since any shape of document can be returned). Mongoose does not cast pipeline stages. The below will not work unless _id is a string in the database


Referencing mongoose's documentation for Model.aggregate:

Arguments are not cast to the model's schema because $project operators allow redefining the "shape" of the documents at any stage of the pipeline, which may leave documents in an incompatible format.

Essentially this means the magic of anything mongoose allows via using a schema is not applied when using aggregates. In fact, you would need to refer directly to MongoDB's documentation for aggregation (specifically $lookup).

const ProjectSchema = new mongoose.Schema({
  projectId: Number,
  description: String
});

const TaskSchema = new mongoose.Schema({
  taskId: Number,
  projectId: Number,
  hours: Number
});

const Project = connection.model('Project', ProjectSchema);
const Task = connection.model('Task', TaskSchema);

const project1 = new Project({ projectId: 1, description: 'Foo'});
const project2 = new Project({ projectId: 2, description: 'Foo'});
const task1 = new Task({ task: 1, projectId: 1, hours: 1});
const task2 = new Task({ task: 2, projectId: 1, hours: 2});
const task3 = new Task({ task: 3, projectId: 2, hours: 1});

Promise.all([
  project1.save(),
  project2.save(),
  task1.save(),
  task2.save(),
  task3.save()
]).then(() => {
  return Project.aggregate([
    { $match: { projectId: 1 }},
    {
      $lookup: {
        from: 'tasks',
        localField: 'projectId',
        foreignField: 'projectId',
        as: 'tasks'
      }
    }
  ]).exec();
}).then((result) => {
  console.dir(result, { depth: null, color: true });
});

Outputs the following:

[{
  _id: ObjectID {/*ID*/},
  projectId: 1,
  description: 'Foo',
  __v: 0,
  tasks: [{
      _id: ObjectID {/*ID*/},
      projectId: 1,
      hours: 2,
      __v: 0
    },
    {
      _id: ObjectID {/*ID*/},
      projectId: 1,
      hours: 1,
      __v: 0
    }
  ]
}]

But you might ask: "That's basically what I have so why does that work?!"

I am not sure if your example code was copy/pasted but in the example provided the $unwind stage appears to have been accidentally included in the 2nd stage ($lookup) of your pipeline. If you were to add it as a 3rd stage as the following code shows then the output would be altered as shown.

return Project.aggregate([
  { $match: { projectId: 1 }},
  {
    $lookup: {
      from: 'tasks',
      localField: 'projectId',
      foreignField: 'projectId',
      as: 'tasks'
    }
  },
  { $unwind: '$tasks' }
]).exec();

Outputs:

[{
    _id: ObjectID {/*ID*/},
    projectId: 1,
    description: 'Foo',
    __v: 0,
    tasks: {
      _id: ObjectID {/*ID*/},
      projectId: 1,
      hours: 1,
      __v: 0
    }
  },
  {

    _id: ObjectID {/*ID*/},
    projectId: 1,
    description: 'Foo',
    __v: 0,
    tasks: {
      _id: ObjectID {/*ID*/},
      projectId: 1,
      hours: 2,
      __v: 0
    }
  }
]

Virtual populates and aggregates with $lookup � Issue #5908 , I'm trying to do aggregations on Mongoose schemas that use the virtual populate functionality (using Mongoose 4.13, Mongo 3.4). const� I actually did post this question to Stackoverflow (mongoose-virtual-populate-and-aggregates but could not get a proper response. I found a post that someone claims to get it working, but I could not find much in the Mongoose docs on the limitations of virtual populate.


mongoose >= 3.6 supports the Model.populate() method.

So, You can pass the result of the aggregate method to the populate method like this.

var opts = [{ path: 'tasks'}];
const populatedResult = await Project.populate(result, opts);

Populate, Virtuals are typically used for computed properties on documents. Your First Virtual; Virtual Setters; Virtuals in JSON; Virtuals with Lean; Limitations; Populate � mongoose-association. A cleaner and faster way to setup mongoose populate with virtual field using normalized associations. This library is built for async await node environment minimum 7.6 This library is built for mongo 3.6, and mongoose version >=4.11.0. Setup. npm install mongoose-association


Mongoose v5.9.25: Mongoose Tutorials: Mongoose Virtuals, Mongoose 4.13 was released last week. The syntax for virtual populate's dynamic properties is slightly different than the one Mongoose 4.13 has 9 other new features, including middleware for .aggregate() , so make sure� Mongoose provides a straight-forward, schema-based solution to model your application data. It includes built-in type casting, validation, query building, business logic hooks and more, out of the box.


What's New In Mongoose 4.13: Dynamic Refs and Fields for Virtual , Here is institute schema 'use strict'; var mongoose But if change nested schema from array After your aggregation, do this: Populate + Aggregate in Mongoose � $match takes a document that specifies the query conditions. The query syntax is identical to the read operation query syntax; i.e. $match does not accept raw


Mongoose populate sub-sub document, Populate. MongoDB has the join-like $lookup aggregation operator in versions > = 3.2. Types.ObjectId, ref: 'Person' }, title: String, fans: [{ type: Schema.Types. Mongoose constructor. The exports object of the mongoose module is an instance of this class. Most apps will only use this one instance. Example: const mongoose = require ('mongoose'); mongoose instanceof mongoose.Mongoose; // true // Create a new Mongoose instance with its own `connect()`, `set()`, `model()`, etc. const m = new mongoose.Mongoose();