进入MongoDB中文手册(4.2版本)目录
在mongo shell中,db.collection.mapReduce() 方法是mapReduce命令的包装。以下示例使用db.collection.mapReduce()方法:
聚合管道作为替代
聚合管道比map-reduce提供更好的性能和更一致的接口。
各种map-reduce表达式可以使用聚合管道运算符重写,诸如$group, $merge等
下面的示例包括聚合管道替换方案。
使用以下文档创建样本集合orders:
db.orders.insertMany([
{ _id: 1, cust_id: "Ant O. Knee", ord_date: new Date("2020-03-01"), price: 25, items: [ { sku: "oranges", qty: 5, price: 2.5 }, { sku: "apples", qty: 5, price: 2.5 } ], status: "A" },
{ _id: 2, cust_id: "Ant O. Knee", ord_date: new Date("2020-03-08"), price: 70, items: [ { sku: "oranges", qty: 8, price: 2.5 }, { sku: "chocolates", qty: 5, price: 10 } ], status: "A" },
{ _id: 3, cust_id: "Busby Bee", ord_date: new Date("2020-03-08"), price: 50, items: [ { sku: "oranges", qty: 10, price: 2.5 }, { sku: "pears", qty: 10, price: 2.5 } ], status: "A" },
{ _id: 4, cust_id: "Busby Bee", ord_date: new Date("2020-03-18"), price: 25, items: [ { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" },
{ _id: 5, cust_id: "Busby Bee", ord_date: new Date("2020-03-19"), price: 50, items: [ { sku: "chocolates", qty: 5, price: 10 } ], status: "A"},
{ _id: 6, cust_id: "Cam Elot", ord_date: new Date("2020-03-19"), price: 35, items: [ { sku: "carrots", qty: 10, price: 1.0 }, { sku: "apples", qty: 10, price: 2.5 } ], status: "A" },
{ _id: 7, cust_id: "Cam Elot", ord_date: new Date("2020-03-20"), price: 25, items: [ { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" },
{ _id: 8, cust_id: "Don Quis", ord_date: new Date("2020-03-20"), price: 75, items: [ { sku: "chocolates", qty: 5, price: 10 }, { sku: "apples", qty: 10, price: 2.5 } ], status: "A" },
{ _id: 9, cust_id: "Don Quis", ord_date: new Date("2020-03-20"), price: 55, items: [ { sku: "carrots", qty: 5, price: 1.0 }, { sku: "apples", qty: 10, price: 2.5 }, { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" },
{ _id: 10, cust_id: "Don Quis", ord_date: new Date("2020-03-23"), price: 25, items: [ { sku: "oranges", qty: 10, price: 2.5 } ], status: "A" }
])
1 返回每个客户的总价
对orders集合执行map-reduce操作,对cust_id进行分组,并计算每个cust_id的price的总和:
- 定义map函数来处理每个输入文档:
* 在函数中,this引用map-reduce操作正在处理的文档。
* 该函数将price映射到cust_id的每个文档,并提交cust_id和price对。var mapFunction1 = function() { emit(this.cust_id, this.price); };
- 用两个参数keyCustId和valuesPrices定义相应的reduce函数 :
* valuesPrices是一个数组,其元素是由map函数提交并根据keyCustId分组的price 值。
* 该函数将valuesPrice数组减少(reduces)为其元素的总和。var reduceFunction1 = function(keyCustId, valuesPrices) { return Array.sum(valuesPrices); };
- 使用map函数mapFunction1和reduce函数reduceFunction1对orders集合中的所有文档执行map-reduce 。
db.orders.mapReduce(
mapFunction1,
reduceFunction1,
{ out: "map_reduce_example" }
)
此操作将结果输出到名为map_reduce_example的集合。如果map_reduce_example集合已经存在,则该操作将用此map-reduce操作的结果替换其内容。
4. 查询map_reduce_example集合来验证结果
db.map_reduce_example.find().sort( { _id: 1 } )
该操作返回以下文档:
{ "_id" : "Ant O. Knee", "value" : 95 }
{ "_id" : "Busby Bee", "value" : 125 }
{ "_id" : "Cam Elot", "value" : 60 }
{ "_id" : "Don Quis", "value" : 155 }
1.1 用聚合替换
使用可用的聚合管道运算符,您可以重写map-reduce操作,而无需定义自定义函数:
db.orders.aggregate([
{ $group: { _id: "$cust_id", value: { $sum: "$price" } } },
{ $out: "agg_alternative_1" }
])
- $group阶段根据cust_id分组并计算value字段(参见$sum)。该value字段包含每个cust_id的price的总和。
该阶段将以下文档输出到下一阶段:
{ "_id" : "Don Quis", "value" : 155 }
{ "_id" : "Ant O. Knee", "value" : 95 }
{ "_id" : "Cam Elot", "value" : 60 }
{ "_id" : "Busby Bee", "value" : 125 }
- 然后,$out将输出写入agg_alternative_1集合。或者,您可以使用$merge代替$out。
- 查询agg_alternative_1集合来验证结果:
db.agg_alternative_1.find().sort( { _id: 1 } )
该操作返回以下文档:
{ "_id" : "Ant O. Knee", "value" : 95 }
{ "_id" : "Busby Bee", "value" : 125 }
{ "_id" : "Cam Elot", "value" : 60 }
{ "_id" : "Don Quis", "value" : 155 }
2 用每项的平均数量计算订单和总数量
在此示例中,您将对在orders集合上ord_date的值大于或等于2020-03-01的所有文档执行map-reduce操作 。操作按item.sku字段分组 ,并计算每个sku的订单数量和总订购量。然后,该操作将为每个sku计算每个订单的平均数量,并将结果合并到输出集合中。合并结果时,如果现有文档的key与新结果相同,则该操作将覆盖现有文档。如果不存在具有相同key的文档,则该操作将插入该文档。
- 定义map-reduce函数来处理每个输入文档:
* 在函数中,this引用map-reduce操作正在处理的文档。
* 对于每一项,该函数将其sku与一个新对象value相关联,该对象value包含订单的计数1和项qty并提交sku和value对。
var reduceFunction2 = function(keySKU, countObjVals) {
reducedVal = { count: 0, qty: 0 };
for (var idx = 0; idx < countObjVals.length; idx++) {
reducedVal.count += countObjVals[idx].count;
reducedVal.qty += countObjVals[idx].qty;
}
return reducedVal;
};
- 用两个参数keySKU和countObjVals定义相应的reduce函数 :
* countObjVals是一个数组,其元素是映射到由map函数传递给reducer函数的分组的keySKU值的对象。
* 该函数将countObjVals数组简化为包含count和qty字段的一个单一的reducedValue对象。
* 在reducedVal中,该count字段包含各个数组元素的count字段的总和,而该qty字段包含各个数组元素的qty字段的总和。
var reduceFunction2 = function(keySKU, countObjVals) {
reducedVal = { count: 0, qty: 0 };
for (var idx = 0; idx < countObjVals.length; idx++) {
reducedVal.count += countObjVals[idx].count;
reducedVal.qty += countObjVals[idx].qty;
}
return reducedVal;
};
- 定义有key和 reducedVal两个参数的finalize函数。该函数修改reducedVal对象来添加一个名为avg的计算字段,并返回修改后的对象:
var finalizeFunction2 = function (key, reducedVal) {
reducedVal.avg = reducedVal.qty/reducedVal.count;
return reducedVal;
};
- 在orders集合上使用mapFunction2,reduceFunction2和 finalizeFunction2函数执行map-reduce操作。
db.orders.mapReduce(
mapFunction2,
reduceFunction2,
{
out: { merge: "map_reduce_example2" },
query: { ord_date: { $gte: new Date("2020-03-01") } },
finalize: finalizeFunction2
}
);
此操作使用该query字段选择ord_date仅大于或等于new Date(“2020-03-01”)的那些文档。然后将结果输出到map_reduce_example2集合 。
如果map_reduce_example2集合已经存在,则该操作会将现有内容与map-reduce操作的结果合并。即,如果现有文档具有与新结果相同的key,则该操作将覆盖现有文档。如果不存在具有相同key的文档,则该操作将插入该文档。
5. 查询map_reduce_example2集合来验证结果:
db.map_reduce_example2.find().sort( { _id: 1 } )
该操作返回以下文档:
{ "_id" : "apples", "value" : { "count" : 3, "qty" : 30, "avg" : 10 } }
{ "_id" : "carrots", "value" : { "count" : 2, "qty" : 15, "avg" : 7.5 } }
{ "_id" : "chocolates", "value" : { "count" : 3, "qty" : 15, "avg" : 5 } }
{ "_id" : "oranges", "value" : { "count" : 6, "qty" : 58, "avg" : 9.666666666666666 } }
{ "_id" : "pears", "value" : { "count" : 1, "qty" : 10, "avg" : 10 } }
2.1 用聚合替换
使用可用的聚合管道运算符,您可以重写map-reduce操作,而无需定义自定义函数:
db.orders.aggregate( [
{ $match: { ord_date: { $gte: new Date("2020-03-01") } } },
{ $unwind: "$items" },
{ $group: { _id: "$items.sku", qty: { $sum: "$items.qty" }, orders_ids: { $addToSet: "$_id" } } },
{ $project: { value: { count: { $size: "$orders_ids" }, qty: "$qty", avg: { $divide: [ "$qty", { $size: "$orders_ids" } ] } } } },
{ $merge: { into: "agg_alternative_3", on: "_id", whenMatched: "replace", whenNotMatched: "insert" } }
] )
- $match阶段仅选择ord_date大于或等于new Date(“2020-03-01”)的这些文档。
- $unwinds阶段按items数组字段细分文档,并为每个数组元素输出一个文档。例如:
{ "_id" : 1, "cust_id" : "Ant O. Knee", "ord_date" : ISODate("2020-03-01T00:00:00Z"), "price" : 25, "items" : { "sku" : "oranges", "qty" : 5, "price" : 2.5 }, "status" : "A" }
{ "_id" : 1, "cust_id" : "Ant O. Knee", "ord_date" : ISODate("2020-03-01T00:00:00Z"), "price" : 25, "items" : { "sku" : "apples", "qty" : 5, "price" : 2.5 }, "status" : "A" }
{ "_id" : 2, "cust_id" : "Ant O. Knee", "ord_date" : ISODate("2020-03-08T00:00:00Z"), "price" : 70, "items" : { "sku" : "oranges", "qty" : 8, "price" : 2.5 }, "status" : "A" }
{ "_id" : 2, "cust_id" : "Ant O. Knee", "ord_date" : ISODate("2020-03-08T00:00:00Z"), "price" : 70, "items" : { "sku" : "chocolates", "qty" : 5, "price" : 10 }, "status" : "A" }
{ "_id" : 3, "cust_id" : "Busby Bee", "ord_date" : ISODate("2020-03-08T00:00:00Z"), "price" : 50, "items" : { "sku" : "oranges", "qty" : 10, "price" : 2.5 }, "status" : "A" }
{ "_id" : 3, "cust_id" : "Busby Bee", "ord_date" : ISODate("2020-03-08T00:00:00Z"), "price" : 50, "items" : { "sku" : "pears", "qty" : 10, "price" : 2.5 }, "status" : "A" }
{ "_id" : 4, "cust_id" : "Busby Bee", "ord_date" : ISODate("2020-03-18T00:00:00Z"), "price" : 25, "items" : { "sku" : "oranges", "qty" : 10, "price" : 2.5 }, "status" : "A" }
{ "_id" : 5, "cust_id" : "Busby Bee", "ord_date" : ISODate("2020-03-19T00:00:00Z"), "price" : 50, "items" : { "sku" : "chocolates", "qty" : 5, "price" : 10 }, "status" : "A" }
...
- $group阶段根据items.sku分组,并计算每个SKU:
* qty字段。该qty字段包含每个items.sku(请参阅$sum)的订单总数qty。
* orders_ids数组。该orders_ids字段包含为items.sku区分订单_id的数组(参见 $addToSet)。
{ "_id" : "chocolates", "qty" : 15, "orders_ids" : [ 2, 5, 8 ] }
{ "_id" : "oranges", "qty" : 63, "orders_ids" : [ 4, 7, 3, 2, 9, 1, 10 ] }
{ "_id" : "carrots", "qty" : 15, "orders_ids" : [ 6, 9 ] }
{ "_id" : "apples", "qty" : 35, "orders_ids" : [ 9, 8, 1, 6 ] }
{ "_id" : "pears", "qty" : 10, "orders_ids" : [ 3 ] }
- 该$project阶段调整输出文档的形状来反映map-reduce的输出,该输出具有两个字段_id和 value。该$project设置:
* 表示orders_ids数组的大小的value.count。(请参阅$size。)
* 表示输入文档的qty字段的value.qty
* 表示每个订单的qty的平均数的value.avg
{ "_id" : "apples", "value" : { "count" : 4, "qty" : 35, "avg" : 8.75 } }
{ "_id" : "pears", "value" : { "count" : 1, "qty" : 10, "avg" : 10 } }
{ "_id" : "chocolates", "value" : { "count" : 3, "qty" : 15, "avg" : 5 } }
{ "_id" : "oranges", "value" : { "count" : 7, "qty" : 63, "avg" : 9 } }
{ "_id" : "carrots", "value" : { "count" : 2, "qty" : 15, "avg" : 7.5 } }
- 最后,$merge将输出写入collection agg_alternative_3。如果现有文档的key _id与新结果的key相同,则该操作将覆盖现有文档。如果不存在具有相同key的文档,则该操作将插入该文档。
- 查询agg_alternative_3集合来验证结果:
db.agg_alternative_3.find().sort( { _id: 1 } )
该操作返回以下文档:
{ "_id" : "apples", "value" : { "count" : 4, "qty" : 35, "avg" : 8.75 } }
{ "_id" : "carrots", "value" : { "count" : 2, "qty" : 15, "avg" : 7.5 } }
{ "_id" : "chocolates", "value" : { "count" : 3, "qty" : 15, "avg" : 5 } }
{ "_id" : "oranges", "value" : { "count" : 7, "qty" : 63, "avg" : 9 } }
{ "_id" : "pears", "value" : { "count" : 1, "qty" : 10, "avg" : 10 } }