How can I get a random record from MongoDB?

I am looking to get a random record from a huge collection (100 million records). What is the fastest and most efficient way to do so? The data is already there and there are no field in which I can generate a random number and obtain a random row.

See also this SO question titled "Ordering a result set randomly in mongo". Thinking about randomly ordering a result set is a more general version of this question -- more powerful and more useful.

This question keeps popping up. The latest information can likely be found at the feature request to get random items from a collection in the MongoDB ticket tracker. If implemented natively, it would likely be the most efficient option. (If you want the feature, go vote it up.)

Does anyone know how much slower this is than just taking the first record? I’m debating whether it’s worth taking a random sample to do something vs just doing it in order.

Actually opposite of the answers $sample might not be fastest solution. Because mongo may do a collection scan for random sorting when using $sample depending on the situation. Please see: Reference: docs.mongodb.com/manual/reference/operator/aggregation/sample Maybe doing counting result set and doing some random skip take will do better.

30 Answers 30

Starting with the 3.2 release of MongoDB, you can get N random docs from a collection using the $sample aggregation pipeline operator:

// Get one random document from the mycoll collection. db.mycoll.aggregate([ < $sample: < size: 1 >>]) 

If you want to select the random document(s) from a filtered subset of the collection, prepend a $match stage to the pipeline:

// Get one random document matching from the mycoll collection. db.mycoll.aggregate([ < $match: < a: 10 >>, < $sample: < size: 1 >> ]) 

As noted in the comments, when size is greater than 1, there may be duplicates in the returned document sample.

This is a good way, but remember that it DO NOT guarantee that there are no copies of the same object in the sample.

Not to be pedantic but the question doesn't specify a MongoDB version, so I'd assume having the most recent version is reasonable.

@brycejl That would have the fatal flaw of not matching anything if the $sample stage didn't select any matching documents.

Do a count of all records, generate a random number between 0 and the count, and then do:

db.yourCollection.find().limit(-1).skip(yourRandomNumber).next() 

Unfortunately skip() is rather inefficient since it has to scan that many documents. Also, there is a race condition if rows are removed between getting the count and running the query.

Note that the random number should be between 0 and the count (exclusive). I.e., if you have 10 items, the random number should be between 0 and 9. Otherwise the cursor could try to skip past the last item, and nothing would be returned.

Thanks, worked perfectly for my purposes. @mstearn, your comments on both efficiency and race conditions are valid, but for collections where neither matters (one-time server-side batch extract in a collection where records aren't deleted), this is vastly superior to the hacky (IMO) solution in the Mongo Cookbook.

@MonkeyBonkey docs.mongodb.org/meta-driver/latest/legacy/… "If numberToReturn is 0, the db will use the default return size. If the number is negative, then the database will return that number and close the cursor."

Update for MongoDB 3.2

3.2 introduced $sample to the aggregation pipeline.

There's also a good blog post on putting it into practice.

For older versions (previous answer)

This was actually a feature request: http://jira.mongodb.org/browse/SERVER-533 but it was filed under "Won't fix."

The cookbook has a very good recipe to select a random document out of a collection: http://cookbook.mongodb.org/patterns/random-attribute/

To paraphrase the recipe, you assign random numbers to your documents:

db.docs.save( < key : 1, . random : Math.random() >) 

Then select a random document:

rand = Math.random() result = db.docs.findOne( < key : 2, random : < $gte : rand >> ) if ( result == null ) < result = db.docs.findOne( < key : 2, random : < $lte : rand >> ) > 

Querying with both $gte and $lte is necessary to find the document with a random number nearest rand .

And of course you'll want to index on the random field:

db.docs.ensureIndex( < key : 1, random :1 >) 

If you're already querying against an index, simply drop it, append random: 1 to it, and add it again.

And here is a simple way to add the random field to every document in the collection. function setRandom() < db.topics.find().forEach(function (obj) ); > db.eval(setRandom);

This selects a document randomly, but if you do it more than once, the lookups are not independent. You are more likely to get the same document twice in a row than random chance would dictate.

Looks like a bad implementation of circular hashing. It's even worse than lacker says: even one lookup is biased because the random numbers aren't evenly distributed. To do this properly, you'd need a set of, say, 10 random numbers per document. The more random numbers you use per document, the more uniform the output distribution becomes.

The MongoDB JIRA ticket is still alive: jira.mongodb.org/browse/SERVER-533 Go comment and vote if you want the feature.

Take note the type of caveat mentioned. This does not work efficiently with small amount of documents. Given two items with random key of 3 and 63. The document #63 will be chosen more frequently where $gte is first. Alternative solution stackoverflow.com/a/9499484/79201 would work better in this case.

You can also use MongoDB's geospatial indexing feature to select the documents 'nearest' to a random number.

First, enable geospatial indexing on a collection:

db.docs.ensureIndex( < random_point: '2d' >) 

To create a bunch of documents with random points on the X-axis:

for ( i = 0; i < 10; ++i ) < db.docs.insert( < key: i, random_point: [Math.random(), 0] >); > 

Then you can get a random document from the collection like this:

db.docs.findOne( < random_point : < $near : [Math.random(), 0] >> ) 

Or you can retrieve several document nearest to a random point:

db.docs.find( < random_point : < $near : [Math.random(), 0] >> ).limit( 4 ) 

This requires only one query and no null checks, plus the code is clean, simple and flexible. You could even use the Y-axis of the geopoint to add a second randomness dimension to your query.

I like this answer, Its the most efficient one I've seen that doesn't require a bunch of messing about server side.

That is true, and there are other problems as well: documents are strongly correlated on their random keys, so it's highly predictable which documents will be returned as a group if you select multiple documents. Also, documents close to the bounds (0 and 1) are less likely to be chosen. The latter could be solved by using spherical geomapping, which wraps around at the edges. However, you should see this answer as an improved version of the cookbook recipe, not as a perfect random selection mechanism. It's random enough for most purposes.

@NicodePoel, I like your answer as well as your comment! And I have a couple of questions for you: 1- How do you know that points close to bounds 0 and 1 are less likely to be chosen, is that based on some mathematical ground?, 2- Can you elaborate more on spherical geomapping, how it will better the random selection, and how to do it in MongoDB? . Appreciated!

The following recipe is a little slower than the mongo cookbook solution (add a random key on every document), but returns more evenly distributed random documents. It's a little less-evenly distributed than the skip( random ) solution, but much faster and more fail-safe in case documents are removed.

function draw(collection, query) < // query: mongodb query object (optional) var query = query || < >; query['random'] = < $lte: Math.random() >; var cur = collection.find(query).sort(< rand: -1 >); if (! cur.hasNext()) < delete query.random; cur = collection.find(query).sort(< rand: -1 >); > var doc = cur.next(); doc.random = Math.random(); collection.update(< _id: doc._id >, doc); return doc; > 

It also requires you to add a random "random" field to your documents so don't forget to add this when you create them : you may need to initialize your collection as shown by Geoffrey

function addRandom(collection) < collection.find().forEach(function (obj) < obj.random = Math.random(); collection.save(obj); >); > db.eval(addRandom, db.things); 

Benchmark results

This method is much faster than the skip() method (of ceejayoz) and generates more uniformly random documents than the "cookbook" method reported by Michael:

For a collection with 1,000,000 elements:

• This method takes less than a millisecond on my machine
• the skip() method takes 180 ms on average

The cookbook method will cause large numbers of documents to never get picked because their random number does not favor them.

• This method will pick all elements evenly over time.
• In my benchmark it was only 30% slower than the cookbook method.
• the randomness is not 100% perfect but it is very good (and it can be improved if necessary)

This recipe is not perfect - the perfect solution would be a built-in feature as others have noted.
However it should be a good compromise for many purposes.

Here is a way using the default ObjectId values for _id and a little math and logic.

// Get the "min" and "max" timestamp values from the _id in the collection and the // diff between. // 4-bytes from a hex string is 8 characters var min = parseInt(db.collection.find() .sort(< "_id": 1 >).limit(1).toArray()[0]._id.str.substr(0,8),16)*1000, max = parseInt(db.collection.find() .sort(< "_id": -1 >)limit(1).toArray()[0]._id.str.substr(0,8),16)*1000, diff = max - min; // Get a random value from diff and divide/multiply be 1000 for The "_id" precision: var random = Math.floor(Math.floor(Math.random(diff)*diff)/1000)*1000; // Use "random" in the range and pad the hex string to a valid ObjectId var _id = new ObjectId(((min + random)/1000).toString(16) + "0000000000000000") // Then query for the single document: var randomDoc = db.collection.find( < "_id": < "$gte": _id >>) .sort(< "_id": 1 >).limit(1).toArray()[0]; 

That's the general logic in shell representation and easily adaptable.

• Find the min and max primary key values in the collection
• Generate a random number that falls between the timestamps of those documents.
• Add the random number to the minimum value and find the first document that is greater than or equal to that value.

This uses "padding" from the timestamp value in "hex" to form a valid ObjectId value since that is what we are looking for. Using integers as the _id value is essentially simplier but the same basic idea in the points.

Now you can use the aggregate. Example:

db.users.aggregate( [ < $sample: < size: 3 >> ] ) 

In Python using pymongo:

import random def get_random_doc(): count = collection.count() return collection.find()[random.randrange(count)] 

Your answer is correct. However, please replace count() with estimated_document_count() as count() is deprecated in Mongdo v4.2.

Using Python (pymongo), the aggregate function also works.

collection.aggregate([>]) 

This approach is a lot faster than running a query for a random number (e.g. collection.find([random_int]). This is especially the case for large collections.

it is tough if there is no data there to key off of. what are the _id field? are they mongodb object id's? If so, you could get the highest and lowest values:

lowest = db.coll.find().sort().limit(1).next()._id; highest = db.coll.find().sort().limit(1).next()._id; 

then if you assume the id's are uniformly distributed (but they aren't, but at least it's a start):

unsigned long long L = first_8_bytes_of(lowest) unsigned long long H = first_8_bytes_of(highest) V = (H - L) * random_from_0_to_1(); N = L + V; oid = N concat random_4_bytes(); randomobj = db.coll.find(>).limit(1); 

Any ideas how would that look like in PHP? or at least what language have you used above? is it Python?

You can pick a random timestamp and search for the first object that was created afterwards. It will only scan a single document, though it doesn't necessarily give you a uniform distribution.

var randRec = function() < // replace with your collection var coll = db.collection // get unixtime of first and last record var min = coll.find().sort().limit(1)[0]._id.getTimestamp() - 0; var max = coll.find().sort().limit(1)[0]._id.getTimestamp() - 0; // allow to pass additional query params return function(query) < if (typeof query === 'undefined') query = <>var randTime = Math.round(Math.random() * (max - min)) + min; var hexSeconds = Math.floor(randTime / 1000).toString(16); var + "0000000000000000"); query._id = return coll.find(query).limit(1) >; >(); 

this is the best method for very large collections, it works at O(1), unline skip() or count() used in the other solutions here

My solution on php:

/** * Get random docs from Mongo * @param $collection * @param $where * @param $fields * @param $limit * @author happy-code * @url happy-code.com */ private function _mongodb_get_random (MongoCollection $collection, $where = array(), $fields = array(), $limit = false) < // Total docs $count = $collection->find($where, $fields)->count(); if (!$limit) < // Get all docs $limit = $count; >$data = array(); for( $i = 0; $i < $limit; $i++ ) < // Skip documents $skip = rand(0, ($count-1) ); if ($skip !== 0) < $doc = $collection->find($where, $fields)->skip($skip)->limit(1)->getNext(); > else < $doc = $collection->find($where, $fields)->limit(1)->getNext(); > if (is_array($doc)) < // Catch document $data[ $doc['_id']-> ] = $doc; // Ignore current document when making the next iteration $where['_id']['$nin'][] = $doc['_id']; > // Every iteration catch document and decrease in the total number of document $count--; > return $data; > 

The best way in Mongoose is to make an aggregation call with $sample. However, Mongoose does not apply Mongoose documents to Aggregation - especially not if populate() is to be applied as well.

For getting a "lean" array from the database:

/* Sample model should be init first const Sample = mongoose … */ const samples = await Sample.aggregate([ < $match: <>>, < $sample: < size: 33 >>, ]).exec(); console.log(samples); //a lean Array 

For getting an array of mongoose documents:

const samples = ( await Sample.aggregate([ < $match: <>>, < $sample: < size: 27 >>, < $project: < _id: 1 >>, ]).exec() ).map(v => v._id); const mongooseSamples = await Sample.find( < _id: < $in: samples >>); console.log(mongooseSamples); //an Array of mongoose documents 

In order to get a determinated number of random docs without duplicates:

1. first get all ids
2. get size of documents
3. loop geting random index and skip duplicated

number_of_docs=7 db.collection('preguntas').find(<>,).toArray(function(err, arr) < count=arr.length idsram=[] rans=[] while(number_of_docs!=0)< var R = Math.floor(Math.random() * count); if (rans.indexOf(R) >-1) < continue >else < ans.push(R) idsram.push(arr[R]._id) number_of_docs-- >> db.collection('preguntas').find(<>).toArray(function(err1, doc1) < if (err1) < console.log(err1); return; >res.send(doc1) >); >); 

My simplest solution to this .

db.coll.find() .limit(1) .skip(Math.floor(Math.random() * 500)) .next() 

Where you have at least 500 items on collections

I would suggest using map/reduce, where you use the map function to only emit when a random value is above a given probability.

function mapf() < if(Math.random() > function reducef(key,values) < return ; > res = db.questions.mapReduce(mapf, reducef, , "scope": < "probability": 0.5>>); printjson(res.results); 

The reducef function above works because only one key ('1') is emitted from the map function.

The value of the "probability" is defined in the "scope", when invoking mapRreduce(. )

Using mapReduce like this should also be usable on a sharded db.

If you want to select exactly n of m documents from the db, you could do it like this:

function mapf() < if(countSubset == 0) return; var prob = countSubset / countTotal; if(Math.random() ); countSubset--; > countTotal--; > function reducef(key,values) < var newArray = new Array(); for(var i=0; i < values.length; i++) < newArray = newArray.concat(values[i].documents); >return ; > res = db.questions.mapReduce(mapf, reducef, , "scope": >) printjson(res.results); 

Where "countTotal" (m) is the number of documents in the db, and "countSubset" (n) is the number of documents to retrieve.

This approach might give some problems on sharded databases.

Doing a full collection scan to return 1 element. this must be the least efficient technique to do it.

The trick is, that it is a general solution for returning an arbitrary number of random elements - in which case it would be faster than the other solutions when getting > 2 random elements.

You can pick random _id and return corresponding object:

 db.collection.count( function(err, count) < db.collection.distinct( "_id" , function( err, result) < if (err) res.send(err) var randomId = result[Math.floor(Math.random() * (count-1))] db.collection.findOne( < _id: randomId >, function( err, result) < if (err) res.send(err) console.log(result) >) >) >) 

Here you dont need to spend space on storing random numbers in collection.

The following aggregation operation randomly selects 3 documents from the collection:

MongoDB now has $rand

To pick n non repeat items, aggregate with < $addFields: < _f: < $rand: <>> > > then $sort by _f and $limit n.

I'd suggest adding a random int field to each object. Then you can just do a

findOne(>) 

to pick a random document. Just make sure you ensureIndex()

If the first record in your collection has a relatively high random_field value, won't it be returned almost all the time?

This solution is completely wrong, adding a random number (let's imagine in between 0 a 2^32-1) doesn't guarantee any good distribution and using $gte makes it even worst, due to your random selection won't be even close to a pseudo-random number. I suggest not to use this concept ever.

When I was faced with a similar solution, I backtracked and found that the business request was actually for creating some form of rotation of the inventory being presented. In that case, there are much better options, which have answers from search engines like Solr, not data stores like MongoDB.

In short, with the requirement to "intelligently rotate" content, what we should do instead of a random number across all of the documents is to include a personal q score modifier. To implement this yourself, assuming a small population of users, you can store a document per user that has the productId, impression count, click-through count, last seen date, and whatever other factors the business finds as being meaningful to compute a q score modifier. When retrieving the set to display, typically you request more documents from the data store than requested by the end user, then apply the q score modifier, take the number of records requested by the end user, then randomize the page of results, a tiny set, so simply sort the documents in the application layer (in memory).

If the universe of users is too large, you can categorize users into behavior groups and index by behavior group rather than user.

If the universe of products is small enough, you can create an index per user.

I have found this technique to be much more efficient, but more importantly more effective in creating a relevant, worthwhile experience of using the software solution.