How to extract time from MongoDB _id's and how to create custom _id's with future dates and times

gistfile1.md

Extracting Date and Time from the MongoDb _id field

Until recently, many of the MongoDB models I worked with contained an extra field "created", which did nothing but state when the document was created. That is fine if you need millisecond precision, but if all you are after is wall clock precision, then you can extract the date from the autogenerated _id field on the document.

var ObjectId = require('mongodb').ObjectID
ObjectId.createFromTime(Date.now()/1000 ).getTimestamp()

This works because the first 4 bytes in the ID are the date. The method getTimestamp converts these 4 bytes to a Date with precision down to a second.

Why is this useful?

If you for instance want to query the database for activity based on hours, or days or weeks, you can use this technique along with the mapReduce function to evaluate a date from _id field. To my knowledge this is not possible with the aggregation framework.

Injecting ID's with custom dates and time for testing

Creating custom ID's is as easy as running the above code, but since you are basing your unique IDs on a coarse time indication you will need a little extra magic. The createFromTime method will only set the first 4 bytes of the ID, and you will quickly find that you creating too many ID's for the time to be different.

The last 8 bytes of the 12 byte unique ID introduce the entrophy that you need. To generate the first 4 and last 8 bytes correctly, I have reused the bson library that the MongoDB driver uses itself.

var mersenne = require('mersenne')
var bson = require('bson')

new mongoose.Types.ObjectId( 
  	bson.BinaryParser.encodeInt((injectedUnixTime/1000), 32, true, true) 
	+ bson.BinaryParser.encodeInt(mersenne.rand(Math.pow(2, 63)), 64, true, true)
)

This sets the first 4 bytes to the injectedUnixTime, and adds an additional 8 bytes of random data based on a random number.

mersenne.rand() is a pseudorandom number generator that delivers a number between 0 and x, you can replace it with Math.random(), but I quite like the npm mersenne module which can be seeded, so test data can be reproduced easily.

Why is this useful?

If you have to create a lot of test data and you want the dates to be different than Date.now(), this is the way to do it.

Using a seeded pseudorandom number generator will make sure you can reproduce the testdata if you have to.