While we're approaching the 1.0 release of Meteor, the need and interest in additional databases support (other than MongoDB) has risen consequently. Thus, SQL support is the second most requested feature on the official meteor roadmap (after server-side rendering) and there have been various demands on the forums about supporting Redis (also on the roadmap), RethinkDB, LevelDB, Neo4J, ElasticSearch, Mysql/MariaDB, PosgreSQL, SQLite and some other. More importantly the Meteor community has started to work on atmosphere packages that bring support for these databases and it appeared to be a very difficult task – most projects were abandoned.
What makes the integration of a new database a lot harder in Meteor than in most other web frameworks (including “full stack” JavaScript frameworks like Derby or Sails) lies in the third principle of Meteor:
Database Everywhere. Use the same transparent API to access your database from the client or the server.
This principle is reflected by the usage of minimongo – an in-memory MongoDB emulator – on the client side of Meteor applications. We all agree it's an amazing feature, making things like latency compensation breeze.
However in the current state of the Meteor platform, this also have a serious consequence for third-party database integrators. Basically they have to emulate the database in a mini{redis|sqlite|...} package. So if I want to bring sqlite support for Meteor, I first need to write a (reactive) in-memory Javascript port of sqlite that works in a browser (IE8+), which may not be the most straightforward task I can think of.
The other solution to implement a sqlite database could be to keep using minimongo on the client – because after all we'll end up writing something that will do more or less the same thing, just with a different public API – and then use the npm/nodejs driver on the server. The problem is that it violates the third principle of Meteor, and thus if we want to have latency compensation for our queries (we do), we'll have to write two versions of them: one using minimongo on the client, and one for “real” sqlite on the server, making this solution far less desirable.
So here we are with one of the best platform for building real-time distributed applications, and we can't use anything but MongoDB.
The idea is quite simple, because we want to use the same API on multiple platforms but we don't want to re-implement a JavaScript database for every database we use, we need to:
- Define a generic enough API to handle most features databases offer (key/value stores, schemas, relations, indexes, etc.);
- Define a way to create adapters, ie an interface between this generic API and the database API for a given platform (currently browser, nodejs, or cordova).
Fortunately enough these are the classical roles ORMs stand for, and so we don't need to reinvent the wheel. We already have plenty of experience about APIs that work and how to connect them to database drivers.
So could we just pick one and move on? Not really. Because of the very nature of Meteor applications, a Meteor ORM would need to have the following characteristics:
- Be written in JavaScript, and working in the browser environment
- Allow synchronous calls on the server
- Allow reactivity
- Allow leader/follower architectures and ensure security (never trust the client)
And as far as I'm aware of, there is no JavaScript ORM with all that features. So after all, we may end up needing to create yet another ORM – a MeteORM.
I'm not an expert but I've found a project called Waterline that could be a very good basis for our MeteORM. It's written is JavaScript, used by SailJS (a real-time nodejs framework), has an adaptors architecture, a reasonable query language (similar to what we use today). It also supports models, models validation, and models relations.
I encourage you to take a look at the Waterline documentation. Assuming we choose it as a basis for our ORM, here is what we'll have to figure out:
- How to remove the
.run(callback)call (I guess we should use.fetch()instead + use fibers on the server) - How adapters export their data in publications (using the low level DDP API) and how a (potentially different) database gets populated from the DDP flow.
- How to implement client side reactivity on cursors (using
Tracker)
Once we have this additional layer between the developer and the database, with a generic API and a way to connect new database engines, like what we have today for “plugins” (CSS/JS pre-processors), I'm pretty confident the Meteor community will create an abundance of adapters for all the databases we use and love.
$ meteor add rethinkdb
downloading rethinkdb...
done!
In Waterline you can write native db queries as you want,
so the point of being closed to the ORM API is not true...