meteorm.md

Thoughts on a Meteor ORM

While we're approaching the 1.0 release of Meteor, the need and interest in additional databases support (other than MongoDB) has risen consequently. Thus, SQL support is the second most requested feature on the official meteor roadmap (after server-side rendering) and there have been various demands on the forums about supporting Redis (also on the roadmap), RethinkDB, LevelDB, Neo4J, ElasticSearch, Mysql/MariaDB, PosgreSQL, SQLite and some other. More importantly the Meteor community has started to work on atmosphere packages that bring support for these databases and it appeared to be a very difficult task – most projects were abandoned.

The problem

What makes the integration of a new database a lot harder in Meteor than in most other web frameworks (including “full stack” JavaScript frameworks like Derby or Sails) lies in the third principle of Meteor:

Database Everywhere. Use the same transparent API to access your database from the client or the server.

This principle is reflected by the usage of minimongo – an in-memory MongoDB emulator – on the client side of Meteor applications. We all agree it's an amazing feature, making things like latency compensation breeze.

However in the current state of the Meteor platform, this also have a serious consequence for third-party database integrators. Basically they have to emulate the database in a mini{redis|sqlite|...} package. So if I want to bring sqlite support for Meteor, I first need to write a (reactive) in-memory Javascript port of sqlite that works in a browser (IE8+), which may not be the most straightforward task I can think of.

The other solution to implement a sqlite database could be to keep using minimongo on the client – because after all we'll end up writing something that will do more or less the same thing, just with a different public API – and then use the npm/nodejs driver on the server. The problem is that it violates the third principle of Meteor, and thus if we want to have latency compensation for our queries (we do), we'll have to write two versions of them: one using minimongo on the client, and one for “real” sqlite on the server, making this solution far less desirable.

So here we are with one of the best platform for building real-time distributed applications, and we can't use anything but MongoDB.

The solution

The idea is quite simple, because we want to use the same API on multiple platforms but we don't want to re-implement a JavaScript database for every database we use, we need to:

• Define a generic enough API to handle most features databases offer (key/value stores, schemas, relations, indexes, etc.);
• Define a way to create adapters, ie an interface between this generic API and the database API for a given platform (currently browser, nodejs, or cordova).

Fortunately enough these are the classical roles ORMs stand for, and so we don't need to reinvent the wheel. We already have plenty of experience about APIs that work and how to connect them to database drivers.

So could we just pick one and move on? Not really. Because of the very nature of Meteor applications, a Meteor ORM would need to have the following characteristics:

• Be written in JavaScript, and working in the browser environment
• Allow synchronous calls on the server
• Allow reactivity
• Allow leader/follower architectures and ensure security (never trust the client)

And as far as I'm aware of, there is no JavaScript ORM with all that features. So after all, we may end up needing to create yet another ORM – a MeteORM.

Great artists steal

I'm not an expert but I've found a project called Waterline that could be a very good basis for our MeteORM. It's written is JavaScript, used by SailJS (a real-time nodejs framework), has an adaptors architecture, a reasonable query language (similar to what we use today). It also supports models, models validation, and models relations.

I encourage you to take a look at the Waterline documentation. Assuming we choose it as a basis for our ORM, here is what we'll have to figure out:

• How to remove the .run(callback) call (I guess we should use .fetch() instead + use fibers on the server)
• How adapters export their data in publications (using the low level DDP API) and how a (potentially different) database gets populated from the DDP flow.
• How to implement client side reactivity on cursors (using Tracker)

---

Once we have this additional layer between the developer and the database, with a generic API and a way to connect new database engines, like what we have today for “plugins” (CSS/JS pre-processors), I'm pretty confident the Meteor community will create an abundance of adapters for all the databases we use and love.

$ meteor add rethinkdb
downloading rethinkdb...
done!

Okay. That's a cool idea.

I can work on this for building a low level DDP layer for the ORM.

I guess we should open a hackpad to discuss the API?

Feeling inspired by most post today, http://joshowens.me/where-is-the-orm-in-meteor/ ?

Btw, I love the idea.

Perhaps some clever naming around mining asteroids:

• Miner
• Frigate // used to haul cargo
• Ore // what you mine

@queso Yep :) Tough your article speak more about the “model” aspect of the ORM whereas my reasoning is more about the need to support other databases. But in the end, the ORM will have both roles (model validation and API unification).

Also I'm a happy collection2 user (and contributor). Do you have an ideal model API, what do you think of waterline models? (I haven't taken a deep look, I've just looked at their adapters)

I can try to take a look at the waterline models tonight and give you my thoughts on them.

I think if we are moving away from just Mongo in the long run, we will have to think through the migration scenarios as well.

Maybe we could have a hangout to talk through what this might look like?

I think this is a great idea! I would love to help out whenever I can with this!

Using an ORM is the wrong path.... I don't think there's going to be much support for this.

@dfischer care to explain?

I didn't read the full article yet (writing this while my code is compiling) but I will comment anyway.

Although ORMs were discussed a lot at Meteor internally, it came up again when we were experimenting with Redis. Eventually we had a conclusion that eventhough ORM solution might be easier to implement for more databases support, in the end of the day ORM will be the lowest common denominator of all of them. Thus users will not be able to leverage all the favorite queries from their chosen database, will need to learn a new API and will constantly get confused by all the behind-the-scenes mappings that ORM does.

I used sails.js with waterline for some time on few projects and its work good. On other side I'm new to meteor.js and dont know too much how its work in background. Anyway this is something that also come to my mind these days, so I'm really happy to read this gist and will like to help as much as I can.

@Slava I completely agree that it's best to stay close to the database api. With that said, a lot of meteor developers are used to ActiveRecord from rails so we expect some kind of model definition... I'm very curious how you are planning to implement the postgres support (how do you abstract the query language ?, and how you do the migrations ?).

https://hackpad.com/WIP-An-ORM-for-Meteor-hflXrUdE4kt

@pinouchon, I think the transform function is the easiest way to decorate an individual object with methods and whatnot.

I worked with @Slava on the Redis support. Personally, I align myself more with the ORM camp. However, the idea was that core Meteor should provide the "direct" interfaces: minimongo, miniredis, minisql. This allows package creators to build ORMs on top, and if you as an app writer don't like ORMs you can still use the direct APIs. Meteor itself shouldn't build "the one true ORM" only to find that most people prefer a different style.

So this proposal is exciting stuff, and exactly what we hoped to see! Let me know how I can help!

If the community all ends up using this ORM, maybe it can be adopted into core meteor for support/versioning reasons (if you agree!) That's actually less important than it used to be, now that we have the new packaging system. But if you are contemplating the fact that Meteor doesn't include an ORM, I don't think the conclusion is that "ORMs are bad", it is that "ORMs are great, and we should build at least one."

Have some of you ever used Bookshelf.js ? Any thoughts ?

@justinsb well “this ORM” does not exist yet, I've just wanted to see what the community (and MDG) think of the ORM idea :-)

Love the concept and approach. THANKS!

Is there any way an ORM could provide a Mongo compatible API?

If you expose all the Mongo selectors/modifiers/etc with which we are familiar, it makes it much easier to adopt.

Sure some other databases may not support all the same functionality, and you could either throw errors, or degrade functionality, or whatever.

Also some other database offer additional functionality currently outside of the Mongo API... those could be abstracted via an ORM layer (client AND server).

Finally, if you "need" direct query access on the ORM you could expose a way to bypass the ORM and hit the DB directly (only from the server).

@zeroasterisk check out the linked hackpad, it seems the methods for creating/using a cursor could be assumed to be "good enough" already.

mongo does have more advanced features that aren't (yet 😁 ) exposed in minimongo such as aggregation pipeline. the API for that is also quite similar to an ORM itself really, so exposing it directly rather than an "alternate syntax" would be great - depending on people's opinions on that as a usable API compared to activeRecord/sequel type ORM apis.

Mongo also has "stored javascript" functionality which though not quite stored procedures, staying compatible with the mongo API would allow us to port code to that layer too.

but things like "joins" aren't really supported by the mongo API atm, so that should be a focus for any ORM imho.

I agree with @Slava. I come from a Rails background as well.

The community can work on an ORM but as a base I think it's bloat.

http://blog.codinghorror.com/object-relational-mapping-is-the-vietnam-of-computer-science/

In a way, what we have today is already an ORM in the sense I used in this article: a common API for different backends. Today the two supported backends are “in-memory” and “mongodb”. Take for instance the observer feature, we use a common API but two different adaptors (oplog+pulling on the server, Tracker on the client). I just say that we should specify a way to interface other databases (=“adaptors”) with this common API. If you take a look at the WIP hackpad, you'll see that an ORM API won't be very different from what we use today.

@zeroasterisk Yep, we'll want a way to expose the raw database driver anyway.

Hi. Going to have to dissent, and speak on behalf of not having an ORM in the core system.

Now, a lot of people have come to Meteor with the view that it's the shiniest, newest NoSQL goodness. And that a single-language document-oriented database is some sort of radical NoSQL alternative to traditional LAMP stacks.

However, others of us have come to Meteor and volunteered countless hours over the past 2 years with the view that it's the 21st century version of MUMPS: a 40 year old single-language document-oriented database technology that powers healthcare EMR systems like VistA and Epic. Disclaimer: I spent nearly a decade as an Oracle Administrator and MUMPS Administrator in the healthcare industry.

So, when I chime in and say that adding an ORMs into the core Meteor infrastructure would be a bad move, I say so from my professional experience having been an administrator for single-language document-oriented database applications. We ran these database apps in mission-critical environments, where people's lives depended on the systems being up-and-running. So, I'm not being critical of this proposal as a hobbyist. But as a professional who's worked with these systems in hospital environments where people's lives depended on their uptime and performance.

For what it's worth, VistA and Epic, which combined account for nearly 35% of all medical records in the US, have been running for 40+ years without an internal ORM. And when compared to Cerner, an Oracle SQL based solution, the difference was night-and-day. The real-world impact is that having an ORM around completely fouls up DevOps, maintenance, and the feature development cycle once an app is in production. ORMs are so bad that people write about them as the Vietnam of Computer Science.

Object Relational Mapping is the Vietnam of Computer Science
The Vietnam of Computer Science

ORMs are like a rubix cube. Great in theory. Illustrates awesome math concepts. Elegant. Everybody has one (lowest common denominator). But why the hell would anybody ever build their mission critical database like a rubix cube? Oh great! We have approximately 5 minutes to pull records and figure out if this person needs blood-thinners before surgery. Quick! Here's a rubix cube! Solve it or somebody bleeds out!

No fracking way.

EMR/PHR systems are such huge clusterfsks because time-and-time again designers keep applying junior level patterns they learned in school, because 'SQL is a best practice'. But VistA and Epic are hold outs, exactly because they were designed in response to real-world needs, before SQL came into fashion. And their data structures actually make sense. Of course, the M language syntax is a complete mess, what with it's method abbreviations, which is why a new language like Javascript is needed. But as far as applications go, MUMPS/VistA/Epic get the data structures right with a document-oriented approach. And an internal ORM would completely foul up the Meteor data model. People may not appreciate it, but Meteor's lack of an ORM is a godsend for development, devops, and operations.

But that's not to say that I don't appreciate why people want an ORM. Data-mapping is a huge issue. And there's a real need to interface with external SQL systems. In my professional opinion, with a decade of database administration experience, the appropriate place for an ORM in the Meteor ecosystem is external to the live-data packages. Something like this:

Anyhow, I bring this up, because there's something of a healthcare working group in the Meteor community. I personally know between one and two dozen developers actively working on healthcare apps with Meteor. And to the extent that I've been coordinating that working group, I'm going to speak up for it and other healthcare professionals using Meteor, and say, 'This ORM suggestion is a really bad idea.' If this moves forward, there would be people interested in forking Meteor at v1.0, and heading it in another direction before having an ORM mess up the internal data model. Bringing an ORM into core Meteor is something of a dealbreaker for some people in the Meteor community.

In the spirit of encouraging more discussion, I'd like to mention a slight alternative to a full-fledged ORM. In a former life doing Java development, I found using iBatis (now MyBatis) to be clean and ops-friendly while working with existing SQL databases. The types of databases that typically don't fit nicely into an ORM's view of the world...

Unlike ORM frameworks, MyBatis does not map Java objects to database tables but Java methods to SQL statements.

MyBatis lets you use all your database functionality like stored procedures, views, queries of any complexity and vendor proprietary features.

I imagine taking this type of approach for Meteor would also be easier to implement than an ORM.

MyBatis provides a mapping engine that maps SQL results to object trees in a declarative way.

<select id="selectUsers" resultType="User">
  select id, username, password
  from users
  where id = #{id}
</select>

<insert id="insertAuthor">
  insert into Author (id,username,password,email,bio)
  values (#{id},#{username},#{password},#{email},#{bio})
</insert>

<update id="updateAuthor">
  update Author set
    username = #{username},
    password = #{password},
    email = #{email},
    bio = #{bio}
  where id = #{id}
</update>

<delete id="deleteAuthor">
  delete from Author where id = #{id}
</delete>

Source: mapping files

Hydrating objects is straightforward while DB Admins and dev-ops folks can still go nuts with their optimizations. It also doesn't hide bad SQL like ORMs would (SELECT N+1 is easier to spot).

What you end up with is an API for your database, independent of vendor, which is also nice since you can switch out your backend db with little impact on the codebase.

Data-caching can be built-in. Code generation can help with the "getting started" process and to avoid writing all SQL by hand. I'd imagine validation would be handled separately on the javascript object side or potentially worked into the mapping.

I'm not sure I can add much to this conversation, but that's never stopped me from trying before. So:

• Having worked at Epic, I tend to agree with @awatson1978 and see things from the same new-MUMPS perspective.
• However, I think different people see "ORM" and think of different things. Epic actually does use an ORM in that they have built their own application-level table/row structure that sits on top of MUMPS and validates the data. So the developers think relationally while the DBAs think post-relationally. So any discussion of ORM needs to also answer the question, "What kind of ORM?"
• Although it is more work to develop and maintain, the approach of having individual "DB/miniDB" packages that understand each other makes the most sense to me. DDP is generic enough to transport the data in whatever way these packages see fit to transport it. We can perhaps agree on a common API/spec (find/findOne/insert/update/remove and cursors, _id is generated on the client, a document is returned, fields, limit, skip, etc.), but each miniDB pkg can and will implement the entire spec separately. Essentially only the constructor options and the query syntax might look different. (Seems similar to what is actually being proposed: "the idea is to have a common API for different database backends". But I'm arguing that this should be in the form of an agreed-upon spec rather than a separate layer or package.)
• Something like @alanning's MyBatis proposal can be implemented as a separate package. Let's say we have sql/minisql packages that simply expect a partial SQL string query: myTable.find('where id = ' + id) (security implications aside). An sql-mybatis package could provide a simple translation function, or an override implementation of find: myTable.find({selectUsers: {id: id}}). My point: Keep everything separate and discrete and loose, so that various people can work on different pieces and do what they need to do to make it work.

In Waterline you can write native db queries as you want,
so the point of being closed to the ORM API is not true...

What about LINQ

I would be pleased to see an alternative to MongoDB, preferrably SQLite as it has indexing, to develop Meteor apps on mobile. Right now to install such an app you need to download 300 MB of packages, including MongoDB. This is too big for any app running on a mobile phone. (Probably this will be overcome by increased capacity and bandwidth, but currently this is a quite blocking issue.)