What's wrong with Ruby libraries for CouchDB?

whats-wrong-with-ruby-libraries-for-couchdb.markdown

What's wrong with Ruby libraries for CouchDB?

During the last year, I have worked on couple of Ruby/Rails based projects where the primary datastore was CouchDB. CouchDB, to me, is a dream database for web developer come true. The simplicity, the HTTP-based API, the abandonment of SQL semantics, the inspiring community, that all reminds me of when I came into Rails years ago.

However, working with Couch in Ruby and Rails is very, very painful, in my opinion. I'd like to briefly summarize some of my frustrations here. Maybe they are shared, maybe not -- if they are, I think we should launch some coordinate effort to make using Couch in Ruby a pleasure and intelectual satisfaction, not endless loops of research and hacks to „make it work“.

Please note, that my interest is solely to stir the debate. I may be severely mistaken in any point. But, I'd like using Couch in a Ruby application to be a joy, not a frustration, which is what I've met more times than I'd have liked.

Plethora of Gems

First and obvious problem is the plethora of gems you can choose from to interact with Couch. There's the granddaddy CouchRest, above which sits CouchPotato and above which sits SimplyStored. There's RelaxDB and Makura with different take on things. There's CouchFoo, aiming for drop-in ActiveRecord replacement. And possibly others -- due to the simple HTTP-based Couch's API, it's relatively simple to create some wrapper around it.

Why the choice is bad? First of all, you have to research your options. You have to read documentation for those gems, noticing that they are almost the same, that they offer almost the same features. So, which of those options are valuable for you? Which less so? Which of those gems seem to be more maintained or used?

The situation reminds me of the state of internationalization/localization in Ruby on Rails some two years ago. You could choose from Globalize, GlobaLITE, Gibberish and loads and loads of other plugins/gems, providing you with relatively simple, and similar funtionality: replacing identifiers in your code with locale-specific text. In the end, it became completely unberable to research and compare the options, and the plugin/gem authors decided to get together and try to nail the commonalities and differences in their approaches, and a generic, higly modular solution for Rails i18n was born: http://github.com/svenfuchs/i18n. In the video from Euruko 2010, Sven Fuchs gives a very good overview of the effort: http://vimeo.com/12665914.

Do you think this is something worthwile to do for Couch-related gems?

Layers of Abstraction

As briefly noted above, some of the solutions have grotesque numbers of layers. Consider using SimplyStored, a very advanced and Rails-compatible library, in your application. Apart from the CouchDB database itself, you deal with:

1. RestClient -- Handling HTTP
2. CouchRest -- Handling creating databases, saving documents, etc
3. CouchPotato -- Handling saving documents, validations, declaring and querying views, etc
4. SimplyStored -- Giving ActiveModel-like API, saving documents, validations, associations, etc

That's rather painful. Whenever you need to adapt, patch, or otherwise change something in your app, you have to constantly shift between these layers, their codebase, their test suites, documentation and quirks.

(Lack of) Modularization

I think current libraries do not do true justice to the flexibility of Couch.

One real world example. On my current contract, we need to save different models into different databases. (And we need to use and interact with the Couch authentication/authorization system, but that's another story.) That's impossible to do eg. in RelaxDB or CouchPotato/SimplyStored. In the end, we had to abandon attempts to hack that into SimplyStored and ended with replicating much of it's functionality in customized code.

But, hell, we need to use different databases for different model instances! (think user „inboxes“). Maybe that's kinda idiosyncratic, but consider that even the good old ActiveRecord has use_table.

And the list could continue. What if you, for example, want to use Typhoeus instead of RestClient? It should be trivial to do that.

Compatibility With The Rest of the World

Whatever our opinion is on Rails and "ActiveRecord", it's the dominant platform for Ruby web applications. CouchPotato has full ActiveModel compatibility -- but, as an example, it magically disappears in SimplyStored (http://github.com/karmi/simply_stored/commit/a168851). CouchPotato does not offer Model#save and such features by itself.

Compare this with the MongoID gem. That is just true drop-in solution. For simple cases, you swap MyModel < ActiveRecord::Base with some includes and you're done. I'd love Couch to be usable on exactly the same level.

I also hear endless talk about Couch „not being the same as relational databases“ and such, repeated over and over. This is truly painful because some of the people who repeat such things confuse „relational“ with has_many associations. Whatever the issue is, ActiveRecord semantics (Model#update_attributes, validations, etc) is (one) valid way for modelling in any data store. Even Ohm for Redis has similar semantics.

Another case in point is pagination. Some of the gems, like RelaxDB, implement pagination, some of them leave that as an exercise for the reader. RelaxDB implements pagination in it's own way, incompatible with the preferred way of pagination in a Rails app -- will_paginate. This way, we lose all the helpers, all the syntax sugar everybody is used to when working with Rails.

And of course, will_paginate's style of pagination is wrong in Couch. But, why should it not be one way of doing it? Why should you think about and implement pagination on your own?

It is my opinion, that anybody should be able to use Couch in Rails or Sinatra or plain Ruby application as easily as using ActiveRecord, or, maybe more importantly, the highly faved MongoDB. Please share your opinion in the comments.

Well.. We can't exactly stop people writing their own gems. This is more of a "problem" with ruby that there tend to be a lot of choices on doing things. Just search for gems with "mongo" in the name to see a huge list. I would classify that as a bit of a luxury problem if anything.

The grand daddy is of course couchrest. If you use Rails 2 you use couchrest_extended_document or couch_potato that try to be similar to ActiveRecord in their own way. For Rails 3 things get better as you have couchrest_model that uses the new ActiveModel API.

Mikeal has a reply to this here: http://www.mikealrogers.com/2010/08/abstracting-couchdb/

I think that you are just abstracting too much for your needs. Why not just use couchrest and couchrest rails gems?

"Maybe that's kinda idiosyncratic, but consider that even the good old ActiveRecord has use_table."

Couchrest has use_database

"CouchPotato does not offer Model#save and such features by itself"

Couchrest has model.save

"Whatever the issue is, ActiveRecord semantics (Model#update_attributes, validations, etc) is (one) valid way for modelling in any data store"

Couchrest's rails gem has support for update_attributes, validations, etc

Hope you find solution in the end that works out well for you ;)

as jongretar says, i also don't think there will (or even should) be one single "official" couch gem. the main points of confusion here seem to be:

• how do i make it easy to get started with couchdb and ruby
• how do i know which gem to use
• do i want an activerecord (style) api or not, do i want a low or a high abstraction

as for the beginner's confusion, i think some better documentation would be nice, maybe set up a web site that explains the options. (something like http://guides.rubyonrails.org/)

imho there are just too many different ways to use couchdb, so there can't be one gem to rule them all. some people prefer to stay really low level, put as much logic into js functions in couch (also using list functions etc.) and only use hashes in ruby, the other extreme wants something that's as close to AR as possible. in between people are experimenting with AR (the pattern) alternatives (e.g. getting rid of callbacks and validations).

p.s. couch potato easily supports your "one database per instance" (CouchPotato::Database.new)

What it really comes down to is how comfortable one is with CouchDB. Because in a way you can say that the CouchDB server is the model && the model related helpers. One avoids using it that way in the beginning because most of us have had bad experiences working with the more "modely" things in SQL databases such as stored procedures.

I started with CouchPotato and CouchRestRails. I find myself using it less and less because unlike with SQL databases I find myself wanting to be closer to the database but not abstract it away.

Thanks for all the feedback. There's been some misunderstanding, which I expected. Thanks @mikeal for the long answer.

I’m not saying you should never use a higher level abstraction but if you want to really understand CouchDB (...) you’re going to need to understand the REST interface (...)

And I'm not saying that there should be "ActiveRecord for Couch". I said true drop-in gem for using Couch in Rails should be 100% ActiveModel compatible, making stuff like form_for etc. work painlessly. (And I still see no real argument against that.) By the way, it seems that current couchrest_model brings that support out of the box. I also think, that in an ideal world a Rubygem for working with Couch would provide a smooth interface to Couch features, optionally even stuff like plugging into authentication/authorization or using the _changes feed. And pagination. Or meaningful exceptions, like most prominently Makura does. Right now, the functionality is difused across a dozen of gems, making it hard to evaluate.

Now, it's not about whether I'll "find solution in the end that works out well", as Jon writes. Of course I will. I have been working with Couch for almost a year, on real, production apps in Ruby (or a couchapp, used by a hosting company to manage invoicing/billing). That's not the point. Compared to how (fuckin') awesome Couch is, working with Couch in Ruby is unnecessarily hard, in my eyes. That's all.

Redis is fuckin’ awesome, but you wouldn’t have a good time trying to map your relational way of doing things to Redis.

Indeed. But this is not about "my relational way of doing things". As said, encountering Couch was probably the most joyful experience in the
"developer" part of my life, during last year, period. I thoroughly enjoyed learning it, researching it, using it. I cannot praise stuff like "no off switch" or trading CPU cycles for HDD space enough. (And yes, Redis is probably on the same level of awesomeness, even sharing the level of design simplicity.)

But, why it's so, that almost every gem out there implements "some" ActiveRecord-like API, even lightweight like Makura? Probably because there's some natural way of thinking about modelling, about imagining entities in Ruby classes? Could we work with that? Could we adapt and change such pattern? (Note that I'm not sold on most, probably all, implementations of has_many and belongs_to. This is the real point where ActiveRecord semantics may not make much sense.)

This point is misleading, however. Most of the commentary is focused on the "abstraction" problem ("close to the metal", etc). Regarding this point, I only said it should be trivially easy to use CouchDB instead of *SQL database in a Rails app. And that in Ruby, a notoriously „smooth“ language, you'd expect elegant, simple and flexible solutions to your problems. I could raise more issues (like storing custom JavaScript in Ruby strings, seriously?, or providing support for storing them on disk, but in a format different from a couchapp?, why?) but that's for another time, maybe.

As it is, it seems that the core of Ruby devs working with Couch think the situation is "good enough" or "great". Really thanks for the various feedback I have received!

Anyway, just imagine what do you say to a guy/gal asking you „so what Rubygem should I use to work with Couch“? „Well, make your pick, they're almost the same“? „First, make sure you know Couch really well, so you know how to evaluate all that stuff“? „Just use a libcurl wrapper, it's about the metal ,,/“?

The gist of my opinion is this, again: it's hard to choose from so many very similar options. Researching and evaluating all the options is just hard, and I don't see any immediate benefit in that.

First and obvious problem is the plethora of gems you can choose from to interact with Couch

I'll simply re-iterate what Mikeal said, it's in the nature of such projects that an explosion of similar-but-different libraries emerge. Despite the inevitable confusion caused, I think in general such efforts are to be lauded.

Quoting Mikeal

but if you want to really understand CouchDB and always be able to debug any higher level library you’re going to need to understand the REST interface as well as you understand the abstractions

I couldn't agree more.

And quoting langalex

imho there are just too many different ways to use couchdb, so there can't be one gem to rule them all

I'm inclined to agree with this too. Ultimately, I suspect the different libraries offer their own take simply because they provide specific solutions to their authors' specific pain points.

Researching and evaluating all the options is just hard, and I don't see any immediate benefit in that.

You're right in that there probably is no immediate benefit. But I guess the appropriate approach to research is the same, irrespective of the number of competing libraries. Work out what you need, then check the test suite of a lib. If it meets your needs, the code looks reasonable, and the lib is maintained, you're done. Else, move on.

Of course, popularity and suitability don't always intersect and so I'll finish by pointing you towards CouchTiny, which you didn't mention. Written by Brian Candler, it offers both database centric and document centric APIs. The code is consise and idiomatic. If you're serious about a desire to produce the de facto Ruby lib for CouchDB, perhaps it would be a good place to start. Or perhaps I've simply added to your pain :)

---

RelaxDB implements pagination in it's own way, incompatible with the preferred way of pagination in a Rails app -- will_paginate... And of course, will_paginate's style of pagination is wrong in Couch

I would have loved to implement pagination in RelaxDB in a manner consitent with will_paginate, but IIRC it wasn't possible, hence the balkanized approach. I don't know what the alternative might have been.

Just to clear some confusion, I agree 100% a) with Mikeal that you have to know the Couch's API to be able to use Couch features in any high-level library, and I even agree b) with Alex that there are many very different ways to use Couch.

As for a), every gem out there provides (sometimes very high-level) abstractions for that, eg. a DSL for view definitions. So while I understand the arguments about "abstractions", I think the argument is being stuck midway somewhere.

As for b), why almost every gem out there has more or less the same semantics, then? I just don't see the „diversity“ in their feture sets, nor in their code. Yes, definitely, different gems provide specific solutions to author's pains, as Paul writes. I myself had to monkeypatch and hack my way on every Couch project I've been working on. But is it really so, that everyone has their own pain, different from others? Wittgenstein would say so. I can't know if you're in pain and what sort of pain. But by observing you, I can think, „he's clearly in pain“. And by talking to you, I can say, „you seem to have some tooth pain, try this, it has helped me, maybe it'll help you as well“.

To re-iterate, this is not about "some lazy jerk's problems with couch gems" or "stopping people making gems". (I appreciate all the efforts, of course!, and may have not stressed that enough.)

This is about Couch being as simple, as elegant, as usable on the high-level, in Ruby, as when using curl, so to speak.

so once again, since we can't and don't want to delete 90% of the gems out there, how about creating some central intro/guides website that helps less experienced people to evaluate their options? couchonrails.org?